DEV Community: Lorenzo Garuti

Install and configure MySQL Server and MySQL InnoDB Cluster

Lorenzo Garuti — Wed, 31 Aug 2022 09:50:30 +0000

This role will install and configure MySQL server or MySQL in HA mode using MySQL InnoDB Cluster or GTID replication

Role Variables
Vagrant up, build the test infrastructure
Ansible setup and pre-flight check
Deploy MySQL InnoDB Cluster
Cluster high availability check
Restore from complete outage
Clean up

Role Variables

This role accept this variables:

Var	Required	Default	Desc
`mysql_subnet`	`yes`	`192.168.25.0/24`	Subnet where MySQL will be listen. If the VM or bare metal server has more than one interface, Ansible will filter the interface and MySQL wil listen only on a specific interface. This variable is also used to calculate the MySQL server ID.
`mysql_root_pw`	`yes`		MySQL root password.
`mysql_authentication`	`no`	`mysql_native_password`	MySQL authentication method.
`disable_firewall`	`no`	`no`	If set to yes Ansible will disable the firewall.
`disable_selinux`	`no`	`no`	Disable SELinux. Default no, if you want to configure SELinux use another Role. You can disable SELinux setting this variable to yes
`resolv_mode`	`no`	`dns`	How MySQL resolve the names, default dns. If set to host the /etc/hosts file will be overwritten
`mysql_listen_all_interfaces`	`no`	`no`	Set this variable to yes to allow MySQL to listen on all interfaces 0.0.0.0/0. Otherwise the listen ip address will be retrieved using mysql_subnet variable
`mysql_user`	`no`	`mysql`	MySQL system user
`mysql_group`	`no`	`mysql`	Group of the MySQL search system user
`mysql_data_dir`	`no`	`/var/lib/mysql`	MySQL data dir
`mysql_log_dir`	`no`	`/var/log/mysql`	MySQL log dir
`mysql_conf_dir`	`no`	`/etc/mysql`	MySQL conf dir
`mysql_pid_dir`	`no`	`/var/run/mysqld`	MySQL pid dir
`mysql_operator_user`	`no`	`operator`	MySQL operator user, used to bootstrap MySQL InnoDB Cluster.
`mysql_operator_password`	`no`	`Op3r4torMyPw`	Password of operator user
`mysql_replica_user`	`no`	`replica`	MySQL replica user. Used for all the replica operations
`mysql_replica_password`	`no`	`rEpL1c4p4Sw0,rd`	Password of replica user
`mysql_replication_mode`	`no`		InnoDB Cluster, GTID, Empty/None (default)
`mysql_gr_name`	`no`		Required if mysql_replication_mode is set to InnoDB Cluster. UUID of the Group Replication
`mysql_gr_vcu`	`no`		Required if mysql_replication_mode is set to InnoDB Cluster. Group Replication view change uuid
`mysql_innodb_cluster_name`	`no`		Required if mysql_replication_mode is set to InnoDB Cluster. The name of MySQL InnoDB Cluster

Vagrant up, build the test infrastructure

To test this role we use Vagrant and Virtualbox, but if you prefer you can also use your own VMs or your baremetal machines.

The first step is to download this repo and birng up all the VMs. But first in the Vagrantfile paste your public ssh key in the CHANGE_ME variable. You can also adjust the number of the vm deployed by changing the NNODES variable (in this exaple we will use 5 nodes). Now we are ready to provision the machines:

git clone https://github.com/garutilorenzo/ansible-role-linux-mysql.git

cd ansible-role-linux-mysql/

vagrant up
Bringing machine 'my-ubuntu-0' up with 'virtualbox' provider...
Bringing machine 'my-ubuntu-1' up with 'virtualbox' provider...
Bringing machine 'my-ubuntu-2' up with 'virtualbox' provider...
Bringing machine 'my-ubuntu-3' up with 'virtualbox' provider...
Bringing machine 'my-ubuntu-4' up with 'virtualbox' provider...

[...]
[...]

    my-ubuntu-4: Inserting generated public key within guest...
==> my-ubuntu-4: Machine booted and ready!
==> my-ubuntu-4: Checking for guest additions in VM...
    my-ubuntu-4: The guest additions on this VM do not match the installed version of
    my-ubuntu-4: VirtualBox! In most cases this is fine, but in rare cases it can
    my-ubuntu-4: prevent things such as shared folders from working properly. If you see
    my-ubuntu-4: shared folder errors, please make sure the guest additions within the
    my-ubuntu-4: virtual machine match the version of VirtualBox you have installed on
    my-ubuntu-4: your host and reload your VM.
    my-ubuntu-4:
    my-ubuntu-4: Guest Additions Version: 6.0.0 r127566
    my-ubuntu-4: VirtualBox Version: 6.1
==> my-ubuntu-4: Setting hostname...
==> my-ubuntu-4: Configuring and enabling network interfaces...
==> my-ubuntu-4: Mounting shared folders...
    my-ubuntu-4: /vagrant => C:/Users/Lorenzo Garuti/workspaces/simple-ubuntu
==> my-ubuntu-4: Running provisioner: shell...
    my-ubuntu-4: Running: inline script
==> my-ubuntu-4: Running provisioner: shell...
    my-ubuntu-4: Running: inline script
    my-ubuntu-4: hello from node 5

Ansible setup and pre-flight check

Now if you don't have Ansible installed, install ansible and all the requirements:

apt-get install python3 python3-pip uuidgen openssl
pip3 install pipenv

pipenv shell
pip install -r requirements.txt

Now with Ansible installed we can download the role directly from GitHub:

ansible-galaxy install git+https://github.com/garutilorenzo/ansible-role-linux-mysql.git

Whit Ansible and the role installed we can setup our inventory file (hosts.ini):

[mysql]
my-ubuntu-0 ansible_host=192.168.25.110
my-ubuntu-1 ansible_host=192.168.25.111
my-ubuntu-2 ansible_host=192.168.25.112
my-ubuntu-3 ansible_host=192.168.25.113
my-ubuntu-4 ansible_host=192.168.25.114

and the vars.yml file:

---

disable_firewall: yes
disable_selinux: yes
mysql_resolv_mode: hosts
mysql_subnet: 192.168.25.0/24
mysql_listen_all_interfaces: yes

mysql_root_pw: '<CHANGE_ME>' # <- openssl rand -base64 32 | sed 's/=//'
mysql_replication_mode: 'InnoDB Cluster'
mysql_gr_name: '<CHANGE_ME>' # <- uuidgen
mysql_gr_vcu: '<CHANGE_ME>' #  <- uuidgen
mysql_innodb_cluster_name: 'cluster_lab'

NOTE mysql_gr_name and mysql_gr_vcu are different uuid, so run uuidgen twice.
With this vars we are going to deploy MySQL in HA Mode with MySQL InnoDB Cluster, the cluster will be created from an existing Group Replication configuration.

The final step before proceed with the installation is to create the site.yml file:

---
- hosts: mysql
  become: yes
  remote_user: vagrant
  roles: 
    - role: ansible-role-linux-mysql
  vars_files:
    - vars.yml

Deploy MySQL InnoDB Cluster

We are finally ready to deploy MySQL InnoDB Cluster using ansible:

export ANSIBLE_HOST_KEY_CHECKING=False # Ansible skip ssh-key validation

ansible-playbook -i hosts.ini site.yml -e mysql_bootstrap_host=my-ubuntu-0

TASK [ansible-role-linux-mysql : render mysql.conf.d/mysqld.cnf] *******************************************************************************************
ok: [my-ubuntu-0]
ok: [my-ubuntu-2]
ok: [my-ubuntu-1]
ok: [my-ubuntu-3]
ok: [my-ubuntu-4]

TASK [ansible-role-linux-mysql : render mysql.conf.d/gtid.cnf] *********************************************************************************************
ok: [my-ubuntu-1]
ok: [my-ubuntu-0]
ok: [my-ubuntu-3]
ok: [my-ubuntu-2]
ok: [my-ubuntu-4]

TASK [ansible-role-linux-mysql : ansible.builtin.fail] *****************************************************************************************************
skipping: [my-ubuntu-0]
skipping: [my-ubuntu-1]
skipping: [my-ubuntu-2]
skipping: [my-ubuntu-3]
skipping: [my-ubuntu-4]

TASK [ansible-role-linux-mysql : ansible.builtin.fail] *****************************************************************************************************
skipping: [my-ubuntu-0]
skipping: [my-ubuntu-1]
skipping: [my-ubuntu-2]
skipping: [my-ubuntu-3]
skipping: [my-ubuntu-4]

TASK [ansible-role-linux-mysql : render innodb_cluster.cnf] ************************************************************************************************
ok: [my-ubuntu-0]
ok: [my-ubuntu-1]
ok: [my-ubuntu-3]
ok: [my-ubuntu-2]
ok: [my-ubuntu-4]

RUNNING HANDLER [ansible-role-linux-mysql : reload systemd] ************************************************************************************************
ok: [my-ubuntu-3]
ok: [my-ubuntu-0]
ok: [my-ubuntu-2]
ok: [my-ubuntu-4]
ok: [my-ubuntu-1]

PLAY RECAP *************************************************************************************************************************************************
my-ubuntu-0               : ok=69   changed=27   unreachable=0    failed=0    skipped=12   rescued=0    ignored=0   
my-ubuntu-1               : ok=71   changed=28   unreachable=0    failed=0    skipped=10   rescued=0    ignored=0   
my-ubuntu-2               : ok=71   changed=28   unreachable=0    failed=0    skipped=10   rescued=0    ignored=0   
my-ubuntu-3               : ok=71   changed=28   unreachable=0    failed=0    skipped=10   rescued=0    ignored=0   
my-ubuntu-4               : ok=71   changed=28   unreachable=0    failed=0    skipped=10   rescued=0    ignored=0

The cluster is now installed, but we have to persist some configurations. Since the cluster is a new cluster, Ansible has started the Group Replication in bootstrap mode. This means that the first instance (in this case my-ubuntu-0) has te value group_replication_bootstrap_group set to ON, and group_replication_group_seeds set to an empty value. With this second run of Ansible, this variables will be set to the correct values:

ansible-playbook -i hosts.ini site.yml

TASK [ansible-role-linux-mysql : ansible.builtin.fail] *****************************************************************************************************
skipping: [my-ubuntu-0]
skipping: [my-ubuntu-1]
skipping: [my-ubuntu-2]
skipping: [my-ubuntu-3]
skipping: [my-ubuntu-4]

TASK [ansible-role-linux-mysql : ansible.builtin.fail] *****************************************************************************************************
skipping: [my-ubuntu-0]
skipping: [my-ubuntu-1]
skipping: [my-ubuntu-2]
skipping: [my-ubuntu-3]
skipping: [my-ubuntu-4]

TASK [ansible-role-linux-mysql : render innodb_cluster.cnf] ************************************************************************************************
ok: [my-ubuntu-2]
ok: [my-ubuntu-4]
ok: [my-ubuntu-1]
ok: [my-ubuntu-3]
changed: [my-ubuntu-0]

PLAY RECAP *************************************************************************************************************************************************
my-ubuntu-0               : ok=30   changed=1    unreachable=0    failed=0    skipped=18   rescued=0    ignored=0   
my-ubuntu-1               : ok=30   changed=0    unreachable=0    failed=0    skipped=18   rescued=0    ignored=0   
my-ubuntu-2               : ok=30   changed=0    unreachable=0    failed=0    skipped=18   rescued=0    ignored=0   
my-ubuntu-3               : ok=30   changed=0    unreachable=0    failed=0    skipped=18   rescued=0    ignored=0   
my-ubuntu-4               : ok=30   changed=0    unreachable=0    failed=0    skipped=18   rescued=0    ignored=0

In this guide mysqlsh is used to make operations on MySQL InnoDB Cluster. Here you can find more information about mysqlsh.

Now we can finally check our cluster:

root@my-ubuntu-0:~# mysqlsh root@my-ubuntu-0
Please provide the password for 'root@my-ubuntu-0': ******************************************
MySQL  localhost:33060+ ssl  JS > clu = dba.getCluster()
MySQL  localhost:33060+ ssl  JS > clu.status()
{
    "clusterName": "cluster_lab", 
    "defaultReplicaSet": {
        "name": "default", 
        "primary": "my-ubuntu-0:3306", 
        "ssl": "DISABLED", 
        "status": "OK", 
        "statusText": "Cluster is ONLINE and can tolerate up to 2 failures.", 
        "topology": {
            "my-ubuntu-0:3306": {
                "address": "my-ubuntu-0:3306", 
                "memberRole": "PRIMARY", 
                "mode": "R/W", 
                "readReplicas": {}, 
                "replicationLag": "applier_queue_applied", 
                "role": "HA", 
                "status": "ONLINE", 
                "version": "8.0.30"
            }, 
            "my-ubuntu-1:3306": {
                "address": "my-ubuntu-1:3306", 
                "memberRole": "SECONDARY", 
                "mode": "R/O", 
                "readReplicas": {}, 
                "replicationLag": "applier_queue_applied", 
                "role": "HA", 
                "status": "ONLINE", 
                "version": "8.0.30"
            }, 
            "my-ubuntu-2:3306": {
                "address": "my-ubuntu-2:3306", 
                "memberRole": "SECONDARY", 
                "mode": "R/O", 
                "readReplicas": {}, 
                "replicationLag": "applier_queue_applied", 
                "role": "HA", 
                "status": "ONLINE", 
                "version": "8.0.30"
            }, 
            "my-ubuntu-3:3306": {
                "address": "my-ubuntu-3:3306", 
                "memberRole": "SECONDARY", 
                "mode": "R/O", 
                "readReplicas": {}, 
                "replicationLag": "applier_queue_applied", 
                "role": "HA", 
                "status": "ONLINE", 
                "version": "8.0.30"
            }, 
            "my-ubuntu-4:3306": {
                "address": "my-ubuntu-4:3306", 
                "memberRole": "SECONDARY", 
                "mode": "R/O", 
                "readReplicas": {}, 
                "replicationLag": "applier_queue_applied", 
                "role": "HA", 
                "status": "ONLINE", 
                "version": "8.0.30"
            }
        }, 
        "topologyMode": "Single-Primary"
    }, 
    "groupInformationSourceMember": "my-ubuntu-0:3306"
}

Cluster high availability check

To test the cluster we can use a sample Docker compose stack, the example uses:

Wordpress as frontend
mysqlrouter will connect WP to MySQL

To run this test you have to install Docker and Docker compose.

User and Database creation

We need to create one db and one user for wordpress, to do this we have to find the primary server (check the cluster status and find the node with -> "mode": "R/W")

root@my-ubuntu-0:~# mysqlsh root@localhost
Please provide the password for 'root@localhost': ******************************************

MySQL  localhost:33060+ ssl  JS > \sql # <- SWITCH TO SQL MODE
Switching to SQL mode... Commands end with ;

create database wordpress;
create user 'wordpress'@'%' identified by 'wordpress';
grant all on wordpress.* TO 'wordpress'@'%';
flush privileges;

Sample Dokcer compose stack

The sample stack can be found in the examples folder, this is the compose file:

version: '3.4'
services:
  wordpress:
    image: wordpress:latest
    ports:
      - 80:80
    restart: always
    environment:
      - WORDPRESS_DB_HOST=mysqlrouter:6446
      - WORDPRESS_DB_USER=wordpress
      - WORDPRESS_DB_PASSWORD=wordpress
      - WORDPRESS_DB_NAME=wordpress

  mysqlrouter:
    image: garutilorenzo/mysqlrouter:8.0.30
    volumes:
      - type: volume
        source: mysqlrouter
        target: /app/mysqlrouter/
        volume:
          nocopy: true
    environment:
     - MYSQL_HOST=my-ubuntu-0
     - MYSQL_PORT=3306
     - MYSQL_USER=root
     - MYSQL_PASSWORD=<CHANGE_ME> # <- the same password in the vars.yml file
     - MYSQL_ROUTER_ACCOUNT=mysql_router_user
     - MYSQL_ROUTER_PASSWORD=<CHANGE_ME> # <- openssl rand -base64 32 | sed 's/=//'
    extra_hosts:
      my-ubuntu-0: 192.168.25.110
      my-ubuntu-1: 192.168.25.111
      my-ubuntu-2: 192.168.25.112
      my-ubuntu-3: 192.168.25.113
      my-ubuntu-4: 192.168.25.114

volumes:
 mysqlrouter:

Now we can start our stack and inspect the logs:

docker-compose -f mysql-router-compose.yml up -d
docker-compose -f mysql-router-compose.yml logs mysqlrouter

examples-mysqlrouter-1  | Succesfully contacted mysql server at my-ubuntu-0. Checking for cluster state.
examples-mysqlrouter-1  | Check if config exist
examples-mysqlrouter-1  | bootstrap mysqlrouter with account mysql_router_user
examples-mysqlrouter-1  | Succesfully contacted mysql server at my-ubuntu-0. Trying to bootstrap.
examples-mysqlrouter-1  | Please enter MySQL password for root: 
examples-mysqlrouter-1  | # Bootstrapping MySQL Router instance at '/app/mysqlrouter'...
examples-mysqlrouter-1  | 
examples-mysqlrouter-1  | Please enter MySQL password for mysql_router_user: 
examples-mysqlrouter-1  | - Creating account(s) (only those that are needed, if any)
examples-mysqlrouter-1  | - Verifying account (using it to run SQL queries that would be run by Router)
examples-mysqlrouter-1  | - Storing account in keyring
examples-mysqlrouter-1  | - Adjusting permissions of generated files
examples-mysqlrouter-1  | - Creating configuration /app/mysqlrouter/mysqlrouter.conf
examples-mysqlrouter-1  | 
examples-mysqlrouter-1  | # MySQL Router configured for the InnoDB Cluster 'cluster_lab'
examples-mysqlrouter-1  | 
examples-mysqlrouter-1  | After this MySQL Router has been started with the generated configuration
examples-mysqlrouter-1  | 
examples-mysqlrouter-1  |     $ mysqlrouter -c /app/mysqlrouter/mysqlrouter.conf
examples-mysqlrouter-1  | 
examples-mysqlrouter-1  | InnoDB Cluster 'cluster_lab' can be reached by connecting to:
examples-mysqlrouter-1  | 
examples-mysqlrouter-1  | ## MySQL Classic protocol
examples-mysqlrouter-1  | 
examples-mysqlrouter-1  | - Read/Write Connections: localhost:6446
examples-mysqlrouter-1  | - Read/Only Connections:  localhost:6447
examples-mysqlrouter-1  | 
examples-mysqlrouter-1  | ## MySQL X protocol
examples-mysqlrouter-1  | 
examples-mysqlrouter-1  | - Read/Write Connections: localhost:6448
examples-mysqlrouter-1  | - Read/Only Connections:  localhost:6449
examples-mysqlrouter-1  | 
examples-mysqlrouter-1  | Starting mysql-router.
examples-mysqlrouter-1  | 2022-08-30 12:03:57 io INFO [7f794f1e0bc0] starting 4 io-threads, using backend 'linux_epoll'
examples-mysqlrouter-1  | 2022-08-30 12:03:57 http_server INFO [7f794f1e0bc0] listening on 0.0.0.0:8443
examples-mysqlrouter-1  | 2022-08-30 12:03:57 metadata_cache_plugin INFO [7f794a606700] Starting Metadata Cache
examples-mysqlrouter-1  | 2022-08-30 12:03:57 metadata_cache INFO [7f794a606700] Connections using ssl_mode 'PREFERRED'
examples-mysqlrouter-1  | 2022-08-30 12:03:57 metadata_cache INFO [7f7948602700] Starting metadata cache refresh thread
examples-mysqlrouter-1  | 2022-08-30 12:03:57 routing INFO [7f790e7fc700] [routing:bootstrap_rw] started: routing strategy = first-available
examples-mysqlrouter-1  | 2022-08-30 12:03:57 routing INFO [7f790e7fc700] Start accepting connections for routing routing:bootstrap_rw listening on 6446
examples-mysqlrouter-1  | 2022-08-30 12:03:57 routing INFO [7f790dffb700] [routing:bootstrap_x_ro] started: routing strategy = round-robin-with-fallback
examples-mysqlrouter-1  | 2022-08-30 12:03:57 routing INFO [7f790dffb700] Start accepting connections for routing routing:bootstrap_x_ro listening on 6449
examples-mysqlrouter-1  | 2022-08-30 12:03:57 routing INFO [7f790effd700] [routing:bootstrap_ro] started: routing strategy = round-robin-with-fallback
examples-mysqlrouter-1  | 2022-08-30 12:03:57 routing INFO [7f790effd700] Start accepting connections for routing routing:bootstrap_ro listening on 6447
examples-mysqlrouter-1  | 2022-08-30 12:03:57 metadata_cache INFO [7f7948602700] Connected with metadata server running on my-ubuntu-2:3306
examples-mysqlrouter-1  | 2022-08-30 12:03:57 routing INFO [7f790d7fa700] [routing:bootstrap_x_rw] started: routing strategy = first-available
examples-mysqlrouter-1  | 2022-08-30 12:03:57 routing INFO [7f790d7fa700] Start accepting connections for routing routing:bootstrap_x_rw listening on 6448
examples-mysqlrouter-1  | 2022-08-30 12:03:57 metadata_cache INFO [7f7948602700] Potential changes detected in cluster 'cluster_lab' after metadata refresh
examples-mysqlrouter-1  | 2022-08-30 12:03:57 metadata_cache INFO [7f7948602700] Metadata for cluster 'cluster_lab' has 5 member(s), single-primary: (view_id=0)
examples-mysqlrouter-1  | 2022-08-30 12:03:57 metadata_cache INFO [7f7948602700]     my-ubuntu-2:3306 / 33060 - mode=RO 
examples-mysqlrouter-1  | 2022-08-30 12:03:57 metadata_cache INFO [7f7948602700]     my-ubuntu-1:3306 / 33060 - mode=RO 
examples-mysqlrouter-1  | 2022-08-30 12:03:57 metadata_cache INFO [7f7948602700]     my-ubuntu-0:3306 / 33060 - mode=RW 
examples-mysqlrouter-1  | 2022-08-30 12:03:57 metadata_cache INFO [7f7948602700]     my-ubuntu-3:3306 / 33060 - mode=RO 
examples-mysqlrouter-1  | 2022-08-30 12:03:57 metadata_cache INFO [7f7948602700]     my-ubuntu-4:3306 / 33060 - mode=RO

Test the frontend

Now if you try to access localhost you can see the Wordpress installation page:

Install and configure WP, and now we are ready for some Chaos Monkey

Simulate disaster

To test WP reachability we can start this simple test:

while true; do curl -s -o /dev/null -w "%{http_code}" http://localhost; echo; sleep 1; done
200
200

now shutdown the RW node (in this case my-ubuntu-0):

root@my-ubuntu-0:~# sudo halt -p
Connection to 192.168.25.110 closed by remote host.
Connection to 192.168.25.110 closed.

and check the output of the test script:

while true; do curl -s -o /dev/null -w "%{http_code}" http://localhost; echo; sleep 1; done
200
500 # <- my-ubuntu-0 shutdown and MySQL primary switch
200
200

Now we check the cluster status from the second node, and we see that the cluster is still ONLINE and can tolerate one more failure:

root@my-ubuntu-1:~# mysqlsh root@localhost
Please provide the password for 'root@localhost': ******************************************

MySQL  localhost:33060+ ssl  JS > clu = dba.getCluster()
MySQL  localhost:33060+ ssl  JS > clu.status()
{
    "clusterName": "cluster_lab", 
    "defaultReplicaSet": {
        "name": "default", 
        "primary": "my-ubuntu-2:3306", 
        "ssl": "DISABLED", 
        "status": "OK_PARTIAL", 
        "statusText": "Cluster is ONLINE and can tolerate up to ONE failure. 1 member is not active.", 
        "topology": {
            "my-ubuntu-0:3306": {
                "address": "my-ubuntu-0:3306", 
                "memberRole": "SECONDARY", 
                "mode": "n/a", 
                "readReplicas": {}, 
                "role": "HA", 
                "shellConnectError": "MySQL Error 2003: Could not open connection to 'my-ubuntu-0:3306': Can't connect to MySQL server on 'my-ubuntu-0:3306' (110)", 
                "status": "(MISSING)"
            }, 
            "my-ubuntu-1:3306": {
                "address": "my-ubuntu-1:3306", 
                "memberRole": "SECONDARY", 
                "mode": "R/O", 
                "readReplicas": {}, 
                "replicationLag": "applier_queue_applied", 
                "role": "HA", 
                "status": "ONLINE", 
                "version": "8.0.30"
            }, 
            "my-ubuntu-2:3306": {
                "address": "my-ubuntu-2:3306", 
                "memberRole": "PRIMARY", 
                "mode": "R/W", 
                "readReplicas": {}, 
                "replicationLag": "applier_queue_applied", 
                "role": "HA", 
                "status": "ONLINE", 
                "version": "8.0.30"
            }, 
            "my-ubuntu-3:3306": {
                "address": "my-ubuntu-3:3306", 
                "memberRole": "SECONDARY", 
                "mode": "R/O", 
                "readReplicas": {}, 
                "replicationLag": "applier_queue_applied", 
                "role": "HA", 
                "status": "ONLINE", 
                "version": "8.0.30"
            }, 
            "my-ubuntu-4:3306": {
                "address": "my-ubuntu-4:3306", 
                "memberRole": "SECONDARY", 
                "mode": "R/O", 
                "readReplicas": {}, 
                "replicationLag": "applier_queue_applied", 
                "role": "HA", 
                "status": "ONLINE", 
                "version": "8.0.30"
            }
        }, 
        "topologyMode": "Single-Primary"
    }, 
    "groupInformationSourceMember": "my-ubuntu-2:3306"
}
 MySQL  localhost:33060+ ssl  JS >

If you check WP at localhost is still available, the master now is the my-ubuntu-2 node.
Now if we bring up again the my-ubuntu-0 node, the node wil rejoin the cluster and will get the updates from the other nodes:

root@my-ubuntu-1:~# mysqlsh root@localhost
Please provide the password for 'root@localhost': ******************************************

MySQL  localhost:33060+ ssl  JS > clu = dba.getCluster()
MySQL  localhost:33060+ ssl  JS > clu.status()
{
    "clusterName": "cluster_lab", 
    "defaultReplicaSet": {
        "name": "default", 
        "primary": "my-ubuntu-2:3306", 
        "ssl": "DISABLED", 
        "status": "OK", 
        "statusText": "Cluster is ONLINE and can tolerate up to 2 failures.", 
        "topology": {
            "my-ubuntu-0:3306": {
                "address": "my-ubuntu-0:3306", 
                "memberRole": "SECONDARY", 
                "mode": "R/O", 
                "readReplicas": {}, 
                "replicationLag": "applier_queue_applied", 
                "role": "HA", 
                "status": "ONLINE", 
                "version": "8.0.30"
            }, 
            "my-ubuntu-1:3306": {
                "address": "my-ubuntu-1:3306", 
                "memberRole": "SECONDARY", 
                "mode": "R/O", 
                "readReplicas": {}, 
                "replicationLag": "applier_queue_applied", 
                "role": "HA", 
                "status": "ONLINE", 
                "version": "8.0.30"
            }, 
            "my-ubuntu-2:3306": {
                "address": "my-ubuntu-2:3306", 
                "memberRole": "PRIMARY", 
                "mode": "R/W", 
                "readReplicas": {}, 
                "replicationLag": "applier_queue_applied", 
                "role": "HA", 
                "status": "ONLINE", 
                "version": "8.0.30"
            }, 
            "my-ubuntu-3:3306": {
                "address": "my-ubuntu-3:3306", 
                "memberRole": "SECONDARY", 
                "mode": "R/O", 
                "readReplicas": {}, 
                "replicationLag": "applier_queue_applied", 
                "role": "HA", 
                "status": "ONLINE", 
                "version": "8.0.30"
            }, 
            "my-ubuntu-4:3306": {
                "address": "my-ubuntu-4:3306", 
                "memberRole": "SECONDARY", 
                "mode": "R/O", 
                "readReplicas": {}, 
                "replicationLag": "applier_queue_applied", 
                "role": "HA", 
                "status": "ONLINE", 
                "version": "8.0.30"
            }
        }, 
        "topologyMode": "Single-Primary"
    }, 
    "groupInformationSourceMember": "my-ubuntu-2:3306"
}

Restore from complete outage

If for any reason all the servers went down, the cluster has to be restored from a complete outage.

To do this we have to connect to one instance, edit /etc/mysql/mysql.conf.d/innodb_cluster.cnf and set group_replication_bootstrap_group to ON and comment group_replication_group_seeds. We have now to restart MySQL:

vagrant@my-ubuntu-0:~$
vi /etc/mysql/mysql.conf.d/innodb_cluster.cnf

group_replication_bootstrap_group=on
#group_replication_group_seeds=my-ubuntu-0:33061,my-ubuntu-1:33061,my-ubuntu-2:33061,my-ubuntu-3:33061,my-ubuntu-4:33061

systemctl restart mysqld

For all the other (four) members we have to start the group replication:

vagrant@my-ubuntu-4:~$ mysqlsh root@localhost
Please provide the password for 'root@localhost': ******************************************

MySQL  localhost:33060+ ssl  JS > \sql # <- SWITCH TO SQL MODE
Switching to SQL mode... Commands end with ;
MySQL  localhost:33060+ ssl  SQL > start group_replication;
Query OK, 0 rows affected (1.7095 sec)

If the traffic on the cluster was low or absent the cluster will be ONLINE:

MySQL  localhost:33060+ ssl  SQL > \js
Switching to JavaScript mode...
MySQL  localhost:33060+ ssl  JS > clu = dba.getCluster()
MySQL  localhost:33060+ ssl  JS > clu.status()
{
    "clusterName": "cluster_lab", 
    "defaultReplicaSet": {
        "name": "default", 
        "primary": "my-ubuntu-0:3306", 
        "ssl": "DISABLED", 
        "status": "OK", 
        "statusText": "Cluster is ONLINE and can tolerate up to 2 failures.", 
        "topology": {
            "my-ubuntu-0:3306": {
                "address": "my-ubuntu-0:3306", 
                "memberRole": "PRIMARY", 
                "mode": "R/W", 
                "readReplicas": {}, 
                "replicationLag": "applier_queue_applied", 
                "role": "HA", 
                "status": "ONLINE", 
                "version": "8.0.30"
            }, 
            "my-ubuntu-1:3306": {
                "address": "my-ubuntu-1:3306", 
                "memberRole": "SECONDARY", 
                "mode": "R/O", 
                "readReplicas": {}, 
                "replicationLag": "applier_queue_applied", 
                "role": "HA", 
                "status": "ONLINE", 
                "version": "8.0.30"
            }, 
            "my-ubuntu-2:3306": {
                "address": "my-ubuntu-2:3306", 
                "memberRole": "SECONDARY", 
                "mode": "R/O", 
                "readReplicas": {}, 
                "replicationLag": "applier_queue_applied", 
                "role": "HA", 
                "status": "ONLINE", 
                "version": "8.0.30"
            }, 
            "my-ubuntu-3:3306": {
                "address": "my-ubuntu-3:3306", 
                "memberRole": "SECONDARY", 
                "mode": "R/O", 
                "readReplicas": {}, 
                "replicationLag": "applier_queue_applied", 
                "role": "HA", 
                "status": "ONLINE", 
                "version": "8.0.30"
            }, 
            "my-ubuntu-4:3306": {
                "address": "my-ubuntu-4:3306", 
                "memberRole": "SECONDARY", 
                "mode": "R/O", 
                "readReplicas": {}, 
                "replicationLag": "applier_queue_applied", 
                "role": "HA", 
                "status": "ONLINE", 
                "version": "8.0.30"
            }
        }, 
        "topologyMode": "Single-Primary"
    }, 
    "groupInformationSourceMember": "my-ubuntu-0:3306"
}

If the cluster has a high volume traffic at the moment of the complete outage. you have to probably run form mysqlsh:

MySQL  localhost:33060+ ssl  JS > var clu = dba.rebootClusterFromCompleteOutage();

Clean up

When you have done you can finally destroy the cluster with:

vagrant destroy

    my-ubuntu-4: Are you sure you want to destroy the 'my-ubuntu-4' VM? [y/N] y
==> my-ubuntu-4: Forcing shutdown of VM...
==> my-ubuntu-4: Destroying VM and associated drives...
    my-ubuntu-3: Are you sure you want to destroy the 'my-ubuntu-3' VM? [y/N] y
==> my-ubuntu-3: Forcing shutdown of VM...
==> my-ubuntu-3: Destroying VM and associated drives...
    my-ubuntu-2: Are you sure you want to destroy the 'my-ubuntu-2' VM? [y/N] y
==> my-ubuntu-2: Forcing shutdown of VM...
==> my-ubuntu-2: Destroying VM and associated drives...
    my-ubuntu-1: Are you sure you want to destroy the 'my-ubuntu-1' VM? [y/N] y
==> my-ubuntu-1: Forcing shutdown of VM...
==> my-ubuntu-1: Destroying VM and associated drives...
    my-ubuntu-0: Are you sure you want to destroy the 'my-ubuntu-0' VM? [y/N] y
==> my-ubuntu-0: Forcing shutdown of VM...
==> my-ubuntu-0: Destroying VM and associated drives...

Install and configure the ELK stack with Ansible

Lorenzo Garuti — Tue, 23 Aug 2022 07:48:26 +0000

This ansible collection will install and configure a high available Elasticsearch cluster.

With this collection you can also install and configure:

Logstash is a free and open server-side data processing pipeline that ingests data from a multitude of sources, transforms it, and then sends it to your favorite "stash."
Kibana is a free and open user interface that lets you visualize your Elasticsearch data and navigate the Elastic Stack. Do anything from tracking query load to understanding the way requests flow through your apps

And also the Elastic Beats:

filebeat - Whether you’re collecting from security devices, cloud, containers, hosts, or OT, Filebeat helps you keep the simple things simple by offering a lightweight way to forward and centralize logs and files.
metricbeat Collect metrics from your systems and services. From CPU to memory, Redis to NGINX, and much more, Metricbeat is a lightweight way to send system and service statistics.
heartbeath Monitor services for their availability with active probing. Given a list of URLs, Heartbeat asks the simple question: Are you alive? Heartbeat ships this information and response time to the rest of the Elastic Stack for further analysis.

In most cases you may prefer ECK or Elastic Cloud but if Kubernetes for you is like kryptonite for Superman or if you are jelous about your data or even more if you don't trust any Cloud provider this is the right place.

Vagrant up, build the test infrastructure
Ansible setup and pre-flight check
Deploy ELK with Ansible
Clean up

Vagrant up, build the test infrastructure

To test this collection we use Vagrant and Virtualbox, but if you prefer you can also use your own VMs or your baremetal machines.

The first step is to download this repo and birng up all the VMs. But first in the Vagrantfile paste your public ssh key in the CHANGE_ME variable. You can also adjust the number of the vm deployed by changing the NNODES variable (in this exaple we will use 6 nodes). Now we are ready to provision the machines:

git clone https://github.com/garutilorenzo/ansible-collection-elk.git

cd nsible-collection-elk/

vagrant up
Bringing machine 'elk-ubuntu-0' up with 'virtualbox' provider...
Bringing machine 'elk-ubuntu-1' up with 'virtualbox' provider...
Bringing machine 'elk-ubuntu-2' up with 'virtualbox' provider...
Bringing machine 'elk-ubuntu-3' up with 'virtualbox' provider...
Bringing machine 'elk-ubuntu-4' up with 'virtualbox' provider...
Bringing machine 'elk-ubuntu-5' up with 'virtualbox' provider...

[...]
[...]

    elk-ubuntu-5: Inserting generated public key within guest...
==> elk-ubuntu-5: Machine booted and ready!
==> elk-ubuntu-5: Checking for guest additions in VM...
    elk-ubuntu-5: The guest additions on this VM do not match the installed version of
    elk-ubuntu-5: VirtualBox! In most cases this is fine, but in rare cases it can
    elk-ubuntu-5: prevent things such as shared folders from working properly. If you see
    elk-ubuntu-5: shared folder errors, please make sure the guest additions within the
    elk-ubuntu-5: virtual machine match the version of VirtualBox you have installed on
    elk-ubuntu-5: your host and reload your VM.
    elk-ubuntu-5:
    elk-ubuntu-5: Guest Additions Version: 6.0.0 r127566
    elk-ubuntu-5: VirtualBox Version: 6.1
==> elk-ubuntu-5: Setting hostname...
==> elk-ubuntu-5: Configuring and enabling network interfaces...
==> elk-ubuntu-5: Mounting shared folders...
    elk-ubuntu-5: /vagrant => C:/Users/Lorenzo Garuti/workspaces/simple-ubuntu
==> elk-ubuntu-5: Running provisioner: shell...
    elk-ubuntu-5: Running: inline script
==> elk-ubuntu-5: Running provisioner: shell...
    elk-ubuntu-5: Running: inline script
    elk-ubuntu-5: hello from node 5

Ansible setup and pre-flight check

Now if you don't have Ansible installed, install ansible and all the requirements:

apt-get install python3 python3-pip
pip3 install pipenv

pipenv shell
pip install -r requirements.txt

Now with Ansible installed we can download the collection directly from GitHub:

ansible-galaxy collection install git+https://github.com/garutilorenzo/ansible-collection-elk

Whit Ansible and the collection installed we can setup our inventory file (hosts.ini):

[elasticsearch_master]
elk-ubuntu-0 ansible_host=192.168.25.110
elk-ubuntu-1 ansible_host=192.168.25.111
elk-ubuntu-2 ansible_host=192.168.25.112

[elasticsearch_data]
elk-ubuntu-3 ansible_host=192.168.25.113
elk-ubuntu-4 ansible_host=192.168.25.114
elk-ubuntu-5 ansible_host=192.168.25.115

[elasticsearch_ca]
elk-ubuntu-0 ansible_host=192.168.25.110

[kibana]
elk-ubuntu-1 ansible_host=192.168.25.111
elk-ubuntu-4 ansible_host=192.168.25.114

[logstash]
elk-ubuntu-2 ansible_host=192.168.25.112
elk-ubuntu-5 ansible_host=192.168.25.115

[elasticsearch:children]
elasticsearch_master
elasticsearch_data

and the vars.yml file:

---

disable_firewall: yes
disable_selinux: yes

elasticsearch_version: 8.3.3
kibana_version: 8.3.3
logstash_version: 8.3.3
beats_version: 8.3.3

elasticsearch_resolv_mode: hosts
elasticsearch_install_mode: local
elasticsearch_local_tar_path: ~/elk_tar_path
elasticsearch_monitoring_enabled: yes
elasticsearch_master_is_also_data_node: yes

kibana_install_mode: local
kibana_local_tar_path: ~/elk_tar_path
setup_kibana_dashboards: yes
kibana_url: http://elk-ubuntu-1:5601

heartbeat_number_of_shards: 3
heartbeat_number_of_replicas: 3

metricbeat_number_of_shards: 3
metricbeat_number_of_replicas: 3

filebeat_number_of_shards: 3
filebeat_number_of_replicas: 3

elasticsearch_hosts:
  - elk-ubuntu-0 
  - elk-ubuntu-1
  - elk-ubuntu-2 
  - elk-ubuntu-3 
  - elk-ubuntu-5 
  - elk-ubuntu-5

The final cluster will be made of:

6 elasticsearch nodes (the master nodes will be also data nodes)
2 kibana instances
2 logstash instances

and:

all VMs will be monitored by the Elastic Beats
all the index templates will have 3 shards and 3 replicas
Since we don't have any DNS available, Ansible will insert all the node names in the /etc/hosts file
firewall and selinux will be disabled

To preserve bandwidth we download elasticsearch and kibana on our Ansible machine:

mkdir -p ~/elk_tar_path # <- you can customize this path by changing elasticsearch_local_tar_path variable
curl  -o ~/elk_tar_path/kibana-8.3.3-linux-x86_64.tar.gz https://artifacts.elastic.co/downloads/kibana/kibana-8.3.3-linux-x86_64.tar.gz
curl  -o ~/elk_tar_path/elk_tar_path/elasticsearch-8.3.3-linux-x86_64.tar.gz https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.3.3-linux-x86_64.tar.gz

and we have to create the certificate directory, where elastic certificates will be stored:

mkdir -p ~/very_secure_dir # <- you can customize this path by changing elasticsearch_local_certs_dir variable

The final step before proceed with the installation is to create the site.yml file:

---
- hosts: elasticsearch 
  become: yes
  remote_user: vagrant
  collections:
    - garutilorenzo.ansible_collection_elk
  vars_files:
    - vars.yml

  tasks:
    - import_role:
        name: elasticsearch

- hosts: kibana 
  become: yes
  remote_user: vagrant
  collections:
    - garutilorenzo.ansible_collection_elk
  vars_files:
    - vars.yml

  tasks:
    - import_role:
        name: kibana

- hosts: logstash 
  become: yes
  remote_user: vagrant
  collections:
    - garutilorenzo.ansible_collection_elk
  vars_files:
    - vars.yml

  tasks:
    - import_role:
        name: logstash

- hosts: elasticsearch 
  become: yes
  remote_user: vagrant
  collections:
    - garutilorenzo.ansible_collection_elk
  vars_files:
    - vars.yml

  tasks:
    - import_role:
        name: beats

Deploy ELK with Ansible

We are finally ready to install the ELK stack with ansible, since we don't have any CA certificate we pass an extra variable generateca to our playbook:

export ANSIBLE_HOST_KEY_CHECKING=False # Ansible skip ssh-key validation
ansible-playbook site.yml -i hosts.ini -e "generateca=yes"

ansible-playbook site.yml -i hosts.ini

PLAY [elasticsearch] ***************************************************************************************************************************************

TASK [Gathering Facts] *************************************************************************************************************************************
ok: [elk-ubuntu-0]
ok: [elk-ubuntu-1]
ok: [elk-ubuntu-2]
ok: [elk-ubuntu-3]
ok: [elk-ubuntu-4]
ok: [elk-ubuntu-5]

TASK [garutilorenzo.ansible_collection_elk.elasticsearch : setup] ******************************************************************************************
ok: [elk-ubuntu-2]
ok: [elk-ubuntu-0]
ok: [elk-ubuntu-3]
ok: [elk-ubuntu-1]
ok: [elk-ubuntu-4]
ok: [elk-ubuntu-5]

TASK [garutilorenzo.ansible_collection_elk.elasticsearch : include_tasks] **********************************************************************************
included: /home/lorenzo/workspaces-local/ansible-test-collection/elk/collections/ansible_collections/garutilorenzo/ansible_collection_elk/roles/elasticsearch/tasks/preflight.yml for elk-ubuntu-0, elk-ubuntu-1, elk-ubuntu-2, elk-ubuntu-3, elk-ubuntu-4, elk-ubuntu-5

TASK [garutilorenzo.ansible_collection_elk.elasticsearch : Put SELinux in permissive mode, logging actions that would be blocked.] *************************
skipping: [elk-ubuntu-0]
skipping: [elk-ubuntu-1]
skipping: [elk-ubuntu-2]
skipping: [elk-ubuntu-3]
skipping: [elk-ubuntu-4]
skipping: [elk-ubuntu-5]

TASK [garutilorenzo.ansible_collection_elk.elasticsearch : Disable SELinux] ********************************************************************************
skipping: [elk-ubuntu-0]
skipping: [elk-ubuntu-1]
skipping: [elk-ubuntu-2]
skipping: [elk-ubuntu-3]
skipping: [elk-ubuntu-4]
skipping: [elk-ubuntu-5]

TASK [garutilorenzo.ansible_collection_elk.elasticsearch : disable firewalld] ******************************************************************************
skipping: [elk-ubuntu-0]
skipping: [elk-ubuntu-1]
skipping: [elk-ubuntu-2]
skipping: [elk-ubuntu-3]
skipping: [elk-ubuntu-4]
skipping: [elk-ubuntu-5]

TASK [garutilorenzo.ansible_collection_elk.elasticsearch : disable ufw] ************************************************************************************
changed: [elk-ubuntu-3]
changed: [elk-ubuntu-0]
changed: [elk-ubuntu-2]
changed: [elk-ubuntu-4]
changed: [elk-ubuntu-1]
changed: [elk-ubuntu-5]

[...]
[...]

This can take a while, but if everything goes well the final Ansible output will be:

[...]
[...]

TASK [garutilorenzo.ansible_collection_elk.beats : enable and start heartbeat] *****************************************************************************
changed: [elk-ubuntu-0]
changed: [elk-ubuntu-1]
changed: [elk-ubuntu-3]
changed: [elk-ubuntu-4]
changed: [elk-ubuntu-5]
changed: [elk-ubuntu-2]

RUNNING HANDLER [garutilorenzo.ansible_collection_elk.beats : reload systemd] ******************************************************************************
ok: [elk-ubuntu-4]
ok: [elk-ubuntu-0]
ok: [elk-ubuntu-3]
ok: [elk-ubuntu-1]
ok: [elk-ubuntu-2]
ok: [elk-ubuntu-5]

RUNNING HANDLER [garutilorenzo.ansible_collection_elk.beats : reload filebeat] *****************************************************************************
changed: [elk-ubuntu-0]
changed: [elk-ubuntu-4]
changed: [elk-ubuntu-1]
changed: [elk-ubuntu-2]
changed: [elk-ubuntu-5]
changed: [elk-ubuntu-3]

RUNNING HANDLER [garutilorenzo.ansible_collection_elk.beats : reload heartbeat] ****************************************************************************
changed: [elk-ubuntu-2]
changed: [elk-ubuntu-1]
changed: [elk-ubuntu-0]
changed: [elk-ubuntu-3]
changed: [elk-ubuntu-4]
changed: [elk-ubuntu-5]

RUNNING HANDLER [garutilorenzo.ansible_collection_elk.beats : reload metricbeat] ***************************************************************************
changed: [elk-ubuntu-1]
changed: [elk-ubuntu-4]
changed: [elk-ubuntu-0]
changed: [elk-ubuntu-3]
changed: [elk-ubuntu-5]
changed: [elk-ubuntu-2]

PLAY RECAP *************************************************************************************************************************************************
elk-ubuntu-0               : ok=119  changed=61   unreachable=0    failed=0    skipped=12   rescued=0    ignored=1   
elk-ubuntu-1               : ok=135  changed=68   unreachable=0    failed=0    skipped=15   rescued=0    ignored=1   
elk-ubuntu-2               : ok=136  changed=72   unreachable=0    failed=0    skipped=14   rescued=0    ignored=1   
elk-ubuntu-3               : ok=114  changed=58   unreachable=0    failed=0    skipped=12   rescued=0    ignored=1   
elk-ubuntu-4               : ok=135  changed=68   unreachable=0    failed=0    skipped=15   rescued=0    ignored=1   
elk-ubuntu-5               : ok=136  changed=72   unreachable=0    failed=0    skipped=14   rescued=0    ignored=1

Now we use Kibana to analyze our Elasticsearch data:

The password is the default bootstrap password changeme. You can customize this variable in the elasticsearch role, you have to modify the elasticsearch_bootstrap_password variable. Remember to set in the vars.yml file the same value form elasticsearch_password and elasticsearch_bootstrap_password.

This is the default Kibana dashboard:

Now we can Discover our data in Analytics -> Discover:

Here we can filter our data using the KQL syntax or use one of the many Kibana features.

We can also inspect our data using one of the many Dashboards provided by the various Beats (Analytics -> Dashboards). We can inspect for example the metricbeat system dashboard:

In Management -> Stack Management -> Index Management we can see all of our indexes. We have to enable the Include hidden indices because in our brand new cluster all the indexes are hidden (filebeat, heartbeat, metricbeat):

In Management -> Stack Monitoring we can enable the self monitoring feature:

And now we can inspect the cluster stats:

This is only a short introduction on the ELK Stack, you can read more on Elastic Docs. To get the best form the ELK stack read also Elastic Observability.

Clean up

When you have done you can finally destroy the cluster with:

vagrant destroy
    elk-ubuntu-5: Are you sure you want to destroy the 'elk-ubuntu-5' VM? [y/N] y
==> elk-ubuntu-5: Forcing shutdown of VM...
==> elk-ubuntu-5: Destroying VM and associated drives...
    elk-ubuntu-4: Are you sure you want to destroy the 'elk-ubuntu-4' VM? [y/N] y
==> elk-ubuntu-4: Forcing shutdown of VM...
==> elk-ubuntu-4: Destroying VM and associated drives...
    elk-ubuntu-3: Are you sure you want to destroy the 'elk-ubuntu-3' VM? [y/N] y
==> elk-ubuntu-3: Forcing shutdown of VM...
==> elk-ubuntu-3: Destroying VM and associated drives...
    elk-ubuntu-2: Are you sure you want to destroy the 'elk-ubuntu-2' VM? [y/N] y
==> elk-ubuntu-2: Forcing shutdown of VM...
==> elk-ubuntu-2: Destroying VM and associated drives...
    elk-ubuntu-1: Are you sure you want to destroy the 'elk-ubuntu-1' VM? [y/N] y
==> elk-ubuntu-1: Forcing shutdown of VM...
==> elk-ubuntu-1: Destroying VM and associated drives...
    elk-ubuntu-0: Are you sure you want to destroy the 'elk-ubuntu-0' VM? [y/N] y
==> elk-ubuntu-0: Forcing shutdown of VM...
==> elk-ubuntu-0: Destroying VM and associated drives...

Install and configure a high available Kubernetes cluster with Ansible

Lorenzo Garuti — Wed, 17 Aug 2022 14:52:08 +0000

This ansible role will install and configure a high available Kubernetes cluster. This repo automate the installation process of Kubernetes using kubeadm.

This repo is only a example on how to use Ansible automation to install and configure a Kubernetes cluster. For a production environment use Kubespray

Requirements

Install ansible, ipaddr and netaddr:

pip install -r requirements.txt

Download the role form GitHub:

ansible-galaxy install git+https://github.com/garutilorenzo/ansible-role-linux-kubernetes.git

Role Variables

This role accept this variables:

Var	Required	Default	Desc
`kubernetes_subnet`	`yes`	`192.168.25.0/24`	Subnet where Kubernetess will be deployed. If the VM or bare metal server has more than one interface, Ansible will filter the interface used by Kubernetes based on the interface subnet
`disable_firewall`	`no`	`no`	If set to yes Ansible will disable the firewall.
`kubernetes_version`	`no`	`1.24.3`	Kubernetes version to install
`kubernetes_cri`	`no`	`containerd`	Kubernetes CRI to install.
`kubernetes_cni`	`no`	`flannel`	Kubernetes CNI to install.
`kubernetes_dns_domain`	`no`	`cluster.local`	Kubernetes default DNS domain
`kubernetes_pod_subnet`	`no`	`10.244.0.0/16`	Kubernetes pod subnet
`kubernetes_service_subnet`	`no`	`10.96.0.0/12`	Kubernetes service subnet
`kubernetes_api_port`	`no`	`6443`	kubeapi listen port
`setup_vip`	`no`	`no`	Setup kubernetes VIP addres using kube-vip
`kubernetes_vip_ip`	`no`	`192.168.25.225`	Required if setup_vip is set to yes. Vip ip address for the control plane
`kubevip_version`	`no`	`v0.4.3`	kube-vip container version
`install_longhorn`	`no`	`no`	Install Longhorn, Cloud native distributed block storage for Kubernetes.
`longhorn_version`	`no`	`v1.3.1`	Longhorn release.
`install_nginx_ingress`	`no`	`no`	Install nginx ingress controller
`nginx_ingress_controller_version`	`no`	`controller-v1.3.0`	nginx ingress controller version
`nginx_ingress_controller_http_nodeport`	`no`	`30080`	NodePort used by nginx ingress controller for the incoming http traffic
`nginx_ingress_controller_https_nodeport`	`no`	`30443`	NodePort used by nginx ingress controller for the incoming https traffic
`enable_nginx_ingress_proxy_protocol`	`no`	`no`	Enable nginx ingress controller proxy protocol mode
`enable_nginx_real_ip`	`no`	`no`	Enable nginx ingress controller real-ip module
`nginx_ingress_real_ip_cidr`	`no`	`0.0.0.0/0`	Required if enable_nginx_real_ip is set to yes Trusted subnet to use with the real-ip module
`nginx_ingress_proxy_body_size`	`no`	`20m`	nginx ingress controller max proxy body size
`sans_base`	`no`	`[list of values, see defaults/main.yml]`	list of ip addresses or FQDN uset to sign the kube-api certificate

Extra Variables

This role accept an extra variable kubernetes_init_host. This variable is used when the cluster is bootstrapped for the first time. The value of this variable must be the hostname of one of the master nodes. When ansible will run on the matched host kubernetes will be initialized.

Cluster resource deployed

Whit this role Nginx ingress controller and Longhorn will be installed.

Nginx ingress controller

Nginx ingress controller is used as ingress controller.

The installation is the bare metal installation, the ingress controller then is exposed via a NodePort Service.
You can customize the ports exposed by the NodePort service, use Role Variables to change this values.

Longhorn

Longhorn is a lightweight, reliable, and powerful distributed block storage system for Kubernetes.

Longhorn implements distributed block storage using containers and microservices. Longhorn creates a dedicated storage controller for each block device volume and synchronously replicates the volume across multiple replicas stored on multiple nodes. The storage controller and replicas are themselves orchestrated using Kubernetes.

Vagrant

To test this role you can use Vagrant and Virtualbox to bring up a example infrastructure. Once you have downloaded this repo use Vagrant to start the virtual machines:

vagrant up

In the Vagrantfile you can inject your public ssh key directly in the authorized_keys of the vagrant user. You have to change the CHANGE_ME placeholder in the Vagrantfile. You can also adjust the number of the vm deployed by changing the NNODES variable (Default: 6)

Using this role

To use this role you follow the example in the examples/ dir. Adjust the hosts.ini file with your hosts and run the playbook:

lorenzo@mint-virtual:~$ ansible-playbook -i hosts-ubuntu.ini site.yml -e kubernetes_init_host=k8s-ubuntu-0

PLAY [kubemaster] ***************************************************************************************************************************************************

TASK [Gathering Facts] **********************************************************************************************************************************************
ok: [k8s-ubuntu-2]
ok: [k8s-ubuntu-1]
ok: [k8s-ubuntu-0]

TASK [ansible-role-kubernetes : include_tasks] **********************************************************************************************************************
included: /home/lorenzo/workspaces-local/ansible-role-kubernetes/tasks/setup_repo_Debian.yml for k8s-ubuntu-0, k8s-ubuntu-1, k8s-ubuntu-2 => (item=/home/lorenzo/workspaces-local/ansible-role-kubernetes/tasks/setup_repo_Debian.yml)

TASK [ansible-role-kubernetes : Install required system packages] ***************************************************************************************************
ok: [k8s-ubuntu-2]
ok: [k8s-ubuntu-1]
ok: [k8s-ubuntu-0]

TASK [ansible-role-kubernetes : Add Google GPG apt Key] *************************************************************************************************************
ok: [k8s-ubuntu-0]
ok: [k8s-ubuntu-1]
ok: [k8s-ubuntu-2]

TASK [ansible-role-kubernetes : Add K8s Repository] *****************************************************************************************************************
ok: [k8s-ubuntu-1]
ok: [k8s-ubuntu-2]
ok: [k8s-ubuntu-0]

TASK [ansible-role-kubernetes : Add Docker GPG apt Key] *************************************************************************************************************
ok: [k8s-ubuntu-1]
ok: [k8s-ubuntu-0]
ok: [k8s-ubuntu-2]

TASK [ansible-role-kubernetes : shell] ******************************************************************************************************************************
changed: [k8s-ubuntu-1]
changed: [k8s-ubuntu-2]
changed: [k8s-ubuntu-0]

TASK [ansible-role-kubernetes : Add Docker Repository] **************************************************************************************************************
ok: [k8s-ubuntu-0]
ok: [k8s-ubuntu-1]
ok: [k8s-ubuntu-2]

TASK [ansible-role-kubernetes : setup] ******************************************************************************************************************************
ok: [k8s-ubuntu-1]
ok: [k8s-ubuntu-0]
ok: [k8s-ubuntu-2]

TASK [ansible-role-kubernetes : include_tasks] **********************************************************************************************************************
included: /home/lorenzo/workspaces-local/ansible-role-kubernetes/tasks/preflight.yml for k8s-ubuntu-0, k8s-ubuntu-1, k8s-ubuntu-2

TASK [ansible-role-kubernetes : disable ufw] ************************************************************************************************************************
ok: [k8s-ubuntu-2]
ok: [k8s-ubuntu-0]
ok: [k8s-ubuntu-1]

TASK [ansible-role-kubernetes : Install iptables-legacy] ************************************************************************************************************
skipping: [k8s-ubuntu-0]
skipping: [k8s-ubuntu-1]
skipping: [k8s-ubuntu-2]

TASK [ansible-role-kubernetes : Remove zram-generator-defaults] *****************************************************************************************************
skipping: [k8s-ubuntu-0]
skipping: [k8s-ubuntu-1]
skipping: [k8s-ubuntu-2]

TASK [ansible-role-kubernetes : disable firewalld] ******************************************************************************************************************
skipping: [k8s-ubuntu-0]
skipping: [k8s-ubuntu-1]
skipping: [k8s-ubuntu-2]

TASK [ansible-role-kubernetes : Put SELinux in permissive mode, logging actions that would be blocked.] *************************************************************
skipping: [k8s-ubuntu-0]
skipping: [k8s-ubuntu-1]
skipping: [k8s-ubuntu-2]

TASK [ansible-role-kubernetes : Disable SELinux] ********************************************************************************************************************
skipping: [k8s-ubuntu-0]
skipping: [k8s-ubuntu-1]
skipping: [k8s-ubuntu-2]

TASK [ansible-role-kubernetes : Install openssl] ********************************************************************************************************************
ok: [k8s-ubuntu-2]
ok: [k8s-ubuntu-1]
ok: [k8s-ubuntu-0]

TASK [ansible-role-kubernetes : load overlay kernel module] *********************************************************************************************************
changed: [k8s-ubuntu-1]
changed: [k8s-ubuntu-0]
changed: [k8s-ubuntu-2]

TASK [ansible-role-kubernetes : load br_netfilter kernel module] ****************************************************************************************************
changed: [k8s-ubuntu-1]
changed: [k8s-ubuntu-0]
changed: [k8s-ubuntu-2]

[...]
[...]
[...]

TASK [ansible-role-kubernetes : Add KUBELET_ROOT_DIR env var] *******************************************************************************************************
skipping: [k8s-ubuntu-3]

TASK [ansible-role-kubernetes : Add KUBELET_ROOT_DIR env var, set value] ********************************************************************************************
skipping: [k8s-ubuntu-3]

TASK [ansible-role-kubernetes : Install longhorn] *******************************************************************************************************************
skipping: [k8s-ubuntu-3]

TASK [ansible-role-kubernetes : Install longhorn storageclass] ******************************************************************************************************
skipping: [k8s-ubuntu-3]

TASK [ansible-role-kubernetes : include_tasks] **********************************************************************************************************************
included: /home/lorenzo/workspaces-local/ansible-role-kubernetes/tasks/install_nginx_ingress.yml for k8s-ubuntu-3, k8s-ubuntu-4, k8s-ubuntu-5

TASK [ansible-role-kubernetes : Check if ingress-nginx is installed] ************************************************************************************************
changed: [k8s-ubuntu-3 -> k8s-ubuntu-0(192.168.25.110)]

TASK [ansible-role-kubernetes : Install ingress-nginx] **************************************************************************************************************
skipping: [k8s-ubuntu-3]

TASK [ansible-role-kubernetes : render nginx_ingress_config.yml] ****************************************************************************************************
skipping: [k8s-ubuntu-3]

TASK [ansible-role-kubernetes : Apply nginx ingress config] *********************************************************************************************************
skipping: [k8s-ubuntu-3]

PLAY RECAP **********************************************************************************************************************************************************
k8s-ubuntu-0               : ok=78   changed=24   unreachable=0    failed=0    skipped=25   rescued=0    ignored=3   
k8s-ubuntu-1               : ok=52   changed=12   unreachable=0    failed=0    skipped=30   rescued=0    ignored=1   
k8s-ubuntu-2               : ok=52   changed=12   unreachable=0    failed=0    skipped=30   rescued=0    ignored=1
k8s-ubuntu-3               : ok=58   changed=30   unreachable=0    failed=0    skipped=35   rescued=0    ignored=1   
k8s-ubuntu-4               : ok=52   changed=28   unreachable=0    failed=0    skipped=27   rescued=0    ignored=1   
k8s-ubuntu-5               : ok=52   changed=28   unreachable=0    failed=0    skipped=27   rescued=0    ignored=1

Now we have a Kubernetes cluster deployed in high available mode, we can check the status of the cluster:

root@k8s-ubuntu-0:~# kubectl get nodes
NAME           STATUS   ROLES           AGE    VERSION
k8s-ubuntu-0   Ready    control-plane   139m   v1.24.3
k8s-ubuntu-1   Ready    control-plane   136m   v1.24.3
k8s-ubuntu-2   Ready    control-plane   136m   v1.24.3
k8s-ubuntu-3   Ready    <none>          117m   v1.24.3
k8s-ubuntu-4   Ready    <none>          117m   v1.24.3
k8s-ubuntu-5   Ready    <none>          117m   v1.24.3

Check the pods status:

root@k8s-ubuntu-0:~# kubectl get pods --all-namespaces
NAMESPACE         NAME                                           READY   STATUS      RESTARTS       AGE
ingress-nginx     ingress-nginx-admission-create-tsc8p           0/1     Completed   0              135m
ingress-nginx     ingress-nginx-admission-patch-48tpn            0/1     Completed   0              135m
ingress-nginx     ingress-nginx-controller-6dc865cd86-kfq88      1/1     Running     0              135m
kube-flannel      kube-flannel-ds-fm4s6                          1/1     Running     0              117m
kube-flannel      kube-flannel-ds-hhvxx                          1/1     Running     0              117m
kube-flannel      kube-flannel-ds-ngdtc                          1/1     Running     0              117m
kube-flannel      kube-flannel-ds-q5ncb                          1/1     Running     0              136m
kube-flannel      kube-flannel-ds-vq4kk                          1/1     Running     0              139m
kube-flannel      kube-flannel-ds-zshpf                          1/1     Running     0              137m
kube-system       coredns-6d4b75cb6d-8dh9h                       1/1     Running     0              139m
kube-system       coredns-6d4b75cb6d-xq98k                       1/1     Running     0              139m
kube-system       etcd-k8s-ubuntu-0                              1/1     Running     0              139m
kube-system       etcd-k8s-ubuntu-1                              1/1     Running     0              136m
kube-system       etcd-k8s-ubuntu-2                              1/1     Running     0              136m
kube-system       kube-apiserver-k8s-ubuntu-0                    1/1     Running     0              139m
kube-system       kube-apiserver-k8s-ubuntu-1                    1/1     Running     0              135m
kube-system       kube-apiserver-k8s-ubuntu-2                    1/1     Running     0              136m
kube-system       kube-controller-manager-k8s-ubuntu-0           1/1     Running     0              139m
kube-system       kube-controller-manager-k8s-ubuntu-1           1/1     Running     0              136m
kube-system       kube-controller-manager-k8s-ubuntu-2           1/1     Running     0              135m
kube-system       kube-proxy-59jqx                               1/1     Running     0              136m
kube-system       kube-proxy-8mjwr                               1/1     Running     0              139m
kube-system       kube-proxy-8nhbw                               1/1     Running     0              117m
kube-system       kube-proxy-j2rrx                               1/1     Running     0              117m
kube-system       kube-proxy-qwd5r                               1/1     Running     0              117m
kube-system       kube-proxy-vcs7g                               1/1     Running     0              137m
kube-system       kube-scheduler-k8s-ubuntu-0                    1/1     Running     0              139m
kube-system       kube-scheduler-k8s-ubuntu-1                    1/1     Running     0              136m
kube-system       kube-scheduler-k8s-ubuntu-2                    1/1     Running     0              135m
kube-system       kube-vip-k8s-ubuntu-0                          1/1     Running     1 (136m ago)   139m
kube-system       kube-vip-k8s-ubuntu-1                          1/1     Running     0              136m
kube-system       kube-vip-k8s-ubuntu-2                          1/1     Running     0              136m
longhorn-system   csi-attacher-dcb85d774-jrggr                   1/1     Running     0              114m
longhorn-system   csi-attacher-dcb85d774-slhqt                   1/1     Running     0              114m
longhorn-system   csi-attacher-dcb85d774-xcbxn                   1/1     Running     0              114m
longhorn-system   csi-provisioner-5d8dd96b57-74x6h               1/1     Running     0              114m
longhorn-system   csi-provisioner-5d8dd96b57-kdzdf               1/1     Running     0              114m
longhorn-system   csi-provisioner-5d8dd96b57-xmpjf               1/1     Running     0              114m
longhorn-system   csi-resizer-7c5bb5fd65-4262v                   1/1     Running     0              114m
longhorn-system   csi-resizer-7c5bb5fd65-mfjgv                   1/1     Running     0              114m
longhorn-system   csi-resizer-7c5bb5fd65-qw944                   1/1     Running     0              114m
longhorn-system   csi-snapshotter-5586bc7c79-bs2xn               1/1     Running     0              114m
longhorn-system   csi-snapshotter-5586bc7c79-d927b               1/1     Running     0              114m
longhorn-system   csi-snapshotter-5586bc7c79-v99t6               1/1     Running     0              114m
longhorn-system   engine-image-ei-766a591b-hrs6g                 1/1     Running     0              114m
longhorn-system   engine-image-ei-766a591b-n9fsn                 1/1     Running     0              114m
longhorn-system   engine-image-ei-766a591b-vxhbb                 1/1     Running     0              114m
longhorn-system   instance-manager-e-3dba6914                    1/1     Running     0              114m
longhorn-system   instance-manager-e-7bd8b1ff                    1/1     Running     0              114m
longhorn-system   instance-manager-e-aca0fdc4                    1/1     Running     0              114m
longhorn-system   instance-manager-r-244c040c                    1/1     Running     0              114m
longhorn-system   instance-manager-r-39bd81b1                    1/1     Running     0              114m
longhorn-system   instance-manager-r-3b7f12b1                    1/1     Running     0              114m
longhorn-system   longhorn-admission-webhook-858d86b96b-j5rcv    1/1     Running     0              135m
longhorn-system   longhorn-admission-webhook-858d86b96b-lphkq    1/1     Running     0              135m
longhorn-system   longhorn-conversion-webhook-576b5c45c7-4p55x   1/1     Running     0              135m
longhorn-system   longhorn-conversion-webhook-576b5c45c7-lq686   1/1     Running     0              135m
longhorn-system   longhorn-csi-plugin-f7zmn                      2/2     Running     0              114m
longhorn-system   longhorn-csi-plugin-hs58p                      2/2     Running     0              114m
longhorn-system   longhorn-csi-plugin-wfpfs                      2/2     Running     0              114m
longhorn-system   longhorn-driver-deployer-96cf98c98-7hzft       1/1     Running     0              135m
longhorn-system   longhorn-manager-92xws                         1/1     Running     0              116m
longhorn-system   longhorn-manager-b6knm                         1/1     Running     0              116m
longhorn-system   longhorn-manager-tg2zc                         1/1     Running     0              116m
longhorn-system   longhorn-ui-86b56b95c8-ctbvf                   1/1     Running     0              135m

we can see, longhorn, nginx ingress and all the kube-system pods.

We can also inspect the service of the nginx ingress controller:

root@k8s-ubuntu-0:~# kubectl get svc -n ingress-nginx
NAME                                    TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
ingress-nginx-controller                NodePort    10.111.203.177   <none>        80:30080/TCP,443:30443/TCP   136m
ingress-nginx-controller-admission      ClusterIP   10.105.11.11     <none>        443/TCP                      136m

we can see the nginx ingress controller listening port, in this case the http port is 30080 and the https port is 30443. From an external machine we can test the ingress controller:

lorenzo@mint-virtual:~$ curl -v http://192.168.25.110:30080
*   Trying 192.168.25.110:30080...
* TCP_NODELAY set
* Connected to 192.168.25.110 (192.168.25.110) port 30080 (#0)
> GET / HTTP/1.1
> Host: 192.168.25.110:30080
> User-Agent: curl/7.68.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 404 Not Found
< Date: Wed, 17 Aug 2022 12:26:17 GMT
< Content-Type: text/html
< Content-Length: 146
< Connection: keep-alive
< 
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>
* Connection #0 to host 192.168.25.110 left intact

Congratulations

You have successfuly deployed a high available Kubernetes cluster, you are now ready to deploy your applications!

Deploy Kubernetes (K8s) on Amazon AWS using mixed on-demand and spot instances

Lorenzo Garuti — Thu, 14 Apr 2022 09:00:52 +0000

Deploy in a few minutes an high available Kubernetes cluster on Amazon AWS using mixed on-demand and spot instances.

Please note, this is only an example on how to Deploy a Kubernetes cluster. For a production environment you should use EKS or ECS.

The scope of this repo is to show all the AWS components needed to deploy a high available K8s cluster.

Requirements
Infrastructure overview
Before you start
Project setup
AWS provider setup
Pre flight checklist
Deploy
Deploy a sample stack
Clean up
Todo

Requirements

Terraform - Terraform is an open-source infrastructure as code software tool that provides a consistent CLI workflow to manage hundreds of cloud services. Terraform codifies cloud APIs into declarative configuration files.
Amazon AWS Account - Amazon AWS account with billing enabled
kubectl - The Kubernetes command-line tool (optional)
aws cli optional

You need also:

one VPC with private and public subnets
one ssh key already uploaded on your AWS account
one bastion host to reach all the private EC2 instances

For VPC and bastion host you can refer to this repository.

Infrastructure overview

The final infrastructure will be made by:

two autoscaling group, one for the kubernetes master nodes and one for the worker nodes
two launch template, used by the asg
one internal load balancer (L4) that will route traffic to Kubernetes servers
one external load balancer (L7) that will route traffic to Kubernetes workers
one security group that will allow traffic from the VPC subnet CIDR on all the k8s ports (kube api, nginx ingress node port etc)
one security group that will allow traffic from all the internet into the public load balancer (L7) on port 80 and 443
one S3 bucket, used to store the cluster join certificates
one IAM role, used to allow all the EC2 instances in the cluster to write on the S3 bucket, used to share the join certificates
one certificate used by the public LB, stored on AWS ACM. The certificate is a self signed certificate.

Kubernetes setup

The installation of K8s id done by kubeadm. In this installation Containerd is used as CRI and flannel is used as CNI.

You can optionally install Nginx ingress controller and Longhorn.

To install Nginx ingress set the variable install_nginx_ingress to yes (default no). To install longhorn set the variable install_longhorn to yes (default no). NOTE if you don't install the nginx ingress, the public Load Balancer and the SSL certificate won't be deployed.

In this installation is used a S3 bucket to store the join certificate/token. At the first startup of the instance, if the cluster does not exist, the S3 bucket is used to get the join certificates/token.

Before you start

Note that this tutorial uses AWS resources that are outside the AWS free tier, so be careful!

Project setup

Clone this repo and go in the example/ directory:

git clone https://github.com/garutilorenzo/k8s-aws-terraform-cluster
cd k8s-aws-terraform-cluster/example/

Now you have to edit the main.tf file and you have to create the terraform.tfvars file. For more detail see AWS provider setup and Pre flight checklist.

Or if you prefer you can create an new empty directory in your workspace and create this three files:

terraform.tfvars
main.tf
provider.tf

The main.tf file will look like:

variable "AWS_ACCESS_KEY" {

}

variable "AWS_SECRET_KEY" {

}

variable "environment" {
  default = "staging"
}

variable "AWS_REGION" {
  default = "<YOUR_REGION>"
}

module "k8s-cluster" {
  ssk_key_pair_name      = "<SSH_KEY_NAME>"
  uuid                   = "<GENERATE_UUID>"
  environment            = var.environment
  vpc_id                 = "<VPC_ID>"
  vpc_private_subnets    = "<PRIVATE_SUBNET_LIST>"
  vpc_public_subnets     = "<PUBLIC_SUBNET_LIST>"
  vpc_subnet_cidr        = "<SUBNET_CIDR>"
  PATH_TO_PUBLIC_LB_CERT = "<PAHT_TO_PUBLIC_LB_CERT>"
  PATH_TO_PUBLIC_LB_KEY  = "<PAHT_TO_PRIVATE_LB_CERT>"
  install_nginx_ingress  = true
  source                 = "github.com/garutilorenzo/k8s-aws-terraform-cluster"
}

output "k8s_dns_name" {
  value = module.k8s-cluster.k8s_dns_name
}

output "k8s_server_private_ips" {
  value = module.k8s-cluster.k8s_server_private_ips
}

output "k8s_workers_private_ips" {
  value = module.k8s-cluster.k8s_workers_private_ips
}

For all the possible variables see Pre flight checklist

The provider.tf will look like:

provider "aws" {
  region     = var.AWS_REGION
  access_key = var.AWS_ACCESS_KEY
  secret_key = var.AWS_SECRET_KEY
}

The terraform.tfvars will look like:

AWS_ACCESS_KEY = "xxxxxxxxxxxxxxxxx"
AWS_SECRET_KEY = "xxxxxxxxxxxxxxxxx"

Now we can init terraform with:

terraform init

Initializing modules...
- k8s-cluster in ..

Initializing the backend...

Initializing provider plugins...
- Finding latest version of hashicorp/template...
- Finding latest version of hashicorp/aws...
- Installing hashicorp/template v2.2.0...
- Installed hashicorp/template v2.2.0 (signed by HashiCorp)
- Installing hashicorp/aws v4.9.0...
- Installed hashicorp/aws v4.9.0 (signed by HashiCorp)

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

Generate self signed SSL certificate for the public LB (L7)

NOTE If you already own a valid certificate skip this step and set the correct values for the variables: PATH_TO_PUBLIC_LB_CERT and PATH_TO_PUBLIC_LB_KEY

We need to generate the certificates (sel signed) for our public load balancer (Layer 7). To do this we need openssl, open a terminal and follow this step:

Generate the key:

openssl genrsa 2048 > privatekey.pem
Generating RSA private key, 2048 bit long modulus (2 primes)
.......+++++
...............+++++
e is 65537 (0x010001)

Generate the a new certificate request:

openssl req -new -key privatekey.pem -out csr.pem
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:IT
State or Province Name (full name) [Some-State]:Italy
Locality Name (eg, city) []:Brescia
Organization Name (eg, company) [Internet Widgits Pty Ltd]:GL Ltd
Organizational Unit Name (eg, section) []:IT
Common Name (e.g. server FQDN or YOUR name) []:testlb.domainexample.com
Email Address []:email@you.com

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:

Generate the public CRT:

openssl x509 -req -days 365 -in csr.pem -signkey privatekey.pem -out public.crt
Signature ok
subject=C = IT, ST = Italy, L = Brescia, O = GL Ltd, OU = IT, CN = testlb.domainexample.com, emailAddress = email@you.com
Getting Private key

This is the final result:

ls

csr.pem  privatekey.pem  public.crt

Now set the variables:

PATH_TO_PUBLIC_LB_CERT: ~/full_path/public.crt
PATH_TO_PUBLIC_LB_KEY: ~/full_path/privatekey.pem

AWS provider setup

Follow the prerequisites step on this link.
In your workspace folder or in the examples directory of this repo create a file named terraform.tfvars:

AWS_ACCESS_KEY = "xxxxxxxxxxxxxxxxx"
AWS_SECRET_KEY = "xxxxxxxxxxxxxxxxx"

Pre flight checklist

Once you have created the terraform.tfvars file edit the main.tf file (always in the example/ directory) and set the following variables:

Var	Required	Desc
`region`	`yes`	set the correct OCI region based on your needs
`environment`	`yes`	Current work environment (Example: staging/dev/prod). This value is used for tag all the deployed resources
`uuid`	`yes`	UUID used to tag all resources
`ssk_key_pair_name`	`yes`	Name of the ssh key to use
`vpc_id`	`yes`	ID of the VPC to use. You can find your vpc_id in your AWS console (Example: vpc-xxxxx)
`vpc_private_subnets`	`yes`	List of private subnets to use. This subnets are used for the public LB You can find the list of your vpc subnets in your AWS console (Example: subnet-xxxxxx)
`vpc_public_subnets`	`yes`	List of public subnets to use. This subnets are used for the EC2 instances and the private LB. You can find the list of your vpc subnets in your AWS console (Example: subnet-xxxxxx)
`vpc_subnet_cidr`	`yes`	Your subnet CIDR. You can find the VPC subnet CIDR in your AWS console (Example: 172.31.0.0/16)
`PATH_TO_PUBLIC_LB_CERT`	`yes`	Path to the public LB certificate. See how to generate the certificate
`PATH_TO_PUBLIC_LB_KEY`	`yes`	Path to the public LB key. See how to generate the key
`ec2_associate_public_ip_address`	`no`	Assign or not a pulic ip to the EC2 instances. Default: false
`s3_bucket_name`	`no`	S3 bucket name used for sharing the kubernetes token used for joining the cluster. Default: my-very-secure-k8s-bucket
`instance_profile_name`	`no`	Instance profile name. Default: K8sInstanceProfile
`iam_role_name`	`no`	IAM role name. Default: K8sIamRole
`ami`	`no`	Ami image name. Default: ami-0a2616929f1e63d91, ubuntu 20.04
`default_instance_type`	`no`	Default instance type used by the Launch template. Default: t3.large
`instance_types`	`no`	Array of instances used by the ASG. Dfault: { asg_instance_type_1 = "t3.large", asg_instance_type_3 = "m4.large", asg_instance_type_4 = "t3a.large" }
`k8s_master_template_prefix`	`no`	Template prefix for the master instances. Default: k8s_master_tpl
`k8s_worker_template_prefix`	`no`	Template prefix for the worker instances. Default: k8s_worker_tpl
`k8s_version`	`no`	Kubernetes version to install
`k8s_pod_subnet`	`no`	Kubernetes pod subnet managed by the CNI (Flannel). Default: 10.244.0.0/16
`k8s_service_subnet`	`no`	Kubernetes pod service managed by the CNI (Flannel). Default: 10.96.0.0/12
`k8s_dns_domain`	`no`	Internal kubernetes DNS domain. Default: cluster.local
`kube_api_port`	`no`	Kubernetes api port. Default: 6443
`k8s_internal_lb_name`	`no`	Internal load balancer name. Default: k8s-server-tcp-lb
`k8s_server_desired_capacity`	`no`	Desired number of k8s servers. Default 3
`k8s_server_min_capacity`	`no`	Min number of k8s servers: Default 4
`k8s_server_max_capacity`	`no`	Max number of k8s servers: Default 3
`k8s_worker_desired_capacity`	`no`	Desired number of k8s workers. Default 3
`k8s_worker_min_capacity`	`no`	Min number of k8s workers: Default 4
`k8s_worker_max_capacity`	`no`	Max number of k8s workers: Default 3
`cluster_name`	`no`	Kubernetes cluster name. Default: k8s-cluster
`install_longhorn`	`no`	Install or not longhorn. Default: false
`longhorn_release`	`no`	longhorn release. Default: v1.2.3
`install_nginx_ingress`	`no`	Install or not nginx ingress controller. Default: false
`k8s_ext_lb_name`	`no`	External load balancer name. Default: k8s-ext-lb
`extlb_listener_http_port`	`no`	HTTP nodeport where nginx ingress controller will listen. Default: 30080
`extlb_listener_https_port`	`no`	HTTPS nodeport where nginx ingress controller will listen. Default 30443
`extlb_http_port`	`no`	External LB HTTP listen port. Default: 80
`extlb_https_port`	`no`	External LB HTTPS listen port. Default 443

Deploy

We are now ready to deploy our infrastructure. First we ask terraform to plan the execution with:

terraform plan

...
...
      + name                   = "k8s-sg"
      + name_prefix            = (known after apply)
      + owner_id               = (known after apply)
      + revoke_rules_on_delete = false
      + tags                   = {
          + "Name"        = "sg-k8s-cluster-staging"
          + "environment" = "staging"
          + "provisioner" = "terraform"
          + "scope"       = "k8s-cluster"
          + "uuid"        = "xxxxx-xxxxx-xxxx-xxxxxx-xxxxxx"
        }
      + tags_all               = {
          + "Name"        = "sg-k8s-cluster-staging"
          + "environment" = "staging"
          + "provisioner" = "terraform"
          + "scope"       = "k8s-cluster"
          + "uuid"        = "xxxxx-xxxxx-xxxx-xxxxxx-xxxxxx"
        }
      + vpc_id                 = "vpc-xxxxxx"
    }

Plan: 25 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + k8s_dns_name            = (known after apply)
  + k8s_server_private_ips  = [
      + (known after apply),
    ]
  + k8s_workers_private_ips = [
      + (known after apply),
    ]

Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you run "terraform apply" now.

now we can deploy our resources with:

terraform apply

...

      + tags_all               = {
          + "Name"        = "sg-k8s-cluster-staging"
          + "environment" = "staging"
          + "provisioner" = "terraform"
          + "scope"       = "k8s-cluster"
          + "uuid"        = "xxxxx-xxxxx-xxxx-xxxxxx-xxxxxx"
        }
      + vpc_id                 = "vpc-xxxxxxxx"
    }

Plan: 25 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + k8s_dns_name            = (known after apply)
  + k8s_server_private_ips  = [
      + (known after apply),
    ]
  + k8s_workers_private_ips = [
      + (known after apply),
    ]

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

...
...

Apply complete! Resources: 25 added, 0 changed, 0 destroyed.

Outputs:

k8s_dns_name = "k8s-ext-<REDACTED>.elb.amazonaws.com"
k8s_server_private_ips = [
  tolist([
    "172.x.x.x",
    "172.x.x.x",
    "172.x.x.x",
  ]),
]
k8s_workers_private_ips = [
  tolist([
    "172.x.x.x",
    "172.x.x.x",
    "172.x.x.x",
  ]),
]

Now on one master node you can check the status of the cluster with:

ssh -j bastion@<BASTION_IP> ubuntu@172.x.x.x

Welcome to Ubuntu 20.04.4 LTS (GNU/Linux 5.13.0-1021-aws x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

  System information as of Wed Apr 13 12:41:52 UTC 2022

  System load:  0.52               Processes:             157
  Usage of /:   17.8% of 19.32GB   Users logged in:       0
  Memory usage: 11%                IPv4 address for cni0: 10.244.0.1
  Swap usage:   0%                 IPv4 address for ens3: 172.68.4.237


0 updates can be applied immediately.


Last login: Wed Apr 13 12:40:32 2022 from 172.68.0.6
ubuntu@i-04d089ed896cfafe1:~$ sudo su -

root@i-04d089ed896cfafe1:~# kubectl get nodes
NAME                  STATUS   ROLES                  AGE     VERSION
i-0033b408f7a1d55f3   Ready    control-plane,master   3m33s   v1.23.5
i-0121c2149821379cc   Ready    <none>                 4m16s   v1.23.5
i-04d089ed896cfafe1   Ready    control-plane,master   4m53s   v1.23.5
i-072bf7de2e94e6f2d   Ready    <none>                 4m15s   v1.23.5
i-09b23242f40eabcca   Ready    control-plane,master   3m56s   v1.23.5
i-0cb1e2e7784768b22   Ready    <none>                 3m57s   v1.23.5

root@i-04d089ed896cfafe1:~# kubectl get ns
NAME              STATUS   AGE
default           Active   5m18s
ingress-nginx     Active   111s # <- ingress controller ns
kube-node-lease   Active   5m19s
kube-public       Active   5m19s
kube-system       Active   5m19s
longhorn-system   Active   109s  # <- longhorn ns

root@i-04d089ed896cfafe1:~# kubectl get pods --all-namespaces
NAMESPACE         NAME                                          READY   STATUS      RESTARTS        AGE
ingress-nginx     ingress-nginx-admission-create-v2fpx          0/1     Completed   0               2m33s
ingress-nginx     ingress-nginx-admission-patch-54d9f           0/1     Completed   0               2m33s
ingress-nginx     ingress-nginx-controller-7fc8d55869-cxv87     1/1     Running     0               2m33s
kube-system       coredns-64897985d-8cg8g                       1/1     Running     0               5m46s
kube-system       coredns-64897985d-9v2r8                       1/1     Running     0               5m46s
kube-system       etcd-i-0033b408f7a1d55f3                      1/1     Running     0               4m33s
kube-system       etcd-i-04d089ed896cfafe1                      1/1     Running     0               5m42s
kube-system       etcd-i-09b23242f40eabcca                      1/1     Running     0               5m
kube-system       kube-apiserver-i-0033b408f7a1d55f3            1/1     Running     1 (4m30s ago)   4m30s
kube-system       kube-apiserver-i-04d089ed896cfafe1            1/1     Running     0               5m46s
kube-system       kube-apiserver-i-09b23242f40eabcca            1/1     Running     0               5m1s
kube-system       kube-controller-manager-i-0033b408f7a1d55f3   1/1     Running     0               4m36s
kube-system       kube-controller-manager-i-04d089ed896cfafe1   1/1     Running     1 (4m50s ago)   5m49s
kube-system       kube-controller-manager-i-09b23242f40eabcca   1/1     Running     0               5m1s
kube-system       kube-flannel-ds-7c65s                         1/1     Running     0               5m2s
kube-system       kube-flannel-ds-bb842                         1/1     Running     0               4m10s
kube-system       kube-flannel-ds-q27gs                         1/1     Running     0               5m21s
kube-system       kube-flannel-ds-sww7p                         1/1     Running     0               5m3s
kube-system       kube-flannel-ds-z8h5p                         1/1     Running     0               5m38s
kube-system       kube-flannel-ds-zrwdq                         1/1     Running     0               5m22s
kube-system       kube-proxy-6rbks                              1/1     Running     0               5m2s
kube-system       kube-proxy-9npgg                              1/1     Running     0               5m21s
kube-system       kube-proxy-px6br                              1/1     Running     0               5m3s
kube-system       kube-proxy-q9889                              1/1     Running     0               4m10s
kube-system       kube-proxy-s5qnv                              1/1     Running     0               5m22s
kube-system       kube-proxy-tng4x                              1/1     Running     0               5m46s
kube-system       kube-scheduler-i-0033b408f7a1d55f3            1/1     Running     0               4m27s
kube-system       kube-scheduler-i-04d089ed896cfafe1            1/1     Running     1 (4m50s ago)   5m58s
kube-system       kube-scheduler-i-09b23242f40eabcca            1/1     Running     0               5m1s
longhorn-system   csi-attacher-6454556647-767p2                 1/1     Running     0               115s
longhorn-system   csi-attacher-6454556647-hz8lj                 1/1     Running     0               115s
longhorn-system   csi-attacher-6454556647-z5ftg                 1/1     Running     0               115s
longhorn-system   csi-provisioner-869bdc4b79-2v4wx              1/1     Running     0               115s
longhorn-system   csi-provisioner-869bdc4b79-4xcv4              1/1     Running     0               114s
longhorn-system   csi-provisioner-869bdc4b79-9q95d              1/1     Running     0               114s
longhorn-system   csi-resizer-6d8cf5f99f-dwdrq                  1/1     Running     0               114s
longhorn-system   csi-resizer-6d8cf5f99f-klvcr                  1/1     Running     0               114s
longhorn-system   csi-resizer-6d8cf5f99f-ptpzb                  1/1     Running     0               114s
longhorn-system   csi-snapshotter-588457fcdf-dlkdq              1/1     Running     0               113s
longhorn-system   csi-snapshotter-588457fcdf-p2c7c              1/1     Running     0               113s
longhorn-system   csi-snapshotter-588457fcdf-p5smn              1/1     Running     0               113s
longhorn-system   engine-image-ei-fa2dfbf0-bkwhx                1/1     Running     0               2m7s
longhorn-system   engine-image-ei-fa2dfbf0-cqq9n                1/1     Running     0               2m8s
longhorn-system   engine-image-ei-fa2dfbf0-lhjjc                1/1     Running     0               2m7s
longhorn-system   instance-manager-e-542b1382                   1/1     Running     0               119s
longhorn-system   instance-manager-e-a5e124bb                   1/1     Running     0               2m4s
longhorn-system   instance-manager-e-acb2a517                   1/1     Running     0               2m7s
longhorn-system   instance-manager-r-11ab6af6                   1/1     Running     0               119s
longhorn-system   instance-manager-r-5b82fba2                   1/1     Running     0               2m4s
longhorn-system   instance-manager-r-c2561fa0                   1/1     Running     0               2m6s
longhorn-system   longhorn-csi-plugin-4br28                     2/2     Running     0               113s
longhorn-system   longhorn-csi-plugin-8gdxf                     2/2     Running     0               113s
longhorn-system   longhorn-csi-plugin-wc6tt                     2/2     Running     0               113s
longhorn-system   longhorn-driver-deployer-7dddcdd5bb-zjh4k     1/1     Running     0               2m31s
longhorn-system   longhorn-manager-cbsh7                        1/1     Running     0               2m31s
longhorn-system   longhorn-manager-d2t75                        1/1     Running     1 (2m9s ago)    2m31s
longhorn-system   longhorn-manager-xqlfv                        1/1     Running     1 (2m9s ago)    2m31s
longhorn-system   longhorn-ui-7648d6cd69-tc6b9                  1/1     Running     0               2m31s

Public LB check

We can now test the public load balancer, nginx ingress controller and the security group ingress rules. On your local PC run:

curl -k -v https://k8s-ext-<REDACTED>.elb.amazonaws.com/
*   Trying 34.x.x.x:443...
* TCP_NODELAY set
* Connected to k8s-ext-<REDACTED>.elb.amazonaws.com (34.x.x.x) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=IT; ST=Italy; L=Brescia; O=GL Ltd; OU=IT; CN=testlb.domainexample.com; emailAddress=email@you.com
*  start date: Apr 11 08:20:12 2022 GMT
*  expire date: Apr 11 08:20:12 2023 GMT
*  issuer: C=IT; ST=Italy; L=Brescia; O=GL Ltd; OU=IT; CN=testlb.domainexample.com; emailAddress=email@you.com
*  SSL certificate verify result: self signed certificate (18), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55c6560cde10)
> GET / HTTP/2
> Host: k8s-ext-<REDACTED>.elb.amazonaws.com
> user-agent: curl/7.68.0
> accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 404 
< date: Tue, 12 Apr 2022 10:08:18 GMT
< content-type: text/html
< content-length: 146
< strict-transport-security: max-age=15724800; includeSubDomains
< 
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>
* Connection #0 to host k8s-ext-<REDACTED>.elb.amazonaws.com left intact

404 is a correct response since the cluster is empty.

Deploy a sample stack

We use the same stack used in this repository.
This stack need longhorn and nginx ingress.

To test all the components of the cluster we can deploy a sample stack. The stack is composed by the following components:

MariaDB
Nginx
Wordpress

Each component is made by: one deployment and one service.
Wordpress and nginx share the same persistent volume (ReadWriteMany with longhorn storage class). The nginx configuration is stored in four ConfigMaps and the nginx service is exposed by the nginx ingress controller.

Deploy the resources with:

kubectl apply -f https://raw.githubusercontent.com/garutilorenzo/k3s-oci-cluster/master/deployments/mariadb/all-resources.yml
kubectl apply -f https://raw.githubusercontent.com/garutilorenzo/k3s-oci-cluster/master/deployments/nginx/all-resources.yml
kubectl apply -f https://raw.githubusercontent.com/garutilorenzo/k3s-oci-cluster/master/deployments/wordpress/all-resources.yml

NOTE to install WP and reach the wp-admin path you have to edit the nginx deployment and change this line:

 env:
  - name: SECURE_SUBNET
    value: 8.8.8.8/32 # change-me

and set your public ip address.

To check the status:

root@i-04d089ed896cfafe1:~# kubectl get pods -o wide
NAME                         READY   STATUS    RESTARTS   AGE     IP            NODE                  NOMINATED NODE   READINESS GATES
mariadb-6cbf998bd6-s98nh     1/1     Running   0          2m21s   10.244.2.13   i-072bf7de2e94e6f2d   <none>           <none>
nginx-68b4dfbcb6-s6zfh       1/1     Running   0          19s     10.244.1.12   i-0121c2149821379cc   <none>           <none>
wordpress-558948b576-jgvm2   1/1     Running   0          71s     10.244.3.14   i-0cb1e2e7784768b22   <none>           <none>

root@i-04d089ed896cfafe1:~# kubectl get deployments
NAME        READY   UP-TO-DATE   AVAILABLE   AGE
mariadb     1/1     1            1           2m32s
nginx       1/1     1            1           30s
wordpress   1/1     1            1           82s

root@i-04d089ed896cfafe1:~# kubectl get svc
NAME            TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
kubernetes      ClusterIP   10.96.0.1       <none>        443/TCP    14m
mariadb-svc     ClusterIP   10.108.78.60    <none>        3306/TCP   2m43s
nginx-svc       ClusterIP   10.103.145.57   <none>        80/TCP     41s
wordpress-svc   ClusterIP   10.103.49.246   <none>        9000/TCP   93s

Now you are ready to setup WP, open the LB public ip and follow the wizard. NOTE nginx and the Kubernetes Ingress rule are configured without virthual host/server name.

To clean the deployed resources:

kubectl delete -f https://raw.githubusercontent.com/garutilorenzo/k3s-oci-cluster/master/deployments/mariadb/all-resources.yml
kubectl delete -f https://raw.githubusercontent.com/garutilorenzo/k3s-oci-cluster/master/deployments/nginx/all-resources.yml
kubectl delete -f https://raw.githubusercontent.com/garutilorenzo/k3s-oci-cluster/master/deployments/wordpress/all-resources.yml

Clean up

Before destroy all the infrastructure DELETE all the object in the S3 bucket.

terraform destroy

Deploy a Kubernetes cluster for free, using K3s and Oracle always free resources

Lorenzo Garuti — Tue, 22 Feb 2022 10:54:42 +0000

Deploy a Kubernetes cluster for free, using K3s and Oracle always free resources.

Important notes
Requirements
Example RSA key generation
Project setup
Oracle provider setup
Pre flight checklist
Notes about OCI always free resources
Notes about K3s
Infrastructure overview
Cluster resource deployed
Deploy
Deploy a sample stack
Clean up

Note choose a region with enough ARM capacity

Important notes

This is tutorial shows only how to use terraform with the Oracle Cloud infrastructure and use only the always free resources. This examples are not for a production environment.
At the end of your trial period (30 days). All the paid resources deployed will be stopped/terminated
At the end of your trial period (30 days), if you have a running compute instance it will be stopped/hibernated

Requirements

To use this tutorial you will need:

an Oracle Cloud account. You can register here

Once you get the account, follow the Before you begin and 1. Prepare step in this document.

you need also:

Terraform - Terraform is an open-source infrastructure as code software tool that provides a consistent CLI workflow to manage hundreds of cloud services. Terraform codifies cloud APIs into declarative configuration files.
kubectl - The Kubernetes command-line tool (optional)
oci cli - Oracle command line interface (optional)

Example RSA key generation

To use terraform with the Oracle Cloud infrastructure you need to generate an RSA key. Generate the rsa key with:

openssl genrsa -out ~/.oci/<your_name>-oracle-cloud.pem 4096
chmod 600 ~/.oci/<your_name>-oracle-cloud.pem
openssl rsa -pubout -in ~/.oci/<your_name>-oracle-cloud.pem -out ~/.oci/<your_name>-oracle-cloud_public.pem

replace with your name or a string you prefer.

NOTE ~/.oci/-oracle-cloud_public.pem this string will be used on the terraform.tfvars used by the Oracle provider plugin, so please take note of this string.

Project setup

You can clone this repository and work in the example/ directory. You have to edit the main.tf file and you have to create the terraform.tfvars file. For more detail see Oracle provider setup and Pre flight checklist.

Or if you prefer you can create an new empty directory in your workspace and create this three files:

terraform.tfvars - More details in Oracle provider setup
main.tf
provider.tf

The main.tf file will look like:

variable "compartment_ocid" {

}

variable "tenancy_ocid" {

}

variable "user_ocid" {

}

variable "fingerprint" {

}

variable "private_key_path" {

}

variable "region" {
  default = "<change_me>"
}

module "k3s_cluster" {
  region              = var.region
  availability_domain = "<change_me>"
  compartment_ocid    = var.compartment_ocid
  my_public_ip_cidr   = "<change_me>"
  cluster_name        = "<change_me>"
  environment         = "staging"
  k3s_token           = "<change_me>"
  source              = "github.com/garutilorenzo/k3s-oci-cluster"
}

output "k3s_servers_ips" {
  value = module.k3s_cluster.k3s_servers_ips
}

output "k3s_workers_ips" {
  value = module.k3s_cluster.k3s_workers_ips
}

output "public_lb_ip" {
  value = module.k3s_cluster.public_lb_ip
}

For all the possible variables see Pre flight checklist

The provider.tf will look like:

provider "oci" {
  tenancy_ocid     = var.tenancy_ocid
  user_ocid        = var.user_ocid
  private_key_path = var.private_key_path
  fingerprint      = var.fingerprint
  region           = var.region
}

Now we can init terraform with:

terraform init

terraform init
Initializing modules...
Downloading git::https://github.com/garutilorenzo/k3s-oci-cluster.git for k3s_cluster...
- k3s_cluster in .terraform/modules/k3s_cluster

Initializing the backend...

Initializing provider plugins...
- Reusing previous version of hashicorp/oci from the dependency lock file
- Reusing previous version of hashicorp/template from the dependency lock file
- Using previously-installed hashicorp/template v2.2.0
- Using previously-installed hashicorp/oci v4.64.0

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

Generate sel signed SSL certificate for the public LB (L7)

NOTE If you already own a valid certificate skip this step and set the correct values for the variables: PATH_TO_PUBLIC_LB_CERT and PATH_TO_PUBLIC_LB_KEY

We need to generate the certificates (sel signed) for our public load balancer (Layer 7). To do this we need openssl, open a terminal and follow this step:

Generate the key:

openssl genrsa 2048 > privatekey.pem
Generating RSA private key, 2048 bit long modulus (2 primes)
.......+++++
...............+++++
e is 65537 (0x010001)

Generate the a new certificate request:

openssl req -new -key privatekey.pem -out csr.pem
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:IT
State or Province Name (full name) [Some-State]:Italy
Locality Name (eg, city) []:Brescia
Organization Name (eg, company) [Internet Widgits Pty Ltd]:GL Ltd
Organizational Unit Name (eg, section) []:IT
Common Name (e.g. server FQDN or YOUR name) []:testlb.domainexample.com
Email Address []:email@you.com

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:

Generate the public CRT:

openssl x509 -req -days 365 -in csr.pem -signkey privatekey.pem -out public.crt
Signature ok
subject=C = IT, ST = Italy, L = Brescia, O = GL Ltd, OU = IT, CN = testlb.domainexample.com, emailAddress = email@you.com
Getting Private key

This is the final result:

ls

csr.pem  privatekey.pem  public.crt

Now set the variables:

PATH_TO_PUBLIC_LB_CERT: ~/full_path/public.crt
PATH_TO_PUBLIC_LB_KEY: ~/full_path/privatekey.pem

Oracle provider setup

This is an example of the terraform.tfvars file:

fingerprint      = "<rsa_key_fingerprint>"
private_key_path = "~/.oci/<your_name>-oracle-cloud_public.pem"
user_ocid        = "<user_ocid>"
tenancy_ocid     = "<tenency_ocid>"
compartment_ocid = "<compartment_ocid>"

To find your tenency_ocid in the Ocacle Cloud console go to: Governance and Administration > Tenency details, then copy the OCID.

To find you user_ocid in the Ocacle Cloud console go to User setting (click on the icon in the top right corner, then click on User settings), click your username and then copy the OCID

The compartment_ocid is the same as tenency_ocid.

The fingerprint is the fingerprint of your RSA key, you can find this vale under User setting > API Keys

Pre flight checklist

Once you have created the terraform.tfvars file edit the main.tf file (always in the example/ directory) and set the following variables:

Var	Required	Desc
`region`	`yes`	set the correct OCI region based on your needs
`availability_domain`	`yes`	Set the correct availability domain. See how to find the availability domain
`compartment_ocid`	`yes`	Set the correct compartment ocid. See how to find the compartment ocid
`cluster_name`	`yes`	the name of your K3s cluster. Default: k3s-cluster
`k3s_token`	`yes`	The token of your K3s cluster. How to generate a random token
`my_public_ip_cidr`	`yes`	your public ip in cidr format (Example: 195.102.xxx.xxx/32)
`environment`	`yes`	Current work environment (Example: staging/dev/prod). This value is used for tag all the deployed resources
`PATH_TO_PUBLIC_LB_CERT`	`yes`	Path to the public LB certificate. See how to generate the certificate
`PATH_TO_PUBLIC_LB_KEY`	`yes`	Path to the public LB key. See how to generate the key
`compute_shape`	`no`	Compute shape to use. Default VM.Standard.A1.Flex. NOTE Is mandatory to use this compute shape for provision 4 always free VMs
`os_image_id`	`no`	Image id to use. Default image: Canonical-Ubuntu-20.04-aarch64-2022.01.18-0. See how to list all available OS images
`oci_core_vcn_dns_label`	`no`	VCN DNS label. Default: defaultvcn
`oci_core_subnet_dns_label10`	`no`	First subnet DNS label. Default: defaultsubnet10
`oci_core_subnet_dns_label11`	`no`	Second subnet DNS label. Default: defaultsubnet11
`oci_core_vcn_cidr`	`no`	VCN CIDR. Default: oci_core_vcn_cidr
`oci_core_subnet_cidr10`	`no`	First subnet CIDR. Default: 10.0.0.0/24
`oci_core_subnet_cidr11`	`no`	Second subnet CIDR. Default: 10.0.1.0/24
`oci_identity_dynamic_group_name`	`no`	Dynamic group name. This dynamic group will contains all the instances of this specific compartment. Default: Compute_Dynamic_Group
`oci_identity_policy_name`	`no`	Policy name. This policy will allow dynamic group 'oci_identity_dynamic_group_name' to read OCI api without auth. Default: Compute_To_Oci_Api_Policy
`k3s_load_balancer_name`	`no`	Internal LB name. Default: k3s internal load balancer
`public_load_balancer_name`	`no`	Public LB name. Default: K3s public LB
`kube_api_port`	`no`	Kube api default port Default: 6443
`public_lb_shape`	`no`	LB shape for the public LB. Default: flexible. NOTE is mandatory to use this kind of shape to provision two always free LB (public and private)
`http_lb_port`	`no`	http port used by the public LB. Default: 80
`https_lb_port`	`no`	http port used by the public LB. Default: 443
`k3s_server_pool_size`	`no`	Number of k3s servers deployed. Default 2
`k3s_worker_pool_size`	`no`	Number of k3s workers deployed. Default 2
`install_nginx_ingress`	`no`	Boolean value, install kubernetes nginx ingress controller instead of Traefik. Default: true. For more information see Nginx ingress controller
`install_longhorn`	`no`	Boolean value, install longhorn "Cloud native distributed block storage for Kubernetes". Default: true
`longhorn_release`	`no`	Longhorn release. Default: v1.2.3
`unique_tag_key`	`no`	Unique tag name used for tagging all the deployed resources. Default: k3s-provisioner
`unique_tag_value`	`no`	Unique value used with unique_tag_key. Default: https://github.com/garutilorenzo/k3s-oci-cluster
`PATH_TO_PUBLIC_KEY`	`no`	Path to your public ssh key (Default: "~/.ssh/id_rsa.pub)
`PATH_TO_PRIVATE_KEY`	`no`	Path to your private ssh key (Default: "~/.ssh/id_rsa)

Generate random token

Generate random k3s token with:

cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 55 | head -n 1

How to find the availability doamin name

To find the list of the availability domains run this command on che Cloud Shell:

oci iam availability-domain list
{
  "data": [
    {
      "compartment-id": "<compartment_ocid>",
      "id": "ocid1.availabilitydomain.oc1..xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
      "name": "iAdc:EU-ZURICH-1-AD-1"
    }
  ]
}

How to list all the OS images

To filter the OS images by shape and OS run this command on che Cloud Shell:

oci compute image list --compartment-id <compartment_ocid> --operating-system "Canonical Ubuntu" --shape "VM.Standard.A1.Flex"
{
  "data": [
    {
      "agent-features": null,
      "base-image-id": null,
      "billable-size-in-gbs": 2,
      "compartment-id": null,
      "create-image-allowed": true,
      "defined-tags": {},
      "display-name": "Canonical-Ubuntu-20.04-aarch64-2022.01.18-0",
      "freeform-tags": {},
      "id": "ocid1.image.oc1.eu-zurich-1.aaaaaaaag2uyozo7266bmg26j5ixvi42jhaujso2pddpsigtib6vfnqy5f6q",
      "launch-mode": "NATIVE",
      "launch-options": {
        "boot-volume-type": "PARAVIRTUALIZED",
        "firmware": "UEFI_64",
        "is-consistent-volume-naming-enabled": true,
        "is-pv-encryption-in-transit-enabled": true,
        "network-type": "PARAVIRTUALIZED",
        "remote-data-volume-type": "PARAVIRTUALIZED"
      },
      "lifecycle-state": "AVAILABLE",
      "listing-type": null,
      "operating-system": "Canonical Ubuntu",
      "operating-system-version": "20.04",
      "size-in-mbs": 47694,
      "time-created": "2022-01-27T22:53:34.270000+00:00"
    },

Note: this setup was only tested with Ubuntu 20.04

Notes about OCI always free resources

In order to get the maximum resources available within the oracle always free tier, the max amount of the k3s servers and k3s workers must be 2. So the max value for k3s_server_pool_size and k3s_worker_pool_size is 2.

In this setup we use two LB, one internal LB and one public LB (Layer 7). In order to use two LB using the always free resources, one lb must be a network load balancer an the other must be a load balancer. The public LB must use the flexible shape (public_lb_shape variable).

Notes about K3s

In this environment the High Availability of the K3s cluster is provided using the Embedded DB. More details here

The default installation of K3s install Traefik as ingress the controller. In this environment Traefik is replaced by Nginx ingress controller. To install Traefik as the ingress controller set the variable install_nginx_ingress to false.
For more details on Nginx ingress controller see the Nginx ingress controller section.

Infrastructure overview

The final infrastructure will be made by:

two instance pool:
- one instance pool for the server nodes named "k3s-servers"
- one instance pool for the worker nodes named "k3s-workers"
one internal load balancer that will route traffic to K3s servers
one external load balancer that will route traffic to K3s workers

The other resources created by terraform are:

two instance configurations (one for the servers and one for the workers) used by the instance pools
one vcn
two public subnets
two security list
one dynamic group
one identity policy

Cluster resource deployed

This setup will automatically install longhorn. Longhorn is a Cloud native distributed block storage for Kubernetes. To disable the longhorn deployment set install_longhorn variable to false

Nginx ingress controller

In this environment Nginx ingress controller is used instead of the standard Traefik ingress controller.

The installation is the bare metal installation, the ingress controller then is exposed via a LoadBalancer Service.

---
apiVersion: v1
kind: Service
metadata:
  name: ingress-nginx-controller-loadbalancer
  namespace: ingress-nginx
spec:
  selector:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
  ports:
    - name: http
      port: 80
      protocol: TCP
      targetPort: 80
    - name: https
      port: 443
      protocol: TCP
      targetPort: 80
  type: LoadBalancer

To properly configure all the Forwarded HTTP Headers (L7 Headers) this parameters are added to che ConfigMap:

---
apiVersion: v1
data:
  allow-snippet-annotations: "true"
  use-forwarded-headers: "true"
  compute-full-forwarded-for: "true"
  enable-real-ip: "true"
  forwarded-for-header: "X-Forwarded-For"
  proxy-real-ip-cidr: "0.0.0.0/0"
kind: ConfigMap
metadata:
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.1.1
    helm.sh/chart: ingress-nginx-4.0.16
  name: ingress-nginx-controller
  namespace: ingress-nginx

Deploy

We are now ready to deploy our infrastructure. First we ask terraform to plan the execution with:

terraform plan

...
...
      + id                             = (known after apply)
      + ip_addresses                   = (known after apply)
      + is_preserve_source_destination = false
      + is_private                     = true
      + lifecycle_details              = (known after apply)
      + nlb_ip_version                 = (known after apply)
      + state                          = (known after apply)
      + subnet_id                      = (known after apply)
      + system_tags                    = (known after apply)
      + time_created                   = (known after apply)
      + time_updated                   = (known after apply)

      + reserved_ips {
          + id = (known after apply)
        }
    }

Plan: 27 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + k3s_servers_ips = [
      + (known after apply),
      + (known after apply),
    ]
  + k3s_workers_ips = [
      + (known after apply),
      + (known after apply),
    ]
  + public_lb_ip    = (known after apply)

──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you run "terraform apply" now.

now we can deploy our resources with:

terraform apply

...
...
      + is_preserve_source_destination = false
      + is_private                     = true
      + lifecycle_details              = (known after apply)
      + nlb_ip_version                 = (known after apply)
      + state                          = (known after apply)
      + subnet_id                      = (known after apply)
      + system_tags                    = (known after apply)
      + time_created                   = (known after apply)
      + time_updated                   = (known after apply)

      + reserved_ips {
          + id = (known after apply)
        }
    }

Plan: 27 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + k3s_servers_ips = [
      + (known after apply),
      + (known after apply),
    ]
  + k3s_workers_ips = [
      + (known after apply),
      + (known after apply),
    ]
  + public_lb_ip    = (known after apply)

  Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.
  Enter a value: yes

...
...

module.k3s_cluster.oci_network_load_balancer_backend.k3s_kube_api_backend[0]: Still creating... [50s elapsed]
module.k3s_cluster.oci_network_load_balancer_backend.k3s_kube_api_backend[0]: Still creating... [1m0s elapsed]
module.k3s_cluster.oci_network_load_balancer_backend.k3s_kube_api_backend[0]: Creation complete after 1m1s [...]

Apply complete! Resources: 27 added, 0 changed, 0 destroyed.

Outputs:

k3s_servers_ips = [
  "X.X.X.X",
  "X.X.X.X",
]
k3s_workers_ips = [
  "X.X.X.X",
  "X.X.X.X",
]
public_lb_ip = tolist([
  "X.X.X.X",
])

Now on one master node you can check the status of the cluster with:

ssh X.X.X.X -lubuntu

ubuntu@inst-iwlqz-k3s-servers:~$ sudo su -
root@inst-iwlqz-k3s-servers:~# kubectl get nodes

NAME                     STATUS   ROLES                       AGE     VERSION
inst-axdzf-k3s-workers   Ready    <none>                      4m34s   v1.22.6+k3s1
inst-hmgnl-k3s-servers   Ready    control-plane,etcd,master   4m14s   v1.22.6+k3s1
inst-iwlqz-k3s-servers   Ready    control-plane,etcd,master   6m4s    v1.22.6+k3s1
inst-lkvem-k3s-workers   Ready    <none>                      5m35s   v1.22.6+k3s1

Public LB check

We can now test the public load balancer, traefik and the security list ingress rules. On your local PC run:

curl -v http://<PUBLIC_LB_IP>

*   Trying PUBLIC_LB_IP:80...
* TCP_NODELAY set
* Connected to PUBLIC_LB_IP (PUBLIC_LB_IP) port 80 (#0)
> GET / HTTP/1.1
> Host: PUBLIC_LB_IP
> User-Agent: curl/7.68.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 404 Not Found
< Date: Fri, 25 Feb 2022 14:03:09 GMT
< Content-Type: text/html
< Content-Length: 146
< Connection: keep-alive
< 
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>
* Connection #0 to host PUBLIC_LB_IP left intact

404 is a correct response since the cluster is empty. We can test also the https listener/backends:

curl -k -v https://<PUBLIC_LB_IP>

curl -k -v https://<PUBLIC_LB_IP>

* Trying PUBLIC_LB_IP:443...
* TCP_NODELAY set
* Connected to PUBLIC_LB_IP (PUBLIC_LB_IP) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: C=IT; ST=Italy; L=Brescia; O=GL Ltd; OU=IT; CN=testlb.domainexample.com; emailAddress=email@you.com
*  start date: Feb 25 10:28:29 2022 GMT
*  expire date: Feb 25 10:28:29 2023 GMT
*  issuer: C=IT; ST=Italy; L=Brescia; O=GL Ltd; OU=IT; CN=testlb.domainexample.com; emailAddress=email@you.com
*  SSL certificate verify result: self signed certificate (18), continuing anyway.
> GET / HTTP/1.1
> Host: PUBLIC_LB_IP
> User-Agent: curl/7.68.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 404 Not Found
< Date: Fri, 25 Feb 2022 13:48:19 GMT
< Content-Type: text/html
< Content-Length: 146
< Connection: keep-alive
< 
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>
* Connection #0 to host PUBLIC_LB_IP left intact

Longhorn check

To check if longhorn was successfully installed run on one master nodes:

kubectl get ns
NAME              STATUS   AGE
default           Active   9m40s
kube-node-lease   Active   9m39s
kube-public       Active   9m39s
kube-system       Active   9m40s
longhorn-system   Active   8m52s   <- longhorn namespace 


root@inst-hmgnl-k3s-servers:~# kubectl get pods -n longhorn-system
NAME                                        READY   STATUS    RESTARTS        AGE
csi-attacher-5f46994f7-8w9sg                1/1     Running   0               7m52s
csi-attacher-5f46994f7-qz7d4                1/1     Running   0               7m52s
csi-attacher-5f46994f7-rjqlx                1/1     Running   0               7m52s
csi-provisioner-6ccbfbf86f-fw7q4            1/1     Running   0               7m52s
csi-provisioner-6ccbfbf86f-gwmrg            1/1     Running   0               7m52s
csi-provisioner-6ccbfbf86f-nsf84            1/1     Running   0               7m52s
csi-resizer-6dd8bd4c97-7l67f                1/1     Running   0               7m51s
csi-resizer-6dd8bd4c97-g66wj                1/1     Running   0               7m51s
csi-resizer-6dd8bd4c97-nksmd                1/1     Running   0               7m51s
csi-snapshotter-86f65d8bc-2gcwt             1/1     Running   0               7m50s
csi-snapshotter-86f65d8bc-kczrw             1/1     Running   0               7m50s
csi-snapshotter-86f65d8bc-sjmnv             1/1     Running   0               7m50s
engine-image-ei-fa2dfbf0-6rpz2              1/1     Running   0               8m30s
engine-image-ei-fa2dfbf0-7l5k8              1/1     Running   0               8m30s
engine-image-ei-fa2dfbf0-7nph9              1/1     Running   0               8m30s
engine-image-ei-fa2dfbf0-ndkck              1/1     Running   0               8m30s
instance-manager-e-31a0b3f5                 1/1     Running   0               8m26s
instance-manager-e-37aa4663                 1/1     Running   0               8m27s
instance-manager-e-9cc7cc9d                 1/1     Running   0               8m20s
instance-manager-e-f39d9f2c                 1/1     Running   0               8m29s
instance-manager-r-1364d994                 1/1     Running   0               8m26s
instance-manager-r-c1670269                 1/1     Running   0               8m20s
instance-manager-r-c20ebeb3                 1/1     Running   0               8m28s
instance-manager-r-c54bf9a5                 1/1     Running   0               8m27s
longhorn-csi-plugin-2qj94                   2/2     Running   0               7m50s
longhorn-csi-plugin-4t8jm                   2/2     Running   0               7m50s
longhorn-csi-plugin-ws82l                   2/2     Running   0               7m50s
longhorn-csi-plugin-zmc9q                   2/2     Running   0               7m50s
longhorn-driver-deployer-784546d78d-s6cd2   1/1     Running   0               8m58s
longhorn-manager-l8sd8                      1/1     Running   0               9m1s
longhorn-manager-r2q5c                      1/1     Running   1 (8m30s ago)   9m1s
longhorn-manager-s6wql                      1/1     Running   0               9m1s
longhorn-manager-zrrf2                      1/1     Running   0               9m
longhorn-ui-9fdb94f9-6shsr                  1/1     Running   0               8m59s

Deploy a sample stack

Finally to test all the components of the cluster we can deploy a sample stack. The stack is composed by the following components:

MariaDB
Nginx
Wordpress

Deploy the resources with:

kubectl apply -f https://raw.githubusercontent.com/garutilorenzo/k3s-oci-cluster/master/deployments/mariadb/all-resources.yml
kubectl apply -f https://raw.githubusercontent.com/garutilorenzo/k3s-oci-cluster/master/deployments/nginx/all-resources.yml
kubectl apply -f https://raw.githubusercontent.com/garutilorenzo/k3s-oci-cluster/master/deployments/wordpress/all-resources.yml

and check the status:

kubectl get deployments
NAME        READY   UP-TO-DATE   AVAILABLE   AGE
mariadb       1/1     1            1           92m
nginx         1/1     1            1           79m
wordpress     1/1     1            1           91m

kubectl get svc
NAME            TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
kubernetes        ClusterIP   10.43.0.1       <none>        443/TCP    5h8m
mariadb-svc       ClusterIP   10.43.184.188   <none>        3306/TCP   92m
nginx-svc         ClusterIP   10.43.9.202     <none>        80/TCP     80m
wordpress-svc     ClusterIP   10.43.242.26    <none>        9000/TCP   91m

Now you are ready to setup WP, open the LB public ip and follow the wizard. NOTE nginx and the Kubernetes Ingress rule are configured without virthual host/server name.

To clean the deployed resources:

kubectl delete -f https://raw.githubusercontent.com/garutilorenzo/k3s-oci-cluster/master/deployments/mariadb/all-resources.yml
kubectl delete -f https://raw.githubusercontent.com/garutilorenzo/k3s-oci-cluster/master/deployments/nginx/all-resources.yml
kubectl delete -f https://raw.githubusercontent.com/garutilorenzo/k3s-oci-cluster/master/deployments/wordpress/all-resources.yml

Clean up

terraform destroy

Build a simple web app using BottlePy

Lorenzo Garuti — Wed, 29 Dec 2021 16:51:10 +0000

Build a simple web application with SQLAlchemy and Redis support using BottlePy

Requirements
The application stack
Data used in this example site
Setup the environmant
Download sample data
Start the environment
Application Overview
App configuration
DB configuration
SQLAlchemy plugin
Redis Cache
Data export
Data import

Requirements

To use this environment you need Docker an Docker compose installed.

The application stack

Backend:

Frontend:

Database:

Webserver:

Nginx - Only Prod env.

Data used

To use this example application we need to import some example data. This example application use the "Stack Exchange Data Dump" available on archive.org.

All the data used by this site is under the cc-by-sa 4.0 license.

Setup the environmant

First of all clone this repository

git clone https://github.com/garutilorenzo/simple-bottlepy-application.git

cd simple-bottlepy-application

Now we are ready to setup our environment. For dev purposes link the docker-compose-dev.yml do docker-compose.yml

ln -s docker-compose-dev.yml docker-compose.yml

For prod environments:

ln -s docker-compose-dev.yml docker-compose.yml

The difference between dev and prod envs are:

Prod	Dev
Nginx is used to expose our example application	Built-in HTTP development server
Http port 80	Http port 8080
Debug mode is disabled	Debug mode is enabled
Reloader is disabled	Reloader is enabled

Now we can download some sample data.

Download sample data

To dwonload some example data run:

./download_samples.sh

By default the archive with the 'meta' attribute will be downloaded. If you want more data remove in download_samples.sh 'meta' form the archive name.

Small data:

for sample in workplace.meta.stackexchange.com.7z unix.meta.stackexchange.com.7z

Big data:

for sample in workplace.stackexchange.com.7z unix.stackexchange.com.7z

Note not all the stackexchange sites where imported on this example. After you choose the archives you will download adjust the network.py schema under src/schema/

class Sites(enum.Enum):
    vi = 'vi.stackexchange.com'
    workplace = 'workplace.stackexchange.com'
    wordpress = 'wordpress.stackexchange.com'
    unix = 'unix.stackexchange.com'
    tex = 'tex.stackexchange.com'

Once the data is downloaded we can import the data:

docker-compose run --rm bottle bash

web@4edf053b7e4f:~/src$  python init_db.py # <- Initialize DB
web@4edf053b7e4f:~/src$  python import_data.py # <- Import sample data

Now for each data sample you have downloaded (Eg. tex, unix, vi) a python subprocess is started and will import in order:

all the tags
all the users
all the posts
all the post history events

Once the import is finished we can start our environment

Start the environment

With our DB populated we can now start our web application:

docker-compose up -d

Creating network "bottle-exchange_default" with the default driver
Creating postgres ... done
Creating redis    ... done
Creating bottle   ... done

The application will be available at http://localhost:8080 for the dev and http://localhost for the prod

Application Overview

Index

Home page with a search form. On every change on the "Network" select, the "Tags" select is populated by an ajax call on /api/autocomplete/form/get_tags (POST) (for more details see src/bottle/static/asset/js/custom.js).

The POST call is authenticated with a random hard coded string (see src/bottle/app.py, api_get_tags)

Users

Table view of all available users with a pagination nav.

Clicking on the username, we enter on the detail's page of the user. In the details user page we see: UP Votes,Views,Down Votes. If the user has populated the "About me" field, we see a button that trigger a modal with the about me details.
If the user ha asked or answered some question we see a list of question in the "Post" section.

Posts

List of all posts with a pagination nav

API/REST endpoint

This application expose one api/rest route: /api/get/tags. You can query this route making a POST call, you have to make the call using a json payload:

curl --header "Content-Type: application/json" \
  --request POST \
  --data '{"auth_key":"dd4d5ff1c13!28356236c402d7ada.aed8b797ebd299b942291bc66,f804492be2009f14"}' \
  http://localhost:8080/api/get/tags | jq

 {
  "data": [
    {
      "clean_name": "html5",
      "created_time": "2021-12-29 11:33:06.517152+00:00",
      "id": "1",
      "name": "html5",
      "network_sites": "Sites.wordpress",
      "questions": "91",
      "tag_id": "2",
      "updated_time": "None"
    },
    ...
    ],
  "errors": [],
  "items": 5431,
  "last_page": 27
}

The auth_key is hard coded in src/bottle/app.py

App configuration

The application's configuration are loaded by the load_config module (src/load_config.py).

This module will load a .yml file under:

/app/src/<BOTTLE_APP_NAME>/config/<BOTTLE_APP_ENVIRONMENT>

BOTTLE_APP_NAME and BOTTLE_APP_ENVIRONMENT are environment variables.

BOTTLE_APP_NAME is the name of the path where our bottle application lives, in this case bottle. BOTTLE_APP_ENVIRONMENT value is prod or env.

An example configuration is:

---
enable_debug: True
enable_reloader: True
http_port: 8080
pgsql_username: "bottle"
pgsql_password: "b0tTl3_Be#"
pgsql_db: "bottle_exchange"
pgsql_host: "pgsql"
pgsql_port: 5432
create_db_schema: True
default_result_limit: 50

DB configuration

The database configuration is defined under the src/schema module.

The base.py file contains the engine configuration:

import load_config # <- See App configuration

from sqlalchemy import create_engine
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker

main_config = load_config.load_config()
conn_string = 'postgresql+psycopg2://{pgsql_username}:{pgsql_password}@{pgsql_host}:{pgsql_port}/{pgsql_db}'.format(**main_config)

engine = create_engine(conn_string, pool_size=80, pool_recycle=60)
Session = sessionmaker(bind=engine)

Base = declarative_base()

All the tables are defined in a separate file always under schema:

schema	Tables	Description
`network.py`	None	Sites class is not a SQLAlchemy object, but is an Enum used by all the other tables
`posts.py`	posts,post_history	Table definition for post and post_history tables. This module contains also three enum definitions: PostType, PostHistoryType, CloseReason
`tags.py`	tags	Tags table
`users.py`	users	Users table

SQLAlchemy plugin

In this example application is used and insalled a SQLAlchemy plugin (src/bottle/bottle_sa.py). This plugin is used to handle the SQLAlchemy session:

from schema.base import Base, engine # <- Base and engine are defined in the schema module, see "DB configuration"
from bottle_sa import SQLAlchemyPlugin
import load_config

main_config = load_config.load_config()

# Main Bottle app/application
app = application = Bottle()

# DB Plugin
saPlugin = SQLAlchemyPlugin(
    engine=engine, metadata=Base.metadata, 
    create=main_config['create_db_schema'], 
    config=main_config,
)
application.install(saPlugin)

This plugin pass an extra parameter on each function defined in src/bottle/app.py. By default this parameter is 'db', but it can be changed by passing the extra parameter 'keyword' on the SQLAlchemyPlugin init.

So an example function will be:

@app.route('/docs')
@view('docs')
def index(db): # <- db is our SQLAlchemy session
    return dict(page_name='docs')

Redis Cache

In this example application we use redis to cache some pages. The caching "approach" is very useful if you have:

a site with few updates
a slow page/route
you have to decrease the load of your DB

The RedisCache is defined in src/bottle/bottle_cache.py and this is an example usage:

from bottle_cache import RedisCache

# Cache
cache = RedisCache()

@app.route('/tags')
@app.route('/tags/<page_nr:int>')
@cache.cached()
@view('tags')
def get_tags(db, page_nr=1):
    do_something()
    return something

You can init RedisCache class with an extra parameter config:

config = {'redis_host': '<redis_hostname'>, 'redis_port': 6379, 'redis_db': 0, 'cache_expiry': 86400}
cache = RedisCache(config=config)

By default the configurations are:

Param	Default	Description
`redis_host`	`redis`	Redis FQDN or ip address
`redis_port`	`6379`	Redis listen port
`redis_db`	`0`	Redis database
`cache_expiry`	`3600`	Global cache expiry time in seconds

The @cached decorator can accept some arguments:

Param	Default	Description
`expiry`	`None`	Route cache expiry time. If not defined is the same value as the global expiry time
`key_prefix`	`bottle_cache_%s`	Redis key prefix
`content_type`	`text/html; charset=UTF-8`	Default content type

Caching json requests

A json caching example would be:

@app.route('/api/get/tags', method='POST')
@cache.cached(content_type='application/json')
def api_get_tags(db):
    do_something()
    return something

Invalidate cache

To invalidate the cache pass invalidate_cache key as query parameter or in the body request if you make a POST call

Skip/bypass cache

To skip or bypass the cache pass skip_cache key as query parameter or in the body request if you make a POST call

Data export

To backup PgSQL data run dump_db.sh

./dump_db.sh

the dump will be placed in the root directory of this repository, the file will be named dump.sql.gz (Gzipped format)

Data import

To import an existing DB uncomment the following line in the docker-compose.yml:

volumes:
    - type: volume
      source: postgres
      target: /var/lib/postgresql/data
    - ./sql:/docker-entrypoint-initdb.d # <- uncomment this line

and plache your dump in gzip or plain text format under sql/ (create the directory first)

Stop the environment

To stop the environment run

docker-compose down

Stopping bottle   ... done
Stopping postgres ... done
Stopping redis    ... done
Removing bottle   ... done
Removing postgres ... done
Removing redis    ... done
Removing network bottle-exchange_default

To clean up all the data (pgsql data) pass the extra argument "-v" to docker-compose down. With this parameter the pgsql volume will be deleted.

Deploy a k3s cluster on Oracle Cloud using terraform

Lorenzo Garuti — Fri, 29 Oct 2021 15:56:20 +0000

Welcome to the last chapter of the series dedicated to the Oracle cloud infrastructure and terraform, if you have missed the previous chapters here you can find the links:

Now we have all the knowledge for deploying a k3s cluster on Oracle Cloud infrastructure.

Notes

NOTE This k3s setup is non high available, the k3s server is a single point of failure. This setup is for testing purposes only

Environment setup

In our repository, change directory and go inside the k3s-cluster directory:

cd oracle-cloud-terraform-examples/k3s-cluster/

Modify the vars.tf in the same way whe have modified the vars.tf file in the simple instance example (to setup the vars.tf file from scratch follow the Variables setup section)

Extra variables

We have some extra variables in this example:

Variable	Default	Description
`fault_domains`	`"FAULT-DOMAIN-1", "FAULT-DOMAIN-2", "FAULT-DOMAIN-3"`	This variable is a list of fault domains where our instance pool will deploy our instances
`instance_pool_size`	`3`	Number of instances to launch in the instance pool. Number of k3s agents to deploy
`k3s_server_private_ip`	`10.0.0.50`	private ip address that will be associated to the k3s-server
`k3s_token`	`2aaf122eed3409ds2c6fagfad4073-92dcdgade664d8c1c7f49z`	token used to install the k3s cluster
`install_longhorn`	`true`	boolean value, if true (default) will install longhorn block storage
`longhorn_release`	`v1.2.2`	longorn release version

Infrastructure overview

The infrastructure is the same as the instance-pool example, but in the network load balancer we have one more listener (port 443, HTTPS).

Notes

Some important notes:

By default firewall on the compute instances is disabled. On some test the firewall has created some problems
k3s will be installed in all the instances
The operating system used is Ubuntu 20.04

Deploy

Now create the terraform.tfvars file (Terraform setup section), and initialize terraform:

terraform init

Initializing the backend...

Initializing provider plugins...
- Finding latest version of hashicorp/oci...
- Installing hashicorp/oci v4.50.0...
- Installed hashicorp/oci v4.50.0 (signed by HashiCorp)

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

we are now ready to deploy our infrastructure:

terraform plan

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # oci_core_default_route_table.default_oci_core_default_route_table will be created
  + resource "oci_core_default_route_table" "default_oci_core_default_route_table" {
      + compartment_id             = (known after apply)
      + defined_tags               = (known after apply)
      + display_name               = (known after apply)
      + freeform_tags              = (known after apply)
      + id                         = (known after apply)
      + manage_default_resource_id = (known after apply)
      + state                      = (known after apply)
      + time_created               = (known after apply)

      + route_rules {
          + cidr_block        = (known after apply)
          + description       = (known after apply)
          + destination       = "0.0.0.0/0"
          + destination_type  = "CIDR_BLOCK"
          + network_entity_id = (known after apply)
        }
    }


<TRUNCATED OUTPUT>

Plan: 21 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + k3s_agents_ips = [
      + (known after apply),
      + (known after apply),
      + (known after apply),
    ]
  + k3s_server_ip  = (known after apply)
  + lb_ip          = (known after apply)

──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you run "terraform apply" now.

if we have no error run:

terraform apply

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create
 <= read (data resources)

Terraform will perform the following actions:

  # data.oci_core_instance.k3s_agents_instances_ips[0] will be read during apply
  # (config refers to values not yet known)
 <= data "oci_core_instance" "k3s_agents_instances_ips"  {
      + agent_config                        = (known after apply)
      + async                               = (known after apply)
      + availability_config                 = (known after apply)
      + availability_domain                 = (known after apply)
      + boot_volume_id                      = (known after apply)
      + capacity_reservation_id             = (known after apply)
      + compartment_id                      = (known after apply)
      + create_vnic_details                 = (known after apply)
      + dedicated_vm_host_id                = (known after apply)
      + defined_tags                        = (known after apply)
      + display_name                        = (known after apply)
      + extended_metadata                   = (known after apply)
      + fault_domain                        = (known after apply)
      + freeform_tags                       = (known after apply)
      + hostname_label                      = (known after apply)
      + id                                  = (known after apply)
      + image                               = (known after apply)
      + instance_id                         = (known after apply)
      + instance_options                    = (known after apply)
      + ipxe_script                         = (known after apply)
      + is_pv_encryption_in_transit_enabled = (known after apply)
      + launch_mode                         = (known after apply)
      + launch_options                      = (known after apply)
      + metadata                            = (known after apply)
      + platform_config                     = (known after apply)
      + preemptible_instance_config         = (known after apply)
      + preserve_boot_volume                = (known after apply)
      + private_ip                          = (known after apply)
      + public_ip                           = (known after apply)
      + region                              = (known after apply)
      + shape                               = (known after apply)
      + shape_config                        = (known after apply)
      + source_details                      = (known after apply)
      + state                               = (known after apply)
      + subnet_id                           = (known after apply)
      + system_tags                         = (known after apply)
      + time_created                        = (known after apply)
      + time_maintenance_reboot_due         = (known after apply)
    }

<TRUNCATED OUTPUT>

oci_network_load_balancer_backend.k3s_http_backend[2]: Still creating... [1m40s elapsed]
oci_network_load_balancer_backend.k3s_https_backend[1]: Still creating... [1m51s elapsed]
oci_network_load_balancer_backend.k3s_https_backend[0]: Still creating... [1m51s elapsed]
oci_network_load_balancer_backend.k3s_http_backend[2]: Still creating... [1m50s elapsed]
oci_network_load_balancer_backend.k3s_http_backend[0]: Still creating... [1m50s elapsed]
oci_network_load_balancer_backend.k3s_https_backend[0]: Still creating... [2m1s elapsed]
oci_network_load_balancer_backend.k3s_https_backend[1]: Still creating... [2m1s elapsed]
oci_network_load_balancer_backend.k3s_http_backend[0]: Still creating... [2m0s elapsed]
oci_network_load_balancer_backend.k3s_http_backend[2]: Still creating... [2m0s elapsed]
oci_network_load_balancer_backend.k3s_http_backend[0]: Creation complete after 2m3s [id=networkLoadBalancers/ocid1.networkloadbalancer.oc1.eu-zurich-1.amaaaaaa5kjm7pyaglmtxbdrio5sp5vetj3b7hwhhxxhd7xtgytvqo4ckfsq/backendSets/k3s%20http%20backend/backends/ocid1.instance.oc1.eu-zurich-1.an5heljr5kjm7pycxldegwyajkogb3hdz6sup45qwpnqirxuck6l5y3jxwqq.80]
oci_network_load_balancer_backend.k3s_https_backend[1]: Creation complete after 2m6s [id=networkLoadBalancers/ocid1.networkloadbalancer.oc1.eu-zurich-1.amaaaaaa5kjm7pyaglmtxbdrio5sp5vetj3b7hwhhxxhd7xtgytvqo4ckfsq/backendSets/k3s%20https%20backend/backends/ocid1.instance.oc1.eu-zurich-1.an5heljr5kjm7pycs67mnhim6h46vimtx6akahatgssfprxkpti6ij5aqm5q.443]
oci_network_load_balancer_backend.k3s_https_backend[0]: Still creating... [2m11s elapsed]
oci_network_load_balancer_backend.k3s_http_backend[2]: Still creating... [2m10s elapsed]
oci_network_load_balancer_backend.k3s_https_backend[0]: Still creating... [2m21s elapsed]
oci_network_load_balancer_backend.k3s_http_backend[2]: Still creating... [2m20s elapsed]
oci_network_load_balancer_backend.k3s_http_backend[2]: Creation complete after 2m21s [id=networkLoadBalancers/ocid1.networkloadbalancer.oc1.eu-zurich-1.amaaaaaa5kjm7pyaglmtxbdrio5sp5vetj3b7hwhhxxhd7xtgytvqo4ckfsq/backendSets/k3s%20http%20backend/backends/ocid1.instance.oc1.eu-zurich-1.an5heljr5kjm7pyca6sjt44vji4axtni5ebyc2mq66hobokh2yfz5qsljnia.80]
oci_network_load_balancer_backend.k3s_https_backend[0]: Still creating... [2m31s elapsed]
oci_network_load_balancer_backend.k3s_https_backend[0]: Creation complete after 2m38s [id=networkLoadBalancers/ocid1.networkloadbalancer.oc1.eu-zurich-1.amaaaaaa5kjm7pyaglmtxbdrio5sp5vetj3b7hwhhxxhd7xtgytvqo4ckfsq/backendSets/k3s%20https%20backend/backends/ocid1.instance.oc1.eu-zurich-1.an5heljr5kjm7pycxldegwyajkogb3hdz6sup45qwpnqirxuck6l5y3jxwqq.443]

Apply complete! Resources: 21 added, 0 changed, 0 destroyed.

Outputs:

k3s_agents_ips = [
  "132.x.x.x",
  "152.x.x.x",
  "152.x.x.x",
]
k3s_server_ip = "132.x.x.x"
lb_ip = tolist([
  {
    "ip_address" = "152.x.x.x"
    "is_public" = true
    "reserved_ip" = tolist([])
  },
])

Now we can ssh in our k3s-server instance:

ssh ubuntu@132.x.x.x

...
35 updates can be applied immediately.
25 of these updates are standard security updates.
To see these additional updates run: apt list --upgradable



The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.

ubuntu@k3s-server:~$

After some minutes (at least one backend must be in HEALTH state) also the network load balancer will respond to our requests:

curl -v http://152.x.x.x/
*   Trying 152.x.x.x:80...
* TCP_NODELAY set
* Connected to 152.x.x.x (152.x.x.x) port 80 (#0)
> GET / HTTP/1.1
> Host: 152.x.x.x
> User-Agent: curl/7.68.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 404 Not Found
< Content-Type: text/plain; charset=utf-8
< X-Content-Type-Options: nosniff
< Date: Wed, 27 Oct 2021 13:20:05 GMT
< Content-Length: 19
< 
404 page not found
* Connection #0 to host 152.x.x.x left intact

NOTE 404 is a correct response since there are no deployment yet

k3s cluster management

To manage the cluster, open a ssh connection to the k3s-server, here some basic kubectl commands:

List the nodes

root@k3s-server:~# kubectl get nodes
NAME                    STATUS   ROLES                  AGE   VERSION
inst-vr4sv-k3s-agents   Ready    <none>                 23m   v1.21.5+k3s2
inst-zkcyl-k3s-agents   Ready    <none>                 23m   v1.21.5+k3s2
k3s-server              Ready    control-plane,master   23m   v1.21.5+k3s2
inst-fhayc-k3s-agents   Ready    <none>                 23m   v1.21.5+k3s2

Get the pods running on kube-system namespace

kubectl get pods -n kube-system
NAME                                      READY   STATUS      RESTARTS   AGE
coredns-7448499f4d-jwgzt                  1/1     Running     0          34m
metrics-server-86cbb8457f-qjgr9           1/1     Running     0          34m
local-path-provisioner-5ff76fc89d-56c7n   1/1     Running     0          34m
helm-install-traefik-crd-9ftr8            0/1     Completed   0          34m
helm-install-traefik-2v48n                0/1     Completed   2          34m
svclb-traefik-2x9q9                       2/2     Running     0          33m
svclb-traefik-d72cf                       2/2     Running     0          33m
svclb-traefik-jq5wv                       2/2     Running     0          33m
svclb-traefik-xnhgs                       2/2     Running     0          33m
traefik-97b44b794-4dz2x                   1/1     Running     0          33m

Get the pods running on longhorn-system namespace (optional)

root@k3s-server:~# kubectl get pods -n longhorn-system
NAME                                        READY   STATUS             RESTARTS   AGE
longhorn-ui-788fd8cf9d-76x84                1/1     Running            0          29m
longhorn-manager-97vzd                      1/1     Running            0          29m
longhorn-driver-deployer-5dff5c7554-c7wbk   1/1     Running            0          29m
longhorn-manager-sq2xn                      1/1     Running            1          29m
csi-attacher-75588bff58-xv9sn               1/1     Running            0          28m
csi-resizer-5c88bfd4cf-ngm2j                1/1     Running            0          28m
engine-image-ei-d4c780c6-ktvs7              1/1     Running            0          28m
csi-provisioner-669c8cc698-mqvjx            1/1     Running            0          28m
longhorn-csi-plugin-9x5wj                   2/2     Running            0          28m
engine-image-ei-d4c780c6-r7r2t              1/1     Running            0          28m
csi-provisioner-669c8cc698-tvs9r            1/1     Running            0          28m
csi-resizer-5c88bfd4cf-h8g6w                1/1     Running            0          28m
instance-manager-e-7aca498c                 1/1     Running            0          28m
instance-manager-r-98153684                 1/1     Running            0          28m
longhorn-csi-plugin-wf24d                   2/2     Running            0          28m
csi-snapshotter-69f8bc8dcf-n85hq            1/1     Running            0          28m
longhorn-csi-plugin-82hv5                   2/2     Running            0          28m
longhorn-csi-plugin-rlcw2                   2/2     Running            0          28m
longhorn-manager-rttww                      1/1     Running            1          29m
instance-manager-e-e43d97f9                 1/1     Running            0          28m
longhorn-manager-47zxl                      1/1     Running            1          29m
instance-manager-r-de0dc83b                 1/1     Running            0          28m
engine-image-ei-d4c780c6-hp4mb              1/1     Running            0          28m
engine-image-ei-d4c780c6-hcwpg              1/1     Running            0          28m
instance-manager-r-464299ad                 1/1     Running            0          28m
instance-manager-e-ccb8666b                 1/1     Running            0          28m
instance-manager-r-3b35070e                 1/1     Running            0          28m
instance-manager-e-9d117ead                 1/1     Running            0          28m

Dploy an example application

We are now deploying an example app (nginx) with:

a persistent storage (longhorn NFS)
a ingress route, exposed by Traefik ingress controller

Create the deployment:

cat <<EOF > test-deployment.yml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: longhorn-volv-pvc
  namespace: default
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: longhorn
  resources:
    requests:
      storage: 2Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-deployment
spec:
  selector:
    matchLabels:
      app: metrics
      department: oci
  replicas: 3
  template:
    metadata:
      labels:
        app: metrics
        department: oci
    spec:
      containers:
      - name: volume-test
        image: nginx:stable-alpine
        imagePullPolicy: IfNotPresent
        livenessProbe:
          exec:
            command:
              - ls
              - /data/
          initialDelaySeconds: 5
          periodSeconds: 5
        volumeMounts:
        - name: volv
          mountPath: /data
        ports:
        - containerPort: 80
      volumes:
      - name: volv
        persistentVolumeClaim:
          claimName: longhorn-volv-pvc
---
apiVersion: v1
kind: Service
metadata:
  name: my-cip-service
spec:
  type: ClusterIP
  selector:
    app: metrics
    department: oci
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80
---
kind: Ingress
apiVersion:  networking.k8s.io/v1
metadata:
  name: "foo"
spec:
  rules:
    - host: example.net
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name:  my-cip-service
                port:
                  number: 80
EOF

apply the deployment with:

kubectl apply -f test-deployment.yml 
persistentvolumeclaim/longhorn-volv-pvc created
deployment.apps/my-deployment created
service/my-cip-service created
ingress.networking.k8s.io/foo created

monitor the deployment status:

kubectl get deployments
NAME            READY   UP-TO-DATE   AVAILABLE   AGE
my-deployment   3/3     3            3           48s

kubectl get pods
NAME                             READY   STATUS    RESTARTS   AGE
my-deployment-5d75fc6b89-xmxzt   1/1     Running   0          46s
my-deployment-5d75fc6b89-xbll2   1/1     Running   0          46s
my-deployment-5d75fc6b89-4g2w8   1/1     Running   0          46s

Now test the reachability of the example.net domain:

curl -H 'Host: example.net' http://152.x.x.x # <- This is the network load balancer IP

<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

Cleanup

To cleanup/destroy our infrastrucure:

terraform destroy

Deploy multiple Oracle compute instances using an instance pool and terraform

Lorenzo Garuti — Fri, 29 Oct 2021 15:49:44 +0000

Welcome to the third chapter of the series dedicated to the Oracle cloud infrastructure and terraform, if you have missed the previous chapters here you can find the links:

After we have successfully launched our first instance we are now ready for a more complicated example.

Environment setup

In our repository, change directory and go inside the instance-pool directory:

cd oracle-cloud-terraform-examples/instance-pool/

Modify the vars.tf in the same way whe have modified the vars.tf file in the simple instance example (to setup the vars.tf file from scratch follow the Variables setup section)

Extra variables

We have some extra variables in this example:

Variable	Default	Description
`fault_domains`	`"FAULT-DOMAIN-1", "FAULT-DOMAIN-2", "FAULT-DOMAIN-3"`	This variable is a list of fault domains where our instance pool will deploy our instances
`instance_pool_size`	`2`	Number of instances to launch in the instance pool

Infrastructure overview

The infrastructure is the same as the simple instance example but we have also:

one network load balancer, that will route the traffic from the internet to our instance pool instances
one instance configuration used by the instance pool
one instance pool
two Oracle compute instances launched by the instance pool

The network load balancer is made by:

one listener (port 80)
one backed set
one backed for each of the instances in the instance pool

Notes

Some important notes:

By default firewall on the compute instances is disabled. On some test the firewall has created some problems
Nginx will be installed by default (nginx is used for testing the security list rules, and for testing the network load balancer setup)
The operating system used is Ubuntu 20.04

Deploy

Now create the terraform.tfvars file (Terraform setup section), and initialize terraform:

terraform init

Initializing the backend...

Initializing provider plugins...
- Finding latest version of hashicorp/oci...
- Installing hashicorp/oci v4.50.0...
- Installed hashicorp/oci v4.50.0 (signed by HashiCorp)

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

we are now ready to deploy our infrastructure:

terraform plan

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # oci_core_default_route_table.default_oci_core_default_route_table will be created
  + resource "oci_core_default_route_table" "default_oci_core_default_route_table" {
      + compartment_id             = (known after apply)
      + defined_tags               = (known after apply)
      + display_name               = (known after apply)
      + freeform_tags              = (known after apply)
      + id                         = (known after apply)
      + manage_default_resource_id = (known after apply)
      + state                      = (known after apply)
      + time_created               = (known after apply)

      + route_rules {
          + cidr_block        = (known after apply)
          + description       = (known after apply)
          + destination       = "0.0.0.0/0"
          + destination_type  = "CIDR_BLOCK"
          + network_entity_id = (known after apply)
        }
    }


<TRUNCATED OUTPUT>

Plan: 14 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + instances_ips = [
      + (known after apply),
      + (known after apply),
    ]
  + lb_ip         = (known after apply)

──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you run "terraform apply" now.

if we have no error run:

terraform apply

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create
 <= read (data resources)


Terraform will perform the following actions:

  # data.oci_core_instance.ubuntu_instance_pool_instances_ips[0] will be read during apply
  # (config refers to values not yet known)
 <= data "oci_core_instance" "ubuntu_instance_pool_instances_ips"  {
      + agent_config                        = (known after apply)
      + async                               = (known after apply)
      + availability_config                 = (known after apply)
      + availability_domain                 = (known after apply)
      + boot_volume_id                      = (known after apply)
      + capacity_reservation_id             = (known after apply)
      + compartment_id                      = (known after apply)
      + create_vnic_details                 = (known after apply)
      + dedicated_vm_host_id                = (known after apply)
      + defined_tags                        = (known after apply)
      + display_name                        = (known after apply)
      + extended_metadata                   = (known after apply)
      + fault_domain                        = (known after apply)
      + freeform_tags                       = (known after apply)
      + hostname_label                      = (known after apply)
      + id                                  = (known after apply)
      + image                               = (known after apply)
      + instance_id                         = (known after apply)
      + instance_options                    = (known after apply)
      + ipxe_script                         = (known after apply)
      + is_pv_encryption_in_transit_enabled = (known after apply)
      + launch_mode                         = (known after apply)
      + launch_options                      = (known after apply)
      + metadata                            = (known after apply)
      + platform_config                     = (known after apply)
      + preemptible_instance_config         = (known after apply)
      + preserve_boot_volume                = (known after apply)
      + private_ip                          = (known after apply)
      + public_ip                           = (known after apply)
      + region                              = (known after apply)
      + shape                               = (known after apply)
      + shape_config                        = (known after apply)
      + source_details                      = (known after apply)
      + state                               = (known after apply)
      + subnet_id                           = (known after apply)
      + system_tags                         = (known after apply)
      + time_created                        = (known after apply)
      + time_maintenance_reboot_due         = (known after apply)
    }

<TRUNCATED OUTPUT>

oci_network_load_balancer_listener.test_listener: Creation complete after 25s [id=networkLoadBalancers/ocid1.networkloadbalancer.oc1.eu-zurich-1.amaaaaaa5kjm7pyarkfapfnqqxrwaowlnmj5mnd3etmig5nfcwd3m5yb7uha/listeners/LB%20test%20listener]
oci_network_load_balancer_backend.test_backend[1]: Still creating... [31s elapsed]
oci_network_load_balancer_backend.test_backend[0]: Still creating... [31s elapsed]
oci_network_load_balancer_backend.test_backend[0]: Still creating... [41s elapsed]
oci_network_load_balancer_backend.test_backend[1]: Still creating... [41s elapsed]
oci_network_load_balancer_backend.test_backend[0]: Creation complete after 42s [id=networkLoadBalancers/ocid1.networkloadbalancer.oc1.eu-zurich-1.amaaaaaa5kjm7pyarkfapfnqqxrwaowlnmj5mnd3etmig5nfcwd3m5yb7uha/backendSets/Backend%20set%20test/backends/ocid1.instance.oc1.eu-zurich-1.an5heljr5kjm7pycu5exolhnubsq5isqo6nveddlmlsblkz7geb6vbwsvbtq.80]
oci_network_load_balancer_backend.test_backend[1]: Still creating... [51s elapsed]
oci_network_load_balancer_backend.test_backend[1]: Still creating... [1m1s elapsed]
oci_network_load_balancer_backend.test_backend[1]: Still creating... [1m11s elapsed]
oci_network_load_balancer_backend.test_backend[1]: Creation complete after 1m14s [id=networkLoadBalancers/ocid1.networkloadbalancer.oc1.eu-zurich-1.amaaaaaa5kjm7pyarkfapfnqqxrwaowlnmj5mnd3etmig5nfcwd3m5yb7uha/backendSets/Backend%20set%20test/backends/ocid1.instance.oc1.eu-zurich-1.an5heljr5kjm7pycft5ixge6ssknpyb5s6q3eihuccogpqrvv2ntqdlww72a.80]

Apply complete! Resources: 14 added, 0 changed, 0 destroyed.

Outputs:

instances_ips = [
  "132.x.x.x",
  "152.x.x.x",
]
lb_ip = tolist([
  {
    "ip_address" = "140.x.x.x"
    "is_public" = true
    "reserved_ip" = tolist([])
  },
])

Now we can ssh in one of the deployed instances:

ssh ubuntu@132.x.x.x

...
35 updates can be applied immediately.
25 of these updates are standard security updates.
To see these additional updates run: apt list --upgradable



The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.

ubuntu@inst-ikudx-ubuntu-instance-pool:~$

After some minutes (at least one backend must be in HEALTH state) also the network load balancer will respond to our requests:

curl -v 140.x.x.x
*   Trying 140.x.x.x:80...
* TCP_NODELAY set
* Connected to 140.x.x.x (140.x.x.x) port 80 (#0)
> GET / HTTP/1.1
> Host: 140.x.x.x
> User-Agent: curl/7.68.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Server: nginx/1.18.0 (Ubuntu)
< Date: Wed, 27 Oct 2021 15:39:51 GMT
< Content-Type: text/html
< Content-Length: 672
< Last-Modified: Wed, 27 Oct 2021 15:33:26 GMT
< Connection: keep-alive
< ETag: "61797146-2a0"
< Accept-Ranges: bytes
...
...
...

Cleanup

To cleanup/destroy our infrastrucure:

terraform destroy

Deploy an Oracle Cloud compute instance using terraform

Lorenzo Garuti — Fri, 29 Oct 2021 15:48:38 +0000

Welcome to the second part of the series dedicated to the Oracle cloud infrastructure and terraform, if you have missed it you can read the first part here.

Once we have setup our terraform Oracle cloud provider, we are now ready to deploy our first instance.

Clone the repository

I you haven't already done so, download this this repository, and enter in the simple-instance directory:

git clone https://github.com/garutilorenzo/oracle-cloud-terraform-examples.git
cd oracle-cloud-terraform-examples/simple-instance/

Infrastructure overview

This example will deploy:

one VCN network (cidr 10.0.0.0/16), you can customize the cidr changing the oci_core_vcn_cidr variable (see Variable setup)
two subnets (cidr 10.0.0.0/24 and cidr 10.0.1.0/24). you can customize the subnets cidr changing oci_core_subnet_cidr10 and oci_core_subnet_cidr11 variables (see Variable setup)
one internet gateway associated with the VCN network
one route table associated with the VCN network
one security list (see notes about security list)
one Ocacle compute instance, VM.Standard.A1.Flex with 6GB of ram and 1 CPU (ARM processor)

notes about security the security list rules are:

By default only the incoming ICMP, SSH and HTTP traffic is allowed from your public ip. You can setup your public ip in my_public_ip_address variable.
By default all the outgoing traffic is allowed
A second security list rule (Custom security list) open all the incoming http traffic
Both default security list and the custom security list are associated on both subnets
Network flow from the private VCN subnet is allowed

Notes

Some important notes:

By default firewall on the compute instances is disabled. On some test the firewall has created some problems
Nginx will be installed by default (nginx is used for testing the security list rules)
The operating system used is Ubuntu 20.04

Variables setup

Before we can proceed we have to modify some variables in the vars.tf, the variables to modify are:

Variable	Default	Description
`region`	`N.D.`	Set the correct region based on your needs
`availability_domain`	`N.D.`	set you availability domain, you can get the availability domain string in the "Create instance form. Once you are in the create instance procedure under the placement section click "Edit" and copy the string that begin with iAdc:. Example iAdc:EU-ZURICH-1-AD-1
`default_fault_domain`	`FAULT-DOMAIN-1`	set de default fault domain, choose one of: FAULT-DOMAIN-1, FAULT-DOMAIN-2, FAULT-DOMAIN-3
`PATH_TO_PUBLIC_KEY`	`~/.ssh/id_rsa.pub`	this variable have to point at your ssh public key
`oci_core_vcn_cidr`	`10.0.0.0/16`	set the default VCN subnet cidr
`oci_core_subnet_cidr10`	`10.0.0.0/24`	set the default subnet cidr
`oci_core_subnet_cidr11`	`10.0.1.0/24`	set the secondary subnet cidr
`tutorial_tag_key`	`oracle-tutorial`	set a key used to tag all the deployed resources
`tutorial_tag_value`	`terraform`	set the value of the tutorial_tag_key
`my_public_ip_address`	`N.D.`	set your public ip address

NOTE this variables have to be set also in the part3 and part4, remember to modify the vars.tf files also in this directories.

Deploy

Now create the terraform.tfvars file (Terraform setup section), and initialize terraform:

terraform init

Initializing the backend...

Initializing provider plugins...
- Finding latest version of hashicorp/oci...
- Installing hashicorp/oci v4.50.0...
- Installed hashicorp/oci v4.50.0 (signed by HashiCorp)

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

we are now ready to launch our first instance:

terraform plan

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # oci_core_default_route_table.default_oci_core_default_route_table will be created
  + resource "oci_core_default_route_table" "default_oci_core_default_route_table" {
      + compartment_id             = (known after apply)
      + defined_tags               = (known after apply)
      + display_name               = (known after apply)
      + freeform_tags              = (known after apply)
      + id                         = (known after apply)
      + manage_default_resource_id = (known after apply)
      + state                      = (known after apply)
      + time_created               = (known after apply)

      + route_rules {
          + cidr_block        = (known after apply)
          + description       = (known after apply)
          + destination       = "0.0.0.0/0"
          + destination_type  = "CIDR_BLOCK"
          + network_entity_id = (known after apply)
        }
    }


<TRUNCATED OUTPUT>

Plan: 8 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + instance_ip = (known after apply)

if we have no error run:

terraform apply

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # oci_core_default_route_table.default_oci_core_default_route_table will be created
  + resource "oci_core_default_route_table" "default_oci_core_default_route_table" {
      + compartment_id             = (known after apply)
      + defined_tags               = (known after apply)
      + display_name               = (known after apply)
      + freeform_tags              = (known after apply)
      + id                         = (known after apply)
      + manage_default_resource_id = (known after apply)
      + state                      = (known after apply)
      + time_created               = (known after apply)

      + route_rules {
          + cidr_block        = (known after apply)
          + description       = (known after apply)
          + destination       = "0.0.0.0/0"
          + destination_type  = "CIDR_BLOCK"
          + network_entity_id = (known after apply)
        }
    }

<TRUNCATED OUTPUT>

oci_core_default_route_table.default_oci_core_default_route_table: Creation complete after 1s [id=ocid1.routetable.oc1.eu-zurich-1.aaaaaaaa6jdgqpdgwsbwbnxcnkml2zhcbgtzpm4jynea6vid56p2ywrit3za]
oci_core_subnet.oci_core_subnet11: Creation complete after 1s [id=ocid1.subnet.oc1.eu-zurich-1.aaaaaaaam3443d65qq7aoc7kf4sftrwj3splrp7wiwzvboufoiu5tc5f7iaq]
oci_core_subnet.default_oci_core_subnet10: Creation complete after 4s [id=ocid1.subnet.oc1.eu-zurich-1.aaaaaaaapyx2u7vih7g6hflzvhe2qhs6utx5qqxezjf2lm5uyswrispjtnpq]
oci_core_instance.ubuntu_oci_instance: Creating...
oci_core_instance.ubuntu_oci_instance: Still creating... [10s elapsed]
oci_core_instance.ubuntu_oci_instance: Still creating... [20s elapsed]
oci_core_instance.ubuntu_oci_instance: Still creating... [30s elapsed]
oci_core_instance.ubuntu_oci_instance: Still creating... [40s elapsed]
oci_core_instance.ubuntu_oci_instance: Still creating... [50s elapsed]
oci_core_instance.ubuntu_oci_instance: Still creating... [1m0s elapsed]
oci_core_instance.ubuntu_oci_instance: Still creating... [1m10s elapsed]
oci_core_instance.ubuntu_oci_instance: Still creating... [1m20s elapsed]
oci_core_instance.ubuntu_oci_instance: Still creating... [1m30s elapsed]
oci_core_instance.ubuntu_oci_instance: Still creating... [1m40s elapsed]
oci_core_instance.ubuntu_oci_instance: Still creating... [1m50s elapsed]
oci_core_instance.ubuntu_oci_instance: Still creating... [2m0s elapsed]
oci_core_instance.ubuntu_oci_instance: Creation complete after 2m8s [id=ocid1.instance.oc1.eu-zurich-1.an5heljr5kjm7pycoborgwavcmh5xrjgd3ozciyvcsjsclxbvbtmrrbpzomq]

Apply complete! Resources: 8 added, 0 changed, 0 destroyed.

Outputs:

instance_ip = "152.x.x.x"

──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you run "terraform apply" now.

Now we can ssh into our instance:

ssh ubuntu@152.x.x.x

...
35 updates can be applied immediately.
25 of these updates are standard security updates.
To see these additional updates run: apt list --upgradable



The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.

ubuntu@ubuntu-instance:~$

If we have setup all the variables correctly we are now inside our Oracle compute instance, we can now check if nginx is running:

ubuntu@ubuntu-instance:~$ systemctl status nginx.service 
● nginx.service - A high performance web server and a reverse proxy server
     Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2021-10-29 09:11:21 UTC; 3h 58min ago
       Docs: man:nginx(8)
   Main PID: 8469 (nginx)
      Tasks: 2 (limit: 6861)
     Memory: 2.3M
     CGroup: /system.slice/nginx.service
             ├─8469 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
             └─8470 nginx: worker process

Oct 29 09:11:21 ubuntu-instance systemd[1]: Starting A high performance web server and a reverse proxy server...
Oct 29 09:11:21 ubuntu-instance systemd[1]: Started A high performance web server and a reverse proxy server.

And from our workstation we can try to reach via browser (or curl) our public ip address on port 80 (HTTP):

curl http://152.x.x.x
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

...
...
...

Cleanup

To cleanup/destroy our infrastrucure:

terraform destroy

<TRUNCATED OUTPUT>

Plan: 0 to add, 0 to change, 8 to destroy.

Changes to Outputs:
  - instance_ip = "152.x.x.x" -> null

Do you really want to destroy all resources?
  Terraform will destroy all your managed infrastructure, as shown above.
  There is no undo. Only 'yes' will be accepted to confirm.

  Enter a value: 

<TRUNCATED OUTPUT>


oci_core_instance.ubuntu_oci_instance: Destruction complete after 1m6s
oci_core_subnet.default_oci_core_subnet10: Destroying... [id=ocid1.subnet.oc1.eu-zurich-1.aaaaaaaapyx2u7vih7g6hflzvhe2qhs6utx5qqxezjf2lm5uyswrispjtnpq]
oci_core_subnet.default_oci_core_subnet10: Destruction complete after 0s
oci_core_security_list.custom_security_list: Destroying... [id=ocid1.securitylist.oc1.eu-zurich-1.aaaaaaaa5qfkat6fldch3jkkderuwruxjnxdgxjq5auennrv2krwqvnsqt3q]
oci_core_default_security_list.default_security_list: Destroying... [id=ocid1.securitylist.oc1.eu-zurich-1.aaaaaaaaklnlfa5y36tmdxyimbfiobgegmrikbt3mnaoaf3kot4i74yxtqga]
oci_core_security_list.custom_security_list: Destruction complete after 1s
oci_core_default_security_list.default_security_list: Destruction complete after 1s
oci_core_vcn.default_oci_core_vcn: Destroying... [id=ocid1.vcn.oc1.eu-zurich-1.amaaaaaa5kjm7pyalfrmvv5hlg5bo35v4pgnqrkdrjaepbaqjt3c32ai7qaq]
oci_core_vcn.default_oci_core_vcn: Destruction complete after 1s

Destroy complete! Resources: 8 destroyed.

Setup Terraform Oracle Cloud provider

Lorenzo Garuti — Fri, 29 Oct 2021 15:48:09 +0000

This is the first post of a series dedicated to Oracle Cloud infrastructure and Terraform.

Oracle Cloud series index

Setup Terraform Oracle Cloud provider
Deploy an Oracle Cloud compute instance using terraform. Go to part 2
Deploy multiple Oracle Cloud compute instances using an instance pool using terraform Go to part 3
Deploy a k3s cluster on Oracle Cloud using terraform part 4

Sign-up to Oracle Cloud

Go to https://cloud.oracle.com/ and create a new account:

Account setup

Once you are logged in we need to create a new user and a new group with limited grants. To do so go to Identity & Security -> Identity:

Under Group, create a new group called terraform:

click "Create"

Under Policys, create a new policy named terraform-policy

set the description to "terraform-users-policy", click on show manual editor and paste this lines:

Allow group terraform to manage virtual-network-family in tenancy
Allow group terraform to manage instance-family in tenancy
Allow group terraform to manage compute-management-family in tenancy
Allow group terraform to manage volume-family in tenancy

This policy could be too open, take it only as example for this tutorial. You can get more details here

Now create a new user, under user create a new user called terraform:

Choose IAM user, set the name to terraform and the description to "terraform user".

Now in the username details (click on terraform in the User table), click "Edit user capabilites" and un-check:

Local Password
SMTP credentials
Customer Secret Keys
OAuth 2.0 Client Credentials

Now click "Add User to Group" and choose the terraform group.

RSA key generation

To use terraform with the Oracle Cloud infrastructure you need to generate an RSA key. Generate the rsa key with:

openssl genrsa -out ~/.oci/terraform-oracle-cloud.pem 4096
chmod 600 ~/.oci/terraform-oracle-cloud.pem
openssl rsa -pubout -in ~/.oci/terraform-oracle-cloud.pem -out ~/.oci/terraform-oracle-cloud_public.pem

NOTE ~/.oci/terraform-oracle-cloud_public.pem this string will be used on the terraform.tfvars used by the Oracle provider plugin, so please take note of this string.

Now copy the content of ~/.oci/terraform-oracle-cloud_public.pem:

cat  ~/.oci/terraform-oracle-cloud_public.pem

In the Oracle Cloud Console, under user select the terraform user. Under API Keys click on "Add API Key" and paste the content of your public RSA key:

Now you should see your configuration details:

[DEFAULT]
user=<user ocid...>
fingerprint=<fingerprint..>
tenancy=<tenecny ocid...>
region=<region>
key_file=<path to your private keyfile> # ~/.oci/terraform-oracle-cloud_public.pem

Terraform setup

The first step is to install terraform. Once terraform is installed download this repository:

git clone https://github.com/garutilorenzo/oracle-cloud-terraform-examples.git
cd oracle-cloud-terraform-examples/

In the root dir of this repository yoi will find three subdirectory, now we move into the simple-instance directory:

cd simple-instance/

on this directory we find all the necessary files for deploying our first instance, we see more in detail in the next session.

Now in this directory we have to create a file named "terraform.tfvars":

touch terraform.tfvars

edit the file and paste your configuration details, the file will look like:

fingerprint      = <fingerprint..>
private_key_path = "~/.oci/terraform-oracle-cloud.pem"
user_ocid        = "<user ocid...>"
tenancy_ocid     = <tenecny ocid...>
compartment_ocid = <compartment ocid...>

NOTE The compartment_ocid is the same as tenency_ocid.

Now we have setup the terraform Oracle provider and we are ready for our first deployment.

A bash solution for docker and iptables conflict

Lorenzo Garuti — Thu, 14 Oct 2021 15:14:30 +0000

If you’ve ever tried to setup firewall rules on the same machine where docker daemon is running you may have noticed that docker (by default) manipulate your iptables chains.
If you want the full control of your iptables rules this might be a problem.

Docker and iptables

Docker is utilizing the iptables "nat" to resolve packets from and to its containers and "filter" for isolation purposes, by default docker creates some chains in your iptables setup:

sudo iptables -L

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy DROP)
target     prot opt source               destination         
DOCKER-USER  all  --  anywhere             anywhere            
DOCKER-ISOLATION-STAGE-1  all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain DOCKER (1 references)
target     prot opt source               destination         

Chain DOCKER-INGRESS (0 references)
target     prot opt source               destination         

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target     prot opt source               destination         
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-ISOLATION-STAGE-2 (1 references)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-USER (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere

now for example we have the need to expose our nginx container to the world:

docker run --name some-nginx -d -p 8080:80 nginx:latest
47a12adff13aa7609020a1aa0863b0dff192fbcf29507788a594e8b098ffe47a

docker ps
CONTAINER ID   IMAGE          COMMAND                  CREATED          STATUS          PORTS                                   NAMES
47a12adff13a   nginx:latest   "/docker-entrypoint.…"   27 seconds ago   Up 24 seconds   0.0.0.0:8080->80/tcp, :::8080->80/tcp   some-nginx

and now we can reach our nginx default page:

curl -v http://192.168.25.200:8080

*   Trying 192.168.25.200:8080...
* TCP_NODELAY set
* Connected to 192.168.25.200 (192.168.25.200) port 8080 (#0)
> GET / HTTP/1.1
> Host: 192.168.25.200:8080
> User-Agent: curl/7.68.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Server: nginx/1.21.1
< Date: Thu, 14 Oct 2021 10:31:38 GMT
< Content-Type: text/html
< Content-Length: 612
< Last-Modified: Tue, 06 Jul 2021 14:59:17 GMT
< Connection: keep-alive
< ETag: "60e46fc5-264"
< Accept-Ranges: bytes
< 
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
...
* Connection #0 to host 192.168.25.200 left intact

NOTE the connection test is made using an external machine, not the same machine where the docker container is running.

The "magic" iptables rules added also allow our containers to reach the outside world:

docker run --rm nginx curl ipinfo.io/ip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    15  100    15    0     0     94      0 --:--:-- --:--:-- --:--:--    94

1.2.3.4

Now check what happened to our iptables rules:

iptables -L

...
Chain DOCKER (1 references)
target     prot opt source               destination         
ACCEPT     tcp  --  anywhere             172.17.0.2           tcp dpt:http
...

a new rule is appeared, but is not the only rule added to our chains.

To get a more detailed view of our iptables chain we can dump the full iptables rules with iptables-save:

# Generated by iptables-save v1.8.4 on Thu Oct 14 12:32:46 2021
*mangle
:PREROUTING ACCEPT [33102:3022248]
:INPUT ACCEPT [33102:3022248]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [32349:12119113]
:POSTROUTING ACCEPT [32357:12120329]
COMMIT
# Completed on Thu Oct 14 12:32:46 2021
# Generated by iptables-save v1.8.4 on Thu Oct 14 12:32:46 2021
*nat
:PREROUTING ACCEPT [1:78]
:INPUT ACCEPT [1:78]
:OUTPUT ACCEPT [13:1118]
:POSTROUTING ACCEPT [13:1118]
:DOCKER - [0:0]
:DOCKER-INGRESS - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 172.17.0.2/32 -d 172.17.0.2/32 -p tcp -m tcp --dport 80 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
-A DOCKER ! -i docker0 -p tcp -m tcp --dport 8080 -j DNAT --to-destination 172.17.0.2:80
COMMIT
# Completed on Thu Oct 14 12:32:46 2021
# Generated by iptables-save v1.8.4 on Thu Oct 14 12:32:46 2021
*filter
:INPUT ACCEPT [4758:361293]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [4622:357552]
:DOCKER - [0:0]
:DOCKER-INGRESS - [0:0]
:DOCKER-ISOLATION-STAGE-1 - [0:0]
:DOCKER-ISOLATION-STAGE-2 - [0:0]
:DOCKER-USER - [0:0]
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A DOCKER -d 172.17.0.2/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 80 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
COMMIT
# Completed on Thu Oct 14 12:32:46 2021

in our dump we can see some other rules added by docker:

DOCKER-INGRESS (nat table)

-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 172.17.0.2/32 -d 172.17.0.2/32 -p tcp -m tcp --dport 80 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
-A DOCKER ! -i docker0 -p tcp -m tcp --dport 8080 -j DNAT --to-destination 172.17.0.2:80

DOCKER-USER (filter table)

-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A DOCKER -d 172.17.0.2/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 80 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN

to explore in detail how iptables and docker work:

Docker docs
Docker forum question
gist from x-yuri
argus-sec.com post

The problem

But what happen if we stop and restart our firewall?

systemctl stop ufw|firewalld # <- the service (ufw or firewalld) may change from distro to distro
systemctl stop ufw|firewalld


curl -v http://192.168.25.200:8080
*   Trying 192.168.25.200:8080...
* TCP_NODELAY set


docker run --rm nginx curl ipinfo.io/ip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:06 --:--:--     0

we can see that:

our container is not reachable from the outside world
our container is not able to reach internet

The solution

The solution for this problem is a simple bash script (combined to an awk script) to manage our iptables rules.
In short the script parse the output of the iptables-save command and preserve a set of chains. The chains preserved are:

for table nat:

POSTROUTING
PREROUTING
DOCKER
DOCKER-INGRESS
OUTPUT

for table filter:

FORWARD
DOCKER-ISOLATION-STAGE-1
DOCKER-ISOLATION-STAGE-2
DOCKER
DOCKER-INGRESS
DOCKER-USER

Install iptables-docker

The first step is to clone this repository

Local install (sh)

NOTE this kind of install use a static file (src/iptables-docker.sh). By default only ssh access to local machine is allowd. To allow specific traffic you have to edit manually this file with your own rules:

    # Other firewall rules
    # insert here your firewall rules
    $IPT -A INPUT -p tcp --dport 1234 -m state --state NEW -s 0.0.0.0/0 -j ACCEPT

NOTE2 if you use a swarm cluster uncomment the lines under Swarm mode - uncomment to enable swarm access (adjust source lan) and adjust your LAN subnet

To install iptables-docker on a local machine, clone this repository and run sudo sh install.sh

sudo sh install.sh 

Set iptables to iptables-legacy
Disable ufw,firewalld
Synchronizing state of ufw.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install disable ufw
Failed to stop firewalld.service: Unit firewalld.service not loaded.
Failed to disable unit: Unit file firewalld.service does not exist.
Install iptables-docker.sh
Create systemd unit
Enable iptables-docker.service
Created symlink /etc/systemd/system/multi-user.target.wants/iptables-docker.service → /etc/systemd/system/iptables-docker.service.
start iptables-docker.service

Automated install (ansible)

You can also use ansible to deploy iptables-docker everywhere. To do this adjust the settings under group_vars/main.yml.

Label	Default	Description
`docker_preserve`	`yes`	Preserve docker iptables rules
`swarm_enabled`	`no`	Tells to ansible to open the required ports for the swarm cluster
`ebable_icmp_messages`	`yes`	Enable response to ping requests
`swarm_cidr`	`192.168.1.0/24`	Local docker swarm subnet
`ssh_allow_cidr`	`0.0.0.0/0`	SSH alloed subnet (default everywhere)
`iptables_allow_rules`	`[]`	List of dict to dynamically open ports. Each dict has the following key: desc, proto, from, port. See group_vars/all.yml for examples
`iptables_docker_uninstall`	`no`	Uninstall iptables-docker

Now create the inventory (hosts.ini file) or use an inline inventory and run the playbook:

ansible-playbook -i hosts.ini site.yml

Usage

To start the service use:

sudo systemctl start iptables-docker

or 

sudo iptables-docker.sh start

To stop the srevice use:

sudo systemctl stop iptables-docker

or 

sudo iptables-docker.sh stop

Test iptables-docker

Now if you turn off the firewall with sudo systemctl stop iptables-docker and if you check the iptable-save output, you will see that the docker rules are still there:

sudo iptables-save

# Generated by iptables-save v1.8.4 on Thu Oct 14 15:52:30 2021
*mangle
:PREROUTING ACCEPT [346:23349]
:INPUT ACCEPT [346:23349]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [340:24333]
:POSTROUTING ACCEPT [340:24333]
COMMIT
# Completed on Thu Oct 14 15:52:30 2021
# Generated by iptables-save v1.8.4 on Thu Oct 14 15:52:30 2021
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:DOCKER - [0:0]
:DOCKER-INGRESS - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 172.17.0.2/32 -d 172.17.0.2/32 -p tcp -m tcp --dport 80 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
-A DOCKER ! -i docker0 -p tcp -m tcp --dport 8080 -j DNAT --to-destination 172.17.0.2:80
COMMIT
# Completed on Thu Oct 14 15:52:30 2021
# Generated by iptables-save v1.8.4 on Thu Oct 14 15:52:30 2021
*filter
:INPUT ACCEPT [357:24327]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [355:26075]
:DOCKER - [0:0]
:DOCKER-INGRESS - [0:0]
:DOCKER-ISOLATION-STAGE-1 - [0:0]
:DOCKER-ISOLATION-STAGE-2 - [0:0]
:DOCKER-USER - [0:0]
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A DOCKER -d 172.17.0.2/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 80 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
COMMIT
# Completed on Thu Oct 14 15:52:30 2021

our container is still accesible form the outside:

 curl -v http://192.168.25.200:8080
*   Trying 192.168.25.200:8080...
* TCP_NODELAY set
* Connected to 192.168.25.200 (192.168.25.200) port 8080 (#0)
> GET / HTTP/1.1
> Host: 192.168.25.200:8080
> User-Agent: curl/7.68.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Server: nginx/1.21.1
< Date: Thu, 14 Oct 2021 13:53:33 GMT
< Content-Type: text/html
< Content-Length: 612
< Last-Modified: Tue, 06 Jul 2021 14:59:17 GMT
< Connection: keep-alive
< ETag: "60e46fc5-264"
< Accept-Ranges: bytes

and our container can reach internet:

docker run --rm nginx curl ipinfo.io/ip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    15  100    15    0     0     94      0 --:--:-- --:--:-- --:--:--    94
my-public-ip-address

Important notes

Before install iptables-docker please read this notes:

both local instal and ansible install configure your system to use iptables-legacy
by default only port 22 is allowed
ufw and firewalld will be permanently disabled
filtering on all docker interfaces is disabled

Docker interfaces are:

vethXXXXXX interfaces
br-XXXXXXXXXXX interfaces
docker0 interface
docker_gwbridge interface

Extending iptables-docker

You can extend or modify iptables-docker by editing:

src/iptables-docker.sh for the local install (sh)
roles/iptables-docker/templates/iptables-docker.sh.j2 template file for the automated install (ansible)

Uninstall

Local install (sh)

Run uninstall.sh

Automated install (ansible)

set the variable "iptables_docker_uninstall" to "yes" into group_vars/all.yml and run the playbook.

Deploy a high available etcd cluster using docker

Lorenzo Garuti — Fri, 08 Oct 2021 12:13:12 +0000

etcd is a strongly consistent, distributed key-value store that provides a reliable way to store data that needs to be accessed by a distributed system or cluster of machines. It gracefully handles leader elections during network partitions and can tolerate machine failure, even in the leader node.

In this post we will see how to deploy an etcd cluster using docker and docker compose.

Prerequisites

docker
docker-compose
docker swarm cluster (optional)

Additional requirements for testing purposes:

python3
pipenv
pip3

Configuration overview

Nginx

The nginx configuration is very simple. We need to create one upstream section and declare our etcd server names.

    upstream etcd_servers {
        least_conn;
        server etcd-00:2379 max_fails=3 fail_timeout=5s;
        server etcd-01:2379 max_fails=3 fail_timeout=5s;
        server etcd-02:2379 max_fails=3 fail_timeout=5s;
    }

NOTE: Since we are running on docker the server names are resolved by the internal dns. The name is the name of our services declared in docker-compose.yml (etcd-00, etcd-01, etcd-02)

now we need to declare the server section, with the listen port (etcd default listen port) and the proxy_pass rule which will route the traffic to our etcd services:

    server {
        listen     2379;
        proxy_pass etcd_servers;
    }

the final configuration will look like:

worker_processes 4;
worker_rlimit_nofile 40000;

events {
    worker_connections 8192;
}

stream {
    log_format  basic   '$time_iso8601 $remote_addr '
                        '$protocol $status $bytes_sent $bytes_received '
                        '$session_time $upstream_addr '
                        '"$upstream_bytes_sent" "$upstream_bytes_received" "$upstream_connect_time"';

    access_log  /dev/stdout basic;

    upstream etcd_servers {
        least_conn;
        server etcd-00:2379 max_fails=3 fail_timeout=5s;
        server etcd-01:2379 max_fails=3 fail_timeout=5s;
        server etcd-02:2379 max_fails=3 fail_timeout=5s;
    }
    server {
        listen     2379;
        proxy_pass etcd_servers;
    }
}

Etcd

To configure etcd in cluster mode, on each container we need to specify the following settings:

    command:
      - etcd
      - --name=etcd-02
      - --data-dir=data.etcd
      - --advertise-client-urls=http://etcd-02:2379
      - --listen-client-urls=http://0.0.0.0:2379
      - --initial-advertise-peer-urls=http://etcd-02:2380
      - --listen-peer-urls=http://0.0.0.0:2380
      - --initial-cluster=etcd-00=http://etcd-00:2380,etcd-01=http://etcd-01:2380,etcd-02=http://etcd-02:2380
      - --initial-cluster-state=new
      - --initial-cluster-token=etcd-cluster-1

where:

--name: Human-readable name for this member.
--data-dir: Path to the data directory.
--advertise-client-urls: List of this member’s client URLs to advertise to the rest of the cluster. These URLs can contain domain names.
--listen-client-urls: List of URLs to listen on for client traffic. This flag tells the etcd to accept incoming requests from the clients on the specified scheme://IP:port combinations.
--initial-advertise-peer-urls: List of this member’s peer URLs to advertise to the rest of the cluster. These addresses are used for communicating etcd data around the cluster. At least one must be routable to all cluster members. These URLs can contain domain names.
--listen-peer-urls: List of URLs to listen on for peer traffic. This flag tells the etcd to accept incoming requests from its peers on the specified scheme://IP:port combinations.
--initial-cluster: Initial cluster configuration for bootstrapping.
--initial-cluster-state: Initial cluster state (“new” or “existing”). Set to new for all members present during initial static or DNS bootstrapping. If this option is set to existing, etcd will attempt to join the existing cluster. If the wrong value is set, etcd will attempt to start but fail safely.
--initial-cluster-token: Initial cluster token for the etcd cluster during bootstrap.

All the configuration flags are available here

Deploy etcd cluster with docker compose

The first step is to clone this repository:

git clone https://github.com/garutilorenzo/docker-etcd-cluster.git

then enter in the repo directory an bring up the environment:

cd docker-etcd-cluster 
docker-compose up -d

now check the status of the environment and wait for the containers to be ready:

docker-compose ps


     Name                   Command               State                        Ports                      
----------------------------------------------------------------------------------------------------------
etcd_etcd-00_1   etcd --name=etcd-00 --data ...   Up      2379/tcp, 2380/tcp                              
etcd_etcd-01_1   etcd --name=etcd-01 --data ...   Up      2379/tcp, 2380/tcp                              
etcd_etcd-02_1   etcd --name=etcd-02 --data ...   Up      2379/tcp, 2380/tcp                              
etcd_nginx_1     /docker-entrypoint.sh ngin ...   Up      0.0.0.0:2379->2379/tcp,:::2379->2379/tcp, 80/tcp

Test the environment

To test the environment you need pipenv installed.
Once you have pipenv installed:

pipenv shell
pip install -r requirements.txt
python test/etcd-test.py 
hey key1
hey key2

Check the log of the nginx service to see the traffic redirected to the etcd hosts:

docker-compose logs -f nginx

Attaching to etcd_nginx_1
nginx_1    | /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
nginx_1    | /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
nginx_1    | /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
nginx_1    | 10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
nginx_1    | 10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
nginx_1    | /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
nginx_1    | /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
nginx_1    | /docker-entrypoint.sh: Configuration complete; ready for start up
nginx_1    | 2021-10-08T09:41:23+00:00 172.28.0.1 TCP 200 422 665 0.052 172.28.0.3:2379 "665" "422" "0.000"
nginx_1    | 2021-10-08T09:41:24+00:00 172.28.0.1 TCP 200 422 665 0.046 172.28.0.2:2379 "665" "422" "0.000"
nginx_1    | 2021-10-08T09:50:56+00:00 172.28.0.1 TCP 200 422 665 0.029 172.28.0.4:2379 "665" "422" "0.000"

Docker swarm stack

To deploy the etcd cluster on a docker swarm cluster:

docker stack deploy -c etcd-stack.yml etcd

Check the status of the deployment:

docker stack ps etcd

mx6fvfwye547   etcd_etcd-00.1       quay.io/coreos/etcd:v3.5.0   node-2    Running         Running 3 hours ago                                        
wybd7n4oitae   etcd_etcd-01.1       quay.io/coreos/etcd:v3.5.0   node-4    Running         Running 3 hours ago                                        
rmlycc3uvc8t   etcd_etcd-02.1       quay.io/coreos/etcd:v3.5.0   node-2    Running         Running 3 hours ago                                        
rexh1smoalpo   etcd_nginx.1         nginx:alpine                 node-2    Running         Running 21 hours ago    

docker service ls

ID             NAME                  MODE         REPLICAS   IMAGE                          PORTS
1u709kzgmo2b   etcd_etcd-00          replicated   1/1        quay.io/coreos/etcd:v3.5.0     
m7ze76xi58ww   etcd_etcd-01          replicated   1/1        quay.io/coreos/etcd:v3.5.0     
1535r562g3az   etcd_etcd-02          replicated   1/1        quay.io/coreos/etcd:v3.5.0     
v8n8qlo3dm30   etcd_nginx            replicated   1/1        nginx:alpine                   *:2379->2379/tcp

If you would like to test the swarm setup, open test/etcd-test.py and change 127.0.0.1 with one ip of a server in your docker swarm cluster (or the ip of your LB) and run the test.

DEV Community: Lorenzo Garuti

Install and configure MySQL Server and MySQL InnoDB Cluster

Table of Contents

Role Variables

Vagrant up, build the test infrastructure

Ansible setup and pre-flight check

Deploy MySQL InnoDB Cluster

Cluster high availability check

User and Database creation

Sample Dokcer compose stack

Test the frontend

Simulate disaster

Restore from complete outage

Clean up

Install and configure the ELK stack with Ansible

Table of Contents

Vagrant up, build the test infrastructure

Ansible setup and pre-flight check

Deploy ELK with Ansible

Clean up

Install and configure a high available Kubernetes cluster with Ansible

Requirements

Role Variables

Extra Variables

Cluster resource deployed

Nginx ingress controller

Longhorn

Vagrant

Using this role

Congratulations

Deploy Kubernetes (K8s) on Amazon AWS using mixed on-demand and spot instances

Table of Contents

Requirements

Infrastructure overview

Kubernetes setup

Before you start

Project setup

Generate self signed SSL certificate for the public LB (L7)

AWS provider setup

Pre flight checklist

Deploy

Public LB check

Deploy a sample stack

Clean up

Deploy a Kubernetes cluster for free, using K3s and Oracle always free resources

Table of Contents

Important notes

Requirements

Example RSA key generation

Project setup

Generate sel signed SSL certificate for the public LB (L7)

Oracle provider setup

Pre flight checklist

Generate random token

How to find the availability doamin name

How to list all the OS images

Notes about OCI always free resources

Notes about K3s

Infrastructure overview

Cluster resource deployed

Nginx ingress controller

Deploy

Public LB check

Longhorn check

Deploy a sample stack

Clean up

Build a simple web app using BottlePy

Table of Contents

Requirements

The application stack

Data used

Setup the environmant

Download sample data

Start the environment

Application Overview

Index

Tags

Users

Posts

API/REST endpoint