DEV Community: richamishra006

Simplifying Kubernetes Native Testing with TestKube

richamishra006 — Wed, 26 Jul 2023 13:44:34 +0000

Testing methodology refers to the process and procedures used to evaluate the quality, correctness, and completeness of software applications. The importance of testing methodology lies in its ability to identify defects and issues early in the development process, thus reducing the time and effort to fix them later. By identifying the defects early, testing can help to improve the functionality and reliability of software applications.

There are numerous software testing tools available in the market that serve different purposes and testing requirements. Popular testing tools including Selenium, JMeter, Appium, SoapUI, Postman, and Cucumber might not be a suitable option to test a cloud native application.

Testing cloud native applications is different from normal testing as:

Cloud native applications are typically built using a distributed and microservices-based architecture. This means testing the interactions and integration between various microservices and ensuring seamless collaboration. In contrast, traditional applications often have a monolithic architecture, so you have to focus on testing individual components.
Cloud native applications abstract the underlying infrastructure using services like Infrastructure as a Service (IaaS) or Platform as a Service (PaaS). Testing cloud native applications requires verifying compatibility and functionality across different cloud providers and configurations. Traditional applications are usually developed and tested within a specific infrastructure environment.

Due to the complex nature of cloud native applications, a cloud native testing framework is a better choice as it enables developers and testers to build and test applications designed to run on cloud platforms. Cloud native testing tools are developed specifically for cloud native environments. Most importantly, they allow you to deploy tests in your clusters and make the executions super scalable. Plus, they are not coupled to any CI/CD framework such as Jenkins, GitHub Actions, etc.

One such cloud native testing framework is Testkube - which will be the focus of this blog post. We will cover what Testkube is, its capabilities, and how you can run tests in your Kubernetes cluster using it.

What is Testkube?

Testkube is an open source, cloud native testing framework that simplifies the testing of Kubernetes applications. It allows users to store, execute and manage tests as part of a Kubernetes cluster. With Testkube, users can utilize multiple testing frameworks, orchestrate tests and perform automated testing. Additionally, it has a user-friendly web-based UI and CLI (command line interface) for better visibility and management.

Now we have an understanding of Testkube and how it simplifies the testing of Kubernetes applications, let’s dive into its capabilities and use cases.

Capabilities of Testkube

Run your tests inside your cluster: Testkube enables users to run their tests directly inside their Kubernetes cluster instead of executing them from a CI pipeline. This provides a significant networking security advantage, as it eliminates the need to expose the cluster to the outside world in order to test its applications. By keeping the testing environment within the cluster, Testkube helps to ensure the integrity and security of the testing process.
Support for multiple testing frameworks: With a flexible test execution framework that comes with pre-built executors for popular testing tools like Postman, Cypress, and K6, Testkube enables users to leverage existing testing assets. It also provides the option to create custom executors for any type of tests users want to run within their cluster.
Test orchestration: Testkube empowers users to orchestrate multiple tests, making it easier to manage complex testing scenarios.
Integration with CI/CD: You can integrate Testkube with CI/CD tool of your choice including Jenkins, TravisCI, CircleCI and make it easier to automate testing.
Centralized dashboard and storage of test artifacts: Testkube provides web-based UI, CLI, and a centralized dashboard for testing which helps in better visibility and management of tests. It enables you to retrieve the results of the tests generated by the testing tools from its storing facilities.

When it comes to use cases, Testkube can be used by developers, DevOps teams, and QA teams for testing Kubernetes applications at various stages of the development process, from local development to production. For instance, developers can use Testkube to test their application logic, while expert DevOps teams can use it to automate tests in their CI/CD pipeline. QA teams can use Testkube to ensure that Kubernetes applications are functioning correctly in different scenarios.

Now we know what Testkube can do, we will see it in action. We will set it up to create and run a few tests, learn about test triggers, and use GitHub Actions that will allow us to enable automated testing with each PR.

Let's proceed with the installation of Testkube.

Testkube installation

First, we will install Testkube CLI.

Step 1: Installation of Testkube CLI

We will use the script installation method:

curl -sSLf https://get.testkube.io | sh

After running the above command, Testkube CLI will be installed.
Next, we will install the cluster component using this CLI.

Step 2: Install Testkube cluster components using Testkube's CLI

We can also use Helm chart to deploy the Testkube cluster component. However, here we will be using Testkube CLI.

testkube init

Just run the above command, and the components will be installed. Make sure that the kubeconfig file points to the desired cluster where the Testkube needs to be installed.

The following components will be installed, once we run the above command:

Testkube API
Testkube namespace
CRDs for Tests, TestSuites, Executors
MongoDB - It is used to store test results and other Testkube configurations.
Minio - default (can be disabled with --no-minio) - Minio is used as a storage server to store the test artifacts
NATS - It is used as a event bus to store all the test related events
Dashboard - default (can be disabled with --no-dashboard flag)

To know more about the setup configurations, refer to the official Testkube installation documentaion.

Now, let’s check if all the components are deployed by running the following command.

➜ kubectl get pods -n testkube
NAME                                                    READY   STATUS    RESTARTS   AGE
testkube-api-server-76cb76dd9c-9tvmr                    1/1     Running   0          3m59s
testkube-dashboard-56c78fdc79-nw7cj                     1/1     Running   0          3m59s
testkube-minio-testkube-bd549c85d-rrwlq                 1/1     Running   0          3m59s
testkube-mongodb-d78699775-48jkm                        1/1     Running   0          3m59s
testkube-nats-0                                         3/3     Running   0          3m59s
testkube-nats-box-5b555bc9c4-w9zfj                      1/1     Running   0          3m59s
testkube-operator-controller-manager-7cb8cdcbb9-smcqr   2/2     Running   0          3m59s

As we can see, all components are running fine. Next, we will be creating and running tests using Testkube.

Create and run tests

I have created a basic Django application which prints “The install worked successfully” available at this repo. For setting up this application, I am using minikube enabled with ingress. So first we will deploy this application in our cluster and then will create tests for the same.

Clone the repository and create a pod and service by running the below commands.

git clone https://github.com/infracloudio/testkube-django-app.git
cd myproject
kubectl apply -f deployment.yaml -f service.yaml -f ingress.yaml

Once we run the above command, a pod, service and ingress will be created.

➜ kubectl get pods | grep myproject
myproject-7777f9f588-xbcnr   1/1     Running   0             60s

After the ingress is created, we will update the /etc/hosts with the hostname and IP of the ingress as shown in the screenshot below so that we can access the application endpoint at django-test.com.
```
➜ kubectl get ingress
NAME                 CLASS   HOSTS             ADDRESS        PORTS   AGE
django-app-ingress   nginx   django-test.com   192.168.49.2   80      17h
```

After the application pod is created, we will create integration and load tests.

Integration tests

I have written a sample integration test in Go, so we will use Ginkgo executor which supports the tests written in Go.

For creating the test, run the below command.

   kubectl testkube create test \
     --git-uri https://github.com/infracloudio/testkube-django-app.git \
     --git-path myproject/examples/ \
     --type ginkgo/test \
     --name django-app-test \
     --git-branch master

git-uri is the url of GitHub repository where the test case exists

Once we run the above command, a test will be created. We can check it by running a command as shown below.

➜ kubectl get tests.tests.testkube.io -A
NAMESPACE   NAME         AGE
testkube    django-app-test   31m

Next, we will access the Testkube UI by running the below command. Once we execute this command, a web page will open on the default browser at localhost:8080
```
testkube dashboard
```

Now we can see that the test is created, so we will run the tests by clicking on Run now option as shown below.

After we click on Run now, the test will be triggered and we can see the output if the test has passed or not.

So we can see that the tests have passed.

Similarly, we can see an overview of the past results on the dashboard, along with other details like execution duration, etc.

Load tests

Now we will use the Testkube K6 executor and perform a load test for the same application we deployed above.
The load test script is available as loadtest.js file in the demo repo.

You can run the following command to create the test.

   testkube create test --name k6-test --type k6/script \
     --test-content-type git-dir \
     --git-uri https://github.com/infracloudio/testkube-django-app.git \
     --git-branch master \
     --git-path myproject/ \
     --executor-args "--vus" \
     --executor-args "5" --executor-args "--duration" \
     --executor-args "10m" --executor-args loadtest.js

Here vus stands for virtual users. We will create the test with 5 users for 10m duration.

After executing the above command, let's check if the test has been created.

➜ kubectl get tests.tests.testkube.io -A
NAMESPACE   NAME              AGE
testkube    django-app-test   5h52m
testkube    k6-test           9s
3. Once the test has been created, we will run it from the UI and check the output, we need to wait for a few minutes to get the load generated for 5 users and then check the output.

So we can see from the screenshot that the load test ran successfully with 5 virtual users.

Test triggers

Testkube allows us to automate running tests and TestSuites by defining triggers on certain events for various Kubernetes resources. A trigger defines an action that will be executed for a given execution when a certain event on a specific resource occurs. For example, we could define a trigger that runs a test when a configmap gets modified or a pod gets restarted.

To test out this feature, we will create a yaml file:



apiVersion: tests.testkube.io/v1
kind: TestTrigger
metadata:
  name: testtrigger-example
  namespace: testkube
spec:
  resource: configmap
  resourceSelector:
    labelSelector:
      matchLabels:
        app: test
  event: modified
  action: run
  execution: test
  testSelector:
    name: django-app-test
    namespace: testkube



kubectl apply -f testtrigger.yaml

We will check if the test trigger is created by running this command:



➜ kubectl get testtriggers.tests.testkube.io -A
NAMESPACE   NAME                  RESOURCE    EVENT      EXECUTION   AGE
testkube    testtrigger-example   configmap   modified   test        48m

We will create a configmap with label app=test and as soon as we modify the configmap, the test will be triggered. So in this way, we can trigger the required tests based on certain conditions like the rollout of deployment, configmap modification, pod restart, etc.

Integration with CI/CD

Here comes the most interesting part of this post – we can trigger tests in Testkube from the CI/CD pipeline. This practice decouples running your tests from the CI/CD, removes all the complexity and speeds up your CI.

In this blog post, we will be using Github Actions which enables us to run the Testkube CLI commands in a GitHub workflow. So first we will talk about setting up access to EKS from GH (GitHub) actions. We are using EKS in this case, but any other cloud platforms can also be used.

Pre-requisites

For using Github Actions to perform tasks on the cluster, we should be ready with the below points:

An EKS cluster
An IAM user with permission of EKS cluster
Update the aws-auth configmap on the cluster with username and role arn as shown in the screenshot below

Note that to keep the post simple, we are mapping testkube user to system:masters group. For an actual setup, consider creating a specific ClusterRole/ClusterRoleBinding for a group and mapping that group with testkube user to grant only minimal access. Take a look at the EKS docs for details, or use a Service Account token based kubeconfig for GitHub Actions.

We need to create a few secrets in GitHub settings under the Secrets section as shown in the screenshot below. The access key and secret key should be of the user generated in the second step. We will be using these secrets as environment variables in the GitHub actions configuration file.

Configuration

We need to create a .github/workflow/testkube.yaml from the Actions tab. Testkube.yaml contains two jobs, job1 will establish connectivity with EKS cluster and job2 will create tests using Testkube.
Here is the content of testkube.yaml



name: Testkube Docker Action
on:
 push:
 pull_request:
   branches:
     - master
env:
 AWS_ACCESS_KEY_ID: $`{{ secrets.AWS_ACCESS_KEY_ID }}`
 AWS_SECRET_ACCESS_KEY: $`{{ secrets.AWS_SECRET_ACCESS_KEY }}`
 CLUSTER_NAME: $`{{ secrets.CLUSTER_NAME }}`
jobs:
 job_1:
   name: test workflow for AWS
   runs-on: ubuntu-latest
   steps:
   - name: AWS cli install action
     uses: chrislennon/action-aws-cli@1.1
   - name: Configure AWS credentials
     uses: aws-actions/configure-aws-credentials@v1
     with:
       aws-access-key-id: $`{{ secrets.AWS_ACCESS_KEY_ID }}`
       aws-secret-access-key: $`{{ secrets.AWS_SECRET_ACCESS_KEY }}`
       aws-region: ap-south-1
   - name: kubeconfig
     run: |
       aws eks update-kubeconfig --name $`{{ env.CLUSTER_NAME }}` --region $`{{ env.AWS_REGION }}`
   - name: Upload kubeconfig file
     uses: actions/upload-artifact@v3
     with:
       name: kubeconfig
       path: /home/runner/.kube/config
 job_2:
   name: create test using testkube
   needs: job_1
   runs-on: ubuntu-latest 
   steps:
   - name: Download kubeconfig from job 1
     uses: actions/download-artifact@v3
     with:
      name: kubeconfig
   - shell: bash
     run: |
       export KUBECONFIG=/home/runner/work/testkube-test/testkube-test/config
   - name: Create test
     id: create_test
     uses: kubeshop/testkube-docker-action@v1.3
     env:
       KUBECONFIG: /home/runner/work/testkube-test/testkube-test/config
       KUBERNETES_MASTER: /home/runner/work/testkube-test/testkube-test/config
     with:
       command: create
       resource: test
       namespace: testkube
       parameters: "--git-uri https://github.com/infracloudio/testkube-django-app.git --git-path myproject/examples/ --type ginkgo/test --name testkube-github-action --git-branch master"

If you wish, you can try creating another test or use the same example used in this blog post. Once you push the changes, the workflow will get triggered. We can check it in the Actions section.

Once the job is completed successfully, you will see that the test has been created in the cluster.



➜ kubectl get tests -A
NAMESPACE   NAME                     AGE
testkube    testkube-github-action   18m

Summary

Testkube is a test execution and orchestration framework for Kubernetes. It has the capability to run any testing tool and integrates with any CI/CD application. It defines tests as Kubernetes CRDs to provide a modern solution for managing all your tests, eliminating CI/CD bottlenecks, and scaling your tests with your needs.

In this blog post, we have focused mainly on the Testkube installation, test creation and execution (for a demo application), test triggers, and integration with CI/CD. With basic groundwork covered, you can run your first test using Testkube.

I hope you found this post informative and engaging. For more posts like this one, do subscribe to our newsletter for a weekly dose of cloud native. I’d love to hear your thoughts on this post, let’s connect and start a conversation on LinkedIn.

Looking to be truly cloud native? Learn why so many startups and enterprise trust InfraCloud as one of the best cloud native consulting services provider for Kubernetes adoption and Day 2 operations.

References

Postgres data migration from VM server to Kubernetes

richamishra006 — Wed, 30 Nov 2022 06:58:53 +0000

PostgreSQL is a powerful, open-source object-relational database system that safely stores and scales data workloads. It uses SQL and is widely used due to its reliability, data integrity, robustness, and many more features. While the world is moving towards Kubernetes to run their entire application ecosystem, migrating the stateful side of the ecosystem to Kubernetes is still not as straightforward. One has to be very sure of how without any data loss and downtime, we can migrate the databases to Kubernetes.

Being one of the widely used and popular relational databases, PostgreSQL DB can run on various platforms such as macOS, Windows, Linux, etc. We can set up PostgreSQL on virtual machines, on cloud or physical machines or use containerized databases .

In this blog post, we will be discussing a scenario, where we will learn how to migrate PostgreSQL data from a virtual machine to Kubernetes setup . While there are various ways to install postgres on Kubernetes, the best approach is to always look for an operator which can take away most of the operational and administration burden. One such operator is Kubergres, which we are going to use due to its simplicity and features. To know more about Kubegres, check out their official documentation

What is Kubegres? [Postgres operator for k8s]

Kubegres is a Kubernetes operator which helps us to deploy clusters of PostgreSQL instances with data replication and failover enabled out of the box. It brings simplicity when using PostgreSQL with Kubernetes.

Features:

It provides data replication, and replicates data from primary PostgreSQL instance to replica instances in real time
It manages failover by promoting the replica instance as primary PostgreSQL in case of failure
It has option to schedule backup of database and dump the data in a separate persistent volume

Having learnt about the features of Kubgres, let's proceed with the installation.

Prerequisites

One needs to have a Postgres database installed and running on a virtual machine. Also, access to a Kubernetes cluster will be required where we'll be migrating the database to.

Installation of Kubegres

Kubegres is an open source tool. We can deploy it on Kubernetes by using the Kubegres operator

Install Kubegres operator

Run the following command in a Kubernetes cluster:

kubectl apply -f https://raw.githubusercontent.com/reactive-tech/kubegres/v1.15/kubegres.yaml

After executing the above command, a kubegres-system namespace will be created, where the controller will be installed. we will see a deployment running in the kubegres-system namespace.

We will see the below output

namespace/kubegres-system created
customresourcedefinition.apiextensions.k8s.io/kubegres.kubegres.reactive-tech.io created
serviceaccount/kubegres-controller-manager created
role.rbac.authorization.k8s.io/kubegres-leader-election-role created
clusterrole.rbac.authorization.k8s.io/kubegres-manager-role created
clusterrole.rbac.authorization.k8s.io/kubegres-metrics-reader created
clusterrole.rbac.authorization.k8s.io/kubegres-proxy-role created
rolebinding.rbac.authorization.k8s.io/kubegres-leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/kubegres-manager-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/kubegres-proxy-rolebinding created
configmap/kubegres-manager-config created
service/kubegres-controller-manager-metrics-service created
deployment.apps/kubegres-controller-manager created

Create a Secret resource

Before creating a cluster for PostgreSQL, we will create a secret with the password of Postgres user, replication user and production database user. We will be using this later in the PostgreSQL configuration

Create a file:

vi postgresql-secret.yaml

Add the following contents:

apiVersion: v1
kind: Secret
metadata:
  name: postgres-production-secret
  namespace: default
type: Opaque
data:
  superUserPassword: S2p4dVhpUmlLVWNBVj0=
  replicationUserPassword: cmVwbGljYXRpb24=
  password: cHJvZHVjdGlvbg==

Apply the changes:

kubectl apply -f postgresql-secret.yaml

Create a ConfigMap

Next we will create a configmap,as Kubegres allows us to override its default configurations and bash scripts based on our requirements. The base configmap contains the default configs of all Kubegres resources in that namespace.
We can override the following configurations by creating our configmap:
postgres.conf- the official PostgreSQL configs used for both primary and replica servers.
pg_hba.conf- the host-based authentication configs for both primary and replica servers.
primary_init_script.sh- a bash script which runs for the first time when a primary PostgreSql container is created. Here we can add instructions to create custom databases.
backup_database.sh- this bash script defines the actions to perform during a backup. It is executed regularly by a dedicated cronjob

Create a file:

vi postgres-configmap.yaml

Add the following contents:

apiVersion: v1
kind: ConfigMap
metadata:
  name: postgres-configmap
  namespace: default
data:

  postgres.conf: |2
    # Replication configs
    listen_addresses = '*'
    max_wal_senders = 10
    max_connections = 5000
    shared_buffers = 128MB
    wal_level= logical
    log_min_duration_statement= 5000


    # Logging
    log_destination = 'stderr,csvlog'
    logging_collector = on
    log_directory = 'pg_log'
    log_filename= 'postgresql-%Y-%m-%d_%H%M%S.log'


  pg_hba.conf: |

    # TYPE  DATABASE        USER            ADDRESS                 METHOD
    # Replication connections by a user with the replication privilege
    host    replication     replication     all                     md5
    # As long as it is authenticated, all connections allowed except from "0.0.0.0/0"
    local   all             all                                     md5
    host    all             all             all                     md5
    host    all             all             0.0.0.0/0               reject
    host production production 127.0.0.1/32 trust
    host production production ::1/128 trust
    host production production 0.0.0.0/0 trust
    host production postgres 127.0.0.1/32 trust

  primary_init_script.sh: |
    #!/bin/bash
    set -e

    # This script assumes that the env-var $POSTGRES_MY_DB_PASSWORD contains the password of the custom user to create.
    # You can add any env-var in your Kubegres resource config YAML.

    dt=$(date '+%d/%m/%Y %H:%M:%S');
    echo "$dt - Running init script the 1st time Primary PostgreSql container is created...";

    customDatabaseName="production"
    customUserName="production"

    echo "$dt - Running: psql -v ON_ERROR_STOP=1 --username $POSTGRES_USER --dbname $POSTGRES_DB ...";

    psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" <<-EOSQL
    CREATE DATABASE $customDatabaseName;
    CREATE USER $customUserName WITH PASSWORD '$POSTGRES_PRODUCTION_PASSWORD';
    GRANT ALL PRIVILEGES ON DATABASE "$customDatabaseName" to $customUserName;
    EOSQL

    echo "$dt - Init script is completed";

  backup_database.sh: |
    #!/bin/bash
    set -e

    dt=$(date '+%d/%m/%Y %H:%M:%S');
    fileDt=$(date '+%d_%m_%Y_%H_%M_%S');
    backUpFileName="$KUBEGRES_RESOURCE_NAME-backup-$fileDt.gz"
    backUpFilePath="$BACKUP_DESTINATION_FOLDER/$backUpFileName"

    echo "$dt - Starting DB backup of Kubegres resource $KUBEGRES_RESOURCE_NAME into file: $backUpFilePath";
    echo "$dt - Running: pg_dumpall -h $BACKUP_SOURCE_DB_HOST_NAME -U postgres -c | gzip > $backUpFilePath"

    pg_dumpall -h $BACKUP_SOURCE_DB_HOST_NAME -U postgres -c | gzip > $backUpFilePath

    if [ $? -ne 0 ]; then
      rm $backUpFilePath
      echo "Unable to execute a BackUp. Please check DB connection settings"
      exit 1
    fi

    echo "$dt - DB backup completed for Kubegres resource $KUBEGRES_RESOURCE_NAME into file: $backUpFilePath";

Apply the changes:

kubectl apply -f postgres-configmap.yaml

In order to create a cluster of PostgreSQL, we need to create a Kubegres resource. We will be creating a file “kubegres.yaml” with the below content. This yaml file will create one primary PostgreSql pod and two replica PostgreSql pods. The data will be replicated in real time from the Primary pod to the 2 Replica pods.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: postgresql-backup-pvc
  namespace: default
spec:
  accessModes:
  - ReadWriteOnce
  storageClassName: standard-lrs
  resources:
    requests:
      storage: 35Gi

---
apiVersion: kubegres.reactive-tech.io/v1
kind: Kubegres
metadata:
  name: production-postgresql
  namespace: default
  annotations:
    imageregistry: "https://hub.docker.com/"
    nginx.ingress.kubernetes.io/affinity: cookie

spec:
  replicas: 3
  image: postgres:13.2
  port: 5432

  database:
     size: 50G
     storageClassName: standard-lrs
     volumeMount: /var/lib/postgresql/data

  customConfig: postgres-configmap

  failover:
     isDisabled: false

  backup:
     schedule: "0 */2 * * *"
     pvcName: postgresql-backup-pvc
     volumeMount: /var/lib/backup

  env:
     - name: POSTGRES_PRODUCTION_PASSWORD
       valueFrom:
         secretKeyRef:
           name: postgres-production-secret
           key: password
     - name: POSTGRES_PASSWORD
       valueFrom:
          secretKeyRef:
             name: postgres-production-secret
             key: superUserPassword

     - name: POSTGRES_REPLICATION_PASSWORD
       valueFrom:
          secretKeyRef:
             name: postgres-production-secret
             key: replicationUserPassword

In the above file, for creating a persistent volume claim, use the storage class as per your setup ( replace standard-lrs with the storage class available in your cluster). Also, you can update the image version of Postgres with the latest image.

Apply the changes:

kubectl apply -f kubegres.yaml

Once we run the above command, three replicas of PostgreSQL will spin up. In our file, we have also scheduled a database backup of the database, which will be stored in a separate persistent volume.

kubectl get pods
NAME                        READY   STATUS    RESTARTS   AGE
production-postgresql-1-0   1/1     Running   0          2m31s
production-postgresql-2-0   1/1     Running   0          2m20s
production-postgresql-3-0   1/1     Running   0          2m6s

Once we have our Kubegres pods up and running, we will create a sample database with few tables in the PostgreSQL VM and try to migrate it to the Kubegres setup in Kubernetes.

Sample Database and data on Postgres(VM) server

Considering PostgreSQL server is already running on a virtual machine, we will create a database “production”, insert some data into it, and then verify the same after migration is done.

First, we will check if the Postgres service is up and running:

postgres@richa-mishra:~$ systemctl status postgresql
● postgresql.service - PostgreSQL RDBMS
     Loaded: loaded (/lib/systemd/system/postgresql.service; enabled; vendor preset: enabled)
     Active: active (exited) since Mon 2022-07-25 08:51:59 IST; 9h ago
    Process: 1233 ExecStart=/bin/true (code=exited, status=0/SUCCESS)
   Main PID: 1233 (code=exited, status=0/SUCCESS)

Now we will create a database called “production”. Switch to user postgres and run:

psql

We will log in into the database with postgres user and \create a database:

postgres=# CREATE DATABASE production;

Let's connect to this production database and insert some data:

postgres=# \c production
You are now connected to database "production" as user "postgres".

create a table and add some data into it:

production=#CREATE TABLE employee (
    employee_id INT PRIMARY KEY,
    first_name VARCHAR (255) NOT NULL,
    last_name VARCHAR (255) NOT NULL,
    manager_id INT,
    FOREIGN KEY (manager_id) 
    REFERENCES employee (employee_id) 
    ON DELETE CASCADE
);

production=#INSERT INTO employee (
    employee_id,
    first_name,
    last_name,
    manager_id
)
VALUES
    (1, 'Sandeep', 'Jain', NULL),
    (2, 'Abhishek ', 'Kelenia', 1),
    (3, 'Harsh', 'Aggarwal', 1),
    (4, 'Raju', 'Kumar', 2),
    (5, 'Nikhil', 'Aggarwal', 2),
    (6, 'Anshul', 'Aggarwal', 2),
    (7, 'Virat', 'Kohli', 3),
    (8, 'Rohit', 'Sharma', 3);

Output:

production=# SELECT * FROM employee;
 employee_id | first_name | last_name | manager_id 
-------------+------------+-----------+------------
           1 | Sandeep    | Jain      |           
           2 | Abhishek   | Kelenia   |          1
           3 | Harsh      | Aggarwal  |          1
           4 | Raju       | Kumar     |          2
           5 | Nikhil     | Aggarwal  |          2
           6 | Anshul     | Aggarwal  |          2
           7 | Virat      | Kohli     |          3
           8 | Rohit      | Sharma    |          3
(8 rows)

Now we have a database called production, a table and some data in it. After migration we will verify if the same data exists on Kubegres. For performing data migration, we will be performing a few steps for establishing network connectivity between PostgreSQL on VM and Kubegres on Kubernetes. Once the connectivity is established, we will be running a job which will dump the data from the VM and restore it to the database on Kubernetes. Lets perform this activity step by step as explained below.

Migration of data from Postgres Server to Kubegres:

We will edit pg_hba.conf file of PostgreSQL(VM) and add an entry to allow connections from Kubegres, basically will try to establish an authentication between kubegres and Postgres on VM
So switch to postgres user and run the below command:

vim /etc/postgresql/13/main/pg_hba.conf

In my case the version of PostgreSQL(VM) is 13, so the folder name is 13.

We will add the following line having IP address of Kubegres:

host    production     all    172.18.0.2/32         trust

After making changes to pg_hba.conf, make sure to restart the postgresql service

systemctl restart postgresql

Next we will create a Service and an endpoint to establish connectivity between Postgres setup on VM and Kubernetes.
Services when used with a corresponding Endpoints object and without a selector can help us to connect the Postgres on VM ( backends outside the cluster). So we will create a service and an endpoint object manually.
When you create an Endpoints object for a Service, you set the name of the endpoint to be the same as that of the Service.
In the Endpoint, under addresses we will mention the IP address of Postgres(VirtualMachine)

vi external-service.yaml

Add the below contents:

apiVersion: v1
kind: Service
metadata:
  name: pg
spec:
  clusterIP: None
  ports:
  - name: postgres
    port: 5432
    targetPort: 5432
  selector: null
---
apiVersion: v1
kind: Endpoints
metadata:
  name: pg
subsets:
- addresses:
  - ip: 192.168.1.41
  ports:
  - name: postgres
    port: 5432
    protocol: TCP

kubectl apply -f external-service.yaml

Now, we will create a configmap, which will contain all details of our setup as follows:
DB: Database name which needs to be migrated. We have already created a new database on Kubernetes setup with the same name “production” at the time of installation.
USERNAME: username/role having access of the DB
PASSWORD: password of the username
SRC: It refers to the host name of Postgres(VM), so here we will add the external service name and namespace which we created in above step
DST: It refers to the hostname of Kubegres setup

vi migration-cm.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: postgres-migration-cm
  labels:
    app: production-postgresql
data:
    DB: production
    USER: postgres
    PGPASSWORD: KjxuXiRiKUcAV=
    #     svcname.namespace
    SRC: "pg.default"
    #     podname.servicename[.namespace]
    DST: "production-postgresql-1-0.production-postgresql"

kubectl apply -f migration-cm.yaml

Since we have all the details of source and destination database in the configmap which we created above, we will run the migration job
Create a file:

vim job.yaml

Add the below contents:

apiVersion: batch/v1
kind: Job
metadata:
  name: populate-db
spec:
  backoffLimit: 0
  template:
    spec:
      restartPolicy: Never
      containers:
      - name: pg
        image: crunchydata/crunchy-postgres:centos8-13.5-4.7.4
        command: ["/bin/sh"]
        args:
        - -c
        - "pg_dump -h $SRC -U $USER --format c $DB | pg_restore --verbose -h $DST -U $USER --format c --dbname $DB"
        envFrom:
        - configMapRef:
            name: postgres-migration-cm

kubectl apply -f job.yaml

Once we run the above command, a job will be created and a container will start spinning, We will see the output as shown below:

kubectl get pods | grep populate
NAME                        READY   STATUS      RESTARTS      AGE
populate-db-txgkt           0/1     Completed   0             2s

we will check the logs of this container. If there is some issue with the connectivity or any errors, you will be able to see them in the logs.

kubectl logs populate-db-txgkt
pg_restore: connecting to database for restore
pg_restore: creating TABLE "public.employee"
pg_restore: processing data for table "public.employee"
pg_restore: creating CONSTRAINT "public.employee employee_pkey"
pg_restore: creating FK CONSTRAINT "public.employee employee_manager_id_fkey"

Here we can see in the logs that database connectivity and restore is successful.

To verify if the same table and data has been created in the Kubegres pod, we will exec into the pod and verify the data.

kubectl exec -it production-postgresql-1-0 bash
root@production-postgresql-1-0:/# su postgres
postgres@production-postgresql-1-0:/$ cd
postgres@production-postgresql-1-0:~$ psql -d production
Password for user postgres: 
psql (14.1 (Debian 14.1-1.pgdg110+1))
Type "help" for help.
production=# \dt
          List of relations
 Schema |   Name   | Type  |  Owner   
--------+----------+-------+----------
 public | employee | table | postgres
(1 row)

production=#  SELECT * FROM employee;
 employee_id | first_name | last_name | manager_id 
-------------+------------+-----------+------------
           1 | Sandeep    | Jain      |           
           2 | Abhishek   | Kelenia   |          1
           3 | Harsh      | Aggarwal  |          1
           4 | Raju       | Kumar     |          2
           5 | Nikhil     | Aggarwal  |          2
           6 | Anshul     | Aggarwal  |          2
           7 | Virat      | Kohli     |          3
           8 | Rohit      | Sharma    |          3
(8 rows)

We will execute the above commands, step by step.
First we will exec into the pod by running:

kubectl exec -it production-postgresql-1-0 bash

Then we will switch to user postgres and type psql -d production to connect to the production database. Once connected after providing the right credentials, we will be running the same query which we ran in the sample database section, to verify the consistency of data.
SELECT * FROM employee;
This query lists the same data which we have inserted in Postgres VM.

So the data has been transferred successfully. Now based on your setup, you can switch the application connectivity from Postgres VM to Kubegres setup which will basically require a merge request with few changes, once the MR is merged the application will start connecting to the kubegres setup, without any downtime and without any data loss.

Conclusion

These were the few steps to transfer the data of a database from a Virtual Machine to Kubernetes setup. Using these steps, we could migrate a production database in VM to kubernetes setup with zero downtime and data loss.
This blog post can be helpful even if you use any other database instead of PostgreSQL and are looking for a migration solution from a VM to Kubernetes setup.
To understand more about Kubegres, Please refer to their official doc and Github page.

That’s all for this post. If you are working on a similar scenario and need some assistance, feel free to reach out to me via LinkedIn, you can also read other blog posts that I’ve written on InfraCloud. I’m always excited to hear thoughts and feedback from my readers!