DEV Community: Naresh Maharaj

How to Copy GCP Storage Buckets Between Accounts Without Using the Internet

Naresh Maharaj — Wed, 12 Feb 2025 17:00:11 +0000

In this article, we’ll walk you through the process of copying a Google Cloud Platform (GCP) storage bucket and its contents from one GCP account to another, entirely within GCP’s internal network—without relying on the internet. This method is ideal for securely transferring large amounts of data between accounts while maintaining high performance and security.

Prerequisites

Before we begin, ensure you have the following:

Two GCP Accounts: You’ll need access to both the source and destination GCP accounts.
Projects in Each Account:
- In the source account, create a project named SourceBucketProject.
- In the destination account, create a project named DestinationBucketProject.

In account #1 I have a project named SourceBucketProject

In the destination project I called it DestinationBucketProject

Step 1: Create a Source Bucket in Account#1

Navigate to Cloud Storage > Buckets in the source account (SourceBucketProject).
Click Create to create a new bucket. Name it my-src-bucket-001.
Inside the bucket, create a folder named folder1 and upload a sample file to it.

Step 2: Set Up a Transfer Job in Account #2

In the destination account (DestinationBucketProject), search for Transfer Job in the GCP Console.

Click Create Transfer Job.
Set both the source and destination as Google Cloud Storage.

Specify the source bucket as my-src-bucket-001 and choose or create a destination bucket with a unique name (e.g., my-dest-bucket-010).
Start the transfer process.

At this point, you may encounter an error related to permissions. This is expected and will be resolved in the next steps.

Step 3: Grant Necessary Permissions

Note the Principal Service Account mentioned in the error message. It will look something like this:

project-1069962656103@storage-transfer-service.iam.gserviceaccount.com.
Go back to the source account (SourceBucketProject) and navigate to IAM & Admin > Roles.
Create a new custom role named GsBucketDataTransferRead. Add the following permissions (which are derived from built-in roles like Storage Object Admin and Storage Legacy Bucket Reader):
- resourcemanager.projects.get
- storage.buckets.get
- storage.folders.get
- storage.folders.list
- storage.managedFolders.get
- storage.managedFolders.list
- storage.multipartUploads.list
- storage.objects.get
- storage.objects.list
Save the custom role.
Navigate to IAM & Admin > IAM and click Grant Access.
Add the Principal Service Account (noted earlier) and assign it the custom role GsBucketDataTransferRead.

Step 4: Complete the Transfer

Return to the destination account (DestinationBucketProject) and retry the transfer job.
Once the transfer is complete, you’ll see a confirmation message.

Step 5: Verify the Data

Navigate to the destination bucket (my-dest-bucket-010) and confirm that the file from the source bucket has been successfully transferred.

Conclusion

And that’s it! You’ve successfully copied a GCP storage bucket and its contents from one account to another without using the internet. This method leverages GCP’s internal network for secure and efficient data transfer, making it an excellent choice for enterprise-level data migrations.

We hope this guide has been helpful. If you have any questions or run into issues, feel free to reach out in the comments below!
New chat

Implementing RBAC in Kubernetes: A Real-World Example

Naresh Maharaj — Tue, 21 Jan 2025 22:08:34 +0000

Implementing RBAC in Kubernetes: A Real-World Example

In this article, we'll dive into how to implement Role-Based Access Control (RBAC) in Kubernetes through a practical, real-world example. While many tutorials cover RBAC concepts, they often remain abstract or overly simplified. By walking through an actual use case, you'll gain a better understanding of how to apply RBAC effectively in a Kubernetes cluster.

What is RBAC?

RBAC stands for Role-Based Access Control. It's a method for regulating access to resources in a system based on the roles of individual users within an organization. Essentially, RBAC is a policy-neutral access control mechanism that assigns permissions to users according to their roles. The key components of RBAC in Kubernetes include:

Roles: Define what actions are allowed on which resources.
RoleBindings: Link roles to users, granting them the associated permissions.
ServiceAccounts: Represent identities for processes running inside pods.

RBAC simplifies user assignments and ensures that access control is clear, consistent, and manageable.

What is a Role?

In Kubernetes, a Role is a collection of permissions that define what actions are allowed on certain resources. Roles can be applied to users, groups, or even other roles (creating hierarchies).

For example:

An "Admin" role might have permissions to create, read, update, and delete resources.
A "Developer" role might only have permissions to create, read, and update resources.
A "Viewer" role might only have permissions to read resources.

Roles are a key part of RBAC as they encapsulate the actions users or processes can perform within Kubernetes.

What is a RoleBinding?

A RoleBinding is the mechanism that binds a Role to a user, a group of users, or even other roles. A RoleBinding effectively grants the permissions defined in a Role to whoever is bound to it.

For example:

You can bind the "Admin" role to a user, which grants them full access to Kubernetes resources.
You can also create a RoleBinding that binds the "Viewer" role to a group of users, allowing them to only read the resources.

RoleBindings are essential for controlling access to Kubernetes resources, ensuring that users have the correct permissions according to their roles.

Real-World Example: A Python Server and Client in Kubernetes

In this example, we will implement RBAC in a Kubernetes environment to control access between two applications: a server and a client.

Server: This application generates a table of random numbers.
Client: This application consumes the random number data from the server.

We will containerize both applications using Docker and then deploy them to a Kubernetes cluster. Finally, we'll create a RoleBinding to allow the client to access the server.

Prerequisites

Before we begin, ensure you have the following installed on your machine:

Docker (e.g., Docker Desktop)
Kubernetes (e.g., Minikube, kind or a local Kubernetes setup)
Python (preferably version 3.x)
pip (Python package manager)
virtualenv (for managing Python environments)
kubectl (Kubernetes command-line tool)
netcat (for network testing)

Clone the Git Repository to Follow Along

Access the source code on GitHub.

Server Code

The server code is responsible for generating random numbers and exposing them over a network. It is located under: kubernetes_rbac_real_example/randomNumberServer/server.py

Create the Docker Image for the Server

First, we need to create a Dockerfile for the server: kubernetes_rbac_real_example /randomNumberServer/Dockerfile

FROM python:3.7
COPY . /app
WORKDIR /app
CMD ["python", "server.py"]

To build the Docker image:

export dockerhub_username=<your_dockerhub_username>
docker build -t ${dockerhub_username}/random-number-server .
docker push ${dockerhub_username}/random-number-server

Client Code

The client code connects to the server and retrieves the random numbers. It is located under: kubernetes_rbac_real_example/randomNumberClient/client.py

Create the Docker Image for the Client

Create a Dockerfile for the client: kubernetes_rbac_real_example /randomNumberClient/Dockerfile

FROM python:3.7
COPY . /app
WORKDIR /app
CMD ["python", "client.py"]

To build the Docker image:

export dockerhub_username=<your_dockerhub_username>
docker build -t ${dockerhub_username}/random-number-client .
docker push ${dockerhub_username}/random-number-client

Running the Docker Containers

Next, we'll set up a Docker network to allow communication between the server and client containers:

docker network create -d bridge random-net

Now, run the server container:

docker run -it -d --name random-number-server -p 3215 --network=random-net --hostname random01 random-number-server

Run the client container:

docker run -it -d --name random-number-client -p 3216:3216 --network=random-net -e RANDOM_SERVER=random01 --hostname=random02 random-number-client

Testing the Setup with `netcat`

In a new shell, use netcat to connect to the client container's port and view the random numbers it receives:

nc localhost 3216

You'll be prompted to enter parameters for the number generation. Here's an example of output you should see:

Enter 3 numbers min,max,cols separated by commas: 1,900,12

565    636    362    538    483    103    898    188    81    432    245    519
120    644    866    487    407    534    156    870    630    418    581    231
174    43     675    9      380    60     555    127    505    471    764    191
...

Kubernetes Environment using Kind

Visit the Kind documentation for detailed instructions on setting up a Kubernetes cluster using Kind.

Once installed and your cluster set up, check the nodes in your cluster:

kind get nodes

kind-worker2
kind-control-plane
kind-worker
kind-worker3

Deploying RBAC to Kubernetes

Once we've confirmed that the Docker containers work correctly, we can deploy both the server and client to a Kubernetes cluster.

Create a pod file to deploy the server and client. If you wish change the image to use your own dockerhub account. E.g. if your account name is johnmcollins, then the image is set to johnmcollins/random-number-server and johnmcollins/random-number-client.

First, create the namespace:

kubectl create namespace random-numbers

Server

cat <<EOF> random-number-server.yaml
apiVersion: v1
kind: Pod
metadata:
  labels:
    app: random-number-server
  name: random-number-server
  namespace: random-numbers
spec:
  containers:
  - name: random-number-server
    image: contactnkm/random-number-server:latest
    ports:
    - containerPort: 3215
EOF

Create the service for this pod:

cat <<EOF> random-number-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: random-number-service
  namespace: random-numbers
spec:
  selector:
    app: random-number-server
  ports:
    - protocol: TCP
      port: 3215
      targetPort: 3215
EOF

Run the following commands to deploy the random number server and random number service:

kubectl apply -f random-number-server.yaml
kubectl apply -f random-number-service.yaml

We need to ensure our service DNS is working correctly. We can do this by creating a pod that will run a nslookup command to the service.

cat <<EOF> dnsutils.yaml
apiVersion: v1
kind: Pod
metadata:
  name: dnsutils
  namespace: default
spec:
  containers:
  - name: dnsutils
    image: gcr.io/kubernetes-e2e-test-images/dnsutils:1.3
    command:
      - sleep
      - "3600"
    imagePullPolicy: IfNotPresent
  restartPolicy: Always
EOF

kubectl apply -f dnsutils.yaml

Run the nslookup command to verify the service:

kubectl exec -ti dnsutils -n random-numbers -- nslookup random-number-service
Server:         10.96.0.10
Address:        10.96.0.10#53

Name:   random-number-service.random-numbers.svc.cluster.local
Address: 10.96.35.141

So we now know for sure our DNS is working correctly, and our FQDN is random-number-service.random-numbers.svc.cluster.local.

In the client pod, set the environment variable RANDOM_NUMBER_SERVER to the FQDN of the server service.

Client

cat <<EOF> random-number-client.yaml
apiVersion: v1
kind: Pod
metadata:
  labels:
    app: random-number-client
  name: random-number-client
  namespace: random-numbers
spec:
  containers:
  - name: random-number-client
    image: contactnkm/random-number-client:latest
    ports:
    - containerPort: 3216
    env:
    - name: RANDOM_SERVER
      value: "random-number-service.random-numbers.svc.cluster.local"
EOF

Create a random number client service:

cat <<EOF> random-number-client-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: random-number-client-service
  namespace: random-numbers
spec:
  selector:
    app: random-number-client
  ports:
    - protocol: TCP
      port: 3216
      targetPort: 3216
EOF

Deploy the client and random number client service to the cluster:

kubectl apply -f random-number-client.yaml
kubectl apply -f random-number-client-service.yaml

Check that all the service endpoints are created; it should not show none:

kubectl get ep -n random-numbers

NAME                           ENDPOINTS          AGE
random-number-client-service   10.244.2.6:3216    12s
random-number-service          10.244.1.12:3215   87s

Testing the Client and Server

Connect to the host using the dnsutils pod and test the client and server:

kubectl exec -ti dnsutils -n random-numbers -- sh

Test the application and use Ctrl-C to exit:

nc random-number-client-service 3216

Enter 3 numbers min,max,cols separated by commas: 10,36,6
16      19      17      30      31      25
28      14      35      32      29      34
20      18      12      33      27      24
36      23      22      26      21      10
15      11      13

Using RBAC in Kubernetes

The next step is to set up RBAC to allow the client to interact with the server.

Create a Service Account.

kubectl create sa random-numbers-sa -n  random-numbers

Step 1: Create a Role for Access

First, we create a Role that defines the permissions for accessing the server. This role grants read access to the server's resources.

For example, the role could be defined in a YAML file:

cat <<EOF> roles.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: random-numbers
  name: client-access-role
rules:
- apiGroups: [""]
  resources:
    - configmaps
    - services # Add other resources as needed
  verbs: 
    - get
    - list # Add other verbs as needed
EOF

Step 2: Create a RoleBinding

Now, we create a RoleBinding to bind the "client-access-role" to the client pod. This will grant the client the necessary permissions to interact with the server pod.

For example, the RoleBinding might look like this:

cat <<EOF> role-binding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: client-access-role-binding
  namespace: random-numbers
subjects:
- kind: ServiceAccount
  name: random-numbers-sa
  namespace: random-numbers
roleRef:
  kind: Role
  name: client-access-role
  apiGroup: rbac.authorization.k8s.io
EOF

Testing in Kubernetes

After deploying the server and client containers to Kubernetes, you can test the connection between them and verify that the RoleBinding is working. We will do this with a simple curl application.

Create the curl application:

cat <<EOF> curlapp.yaml
apiVersion: v1
kind: Pod
metadata:   
  name: curlo
  namespace: random-numbers
  labels:
    app: curlo
spec:
  serviceAccountName: random-numbers-sa
  containers:
  - name: curlo
    image: curlimages/curl
    command: ["sleep","999999"]
EOF

kubectl apply -f curlapp.yaml

Connect to the host using the following command

kubectl exec -it curlo -n random-numbers -- /bin/sh

In order to use curl with https we will need to use the certificate file ca.crt file and token in the curl command.

cat /var/run/secrets/kubernetes.io/serviceaccount/token > TOKEN
export TOKEN=$(cat TOKEN)
curl -k --header "Authorization: Bearer $TOKEN" --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt https://kubernetes.default.svc

For various reasons which I will let you discover, you will notice how this can be shortened to:

export CURL_CA_BUNDLE=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
curl -k --header "Authorization: Bearer $TOKEN" https://kubernetes.default.svc

With the following command you should be able to see the service we had created earlier

curl -k --header "Authorization: Bearer $TOKEN" https://kubernetes.default.svc/api/v1/namespaces/random-numbers/services/random-number-service

Result:

{
  "kind": "Service",
  "apiVersion": "v1",
  "metadata": {
    "name": "random-number-service",
    "namespace": "random-numbers",
    "uid": "504cd44b-7e10-4ea2-9ee8-a912909f6dee",
    "resourceVersion": "183093",
    "creationTimestamp": "2025-01-19T19:52:28Z",
    "managedFields": [
  ...
  "status": {
    "loadBalancer": {}
  }

Try to list the other pods that are running in the cluster within the same namespace using the following curl command. You should get a 403 Forbidden error.

curl -k --header "Authorization: Bearer $TOKEN" https://kubernetes.default.svc/api/v1/namespaces/random-numbers/pods

Result

{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "pods is forbidden: User \"system:serviceaccount:random-numbers:random-numbers-sa\" cannot list resource \"pods\" in API group \"\" in the namespace \"random-numbers\"",
  "reason": "Forbidden",
  "details": {
    "kind": "pods"
  },
  "code": 403
}

Great! Now we have achieved the exact level of role-based access control needed to limit access exclusively to services and configmaps within the random-numbers namespace.

Next, create a ConfigMap to store the min, max, and table -size values for our random number generator. These values will be used by the client pod to generate the random number table.

kubectl create configmap random-number-config --from-literal=min=1 --from-literal=max=100 --from-literal=table-size=10 -n random-numbers

Here is the yaml file version of the same command:

cat <<EOF> random-number-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: random-number-config
  namespace: random-numbers
data:
  min: "1"
  max: "100"
  table-size: "10"
  table-name: "random-numbers"
EOF

kubectl apply -f random-number-configmap.yaml

Python Kubernetes API Client

We need to modify the client pod to use the Kubernetes API to get the configmap values and use them to generate the random number table.

We will use the Kubernetes API client for Python to get the configmap values.

The source code can be found here: kubernetes_rbac_real_example/randomNumberClientCM/client-cm.py

We will need to add the kubernetes library to our Docker image. Use pip to install the Kubernetes API client for Python in your Dockerfile ( note we have already done this for you ).

pip install kubernetes

Here is the code required to get the configmap values:

import os
from kubernetes import client, config
def get_configmap_values():
    config.load_incluster_config()
    v1 = client.CoreV1Api()
    configmap = v1.read_namespaced_config_map(name="random-number-config", namespace="random-numbers")
    min_value = int(configmap.data["min"])
    max_value = int(configmap.data["max"])
    table_size = int(configmap.data["table-size"])
    return min_value, max_value, table_size

Build a new docker image called {your-username}/random-number-client-cm and push it to docker hub.

docker build -t {your-username}/random-number-client-cm .
docker push {your-username}/random-number-client-cm

Lets double check we have our service account added the client pod definition and the lastest docker image for the client pod. Remember, the service account is used to restrict access to the Kubernetes API using RBAC.

apiVersion: v1
kind: Pod
metadata:
  labels:
    app: random-number-client
  name: random-number-client
  namespace: random-numbers
spec:
  serviceAccountName: random-numbers-sa
  containers:
  - name: random-number-client
    image: {your-docker-username}/random-number-client-cm:latest
    ports:
    - containerPort: 3216
    env:
    - name: RANDOM_SERVER
      value: "random-number-service.random-numbers.svc.cluster.local"

kubectl delete -f random-number-client.yaml
kubectl apply -f random-number-client.yaml

If everything is set up correctly, the client should be able to access the server and receive the random number table, just like it did in the Docker environment but can't do very much else. Let's test it out:

kubectl exec -ti dnsutils -n random-numbers -- sh

/ # nc random-number-client-service 3216

Press [Enter] to get a new set of values using the kubernetes config map: 
Will send these values to server: 1,100,10
Waiting for server response...
74      80      40      26      17      72      87      38      64      79
97      96      63      86      7       58      75      52      76      32
47      29      5       83      68      90      60      100     16      70
84      9       44      37      48      49      98      81      27      54
45      61      51      25      43      57      65      89      92      19
22      24      55      28      53      15      13      8       10      39
20      77      31      93      91      12      62      94      34      42
35      1       67      85      78      59      14      99      11      33
21      30      95      2       6       82      4       3       56      66
50      69      18      23      46      71      36      88      41      73

Press [Enter] to get a new set of values using the kubernetes config map:

Now try changing the min, max and table-size values in the config map and see if the client can pick up the new values.

apiVersion: v1
kind: ConfigMap
metadata:
  name: random-number-config
  namespace: random-numbers
data:
  min: "64"
  max: "256"
  table-size: "8"
  table-name: "random-numbers"
...

Save the file apply the changes.

kubectl apply -f random-number-configmap.yaml

Visit the window where you are running the dnsutils pod and hit enter again in the terminal. This should now fetch a new table using the updated min,max and table-size values.

Press [Enter] to get a new set of values using the kubernetes config map: 
Will send these values to server: 64,256,8
Waiting for server response...
219     195     91      88      104     93      134     144
246     131     241     239     128     210     190     114
163     183     191     161     115     145     160     70
136     218     171     67      98      237     248     256
227     173     80      202     206     71      129     126
249     154     245     216     193     204     92      66
75      169     106     189     155     185     238     179
103     230     99      221     149     137     152     138
117     174     100     247     215     170     105     68
120     74      180     141     231     186     139     250
79      122     209     121     127     123     83      157
94      235     199     211     201     125     188     95
229     205     220     198     147     236     217     176
97      167     213     234     203     225     84      172
208     142     254     150     175     148     87      228
158     253     135     64      242     81      233     212
133     72      118     73      243     143     226     159
146     140     178     77      124     151     112     109
164     184     251     153     96      197     65      240
78      132     166     252     165     194     102     244
86      187     116     232     222     196     110     111
119     90      168     255     101     76      82      85
162     89      108     69      223     200     214     182
207     192     113     177     130     156     107     181
224

Conclusion

In this article, we've walked through a real-world example of using RBAC in a Kubernetes environment. We demonstrated how to set up roles and RoleBindings to control access between two applications—one acting as a server and the other as a client. By building the applications with Docker, deploying them to Kubernetes, and using RBAC, you now have a solid understanding of how to implement access control in a Kubernetes cluster.

Remember, RBAC is a powerful feature that helps you ensure that users and services in your cluster have the appropriate permissions based on their roles. With RBAC, you can implement fine-grained access control to safeguard your resources.

MongoDB Sharding - Single Node Macbook - Quick Setup

Naresh Maharaj — Sat, 05 Oct 2024 18:09:48 +0000

MongoDB Sharding - Single Node Macbook - Quick Setup

There are times when I need to quickly set up a shard, or even a MongoDB cluster with 2 or 3 shards. This guide makes it simple and configurable to do so with a single command. No Docker is involved—just plain old MongoDB, specifically version 8, which is the latest at the time of writing.

Git repo here in case you want to skip reading this:
https://github.com/nareshmaharaj-consultant/mongodb-sharding-bash

Overview

Let's begin by installing all the necessary binaries.

mongosh: Download the MongoDB shell, as we'll need this multiple times throughout the setup.
mongod: Grab the latest MongoDB server, required for each instance.

After downloading, unzip the contents and move the bin folder contents to your system’s environment paths.

You should now be able to run the following commands to verify the versions of both.

mongosh --version
2.3.1

mongod --version
db version v8.0.0
Build Info: {
    "version": "8.0.0",
    "gitVersion": "d7cd03b239ac39a3c7d63f7145e91aca36f93db6",
    "modules": [
        "enterprise"
    ],
    "allocator": "system",
    "environment": {
        "distarch": "x86_64",
        "target_arch": "x86_64"
    }
}
mongos --version
mongos version v8.0.0
Build Info: {
    "version": "8.0.0",
    "gitVersion": "d7cd03b239ac39a3c7d63f7145e91aca36f93db6",
    "modules": [
        "enterprise"
    ],
    "allocator": "system",
    "environment": {
        "distarch": "x86_64",
        "target_arch": "x86_64"
    }
}

Once everything is confirmed, we're good to go. We'll follow the sequence below:

Replica Sets: First, we'll create a simple replica set and then tear it down.
Shards: Set up a sharded cluster with 2 shards.
Data: Insert some sample data.
Collections: Shard the collection.
Config Shard: Convert the config server(s) into a data shard.
Wrap up: What’s next?

Replica Set

Start by creating a file named mongo-replicaset-create to set up the replica set.

NAME=$1
PORT=$2
DIR=~/data/$NAME
NODES=$3
USAGE="\nUsage:  mongo-replicaset-create {name} {port} {numNodes} { Optional:[ --shardsvr, --configsvr ] }\nSet env variable \$HOST for all nodes to use this domain name."
SLEEPTIME=8
CONFIGSVR=""
SHARDSVR=""
GIVEN_HOSTNAME=localhost

if [ -z "$HOST" ]
then
      echo "\$HOST not set using localhost"
else
      GIVEN_HOSTNAME=$HOST
fi

if [ "$1" == "-h" ]; then
  echo -e $USAGE
  exit 0
fi

if [ -z "$1" ]
  then
    echo -e "No NAME argument supplied $USAGE"
    exit 0
fi

if [ -z "$2" ]
  then
    echo -e "No PORT argument supplied $USAGE"
    exit 0
fi

if [ -z "$3" ]
  then
    echo -e "Number of NODES  argument supplied $USAGE"
    exit 0
fi

# idiomatic parameter and option handling in sh
while test $# -gt 0
do
    case "$4" in
        --shardsvr) echo "Using option --shardsvr"
        SHARDSVR="--shardsvr"
            ;;
        --configsvr) echo "Using option --configsvr"
        CONFIGSVR="--configsvr"
            ;;
        --*) echo "bad option $1"
            ;;
    esac
    shift
done

rm -fr $DIR
mkdir -p $DIR

COUNTER=0
  until [  $COUNTER -ge $NODES ]; do
    LOC=$DIR/$COUNTER
    mkdir -p $LOC
    PC=$(($PORT+$COUNTER))
    echo "[ ++ Executing:  mongod --port $PC --fork --dbpath $LOC --logpath $LOC/log.out -replSet $NAME $CONFIGSVR $SHARDSVR ]"
    mongod --port $PC --fork --bind_ip localhost,$GIVEN_HOSTNAME --dbpath $LOC --logpath $LOC/log.out --replSet $NAME $CONFIGSVR $SHARDSVR
    let COUNTER+=1
  done

echo "Sleeping for $SLEEPTIME seconds"
sleep $SLEEPTIME

echo "[ ++ Executing:  mongosh --port $PORT -eval \"rs.initiate()\" ]"
mongosh --port $PORT -eval "rs.initiate()"
sleep $SLEEPTIME

COUNTER=1
  until [  $COUNTER -ge $NODES ]; do
    PC=$(($PORT+$COUNTER))
    if [ $COUNTER -gt 6 ]; then
    echo "[ ++ Executing: mongosh --port $PORT -eval \"rs.add({host: '$GIVEN_HOSTNAME:$PC', priority:0, votes:0})\" ]"
    mongosh --port $PORT -eval "rs.add({host: '$GIVEN_HOSTNAME:$PC', priority:0, votes:0})"
    else
        echo "[ ++ Executing:  mongosh --port $PORT -eval \"rs.add('$GIVEN_HOSTNAME:$PC')\" ]"
        mongosh --port $PORT -eval "rs.add('$GIVEN_HOSTNAME:$PC')"
    fi
    let COUNTER+=1
  done

sleep $SLEEPTIME
mongosh --port $PORT -eval "rs.status()"

Create the replica set and capture the stdout if you want to inspect the output later. It makes sense to set the HOST variable so that it can be used as a Fully Qualified Domain Name (FQDN) during the creation process.

# Set the host
export HOST=`hostname`

# Check we set it correctly
env | grep HOST
HOST=nmaharaj-MBP-MMD6T.local

Run the script

mongo-replicaset-create myrs 27017 3

Here is the result and output

[ ++ Executing:  mongod --port 27017 --fork --dbpath /Users/nareshmaharaj/data/myrs/0 --logpath /Users/nareshmaharaj/data/myrs/0/log.out -replSet myrs   ]
about to fork child process, waiting until server is ready for connections.
forked process: 43633
child process started successfully, parent exiting
[ ++ Executing:  mongod --port 27018 --fork --dbpath /Users/nareshmaharaj/data/myrs/1 --logpath /Users/nareshmaharaj/data/myrs/1/log.out -replSet myrs   ]
about to fork child process, waiting until server is ready for connections.
forked process: 43664
child process started successfully, parent exiting
[ ++ Executing:  mongod --port 27019 --fork --dbpath /Users/nareshmaharaj/data/myrs/2 --logpath /Users/nareshmaharaj/data/myrs/2/log.out -replSet myrs   ]
about to fork child process, waiting until server is ready for connections.
forked process: 43689
child process started successfully, parent exiting
Sleeping for 8 seconds
[ ++ Executing:  mongosh --port 27017 -eval "rs.initiate()" ]
{
  info2: 'no configuration specified. Using a default configuration for the set',
  me: 'nmaharaj-MBP-MMD6T.local:27017',
  ok: 1,
  '$clusterTime': {
    clusterTime: Timestamp({ t: 1728146621, i: 1 }),
    signature: {
      hash: Binary.createFromBase64('AAAAAAAAAAAAAAAAAAAAAAAAAAA=', 0),
      keyId: Long('0')
    }
  },
  operationTime: Timestamp({ t: 1728146621, i: 1 })
}
[ ++ Executing:  mongosh --port 27017 -eval "rs.add('nmaharaj-MBP-MMD6T.local:27018')" ]
{
  ok: 1,
  '$clusterTime': {
    clusterTime: Timestamp({ t: 1728146631, i: 1 }),
    signature: {
      hash: Binary.createFromBase64('AAAAAAAAAAAAAAAAAAAAAAAAAAA=', 0),
      keyId: Long('0')
    }
  },
  operationTime: Timestamp({ t: 1728146631, i: 1 })
}
[ ++ Executing:  mongosh --port 27017 -eval "rs.add('nmaharaj-MBP-MMD6T.local:27019')" ]
{
  ok: 1,
  '$clusterTime': {
    clusterTime: Timestamp({ t: 1728146633, i: 1 }),
    signature: {
      hash: Binary.createFromBase64('AAAAAAAAAAAAAAAAAAAAAAAAAAA=', 0),
      keyId: Long('0')
    }
  },
  operationTime: Timestamp({ t: 1728146633, i: 1 })
}
{
  set: 'myrs',
  date: ISODate('2024-10-05T16:44:02.989Z'),
  myState: 1,
  term: Long('1'),
  syncSourceHost: '',
  syncSourceId: -1,
  heartbeatIntervalMillis: Long('2000'),
  majorityVoteCount: 2,
  writeMajorityCount: 2,
  votingMembersCount: 3,
  writableVotingMembersCount: 3,
  optimes: {
    lastCommittedOpTime: { ts: Timestamp({ t: 1728146641, i: 1 }), t: Long('1') },
    lastCommittedWallTime: ISODate('2024-10-05T16:44:01.930Z'),
    readConcernMajorityOpTime: { ts: Timestamp({ t: 1728146641, i: 1 }), t: Long('1') },
    appliedOpTime: { ts: Timestamp({ t: 1728146641, i: 1 }), t: Long('1') },
    durableOpTime: { ts: Timestamp({ t: 1728146641, i: 1 }), t: Long('1') },
    writtenOpTime: { ts: Timestamp({ t: 1728146641, i: 1 }), t: Long('1') },
    lastAppliedWallTime: ISODate('2024-10-05T16:44:01.930Z'),
    lastDurableWallTime: ISODate('2024-10-05T16:44:01.930Z'),
    lastWrittenWallTime: ISODate('2024-10-05T16:44:01.930Z')
  },
  lastStableRecoveryTimestamp: Timestamp({ t: 1728146621, i: 1 }),
  electionCandidateMetrics: {
    lastElectionReason: 'electionTimeout',
    lastElectionDate: ISODate('2024-10-05T16:43:42.053Z'),
    electionTerm: Long('1'),
    lastCommittedOpTimeAtElection: { ts: Timestamp({ t: 1728146621, i: 1 }), t: Long('-1') },
    lastSeenWrittenOpTimeAtElection: { ts: Timestamp({ t: 1728146621, i: 1 }), t: Long('-1') },
    lastSeenOpTimeAtElection: { ts: Timestamp({ t: 1728146621, i: 1 }), t: Long('-1') },
    numVotesNeeded: 1,
    priorityAtElection: 1,
    electionTimeoutMillis: Long('10000'),
    newTermStartDate: ISODate('2024-10-05T16:43:42.465Z'),
    wMajorityWriteAvailabilityDate: ISODate('2024-10-05T16:43:42.735Z')
  },
  members: [
    {
      _id: 0,
      name: 'nmaharaj-MBP-MMD6T.local:27017',
      health: 1,
      state: 1,
      stateStr: 'PRIMARY',
      uptime: 40,
      optime: { ts: Timestamp({ t: 1728146641, i: 1 }), t: Long('1') },
      optimeDate: ISODate('2024-10-05T16:44:01.000Z'),
      optimeWritten: { ts: Timestamp({ t: 1728146641, i: 1 }), t: Long('1') },
      optimeWrittenDate: ISODate('2024-10-05T16:44:01.000Z'),
      lastAppliedWallTime: ISODate('2024-10-05T16:44:01.930Z'),
      lastDurableWallTime: ISODate('2024-10-05T16:44:01.930Z'),
      lastWrittenWallTime: ISODate('2024-10-05T16:44:01.930Z'),
      syncSourceHost: '',
      syncSourceId: -1,
      infoMessage: 'Could not find member to sync from',
      electionTime: Timestamp({ t: 1728146622, i: 1 }),
      electionDate: ISODate('2024-10-05T16:43:42.000Z'),
      configVersion: 5,
      configTerm: 1,
      self: true,
      lastHeartbeatMessage: ''
    },
    {
      _id: 1,
      name: 'nmaharaj-MBP-MMD6T.local:27018',
      health: 1,
      state: 2,
      stateStr: 'SECONDARY',
      uptime: 11,
      optime: { ts: Timestamp({ t: 1728146641, i: 1 }), t: Long('1') },
      optimeDurable: { ts: Timestamp({ t: 1728146641, i: 1 }), t: Long('1') },
      optimeWritten: { ts: Timestamp({ t: 1728146641, i: 1 }), t: Long('1') },
      optimeDate: ISODate('2024-10-05T16:44:01.000Z'),
      optimeDurableDate: ISODate('2024-10-05T16:44:01.000Z'),
      optimeWrittenDate: ISODate('2024-10-05T16:44:01.000Z'),
      lastAppliedWallTime: ISODate('2024-10-05T16:44:01.930Z'),
      lastDurableWallTime: ISODate('2024-10-05T16:44:01.930Z'),
      lastWrittenWallTime: ISODate('2024-10-05T16:44:01.930Z'),
      lastHeartbeat: ISODate('2024-10-05T16:44:02.075Z'),
      lastHeartbeatRecv: ISODate('2024-10-05T16:44:02.075Z'),
      pingMs: Long('9'),
      lastHeartbeatMessage: '',
      syncSourceHost: 'nmaharaj-MBP-MMD6T.local:27017',
      syncSourceId: 0,
      infoMessage: '',
      configVersion: 5,
      configTerm: 1
    },
    {
      _id: 2,
      name: 'nmaharaj-MBP-MMD6T.local:27019',
      health: 1,
      state: 2,
      stateStr: 'SECONDARY',
      uptime: 9,
      optime: { ts: Timestamp({ t: 1728146637, i: 1 }), t: Long('1') },
      optimeDurable: { ts: Timestamp({ t: 1728146637, i: 1 }), t: Long('1') },
      optimeWritten: { ts: Timestamp({ t: 1728146637, i: 1 }), t: Long('1') },
      optimeDate: ISODate('2024-10-05T16:43:57.000Z'),
      optimeDurableDate: ISODate('2024-10-05T16:43:57.000Z'),
      optimeWrittenDate: ISODate('2024-10-05T16:43:57.000Z'),
      lastAppliedWallTime: ISODate('2024-10-05T16:44:01.930Z'),
      lastDurableWallTime: ISODate('2024-10-05T16:44:01.930Z'),
      lastWrittenWallTime: ISODate('2024-10-05T16:44:01.930Z'),
      lastHeartbeat: ISODate('2024-10-05T16:44:02.078Z'),
      lastHeartbeatRecv: ISODate('2024-10-05T16:44:02.579Z'),
      pingMs: Long('3'),
      lastHeartbeatMessage: '',
      syncSourceHost: '',
      syncSourceId: -1,
      infoMessage: '',
      configVersion: 5,
      configTerm: 1
    }
  ],
  ok: 1,
  '$clusterTime': {
    clusterTime: Timestamp({ t: 1728146641, i: 1 }),
    signature: {
      hash: Binary.createFromBase64('AAAAAAAAAAAAAAAAAAAAAAAAAAA=', 0),
      keyId: Long('0')
    }
  },
  operationTime: Timestamp({ t: 1728146641, i: 1 })
}

All the commands that were run are shown with the prefix: ++ Executing: ...

Next, log in to the host.

mongosh nmaharaj-MBP-MMD6T.local:27019

Now, tear everything down by running the following commands. Make sure to wait for all processes to be completely removed.

kill `ps -elf | grep mongo | awk '{print $2}' | tr '\n' ' ' | awk '{$1=$1};1'` > /dev/null 2>&1

Sharded Cluster

To create a sharded cluster with a config server replicaset and a mongos router, create a file named mongo-shard-create with the following contents.

PORT=$1
SHARDS=$2
LISTNAMES=$3
DIR=~/data/mongos/
USAGE="Usage:  mongo-shard-create {port} {numShards} { shardName1,shardName2,..}"
GIVEN_HOSTNAME=localhost
SLEEPTIME=1

if [ -z "$HOST" ]
then
      echo "\$HOST is empty but is optional - will use localhost"
else
      GIVEN_HOSTNAME=$HOST
fi


if [ -z "$1" ]
  then
    echo "No PORT argument supplied - $USAGE"
    exit 0
fi

if [ -z "$2" ]
  then
    echo "Number of Shards required - $USAGE"
    exit 0
fi

if [ -z "$3" ]
  then
    echo "Name of each shard csv required - $USAGE"
    exit 0
fi

# Split the names for each RS
rs_name_arr=($(echo "$3" | sed "s/ //g" | tr ',' '\n'))
count=${#rs_name_arr[@]}
if [ $count -lt $SHARDS ]
  then
    echo "Shard names(s) count does not match number of Shards required - $USAGE"
    exit 0
fi

echo "##########################################################################################################"
echo "######################################### Creating Replica Sets ##########################################"
echo "##########################################################################################################"
COUNTER=1
  until [  $COUNTER -gt $SHARDS ]; do
  name=${rs_name_arr[$COUNTER-1]}
    PC=$(($PORT+1000*$COUNTER))
    #PC=$(($PORT+17+1000*$COUNTER))
    mongo-replicaset-create $name $PC 3 --shardsvr
    let COUNTER+=1
  done

echo "##########################################################################################################"
echo "######################################### Creating Config Sets ###########################################"
echo "##########################################################################################################"
# Config Servers
#PC=$(($PORT+17+$SHARDS*1000+1000))
PC=$(($PORT+$SHARDS*1000+1000))
mongo-replicaset-create csrs $PC 3 --configsvr

echo " *** Adding user: 'admin', pwd: 'password' to Config Server - please change password *** "
CSRS_HOSTS="csrs/$GIVEN_HOSTNAME:$PC,$GIVEN_HOSTNAME:$(($PC+1)),$GIVEN_HOSTNAME:$(($PC+2))"
mongosh --host $CSRS_HOSTS --eval "db.getSiblingDB('admin').createUser({ user: 'admin', pwd: 'password', roles: [ {role: 'root', db: 'admin'} ] })"


echo "##########################################################################################################"
echo "######################################### Creating mongos #### ###########################################"
echo "##########################################################################################################"
LOG_DIR_MONGOS=~/data/mongos/
mkdir -p $LOG_DIR_MONGOS
#echo "[ ++ Executing:  mongos --port $(($PORT+17)) --bind_ip_all --fork --logpath $LOG_DIR_MONGOS/mongos.log --configdb $CSRS_HOSTS ]"
#mongos --port $(($PORT+17)) --bind_ip_all --fork --logpath $LOG_DIR_MONGOS/mongos.log --configdb $CSRS_HOSTS
echo "[ ++ Executing:  mongos --port $(($PORT)) --bind_ip_all --fork --logpath $LOG_DIR_MONGOS/mongos.log --configdb $CSRS_HOSTS ]"
mongos --port $(($PORT)) --bind_ip_all --fork --logpath $LOG_DIR_MONGOS/mongos.log --configdb $CSRS_HOSTS

COUNTER=1
  until [  $COUNTER -gt $SHARDS ]; do
  name=${rs_name_arr[$COUNTER-1]}
    #PC=$(($PORT+17+1000*$COUNTER))
    #echo "[ ++ Executing: mongosh --port $(($PORT+17)) --eval \"sh.addShard('$name/$GIVEN_HOSTNAME:$PC')\" ]"
    #mongosh --port $(($PORT+17)) --eval "sh.addShard('$name/$GIVEN_HOSTNAME:$PC')"
    PC=$(($PORT+1000*$COUNTER))
    echo "[ ++ Executing: mongosh --port $(($PORT)) --eval \"sh.addShard('$name/$GIVEN_HOSTNAME:$PC')\" ]"
    mongosh --port $(($PORT)) --eval "sh.addShard('$name/$GIVEN_HOSTNAME:$PC')"
    let COUNTER+=1
  done

sleep 5
#mongosh --port $(($PORT+17)) --eval "sh.status()"
mongosh --port $(($PORT)) --eval "sh.status()"

Run the following command to create a MongoDB sharded cluster with 2 shards.

mongo-shard-create 27017 2 sh1,sh2 | tee ~/out.log

Here is the output; however, we have omitted most of it due to its large size. We have included all the executed commands and the last part of the stdout.

# commands executed
cat ~/out.log | grep -e Exec -e pass
[ ++ Executing:  mongod --port 28017 --fork --dbpath /Users/nareshmaharaj/data/sh1/0 --logpath /Users/nareshmaharaj/data/sh1/0/log.out -replSet sh1  --shardsvr ]
[ ++ Executing:  mongod --port 28018 --fork --dbpath /Users/nareshmaharaj/data/sh1/1 --logpath /Users/nareshmaharaj/data/sh1/1/log.out -replSet sh1  --shardsvr ]
[ ++ Executing:  mongod --port 28019 --fork --dbpath /Users/nareshmaharaj/data/sh1/2 --logpath /Users/nareshmaharaj/data/sh1/2/log.out -replSet sh1  --shardsvr ]
[ ++ Executing:  mongosh --port 28017 -eval "rs.initiate()" ]
[ ++ Executing:  mongosh --port 28017 -eval "rs.add('nmaharaj-MBP-MMD6T.local:28018')" ]
[ ++ Executing:  mongosh --port 28017 -eval "rs.add('nmaharaj-MBP-MMD6T.local:28019')" ]
[ ++ Executing:  mongod --port 29017 --fork --dbpath /Users/nareshmaharaj/data/sh2/0 --logpath /Users/nareshmaharaj/data/sh2/0/log.out -replSet sh2  --shardsvr ]
[ ++ Executing:  mongod --port 29018 --fork --dbpath /Users/nareshmaharaj/data/sh2/1 --logpath /Users/nareshmaharaj/data/sh2/1/log.out -replSet sh2  --shardsvr ]
[ ++ Executing:  mongod --port 29019 --fork --dbpath /Users/nareshmaharaj/data/sh2/2 --logpath /Users/nareshmaharaj/data/sh2/2/log.out -replSet sh2  --shardsvr ]
[ ++ Executing:  mongosh --port 29017 -eval "rs.initiate()" ]
[ ++ Executing:  mongosh --port 29017 -eval "rs.add('nmaharaj-MBP-MMD6T.local:29018')" ]
[ ++ Executing:  mongosh --port 29017 -eval "rs.add('nmaharaj-MBP-MMD6T.local:29019')" ]
[ ++ Executing:  mongod --port 30017 --fork --dbpath /Users/nareshmaharaj/data/csrs/0 --logpath /Users/nareshmaharaj/data/csrs/0/log.out -replSet csrs --configsvr  ]
[ ++ Executing:  mongod --port 30018 --fork --dbpath /Users/nareshmaharaj/data/csrs/1 --logpath /Users/nareshmaharaj/data/csrs/1/log.out -replSet csrs --configsvr  ]
[ ++ Executing:  mongod --port 30019 --fork --dbpath /Users/nareshmaharaj/data/csrs/2 --logpath /Users/nareshmaharaj/data/csrs/2/log.out -replSet csrs --configsvr  ]
[ ++ Executing:  mongosh --port 30017 -eval "rs.initiate()" ]
[ ++ Executing:  mongosh --port 30017 -eval "rs.add('nmaharaj-MBP-MMD6T.local:30018')" ]
[ ++ Executing:  mongosh --port 30017 -eval "rs.add('nmaharaj-MBP-MMD6T.local:30019')" ]
 *** Adding user: 'admin', pwd: 'password' to Config Server - please change password ***
[ ++ Executing:  mongos --port 27017 --bind_ip_all --fork --logpath /Users/nareshmaharaj/data/mongos//mongos.log --configdb csrs/nmaharaj-MBP-MMD6T.local:30017,nmaharaj-MBP-MMD6T.local:30018,nmaharaj-MBP-MMD6T.local:30019 ]
[ ++ Executing: mongosh --port 27017 --eval "sh.addShard('sh1/nmaharaj-MBP-MMD6T.local:28017')" ]
[ ++ Executing: mongosh --port 27017 --eval "sh.addShard('sh2/nmaharaj-MBP-MMD6T.local:29017')" ]

# stdout
...
##########################################################################################################
######################################### Creating mongos #### ###########################################
##########################################################################################################
[ ++ Executing:  mongos --port 27017 --bind_ip_all --fork --logpath /Users/nareshmaharaj/data/mongos//mongos.log --configdb csrs/nmaharaj-MBP-MMD6T.local:30017,nmaharaj-MBP-MMD6T.local:30018,nmaharaj-MBP-MMD6T.local:30019 ]
about to fork child process, waiting until server is ready for connections.
forked process: 51366
child process started successfully, parent exiting
[ ++ Executing: mongosh --port 27017 --eval "sh.addShard('sh1/nmaharaj-MBP-MMD6T.local:28017')" ]
{
  shardAdded: 'sh1',
  ok: 1,
  '$clusterTime': {
    clusterTime: Timestamp({ t: 1728147532, i: 6 }),
    signature: {
      hash: Binary.createFromBase64('AAAAAAAAAAAAAAAAAAAAAAAAAAA=', 0),
      keyId: Long('0')
    }
  },
  operationTime: Timestamp({ t: 1728147532, i: 6 })
}
[ ++ Executing: mongosh --port 27017 --eval "sh.addShard('sh2/nmaharaj-MBP-MMD6T.local:29017')" ]
{
  shardAdded: 'sh2',
  ok: 1,
  '$clusterTime': {
    clusterTime: Timestamp({ t: 1728147535, i: 16 }),
    signature: {
      hash: Binary.createFromBase64('AAAAAAAAAAAAAAAAAAAAAAAAAAA=', 0),
      keyId: Long('0')
    }
  },
  operationTime: Timestamp({ t: 1728147535, i: 10 })
}
shardingVersion
{ _id: 1, clusterId: ObjectId('670170310cdeaa5de24f0045') }
---
shards
[
  {
    _id: 'sh1',
    host: 'sh1/nmaharaj-MBP-MMD6T.local:28017,nmaharaj-MBP-MMD6T.local:28018,nmaharaj-MBP-MMD6T.local:28019',
    state: 1,
    topologyTime: Timestamp({ t: 1728147531, i: 6 }),
    replSetConfigVersion: Long('-1')
  },
  {
    _id: 'sh2',
    host: 'sh2/nmaharaj-MBP-MMD6T.local:29017,nmaharaj-MBP-MMD6T.local:29018,nmaharaj-MBP-MMD6T.local:29019',
    state: 1,
    topologyTime: Timestamp({ t: 1728147535, i: 1 }),
    replSetConfigVersion: Long('-1')
  }
]
---
active mongoses
[ { '8.0.0': 1 } ]
---
autosplit
{ 'Currently enabled': 'yes' }
---
balancer
{
  'Currently enabled': 'yes',
  'Currently running': 'no',
  'Failed balancer rounds in last 5 attempts': 0,
  'Migration Results for the last 24 hours': 'No recent migrations'
}
---
databases
[
  {
    database: { _id: 'config', primary: 'config', partitioned: true },
    collections: {}
  }
]

As you can see, we now have a fully configured sharded cluster, and it only takes about a minute to set up.

Log into mongos using mongosh on port 27017 and check the status of the shards.

mongosh --port 27017 --eval "sh.status()"

Note that there are only 2 shards. As of version 8.0, you can transition the config servers to data shards. Let's see how this is done.

mongosh --port 27017"

Run the following command:

db.adminCommand({transitionFromDedicatedConfigServer:1})
{
  ok: 1,
  '$clusterTime': {
    clusterTime: Timestamp({ t: 1728148222, i: 12 }),
    signature: {
      hash: Binary.createFromBase64('AAAAAAAAAAAAAAAAAAAAAAAAAAA=', 0),
      keyId: Long('0')
    }
  },
  operationTime: Timestamp({ t: 1728148222, i: 12 })
}

Now, check to see how many data shards are present.

mongosh --port 27017 --eval "sh.status()"

You should now see all 3

shards
[
  {
    _id: 'config',
    host: 'csrs/nmaharaj-MBP-MMD6T.local:30017,nmaharaj-MBP-MMD6T.local:30018,nmaharaj-MBP-MMD6T.local:30019',
    state: 1,
    topologyTime: Timestamp({ t: 1728148222, i: 3 }),
    replSetConfigVersion: Long('-1')
  },
  {
    _id: 'sh1',
    host: 'sh1/nmaharaj-MBP-MMD6T.local:28017,nmaharaj-MBP-MMD6T.local:28018,nmaharaj-MBP-MMD6T.local:28019',
    state: 1,
    topologyTime: Timestamp({ t: 1728147531, i: 6 }),
    replSetConfigVersion: Long('-1')
  },
  {
    _id: 'sh2',
    host: 'sh2/nmaharaj-MBP-MMD6T.local:29017,nmaharaj-MBP-MMD6T.local:29018,nmaharaj-MBP-MMD6T.local:29019',
    state: 1,
    topologyTime: Timestamp({ t: 1728147535, i: 1 }),
    replSetConfigVersion: Long('-1')
  }
]

VPC Peering, Split Brain with Distributed Cross Region NoSQL-DB

Naresh Maharaj — Sat, 31 Aug 2024 09:52:39 +0000

Network Partitioning - confidence and knowledge read.

Motivation

In this experiment, we knowingly and abruptly create a network separation within a live distributed database. The system is split evenly across two geo-regions in equal proportion. Each region maintains its partition subset. By doing this, we can clearly understand the robustness and behavior of a distributed system under partitioned conditions.Understanding these scenarios can help solution architects design resilient systems that handle various partitioning cases effectively.

While this article focuses on an equal partition split, it's also crucial to test unequal splits. In a minority-majority scenario, the majority partition will continue to handle operations with quorum, while the minority partition may face availability issues. I will discuss this in detail in a separate article.

I've structured this blog into a series of five articles, each corresponding to a different phase of the experiment. Each phase will demand a unique set of skills. For instance, part 1, you'll be exploring Virtual Private Networks and Peering, while on part 2, you'll take on the role of a DBA, each part introducing you to different challenges.

Part 1 - Simple cross region messaging application
Part 2 - Installing an Aerospike NoSQL DB as a Stretch Cluster
Part 3 - Building a Python Application to insert test Data
Part 4 - Split Brain network partitioning
Part 5 - Partition Management Strong Consistency
Wrap up - What's next

Overview

This is what we plan to do over the next 5 parts.

Create 2 unrelated VPCs in AWS, each VPC will be in a different region.
Establish basic communication using a simple message app over the private network.
Demonstrate traffic blocking by splitting the cross-regional connection.
Install a Distributed Database spanning the 2 regions and treat it as a single system.
Verify data integrity by enabling strong consistency features and rules.
Simulate real-world traffic using a simple Python data loader
Enforce a network partition that will create a well known split brain scenario.
Evaluate the Results.

Part 1: Talking Cross Region

Selecting Regions

In a browser navigate to the top right hand corner of the AWS console
Select a unique region.
1. For this example we selected eu-west-2, being London.
Verify you have the correct key pairs downloaded, as you will need these later to log into the host.

Open a new tab and select a different region.
1. For the second region, we will use eu-west-3, which is Paris.
Again, verify you have the correct key pairs downloaded for logging into the host.

By following these steps, we will demonstrate the impact of a network split on a distributed, cross-regional NoSQL database.But before that, we will test our cross-regional connections with a simple messaging tool with no application coding required.

VPC in Region London 🏴󠁧󠁢󠁥󠁮󠁧󠁿 󠁧󠁢󠁥󠁮󠁧󠁿󠁧󠁢󠁥

From the AWS console, visit the VPC Dashboard and create a new VPC named 'my-vpc-london-2' with the IPv4 CIDR block 172.32.0.0/16.

Next, we will add subnets for the various availability zones and attach a new internet gateway, linking all of these to a new routing table.

Create the subnets for each availability zone in the VPC we just created:

First Availability Zone
- Set the IPv4 subnet CIDR block to 172.32.32.0/20.
- Subnet name: my-subnet-2a
- Select the availability zone: eu-west-2a
Second Availability Zone
- Set the IPv4 subnet CIDR block to 172.32.16.0/20.
- Subnet name: my-subnet-2b
- Select the availability zone: eu-west-2b
Third Availability Zone
- Set the IPv4 subnet CIDR block to 172.32.0.0/20.
- Subnet name: my-subnet-2c
- Select the availability zone: eu-west-2c

Under Your VPCs → Resource Map, you should now see the subnets added.

Create a new Internet Gateway and then add it to the Routing Table. Check that you can see this in the Resource Map.

EC2 Host in Region London 🏴󠁧󠁢󠁥󠁮󠁧󠁿

Launch an EC2 instance with the following settings:

Parameter	Value
Image	Canonical, Ubuntu, 22.04 LTS, amd64 jammy image built on 2024-07-01
Instance Type	t2-micro
Key Pair	Select the key you created earlier and downloaded safely.
VPC	Select the VPC we created earlier.
Subnet	Choose the subnet we created earlier for the availability zone this host will be placed in.
Auto Assign Public IP	In production, you would probably disable this and use a jump box. For simplicity, we will SSH directly using the public IP.
Security Group	Create a new security group named my-sg-1.
Security Group Rule	Add a custom TCP rule for ports 3000-3003, source from anywhere.

ssh -o IdentitiesOnly=yes -i aws-instance-key-london-2.pem ubuntu@35.177.110.209

Congratulations, your first region is complete. Let's move on to the second region, Paris!

VPC in Region Paris 🇫🇷

From the browser tab with the Paris region selected, go to the VPC Dashboard and create a new VPC. Verify the CIDR blocks do not overlap with the London VPC. Use the IPv4 CIDR block 172.33.0.0/16.

Next, we will add subnets for the various availability zones and attach a new internet gateway, linking all of these to a new routing table, just as we did before.

Create the subnets for each availability zone in the VPC we just created:

First Availability Zone
- Set the IPv4 subnet CIDR block to 172.33.16.0/20.
- Subnet name: my-subnet-3a
- Select the availability zone: eu-west-3a
Second Availability Zone
- Set the IPv4 subnet CIDR block to 172.33.0.0/20.
- Subnet name: my-subnet-3b
- Select the availability zone: eu-west-3b
Third Availability Zone
- Set the IPv4 subnet CIDR block to 172.33.32.0/20.
- Subnet name: my-subnet-3c
- Select the availability zone: eu-west-3c

Create a new Internet Gateway and then add it to the Routing Table. Check that you can see this in the Resource Map.

EC2 Host in Region Paris 🇫🇷

Launch an EC2 instance with the following settings:

Parameter	Value
Image	ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-amd64-server-20240701
Instance Type	t2-micro
Key Pair	Select the key you created earlier and downloaded safely.
VPC	Select the VPC we created earlier.
Subnet	Choose the subnet we created earlier for the availability zone this host will be placed in.
Auto Assign Public IP	In production, you would probably disable this and use a secure jump box. For simplicity, we will SSH directly using the public IP.
Security Group	Create a new security group named my-sg-1.
Security Group Rule	Add a custom TCP rule for ports 3000-3003, source from anywhere.

ssh -o IdentitiesOnly=yes -i aws-instance-key-paris-1.pem ubuntu@13.38.38.248

Congratulations, your second region is complete.

Network Peering - Stretched Network

The following diagram shows what we intend to achieve with our cross-regional network. We will use AWS's VPC peering to achieve this seamlessly. We will test that we can reach each region with a simple yet powerful networking tool.

Paris Region 🇫🇷
- Under Your VPCs → Peering Connections, create a new peering connection.
- Name it 'my-pc-to-london-1'.
- As the VPC ID (Requester), select the VPC we created earlier.
- Select another VPC to peer with in another region; in our example, it's London (eu-west-2). Enter the VPC ID for the VPC in London.
- Go to the Paris VPCs
  - Update the routing table:
  - Target: VPC peering
  - Destination: London CIDR 172.32.0.0/16

London Region 🏴󠁧󠁢󠁥󠁮󠁧󠁿
- Go to the London VPCs → Peering Connections and accept the request made from the Paris VPC. You might be prompted to update the routing table. If so, accept it.
- Update the routing table:
  - Target: VPC peering
  - Destination: Paris CIDR 172.33.0.0/16

Messaging 'chat' (using `nc` Netcat)

From the London EC2 instance, start an nc server on port 3000 with the following switches:

nc -l -k -p 3000

From the Paris EC2 instance, establish a client connection to the London server:

nc 172.32.34.147 3000

You can now start chatting as all your messages are being sent across the channel literally!

Part 2: Aerospike NoSQL DB Stretch Cluster

In this section, we will create a 6-node stretch cluster NoSQL DB, with each region hosting 3 nodes. The following diagram illustrates the stretch cluster configuration, where every node interconnects with each other node. Due to VPC peering, additional latencies may be observed for replica updates, but this is not a concern for this topic.

Create Database Hosts

For each region, create three nodes. Select the VPC you previously set up, enable public IP assignments, and use the same security group.

EC2 Database Hosts Region London 🏴󠁧󠁢󠁥󠁮󠁧󠁿

Launch 3 x EC2 instance with the following settings:

Parameter	Value
Image	Rocky-8-EC2-Base-8.7-20230215.0.x86_64 (ami-07d2b4d8d9980a125)
Instance Type	t3a.medium (not what you would use in production)
Key Pair	Select the key you created earlier and downloaded safely.
VPC	Select the VPC we created earlier.
Subnet	Choose the subnet we created earlier for the availability zone this host will be placed in.
Auto Assign Public IP	In production, you would probably disable this and use a jump box. For simplicity, we will SSH directly using the public IP.
Security Group	Use the same security group from earlier.
Security Group Rule	None so far
Volumes	Root Volume: 1x10GB-gp2, 1x8GB-gp3

EC2 Database Hosts Region Paris 🇫🇷

Launch 3 x EC2 instance with the following settings:

Parameter	Value
Image	Rocky-8-EC2-LVM-8.7-20230215.0.x86_64 (ami-064a83a6b9c2edb23)
Instance Type	t3a.medium (not what you would use in production)
Key Pair	Select the key you created earlier and downloaded safely.
VPC	Select the VPC we created earlier.
Subnet	Choose the subnet we created earlier for the availability zone this host will be placed in.
Auto Assign Public IP	In production, you would probably disable this and use a jump box. For simplicity, we will SSH directly using the public IP.
Security Group	Use the same security group from earlier.
Security Group Rule	None so far
Volumes	Root Volume: 1x10GB-gp2, 1x8GB-gp3

Installing Aerospike NoSQL DB

Log into each host in a single region using SSH and install Aerospike. There are comments in the file below to remind you about specific changes required for each host. Although there are several automation tools available, we will manually configure each of the six nodes to keep things simple and aid learning.

If you are interested in knowing more about Aerospike, visit the Developer website.

If you have a license key, also known as a feature file, copy this across to the host.
SSH into each host machine and run the following.

EC2 database hosts in Region London 🏴󠁧󠁢󠁥󠁮󠁧󠁿

export VER="7.1.0.0"
export TOOLS=11.0.0
export OS=el8
export ARCH=x86_64
sudo yum install java-11-openjdk.x86_64 java-11-openjdk java-11-openjdk-devel python3 openssl-devel wget git gcc maven bind-utils sysstat nc -y
SERVER_BIN=aerospike-server-enterprise_${VER}_tools-${TOOLS}_${OS}_${ARCH}
LINK=https://download.aerospike.com/artifacts/aerospike-server-enterprise/${VER}/${SERVER_BIN}.tgz
wget -q $LINK
tar -xvf ${SERVER_BIN}.tgz
NS=mydata
sudo mkdir -p /var/log/aerospike/
sudo mkdir -p /etc/aerospike/
# Zero the data disks
ls -l /dev/nvme1n1 | awk '{print $NF}' | while read -r line; do sudo dd if=/dev/zero of=$line bs=1024 count=8192 oflag=direct; done

export ID=$((1 + $RANDOM % 1000))
export INDEX=A$ID
export IP=`ip a | grep 172 | awk {'print $2'} | cut -f1 -d'/'`
export PIP=`dig +short myip.opendns.com @resolver1.opendns.com`
export S1IP=172.32.15.231 # another london node for IP seeding <<--------------- Change this
export S2IP=172.33.16.172 # another paris node for IP seeding <<--------------- Change this
export DEV1=/dev/nvme1n1 # <<--------------- Maybe Change this

cat <<EOF> aerospike.conf
service {
        proto-fd-max 15000

        node-id $INDEX
        cluster-name test-aerocluster.eu
        transaction-max-ms 1500
        log-local-time true
}

logging {
        file /var/log/aerospike/aerospike.log {
                context any info
        }
}
network {
        service {
               address any
               access-address $IP
               alternate-access-address $PIP
               port 3000
       }

       heartbeat {
              mode mesh
              address $IP
              port 3002 # Heartbeat port for this node.
              mesh-seed-address-port $S1IP 3002
              mesh-seed-address-port $S2IP 3002
              interval 150 # controls how often to send a heartbeat packet
              timeout 10 # number of intervals after which a node is considered to be missing
       }

        fabric {
              port 3001
              channel-meta-recv-threads 8
        }

}
security {
        # enable-security true

        log {
                report-authentication true
                report-sys-admin true
                report-user-admin true
                report-violation true
        }
}

namespace mydata {
        # How many copies of the data
        replication-factor 2

        # How full may the memory become before the server begins eviction
        # (expiring records early)
        evict-sys-memory-pct 50

        # How full may the memory become before the server goes read only
        stop-writes-sys-memory-pct 60

        # How long (in seconds) to keep data after it is written Default days,
        # use 0 to never expire/evict.
        default-ttl 0

  # Specify a percentage of record expiration time, read within this interval of the record’s end of life will generate a touch
        # e.g. with default-ttl of 60s, a read with 12 seconds remaining will touch the record. [ 60 x ( 1 - default-read-touch-ttl-pct ) = 12 ]
        default-read-touch-ttl-pct 20

        # The interval at which the main expiration/eviction thread wakes up,
        # to process the namespace.
        nsup-period 120

        # Disables eviction that may occur at cold start for this namespace only
        disable-cold-start-eviction True

  # Data high availability across racks
        rack-id ${ID}

  # SC Mode
        strong-consistency true

        # (optional) write-block is 8MiB in server 7.0 or later so max-record-size can be used to limit the record size.
        max-record-size 128K

#        storage-engine device {
#                device $DEV1
#
#                post-write-cache 64
#                read-page-cache true
#
#         # How full may the disk become before the server begins eviction
#         # (expiring records early)
#                evict-used-pct 45
#        }
        storage-engine memory {
                file /opt/aerospike/ns1.dat   # Location of a namespace data file on server
                filesize 1G                   # Max size of each file in GiB. Maximum size is 2TiB
                stop-writes-avail-pct 5       # (optional) stop-writes threshold as a percentage of
                                              # devices/files size or data-size.
                stop-writes-used-pct 70       # (optional) stop-writes threshold as a percentage of
                                              # devices/files size, or data-size.
                evict-used-pct 60             # (optional) eviction threshold, as a percentage of
                                              # devices/files size, or data-size.
        }
}
EOF
sudo cp aerospike.conf /etc/aerospike/aerospike.conf


cat <<EOF> aerospike.log.rotation
/var/log/aerospike/aerospike.log {
    daily
    rotate 90
    dateext
    compress
    olddir /var/log/aerospike/
    sharedscripts
    postrotate
        /bin/kill -HUP `pgrep -x asd`
    endscript
}
EOF
sudo cp aerospike.log.rotation /etc/logrotate.d/aerospike

sudo cp features.conf /etc/aerospike/features.conf

cd $SERVER_BIN
sudo ./asinstall

sudo systemctl start aerospike
sudo systemctl status aerospike

ACL User Authentication

You can run the following command once on a single host to allow the admin user to add records to the db. Typically, you would set up different users and roles for such tasks, but for simplicity, we are using the admin role (which is not recommended for production environments).

asadm -Uadmin -Padmin -e "enable; manage acl grant user admin roles read-write"
asadm -Uadmin -Padmin -e "enable; manage acl grant user admin roles sys-admin"
asadm -Uadmin -Padmin -e "enable; show user"

To get a view of your current Aerospike cluster, including the nodes you've added, you can run the following command. At this stage, you should have added at least three nodes in the London region.

asadm -e i -Uadmin -Padmin

Seed:        [('127.0.0.1', 3000, None)]
Config_file: /home/rocky/.aerospike/astools.conf, /etc/aerospike/astools.conf
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Network Information (2024-08-12 09:44:44 UTC)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                                           Node| Node|                IP|    Build|Migrations|~~~~~~~~~~~~~~Cluster~~~~~~~~~~~~~~~|Client|  Uptime
                                               |   ID|                  |         |          |Size|        Key|Integrity|Principal| Conns|
172.32.15.231:3000                             | A352|172.32.15.231:3000|E-7.1.0.0|   0.000  |   3|155A640ADB8|True     |     A600|     7|00:08:16
172.32.4.2:3000                                |*A600|172.32.4.2:3000   |E-7.1.0.0|   0.000  |   3|155A640ADB8|True     |     A600|     7|00:07:43
ip-172-32-5-239.eu-west-2.compute.internal:3000| A129|172.32.5.239:3000 |E-7.1.0.0|   0.000  |   3|155A640ADB8|True     |     A600|     7|00:09:38
Number of rows: 3

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Namespace Usage Information (2024-08-12 09:44:44 UTC)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Namespace|                                           Node|Evictions|  Stop|~System Memory~|~~~~Primary Index~~~~|~~Secondary~~~|~~~~~~~~~~~~~~~~~Storage Engine~~~~~~~~~~~~~~~~~
         |                                               |         |Writes| Avail%| Evict%| Type|    Used|Evict%|~~~~Index~~~~~|  Type|    Used|Used%|Evict%|  Used|Avail%|Avail
         |                                               |         |      |       |       |     |        |      | Type|    Used|      |        |     |      | Stop%|      |Stop%
mydata   |172.32.15.231:3000                             |  0.000  |False |     82|     50|shmem|0.000 B | 0.0 %|shmem|0.000 B |memory|0.000 B |0.0 %|60.0 %|70.0 %|99.0 %|5.0 %
mydata   |172.32.4.2:3000                                |  0.000  |False |     82|     50|shmem|0.000 B | 0.0 %|shmem|0.000 B |memory|0.000 B |0.0 %|60.0 %|70.0 %|99.0 %|5.0 %
mydata   |ip-172-32-5-239.eu-west-2.compute.internal:3000|  0.000  |False |     81|     50|shmem|0.000 B | 0.0 %|shmem|0.000 B |memory|0.000 B |0.0 %|60.0 %|70.0 %|99.0 %|5.0 %
mydata   |                                               |  0.000  |      |       |       |     |0.000 B |      |     |0.000 B |      |0.000 B |0.0 %|      |      |      |
Number of rows: 3

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Namespace Object Information (2024-08-12 09:44:44 UTC)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Namespace|                                           Node|Rack|  Repl|Expirations|  Total|~~~~~~~~~~Objects~~~~~~~~~~|~~~~~~~~~Tombstones~~~~~~~~|~~~~Pending~~~~
         |                                               |  ID|Factor|           |Records| Master|  Prole|Non-Replica| Master|  Prole|Non-Replica|~~~~Migrates~~~
         |                                               |    |      |           |       |       |       |           |       |       |           |     Tx|     Rx
mydata   |172.32.15.231:3000                             | 352|     0|    0.000  |0.000  |0.000  |0.000  |    0.000  |0.000  |0.000  |    0.000  |0.000  |0.000
mydata   |172.32.4.2:3000                                | 600|     0|    0.000  |0.000  |0.000  |0.000  |    0.000  |0.000  |0.000  |    0.000  |0.000  |0.000
mydata   |ip-172-32-5-239.eu-west-2.compute.internal:3000| 129|     0|    0.000  |0.000  |0.000  |0.000  |    0.000  |0.000  |0.000  |    0.000  |0.000  |0.000
mydata   |                                               |    |      |    0.000  |0.000  |0.000  |0.000  |    0.000  |0.000  |0.000  |    0.000  |0.000  |0.000
Number of rows: 3

Congratulations! You should have added all six nodes in your stretch cluster. This setup includes the three nodes you configured in the London region and the three nodes in the Paris region.

asadm -e i -Uadmin -Padmin
Seed:        [('127.0.0.1', 3000, None)]
Config_file: /home/rocky/.aerospike/astools.conf, /etc/aerospike/astools.conf
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Network Information (2024-08-12 09:56:34 UTC)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                                          Node| Node|                IP|    Build|Migrations|~~~~~~~~~~~~~~~Cluster~~~~~~~~~~~~~~~|Client|  Uptime
                                              |   ID|                  |         |          |Size|         Key|Integrity|Principal| Conns|
172.32.15.231:3000                            | A352|172.32.15.231:3000|E-7.1.0.0|   0.000  |   6|ECACBC564992|True     |     A882|     7|00:20:06
172.32.4.2:3000                               | A600|172.32.4.2:3000   |E-7.1.0.0|   0.000  |   6|ECACBC564992|True     |     A882|     7|00:19:33
172.32.5.239:3000                             | A129|172.32.5.239:3000 |E-7.1.0.0|   0.000  |   6|ECACBC564992|True     |     A882|     7|00:21:28
172.33.11.90:3000                             |  A70|172.33.11.90:3000 |E-7.1.0.0|   0.000  |   6|ECACBC564992|True     |     A882|     6|00:00:39
172.33.8.38:3000                              |*A882|172.33.8.38:3000  |E-7.1.0.0|   0.000  |   6|ECACBC564992|True     |     A882|     7|00:00:46
ip-172-33-7-44.eu-west-3.compute.internal:3000| A517|172.33.7.44:3000  |E-7.1.0.0|   0.000  |   6|ECACBC564992|True     |     A882|     7|00:00:39
Number of rows: 6

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Namespace Usage Information (2024-08-12 09:56:34 UTC)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Namespace|                                          Node|Evictions|  Stop|~System Memory~|~~~~Primary Index~~~~|~~Secondary~~~|~~~~~~~~~~~~~~~~~Storage Engine~~~~~~~~~~~~~~~~~
         |                                              |         |Writes| Avail%| Evict%| Type|    Used|Evict%|~~~~Index~~~~~|  Type|    Used|Used%|Evict%|  Used|Avail%|Avail
         |                                              |         |      |       |       |     |        |      | Type|    Used|      |        |     |      | Stop%|      |Stop%
mydata   |172.32.15.231:3000                            |  0.000  |False |     78|     50|shmem|0.000 B | 0.0 %|shmem|0.000 B |memory|0.000 B |0.0 %|60.0 %|70.0 %|99.0 %|5.0 %
mydata   |172.32.4.2:3000                               |  0.000  |False |     78|     50|shmem|0.000 B | 0.0 %|shmem|0.000 B |memory|0.000 B |0.0 %|60.0 %|70.0 %|99.0 %|5.0 %
mydata   |172.32.5.239:3000                             |  0.000  |False |     78|     50|shmem|0.000 B | 0.0 %|shmem|0.000 B |memory|0.000 B |0.0 %|60.0 %|70.0 %|99.0 %|5.0 %
mydata   |172.33.11.90:3000                             |  0.000  |False |     78|     50|shmem|0.000 B | 0.0 %|shmem|0.000 B |memory|0.000 B |0.0 %|60.0 %|70.0 %|99.0 %|5.0 %
mydata   |172.33.8.38:3000                              |  0.000  |False |     78|     50|shmem|0.000 B | 0.0 %|shmem|0.000 B |memory|0.000 B |0.0 %|60.0 %|70.0 %|99.0 %|5.0 %
mydata   |ip-172-33-7-44.eu-west-3.compute.internal:3000|  0.000  |False |     77|     50|shmem|0.000 B | 0.0 %|shmem|0.000 B |memory|0.000 B |0.0 %|60.0 %|70.0 %|99.0 %|5.0 %
mydata   |                                              |  0.000  |      |       |       |     |0.000 B |      |     |0.000 B |      |0.000 B |0.0 %|      |      |      |
Number of rows: 6

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Namespace Object Information (2024-08-12 09:56:34 UTC)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Namespace|                                          Node|Rack|  Repl|Expirations|  Total|~~~~~~~~~~Objects~~~~~~~~~~|~~~~~~~~~Tombstones~~~~~~~~|~~~~Pending~~~~
         |                                              |  ID|Factor|           |Records| Master|  Prole|Non-Replica| Master|  Prole|Non-Replica|~~~~Migrates~~~
         |                                              |    |      |           |       |       |       |           |       |       |           |     Tx|     Rx
mydata   |172.32.15.231:3000                            | 352|     0|    0.000  |0.000  |0.000  |0.000  |    0.000  |0.000  |0.000  |    0.000  |0.000  |0.000
mydata   |172.32.4.2:3000                               | 600|     0|    0.000  |0.000  |0.000  |0.000  |    0.000  |0.000  |0.000  |    0.000  |0.000  |0.000
mydata   |172.32.5.239:3000                             | 129|     0|    0.000  |0.000  |0.000  |0.000  |    0.000  |0.000  |0.000  |    0.000  |0.000  |0.000
mydata   |172.33.11.90:3000                             |  70|     0|    0.000  |0.000  |0.000  |0.000  |    0.000  |0.000  |0.000  |    0.000  |0.000  |0.000
mydata   |172.33.8.38:3000                              | 882|     0|    0.000  |0.000  |0.000  |0.000  |    0.000  |0.000  |0.000  |    0.000  |0.000  |0.000
mydata   |ip-172-33-7-44.eu-west-3.compute.internal:3000| 517|     0|    0.000  |0.000  |0.000  |0.000  |    0.000  |0.000  |0.000  |    0.000  |0.000  |0.000
mydata   |                                              |    |      |    0.000  |0.000  |0.000  |0.000  |    0.000  |0.000  |0.000  |    0.000  |0.000  |0.000

Strong Consistency

To enable Strong Consistency (SC) rules in Aerospike, you will need to run a few administrative commands. This will reinforce that your database maintains strict consistency across all nodes in the cluster.

enable
manage roster stage observed ns mydata
manage recluster

Run the following to verify that the nodes are part of the roster

show racks
~Racks (2024-08-12 10:00:00 UTC)~
Namespace|Rack|Nodes
         |  ID|
mydata   |  70|A70
mydata   | 129|A129
mydata   | 352|A352
mydata   | 517|A517
mydata   | 600|A600
mydata   | 882|A882
Number of rows: 6

show roster
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Roster (2024-08-12 10:00:03 UTC)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                                          Node| Node|Namespace|                                     Current Roster|                                     Pending Roster|                                     Observed Nodes
                                              |   ID|         |                                                   |                                                   |
172.32.5.239:3000                             |A129 |mydata   |A882@882,A600@600,A517@517,A352@352,A129@129,A70@70|A882@882,A600@600,A517@517,A352@352,A129@129,A70@70|A882@882,A600@600,A517@517,A352@352,A129@129,A70@70
172.32.15.231:3000                            |A352 |mydata   |A882@882,A600@600,A517@517,A352@352,A129@129,A70@70|A882@882,A600@600,A517@517,A352@352,A129@129,A70@70|A882@882,A600@600,A517@517,A352@352,A129@129,A70@70
ip-172-33-7-44.eu-west-3.compute.internal:3000|A517 |mydata   |A882@882,A600@600,A517@517,A352@352,A129@129,A70@70|A882@882,A600@600,A517@517,A352@352,A129@129,A70@70|A882@882,A600@600,A517@517,A352@352,A129@129,A70@70
172.32.4.2:3000                               |A600 |mydata   |A882@882,A600@600,A517@517,A352@352,A129@129,A70@70|A882@882,A600@600,A517@517,A352@352,A129@129,A70@70|A882@882,A600@600,A517@517,A352@352,A129@129,A70@70
172.33.11.90:3000                             |A70  |mydata   |A882@882,A600@600,A517@517,A352@352,A129@129,A70@70|A882@882,A600@600,A517@517,A352@352,A129@129,A70@70|A882@882,A600@600,A517@517,A352@352,A129@129,A70@70
172.33.8.38:3000                              |*A882|mydata   |A882@882,A600@600,A517@517,A352@352,A129@129,A70@70|A882@882,A600@600,A517@517,A352@352,A129@129,A70@70|A882@882,A600@600,A517@517,A352@352,A129@129,A70@70
Number of rows: 6

Edit RACK-IDs

All six nodes (three in London and three in Paris) should appear in the output, indicating they are part of the current cluster roster and are correctly configured.

But wait!

It appears there are currently six racks displayed in your Aerospike cluster configuration, which doesn’t align with your setup of 3 nodes per region and a total of 2 regions. Since all nodes in each region are in a single subnet, they could be grouped into two logical racks.

To correct this, you need to edit the cluster configuration to consolidate the existing racks into the appropriate number of rack-id(s). See the diagram below.

Action: Edit the cluster so we have only 2 logical racks.

# Login to the admin tool
asadm -Uadmin -Padmin

# show the roster
show roster

# Output
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Roster (2024-08-12 10:00:03 UTC)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                                          Node| Node|Namespace|                                     Current Roster|                                     Pending Roster|                                     Observed Nodes
                                              |   ID|         |                                                   |                                                   |
172.32.5.239:3000                             |A129 |mydata   |A882@882,A600@600,A517@517,A352@352,A129@129,A70@70|A882@882,A600@600,A517@517,A352@352,A129@129,A70@70|A882@882,A600@600,A517@517,A352@352,A129@129,A70@70
172.32.15.231:3000                            |A352 |mydata   |A882@882,A600@600,A517@517,A352@352,A129@129,A70@70|A882@882,A600@600,A517@517,A352@352,A129@129,A70@70|A882@882,A600@600,A517@517,A352@352,A129@129,A70@70
ip-172-33-7-44.eu-west-3.compute.internal:3000|A517 |mydata   |A882@882,A600@600,A517@517,A352@352,A129@129,A70@70|A882@882,A600@600,A517@517,A352@352,A129@129,A70@70|A882@882,A600@600,A517@517,A352@352,A129@129,A70@70
172.32.4.2:3000                               |A600 |mydata   |A882@882,A600@600,A517@517,A352@352,A129@129,A70@70|A882@882,A600@600,A517@517,A352@352,A129@129,A70@70|A882@882,A600@600,A517@517,A352@352,A129@129,A70@70
172.33.11.90:3000                             |A70  |mydata   |A882@882,A600@600,A517@517,A352@352,A129@129,A70@70|A882@882,A600@600,A517@517,A352@352,A129@129,A70@70|A882@882,A600@600,A517@517,A352@352,A129@129,A70@70
172.33.8.38:3000                              |*A882|mydata   |A882@882,A600@600,A517@517,A352@352,A129@129,A70@70|A882@882,A600@600,A517@517,A352@352,A129@129,A70@70|A882@882,A600@600,A517@517,A352@352,A129@129,A70@70
Number of rows: 6

# remove the n-1 nodes from the cluster
manage roster remove nodes A882@882 A600@600 A517@517 A352@352 A129@129 ns mydata

# check the current roster should only be a single node
show roster

# Output
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Roster (2024-08-12 11:25:26 UTC)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                                           Node| Node|Namespace|Current|Pending|                          Observed Nodes
                                               |   ID|         | Roster| Roster|
ip-172-32-5-239.eu-west-2.compute.internal:3000|A129 |mydata   |A70@70  |A70@70  |A882@882,A600@600,A517@517,A352@352,A129@129,A70@70
172.32.15.231:3000                             |A352 |mydata   |A70@70  |A70@70  |A882@882,A600@600,A517@517,A352@352,A129@129,A70@70
172.33.7.44:3000                               |A517 |mydata   |A70@70  |A70@70  |A882@882,A600@600,A517@517,A352@352,A129@129,A70@70
172.32.4.2:3000                                |A600 |mydata   |A70@70  |A70@70  |A882@882,A600@600,A517@517,A352@352,A129@129,A70@70
172.33.11.90:3000                              |*A70 |mydata   |A70@70  |A70@70  |A882@882,A600@600,A517@517,A352@352,A129@129,A70@70
172.33.8.38:3000                               |A882 |mydata   |A70@70  |A70@70  |A882@882,A600@600,A517@517,A352@352,A129@129,A70@70
Number of rows: 6

# change the rack ids
manage config namespace mydata param rack-id to 32 with A129 A352 A600
manage recluster
info

manage config namespace mydata param rack-id to 33 with A70 A517 A882
manage recluster
info

manage roster stage observed A882@33,A600@32,A517@33,A352@32,A129@32,A70@33 ns mydata
manage recluster
show roster

# Output
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Roster (2024-08-12 11:31:08 UTC)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                                           Node| Node|Namespace|                                Current Roster|                                Pending Roster|                                Observed Nodes
                                               |   ID|         |                                              |                                              |
ip-172-32-5-239.eu-west-2.compute.internal:3000|A129 |mydata   |A882@33,A600@32,A517@33,A352@32,A129@32,A70@33|A882@33,A600@32,A517@33,A352@32,A129@32,A70@33|A882@33,A600@32,A517@33,A352@32,A129@32,A70@33
172.32.15.231:3000                             |A352 |mydata   |A882@33,A600@32,A517@33,A352@32,A129@32,A70@33|A882@33,A600@32,A517@33,A352@32,A129@32,A70@33|A882@33,A600@32,A517@33,A352@32,A129@32,A70@33
172.33.7.44:3000                               |A517 |mydata   |A882@33,A600@32,A517@33,A352@32,A129@32,A70@33|A882@33,A600@32,A517@33,A352@32,A129@32,A70@33|A882@33,A600@32,A517@33,A352@32,A129@32,A70@33
172.32.4.2:3000                                |A600 |mydata   |A882@33,A600@32,A517@33,A352@32,A129@32,A70@33|A882@33,A600@32,A517@33,A352@32,A129@32,A70@33|A882@33,A600@32,A517@33,A352@32,A129@32,A70@33
172.33.11.90:3000                              |*A70 |mydata   |A882@33,A600@32,A517@33,A352@32,A129@32,A70@33|A882@33,A600@32,A517@33,A352@32,A129@32,A70@33|A882@33,A600@32,A517@33,A352@32,A129@32,A70@33
172.33.8.38:3000                               |A882 |mydata   |A882@33,A600@32,A517@33,A352@32,A129@32,A70@33|A882@33,A600@32,A517@33,A352@32,A129@32,A70@33|A882@33,A600@32,A517@33,A352@32,A129@32,A70@33
Number of rows: 6

show racks

# Output
~Racks (2024-08-12 11:31:34 UTC)~
Namespace|Rack|         Nodes
         |  ID|
mydata   |  32|A600,A352,A129
mydata   |  33|A882,A517,A70
Number of rows: 2

Congratulations! You have successfully updated the rack configuration for your cross-regional Aerospike cluster. The cluster now accurately reflects two logical racks—one for each region. Don’t forget to verify and update the rack-id values in your Aerospike configuration file to match the revised rack setup. This will make sure that the configuration aligns with your intended architecture.

Part 3: Insert some records

You will want to verify that data is being written to your Aerospike database while performing the split brain scenarios. To achieve this you will create a basic Python application to insert data. This will help you verify the cluster's behavior and data consistency under test conditions. Below is a simple Python script that inserts some data into the Aerospike database. This script uses the aerospike client library to connect to the cluster and perform data operations.

When you run the provided Python script to insert data into your Aerospike database, the data should be structured and stored as follows. Here’s an example of how the inserted data might look:

Create an additional ec2 host in one of the subnets to run your code.

aql> select * from mydata.dummy limit 10
+------+------+--------------+------------------------+--------------------------------------------------------------------------------------------------------------------+
| PK   | freq | country_code | logged_by              | report                                                                                                             |
+------+------+--------------+------------------------+--------------------------------------------------------------------------------------------------------------------+
| 134  | 340  | 199          | "FJ4Z50qm1YFKLC5g98T2" | MAP('{"colour-scheme":["purple", "magenta"], "date_mfg":"2024-02-09", "machine":["Surface"], "pixels":524288}')    |
| 3758 | 408  | 121          | "rMyHrqM6eZcfYQCcCQFC" | MAP('{"colour-scheme":["cyan", "brown"], "date_mfg":"2023-10-07", "machine":["Inspiron"], "pixels":65536}')        |
| 2297 | 323  | 81           | "kDeHYVgb4QqzCPj1RkOw" | MAP('{"colour-scheme":["orange", "green"], "date_mfg":"2021-08-18", "machine":["MacBook"], "pixels":16777216}')    |
| 1841 | 833  | 224          | "2bedyAaZll3nPGKyty44" | MAP('{"colour-scheme":["green", "purple"], "date_mfg":"2022-07-02", "machine":["Chromebook"], "pixels":16777216}') |
| 3017 | 898  | 213          | "qwGXGe6BdbUHh8ZBGGit" | MAP('{"colour-scheme":["purple", "cyan"], "date_mfg":"2024-06-22", "machine":["ZenBook"], "pixels":32768}')        |
| 3589 | 250  | 165          | "Od4R4ADltbWCD8budaco" | MAP('{"colour-scheme":["yellow", "green"], "date_mfg":"2018-08-02", "machine":["ThinkPad"], "pixels":65536}')      |
| 2432 | 796  | 133          | "DD1Evor4WGFX9yr9WVuc" | MAP('{"colour-scheme":["brown", "cyan"], "date_mfg":"2022-02-04", "machine":["ThinkPad"], "pixels":4194304}')      |
| 1652 | 623  | 1            | "HTkLNYHIPyYwUqtlZ883" | MAP('{"colour-scheme":["blue", "magenta"], "date_mfg":"2019-08-06", "machine":["Latitude"], "pixels":4096}')       |
| 970  | 348  | 91           | "Cao8qtth9x981pjkpp9M" | MAP('{"colour-scheme":["red", "magenta"], "date_mfg":"2019-09-14", "machine":["Latitude"], "pixels":1048576}')     |
| 2683 | 442  | 12           | "W9U9PBvCWodrTvf59FMz" | MAP('{"colour-scheme":["brown", "blue"], "date_mfg":"2024-01-14", "machine":["Latitude"], "pixels":2097152}')      |
+------+------+--------------+------------------------+--------------------------------------------------------------------------------------------------------------------+
10 rows in set (0.033 secs)

To control how long your Python application should run and insert data into the Aerospike database, modify the script's timeout execution period. This allows you to set a specific duration for the script to run.

# Set a timeout value in seconds
run_for_sec = 30  # Adjust this value based on your needs

Edit the seed hosts for your Aerospike cluster. I have chosen 1 node from London and 1 from Paris.

hosts = [ ('172.33.7.44', 3000), ('172.32.5.239', 3000) ]

To successfully install the Aerospike Python client library, you need to ensure that certain dependencies are met.

python3-devel
python3.8
make

e.g. use the following to install

sudo yum install python3-devel python3.8 make -y

Next install Aerospike Client Lib using pip.

sudo pip3.8 install aerospike

Here is the code:

import aerospike
from aerospike import exception as ex
import sys
import random
import string
from datetime import datetime, timedelta
import time

hosts = [ ('172.33.7.44', 3000), ('172.32.5.239', 3000) ]
run_for_sec = 60

# Sleep function to pause execution for a specified number of milliseconds
def sleep_ms(milliseconds):
    time.sleep(milliseconds / 1000.0)

# Function to generate a list of random dates within a specified range
def generate_random_dates(num_dates):
    start_date = datetime(2018, 1, 1)  # Start date
    end_date = datetime(2024, 8, 31)  # End date
    date_range = end_date - start_date  # Calculate date range

    random_dates = []
    for _ in range(num_dates):
        random_days = random.randint(0, date_range.days)  # Generate random number of days
        random_date = start_date + timedelta(days=random_days)  # Add random days to start date
        random_dates.append(random_date)

    return random_dates

# Function to generate a random username of a given length
def generate_username(length):
    characters = string.ascii_letters + string.digits  # Pool of characters
    username = ''.join(random.choice(characters) for _ in range(length))
    return username

# Function to generate a list of random colors from a predefined set
def generate_random_colors(num_colors):
    colors = ['red', 'blue', 'green', 'yellow', 'orange', 'purple', 'pink', 'brown', 'cyan', 'magenta']
    random_colors = random.choices(colors, k=num_colors)
    return random_colors

# Function to generate a list of random computer names from a predefined set
def generate_computer_names(num_computers):
    computer_types = ['MacBook', 'ThinkPad', 'Chromebook', 'Surface', 'Latitude', 'Surface Book', 'Alienware', 'ZenBook', 'Inspiron', 'Pavilion']
    names = random.sample(computer_types, num_computers)
    return names

# Configuration for Aerospike client
config = {
  'hosts': hosts,  # Aerospike cluster hosts
  'user': "admin",
  'password': "admin"
}

namespace = 'mydata'
set = 'dummy'

try:
    # Connect to Aerospike client
    client = aerospike.client(config).connect()
    print("Connected to Server")

    # Create new write policy
    write_policy = {'key': aerospike.POLICY_KEY_SEND}

    # Set a timeout value in seconds
    timeout = run_for_sec  # Adjust this value based on your needs

    # Define the start time
    start_time = time.time()
    count = 0
    while True:
        # Generate a random key
        key = (namespace, set, random.randint(0, 4095))

        # Generate random data for bins
        number_sightings = random.randint(0, 1000)
        cc = random.randint(0, 252)
        user = generate_username(20)
        date_made = generate_random_dates(1)

        data = {
            'machine': generate_computer_names(1),
            'pixels': 2 ** random.randint(12, 24),
            'colour-scheme': generate_random_colors(2),
            'date_mfg': date_made[0].strftime("%Y-%m-%d")
        }

        # Create the bins
        bins = {
            'freq': number_sightings,
            'country_code': cc,
            'logged_by': user,
            'report': data,
        }

        # Put the record into the Aerospike database
        client.put(key, bins, policy=write_policy)
        count = count + 1
        # Check if the current time has exceeded the timeout
        if time.time() - start_time > timeout:
            print("Duration reached. Records[r/u]:",count)
            break

        # Sleep for 200 milliseconds
        sleep_ms(200)

    # Close the client connection
    client.close()
except ex.ClientError as e:
    # Handle client errors
    print("Error: {0} [{1}]".format(e.msg, e.code))
    sys.exit(1)

Ensure that your Python application continues to run in the background for an extended period, allowing you to perform tests and simulate various scenarios.

python3.8 clientApp.py

Part 4: Split Brain

Here is an illustration of a split brain scenario in a distributed data system.This image depicts how two halves of a data system (Paris Region and London Region) might split up and start operating independently due to network separation. Some might even think thats cool!

Understanding Strong Consistency in Aerospike

In the context of database systems, Strong Consistency (SC) is a critical concept, especially when dealing with scenarios such as split-brain conditions. A split-brain scenario occurs when a network partition divides the database cluster into two or more partitions, potentially leading to data inconsistency.

Aerospike's Strong Consistency mode is designed to address such challenges by ensuring that all writes to a single record are applied in a strict, sequential order. This guarantees that no writes are reordered or skipped, and thus, no data is lost.

Here’s a deeper look at how Strong Consistency works and its importance:

Key Features of Strong Consistency

Sequential Writes:
All writes to a single record are applied in the order they are received. This ensures that the state of the record is predictable and consistent across all nodes in the cluster.
No Data Loss: Aerospike SC mode ensures that data is not lost, even in the event of network partitions or node failures. However, there are rare exceptions, such as simultaneous hardware failures on different nodes, which could potentially result in data loss.
Quorum-based Commit: Writes are only considered successful when a quorum of nodes acknowledge the write operation. This means that a majority of nodes must agree on the write, ensuring data consistency even in the presence of node failures or network issues.
Immediate Consistency: As soon as a write operation is confirmed, all subsequent read operations will reflect this write. This contrasts with eventual consistency, where reads might temporarily return stale data.

Evaluating Strong Consistency in Split-Brain Scenarios

During a split-brain scenario, the network partition can lead to isolated clusters of nodes. Evaluating the behavior of Aerospike under such conditions is crucial to understanding the robustness of its SC mode. Here’s how SC mode helps:

Write Operations: In a split-brain scenario, writes can only be processed by the nodes that form a majority partition. This prevents conflicting writes to the same record from different partitions, maintaining data integrity.
Read Operations: Reads will always return the most recent write that was acknowledged by a majority of nodes. If a node is isolated in a minority partition, it will not serve stale data to clients.
Reconciliation Post-Recovery: Once the network partition is resolved, Aerospike uses SC mode to reconcile any divergent states. The system ensures that the state of records is consistent across all nodes, based on the majority writes that were committed during the partition.

How to create the Split Brain

The image below shows the overall network has been split between the 2 regions and their corresponding subnets. Within each subnet, the subcluster now has its own view of the world. Each subcluster is communicating only with nodes inside its own subnet.

To simulate a network split between regions in AWS, Log into the Paris AWS Console. Observe that the source is set to 0.0.0.0/0, which means that connections can be established from anywhere.
This configuration allows traffic from any IP address to access these ports.

To simulate a network split, you would need to restrict the traffic so that only nodes within the same subnet or region can communicate. For example, you can change the source to the specific CIDR block of your subnet or reference the security group itself. By applying these changes, will ensure that:

Nodes in the Paris region can communicate with nodes in both the Paris and London regions.
Nodes in the London region can only communicate with other nodes within the London region.

This configuration effectively simulates a network split where the London subcluster is isolated from Paris, while Paris can still interact with both regions.

By examining these details, it becomes evident that the number of records in London has been reduced by half following the network separation.

Before 940 records 🏴󠁧󠁢󠁥󠁮󠁧󠁿

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Namespace Object Information (2024-08-14 15:17:18 UTC)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Namespace|                                            Node|Rack|  Repl|Expirations|    Total|~~~~~~~~~~~~Objects~~~~~~~~~~~~|~~~~~~~~~Tombstones~~~~~~~~|~~~~Pending~~~~
         |                                                |  ID|Factor|           |  Records|   Master|    Prole|Non-Replica| Master|  Prole|Non-Replica|~~~~Migrates~~~
         |                                                |    |      |           |         |         |         |           |       |       |           |     Tx|     Rx
mydata   |172.32.4.2:3000                                 |  32|     2|    0.000  |165.000  | 92.000  | 73.000  |    0.000  |0.000  |0.000  |    0.000  |0.000  |0.000
mydata   |172.32.5.239:3000                               |  32|     2|    0.000  |162.000  | 88.000  | 74.000  |    0.000  |0.000  |0.000  |    0.000  |0.000  |0.000
mydata   |172.33.11.90:3000                               |  33|     2|    0.000  |148.000  | 77.000  | 71.000  |    0.000  |0.000  |0.000  |    0.000  |0.000  |0.000
mydata   |172.33.7.44:3000                                |  33|     2|    0.000  |152.000  | 65.000  | 87.000  |    0.000  |0.000  |0.000  |    0.000  |0.000  |0.000
mydata   |172.33.8.38:3000                                |  33|     2|    0.000  |170.000  | 82.000  | 88.000  |    0.000  |0.000  |0.000  |    0.000  |0.000  |0.000
mydata   |ip-172-32-15-231.eu-west-2.compute.internal:3000|  32|     2|    0.000  |143.000  | 66.000  | 77.000  |    0.000  |0.000  |0.000  |    0.000  |0.000  |0.000
mydata   |                                                |    |      |    0.000  |940.000  |470.000  |470.000  |    0.000  |0.000  |0.000  |    0.000  |0.000  |0.000
Number of rows: 6

After 470 records 🏴󠁧󠁢󠁥󠁮󠁧󠁿

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Namespace Object Information (2024-08-14 15:33:09 UTC)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Namespace|                                            Node|Rack|  Repl|Expirations|    Total|~~~~~~~~~~~~Objects~~~~~~~~~~~~|~~~~~~~~~Tombstones~~~~~~~~|~~~~Pending~~~~
         |                                                |  ID|Factor|           |  Records|   Master|    Prole|Non-Replica| Master|  Prole|Non-Replica|~~~~Migrates~~~
         |                                                |    |      |           |         |         |         |           |       |       |           |     Tx|     Rx
~~       |172.33.11.90:3000                               |  ~~|    ~~|         ~~|       ~~|       ~~|       ~~|         ~~|     ~~|     ~~|         ~~|     ~~|     ~~
~~       |172.33.7.44:3000                                |  ~~|    ~~|         ~~|       ~~|       ~~|       ~~|         ~~|     ~~|     ~~|         ~~|     ~~|     ~~
~~       |172.33.8.38:3000                                |  ~~|    ~~|         ~~|       ~~|       ~~|       ~~|         ~~|     ~~|     ~~|         ~~|     ~~|     ~~
~~       |                                                |    |      |         ~~|       ~~|       ~~|       ~~|         ~~|     ~~|     ~~|         ~~|     ~~|     ~~
mydata   |172.32.4.2:3000                                 |  32|     2|    0.000  |165.000  | 92.000  | 73.000  |    0.000  |0.000  |0.000  |    0.000  |0.000  |0.000
mydata   |172.32.5.239:3000                               |  32|     2|    0.000  |162.000  | 88.000  | 74.000  |    0.000  |0.000  |0.000  |    0.000  |0.000  |0.000
mydata   |ip-172-32-15-231.eu-west-2.compute.internal:3000|  32|     2|    0.000  |143.000  | 66.000  | 77.000  |    0.000  |0.000  |0.000  |    0.000  |0.000  |0.000
mydata   |                                                |    |      |    0.000  |470.000  |246.000  |224.000  |    0.000  |0.000  |0.000  |    0.000  |0.000  |0.000
Number of rows: 6

As we can see, the Paris 🇫🇷 nodes have not experienced any changes in the number of records or their status. This is because the London ports are still open to sources 0.0.0.0/0, allowing communication between the regions.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Namespace Object Information (2024-08-14 15:53:59 UTC)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Namespace|                                          Node|Rack|  Repl|Expirations|    Total|~~~~~~~~~~~~Objects~~~~~~~~~~~~|~~~~~~~~~Tombstones~~~~~~~~|~~~~Pending~~~~
         |                                              |  ID|Factor|           |  Records|   Master|    Prole|Non-Replica| Master|  Prole|Non-Replica|~~~~Migrates~~~
         |                                              |    |      |           |         |         |         |           |       |       |           |     Tx|     Rx
mydata   |172.32.15.231:3000                            |  32|     2|    0.000  |143.000  | 66.000  | 77.000  |    0.000  |0.000  |0.000  |    0.000  |0.000  |0.000
mydata   |172.32.4.2:3000                               |  32|     2|    0.000  |165.000  | 92.000  | 73.000  |    0.000  |0.000  |0.000  |    0.000  |0.000  |0.000
mydata   |172.32.5.239:3000                             |  32|     2|    0.000  |162.000  | 88.000  | 74.000  |    0.000  |0.000  |0.000  |    0.000  |0.000  |0.000
mydata   |172.33.11.90:3000                             |  33|     2|    0.000  |148.000  | 77.000  | 71.000  |    0.000  |0.000  |0.000  |    0.000  |0.000  |0.000
mydata   |172.33.8.38:3000                              |  33|     2|    0.000  |170.000  | 82.000  | 88.000  |    0.000  |0.000  |0.000  |    0.000  |0.000  |0.000
mydata   |ip-172-33-7-44.eu-west-3.compute.internal:3000|  33|     2|    0.000  |152.000  | 65.000  | 87.000  |    0.000  |0.000  |0.000  |    0.000  |0.000  |0.000
mydata   |                                              |    |      |    0.000  |940.000  |470.000  |470.000  |    0.000  |0.000  |0.000  |    0.000  |0.000  |0.000
Number of rows: 6

Admin>

Run your Python application from a subnet in London to simulate more inbound data from another region. Recall our previous data was created from Paris.

Modify the London security group inbound rules to allow traffic only from the London subnet and block all external traffic, including traffic from Paris. By isolating London from Paris, you have successfully created a full split-brain scenario. This setup helps in understanding how such network partitions affect data distribution and cluster communication in a distributed database environment like Aerospike.

Let's see the results from asadm, Aerospike CLI managemnt tool.

London 🏴󠁧󠁢󠁥󠁮󠁧󠁿

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Namespace Object Information (2024-08-14 16:18:29 UTC)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Namespace|                                            Node|Rack|  Repl|Expirations|    Total|~~~~~~~~~~~~Objects~~~~~~~~~~~~|~~~~~~~~~Tombstones~~~~~~~~|~~~~Pending~~~~
         |                                                |  ID|Factor|           |  Records|   Master|    Prole|Non-Replica| Master|  Prole|Non-Replica|~~~~Migrates~~~
         |                                                |    |      |           |         |         |         |           |       |       |           |     Tx|     Rx
~~       |172.33.11.90:3000                               |  ~~|    ~~|         ~~|       ~~|       ~~|       ~~|         ~~|     ~~|     ~~|         ~~|     ~~|     ~~
~~       |172.33.7.44:3000                                |  ~~|    ~~|         ~~|       ~~|       ~~|       ~~|         ~~|     ~~|     ~~|         ~~|     ~~|     ~~
~~       |172.33.8.38:3000                                |  ~~|    ~~|         ~~|       ~~|       ~~|       ~~|         ~~|     ~~|     ~~|         ~~|     ~~|     ~~
~~       |                                                |    |      |         ~~|       ~~|       ~~|       ~~|         ~~|     ~~|     ~~|         ~~|     ~~|     ~~
mydata   |172.32.4.2:3000                                 |  32|     2|    0.000  |481.000  |155.000  |171.000  |  155.000  |0.000  |0.000  |    0.000  |0.000  |0.000
mydata   |172.32.5.239:3000                               |  32|     2|    0.000  |485.000  |178.000  |149.000  |  158.000  |0.000  |0.000  |    0.000  |0.000  |0.000
mydata   |ip-172-32-15-231.eu-west-2.compute.internal:3000|  32|     2|    0.000  |454.000  |147.000  |160.000  |  147.000  |0.000  |0.000  |    0.000  |0.000  |0.000
mydata   |                                                |    |      |    0.000  |  1.420 K|480.000  |480.000  |  460.000  |0.000  |0.000  |    0.000  |0.000  |0.000
Number of rows: 6

Paris 🇫🇷

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Namespace Object Information (2024-08-14 16:18:16 UTC)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Namespace|                                          Node|Rack|  Repl|Expirations|    Total|~~~~~~~~~~~~Objects~~~~~~~~~~~~|~~~~~~~~~Tombstones~~~~~~~~|~~~~Pending~~~~
         |                                              |  ID|Factor|           |  Records|   Master|    Prole|Non-Replica| Master|  Prole|Non-Replica|~~~~Migrates~~~
         |                                              |    |      |           |         |         |         |           |       |       |           |     Tx|     Rx
~~       |172.32.15.231:3000                            |  ~~|    ~~|         ~~|       ~~|       ~~|       ~~|         ~~|     ~~|     ~~|         ~~|     ~~|     ~~
~~       |172.32.4.2:3000                               |  ~~|    ~~|         ~~|       ~~|       ~~|       ~~|         ~~|     ~~|     ~~|         ~~|     ~~|     ~~
~~       |172.32.5.239:3000                             |  ~~|    ~~|         ~~|       ~~|       ~~|       ~~|         ~~|     ~~|     ~~|         ~~|     ~~|     ~~
~~       |                                              |    |      |         ~~|       ~~|       ~~|       ~~|         ~~|     ~~|     ~~|         ~~|     ~~|     ~~
mydata   |172.33.11.90:3000                             |  33|     2|    0.000  |471.000  |165.000  |156.000  |  150.000  |0.000  |0.000  |    0.000  |0.000  |0.000
mydata   |172.33.8.38:3000                              |  33|     2|    0.000  |463.000  |153.000  |154.000  |  156.000  |0.000  |0.000  |    0.000  |0.000  |0.000
mydata   |ip-172-33-7-44.eu-west-3.compute.internal:3000|  33|     2|    0.000  |466.000  |142.000  |150.000  |  174.000  |0.000  |0.000  |    0.000  |0.000  |0.000
mydata   |                                              |    |      |    0.000  |  1.400 K|460.000  |460.000  |  480.000  |0.000  |0.000  |    0.000  |0.000  |0.000
Number of rows: 6

Run the client application based in London now we have a full network partition. Notice how it writes a couple of records for the partitions it has before failing.

python3.6 aerospike-client.py
Connected to Server
Error: Node not found for partition mydata:3773 [-8]

London 🏴󠁧󠁢󠁥󠁮󠁧󠁿

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Namespace Object Information (2024-08-14 16:33:08 UTC)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Namespace|                                            Node|Rack|  Repl|Expirations|    Total|~~~~~~~~~~~~Objects~~~~~~~~~~~~|~~~~~~~~~Tombstones~~~~~~~~|~~~~Pending~~~~
         |                                                |  ID|Factor|           |  Records|   Master|    Prole|Non-Replica| Master|  Prole|Non-Replica|~~~~Migrates~~~
         |                                                |    |      |           |         |         |         |           |       |       |           |     Tx|     Rx
~~       |172.33.11.90:3000                               |  ~~|    ~~|         ~~|       ~~|       ~~|       ~~|         ~~|     ~~|     ~~|         ~~|     ~~|     ~~
~~       |172.33.7.44:3000                                |  ~~|    ~~|         ~~|       ~~|       ~~|       ~~|         ~~|     ~~|     ~~|         ~~|     ~~|     ~~
~~       |172.33.8.38:3000                                |  ~~|    ~~|         ~~|       ~~|       ~~|       ~~|         ~~|     ~~|     ~~|         ~~|     ~~|     ~~
~~       |                                                |    |      |         ~~|       ~~|       ~~|       ~~|         ~~|     ~~|     ~~|         ~~|     ~~|     ~~
mydata   |172.32.4.2:3000                                 |  32|     2|    0.000  |483.000  |156.000  |172.000  |  155.000  |0.000  |0.000  |    0.000  |0.000  |0.000
mydata   |172.32.5.239:3000                               |  32|     2|    0.000  |487.000  |179.000  |150.000  |  158.000  |0.000  |0.000  |    0.000  |0.000  |0.000
mydata   |ip-172-32-15-231.eu-west-2.compute.internal:3000|  32|     2|    0.000  |454.000  |147.000  |160.000  |  147.000  |0.000  |0.000  |    0.000  |0.000  |0.000
mydata   |                                                |    |      |    0.000  |  1.424 K|482.000  |482.000  |  460.000  |0.000  |0.000  |    0.000  |0.000  |0.000
Number of rows: 6

Paris 🇫🇷 - we dont expect any record count changes as we cannot reach Paris

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Namespace Object Information (2024-08-14 16:34:00 UTC)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Namespace|                                          Node|Rack|  Repl|Expirations|    Total|~~~~~~~~~~~~Objects~~~~~~~~~~~~|~~~~~~~~~Tombstones~~~~~~~~|~~~~Pending~~~~
         |                                              |  ID|Factor|           |  Records|   Master|    Prole|Non-Replica| Master|  Prole|Non-Replica|~~~~Migrates~~~
         |                                              |    |      |           |         |         |         |           |       |       |           |     Tx|     Rx
~~       |172.32.15.231:3000                            |  ~~|    ~~|         ~~|       ~~|       ~~|       ~~|         ~~|     ~~|     ~~|         ~~|     ~~|     ~~
~~       |172.32.4.2:3000                               |  ~~|    ~~|         ~~|       ~~|       ~~|       ~~|         ~~|     ~~|     ~~|         ~~|     ~~|     ~~
~~       |172.32.5.239:3000                             |  ~~|    ~~|         ~~|       ~~|       ~~|       ~~|         ~~|     ~~|     ~~|         ~~|     ~~|     ~~
~~       |                                              |    |      |         ~~|       ~~|       ~~|       ~~|         ~~|     ~~|     ~~|         ~~|     ~~|     ~~
mydata   |172.33.11.90:3000                             |  33|     2|    0.000  |471.000  |165.000  |156.000  |  150.000  |0.000  |0.000  |    0.000  |0.000  |0.000
mydata   |172.33.8.38:3000                              |  33|     2|    0.000  |463.000  |153.000  |154.000  |  156.000  |0.000  |0.000  |    0.000  |0.000  |0.000
mydata   |ip-172-33-7-44.eu-west-3.compute.internal:3000|  33|     2|    0.000  |466.000  |142.000  |150.000  |  174.000  |0.000  |0.000  |    0.000  |0.000  |0.000
mydata   |                                              |    |      |    0.000  |  1.400 K|460.000  |460.000  |  480.000  |0.000  |0.000  |    0.000  |0.000  |0.000
Number of rows: 6

To summarise, we have an even split of nodes in each sub-cluster and each is up and running but only for the partitions it owns.

Key Observations:

Even Split: Each region is operating independently, handling only the partitions it owns.
Partition Ownership: Each region manages 50% of the total partitions. This means that Paris handles half of the partitions, and London handles the other half.

Paris 🇫🇷

Admin> show pmap
~~~~~~~~~~~~~~~~~~~~~~~~~~~~Partition Map Analysis (2024-08-14 16:35:12 UTC)~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Namespace|                                          Node| Cluster Key|~~~~~~~~~~~~Partitions~~~~~~~~~~~~
         |                                              |            |Primary|Secondary|Unavailable|Dead
~~       |172.32.15.231:3000                            |          ~~|     ~~|       ~~|         ~~|  ~~
~~       |172.32.4.2:3000                               |          ~~|     ~~|       ~~|         ~~|  ~~
~~       |172.32.5.239:3000                             |          ~~|     ~~|       ~~|         ~~|  ~~
~~       |                                              |            |     ~~|       ~~|         ~~|  ~~
mydata   |172.33.11.90:3000                             |3CF08B51D327|    682|      700|       2048|   0
mydata   |172.33.8.38:3000                              |3CF08B51D327|    683|      669|       2048|   0
mydata   |ip-172-33-7-44.eu-west-3.compute.internal:3000|3CF08B51D327|    683|      679|       2048|   0
mydata   |                                              |            |   2048|     2048|       6144|   0
Number of rows: 6

Partition Distribution: The show pmap command confirms that the partitions are evenly split between the Paris and London regions. Each region’s nodes manage an equal share of the partitions, which aligns with the network isolation we’ve implemented.
Cluster Operation: Both clusters (Paris and London) are fully operational but only for the partitions they currently own. This demonstrates how partition ownership and data distribution are maintained even during a network split.

By analyzing this command output, it’s clear that each subcluster is functioning correctly within its partition scope, illustrating the impact of the network partition on the Aerospike database’s partition management.

Restoring Network Partition Configuration

To resolve the network partition and restore full connectivity, you need to undo the previous security group rule changes made and set the inbound rules back to allow traffic from all sources (0.0.0.0/0).

Partition Map: After removing the restrictions, the show pmap command should show all 4096 partitions being managed correctly across the cluster, indicating that the data is now fully distributed and accessible.
Node Communication: All nodes should be active and successfully heartbeating with each other.

By following these steps, you have restored the Aerospike cluster to it's full operational state, ensuring all nodes can communicate and data distribution is consistent across the entire system.

Admin> show pmap
~~~~~~~~~~~~~~~~~~~~~~~~~~~~Partition Map Analysis (2024-08-14 16:46:53 UTC)~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Namespace|                                          Node| Cluster Key|~~~~~~~~~~~~Partitions~~~~~~~~~~~~
         |                                              |            |Primary|Secondary|Unavailable|Dead
mydata   |172.32.15.231:3000                            |4A4F30116D58|    683|      682|          0|   0
mydata   |172.32.4.2:3000                               |4A4F30116D58|    683|      683|          0|   0
mydata   |172.32.5.239:3000                             |4A4F30116D58|    682|      683|          0|   0
mydata   |172.33.11.90:3000                             |4A4F30116D58|    682|      683|          0|   0
mydata   |172.33.8.38:3000                              |4A4F30116D58|    683|      683|          0|   0
mydata   |ip-172-33-7-44.eu-west-3.compute.internal:3000|4A4F30116D58|    683|      682|          0|   0
mydata   |                                              |            |   4096|     4096|          0|   0
Number of rows: 6

Admin> i
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Namespace Object Information (2024-08-14 16:46:58 UTC)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Namespace|                                          Node|Rack|  Repl|Expirations|    Total|~~~~~~~~~~~~Objects~~~~~~~~~~~~|~~~~~~~~~Tombstones~~~~~~~~|~~Pending Migrates~
         |                                              |  ID|Factor|           |  Records|   Master|    Prole|Non-Replica| Master|  Prole|Non-Replica|       Tx|       Rx
mydata   |172.32.15.231:3000                            |  32|     2|    0.000  |434.000  |147.000  |147.000  |  140.000  |0.000  |0.000  |    0.000  |431.000  |102.000
mydata   |172.32.4.2:3000                               |  32|     2|    0.000  |461.000  |156.000  |155.000  |  150.000  |0.000  |0.000  |    0.000  |423.000  |593.000
mydata   |172.32.5.239:3000                             |  32|     2|    0.000  |434.000  |179.000  |158.000  |   97.000  |0.000  |0.000  |    0.000  |425.000  |593.000
mydata   |172.33.11.90:3000                             |  33|     2|    0.000  |436.000  |165.000  |150.000  |  121.000  |0.000  |0.000  |    0.000  |440.000  |422.000
mydata   |172.33.8.38:3000                              |  33|     2|    0.000  |446.000  |153.000  |156.000  |  137.000  |0.000  |0.000  |    0.000  |426.000  |440.000
mydata   |ip-172-33-7-44.eu-west-3.compute.internal:3000|  33|     2|    0.000  |446.000  |142.000  |174.000  |  130.000  |0.000  |0.000  |    0.000  |425.000  |418.000
mydata   |                                              |    |      |    0.000  |  2.657 K|942.000  |940.000  |  775.000  |0.000  |0.000  |    0.000  |  2.570 K|  2.568 K
Number of rows: 6

Part 5: Understanding Partition Management in Strong Consistency

By understanding how Aerospike maintains partitions under Strong Consistency (SC), application developers and solution architects can design their systems to handle network partitions and maintain data integrity effectively. Here's how this knowledge can be applied:

Key Points About Aerospike’s Partition Management

Strong Consistency Guarantees:
- Aerospike’s SC mode ensures that all writes to a single record are applied in a specific order, sequentially.
- This means that even in a network partition or split-brain scenario, data consistency is preserved as long as the partition ownership and quorum requirements are met.
Partition Ownership:
- Each node in the Aerospike cluster manages a portion of the partitions. During a network split, nodes in each region will only manage the partitions they own.

This partition ownership helps ensure that data is consistently handled within each partition subset.

Data Distribution:
- In a distributed system like Aerospike, data is divided into partitions and distributed across nodes. During a split, nodes in each region will continue to manage and serve the partitions they own. This partition-centric approach helps in maintaining operational continuity even when parts of the network are isolated.
Handling Network Partitions:
- When designing systems with Aerospike, it's important to account for the possibility of network partitions. Understanding how partitions are managed and how strong consistency is maintained allows for better planning and mitigation strategies to handle such scenarios.
Application Design Considerations:
- Data Redundancy: Verify that data is replicated across multiple nodes to prevent data loss in case of node or region failures.
- Quorum Configuration: Configure the quorum settings to balance between performance and data consistency, considering the potential for network partitions.
- Monitoring and Alerts: Implement robust monitoring and alerting mechanisms to detect and respond to network partitions and split-brain scenarios promptly.
Solution Architecture:
- Design the architecture to minimize the impact of network partitions.
- This includes configuring the network and security settings to control access between regions and ensuring that the system can handle partitions gracefully without significant data inconsistencies.

By incorporating these considerations into your application design and solution architecture, you can leverage Aerospike’s strong consistency features to build robust, fault-tolerant systems that maintain data integrity even in complex network conditions.

Finally: What's next

Exploring Different Network Partition Scenarios

In this article, we explored how a network split (split brain) can affect a distributed data system. We initially focused on an equal split across two regions, but there are numerous permutations of network partitions that can yield interesting and varied results.
Here, we discuss a few of these scenarios.

A: Uneven subnet split within a region

Configuration:
- London Region: 6 nodes (2 in each of 3 subnets)
- Paris Region: 6 nodes (2 in each of 3 subnets)
Network Partition:
- Partition localized to a single subnet within the London region.
Expected Outcome:
- The majority subnet with the network partition will operate independently, leading to an uneven split within the region and will most likely have all active database partitions.
- With replication factor > 1 and a properly configured logical rack layout all the database partitions will be available in the sub-cluster of 5 subnets.
- Nodes in the single subnet will continue to function normally but may not have any active database partitions.

B: Uneven split across regions - 2 racks per region

Each region contains 2 racks, configured for strong consistency with a replication factor of 3. This replication factor ensures that a copy of the data is always written to the alternate data center. Having more than 1 rack allows for resilience against a data center or rack failure, which is a more realistic scenario.

Configuration:
- London Region: 4 nodes ( 2 racks )
- Paris Region: 3 nodes ( 2 racks )
Network Partition:
- Split between the two regions.
Expected Outcome:
- Data will be at rest in the minorty Paris cluster but unavailable
- With replication factor 3 or more and strong consistency manually resetting the roster to the available nodes will bring the system back online without data loss.

Under normal circumstances, when writing a record with a replication factor of 3 (rf:3), the master write and replicas can be distributed across any of the 4 racks. Since each region has only 2 racks, at least one copy is guaranteed to be written to a different rack within the other region.

See the possible combinations below:

[100,101,102]

[100,101,103]

[100,102,103]

[101,102,103]

To bring the minority cluster online, you will need to reassign the 3 nodes in Paris.

After manually re-rostering to the new remaining nodes, the cluster is now operational as below.

C: Uneven split across regions

Catastrophic event where the majority of the London region's 4 nodes go permanently offline. What would the cluster state look like, and how would client connectivity be affected?

Configuration:
- London Region: 4 nodes
- Paris Region: 3 nodes
Network Partition:
- Split between the two regions.
Expected Outcome:
- Data will remain at rest in the minority Paris cluster but will be unavailable.
- With a replication factor of 2 or more and strong consistency, manually resetting the roster to the available nodes will restore the system without any data loss.

We'll have a detailed focus on this particular scenario as it presents an intriguing challenge.

By now, you should be familiar with setting up VPCs across two regions and establishing a peering connection between them. If you need a refresher, refer to Part 1 of the series titled "Simple Cross-Region Messaging Application."

Next, create 4 EC2 instances in each region, but only install Aerospike on 7 of the nodes. This ensures an odd-sized cluster, which is crucial for this experiment. If you need guidance on this step, visit Part 2 - "Installing an Aerospike NoSQL DB as a Stretch Cluster." You'll find everything you need there.

Once completed, you should have strong consistency enabled with 2 racks numbered 32 and 33.

Now it's time to insert some test data. We covered this on Part 3, so feel free to reuse the same code, updating only the seed host addresses.

Before adding any data, verify the current state of the cluster. There should be no data present at this stage.

# 4:3 split over 2 racks '32' and '33'

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Namespace Object Information (2024-08-21 12:18:31 UTC)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Namespace|                                           Node|Rack|  Repl|Expirations|  Total|~~~~~~~~~~Objects~~~~~~~~~~|~~~~~~~~~Tombstones~~~~~~~~|~~~~Pending~~~~
         |                                               |  ID|Factor|           |Records| Master|  Prole|Non-Replica| Master|  Prole|Non-Replica|~~~~Migrates~~~
         |                                               |    |      |           |       |       |       |           |       |       |           |     Tx|     Rx
mydata   |172.32.0.79:3000                               |  32|     2|    0.000  |0.000  |0.000  |0.000  |    0.000  |0.000  |0.000  |    0.000  |0.000  |0.000
mydata   |172.32.1.105:3000                              |  32|     2|    0.000  |0.000  |0.000  |0.000  |    0.000  |0.000  |0.000  |    0.000  |0.000  |0.000
mydata   |172.32.7.25:3000                               |  32|     2|    0.000  |0.000  |0.000  |0.000  |    0.000  |0.000  |0.000  |    0.000  |0.000  |0.000
mydata   |172.32.8.190:3000                              |  32|     2|    0.000  |0.000  |0.000  |0.000  |    0.000  |0.000  |0.000  |    0.000  |0.000  |0.000
mydata   |172.33.35.102:3000                             |  33|     2|    0.000  |0.000  |0.000  |0.000  |    0.000  |0.000  |0.000  |    0.000  |0.000  |0.000
mydata   |172.33.42.240:3000                             |  33|     2|    0.000  |0.000  |0.000  |0.000  |    0.000  |0.000  |0.000  |    0.000  |0.000  |0.000
mydata   |ip-172-33-46-58.eu-west-3.compute.internal:3000|  33|     2|    0.000  |0.000  |0.000  |0.000  |    0.000  |0.000  |0.000  |    0.000  |0.000  |0.000
mydata   |

Insert some test records using your Python Application

python3.8 app.py
Connected to Server
Timeout reached. Exiting loop.

For our first run we had 271 records split across all nodes and regions. Interestingly if you sum up the master records in rack '32' [ 35+40+33+38=146] this should equal the prole records in rack '33' [52+53+41=146].

This is because we have 2 racks with replication factor of 2. We expect under normal circumstances a write will be written to rack rx with a copy written to rack ry and in strong consistency we guarantee no ambiguity about whether a record is written or not written.( Note: with replication factor 2 and a single rack available with 2 or more nodes, clients will still be able write data successfully)

# 271 master records
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Namespace Object Information (2024-08-21 12:21:31 UTC)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Namespace|                                           Node|Rack|  Repl|Expirations|    Total|~~~~~~~~~~~~Objects~~~~~~~~~~~~|~~~~~~~~~Tombstones~~~~~~~~|~~~~Pending~~~~
         |                                               |  ID|Factor|           |  Records|   Master|    Prole|Non-Replica| Master|  Prole|Non-Replica|~~~~Migrates~~~
         |                                               |    |      |           |         |         |         |           |       |       |           |     Tx|     Rx
mydata   |172.32.0.79:3000                               |  32|     2|    0.000  | 71.000  | 35.000  | 36.000  |    0.000  |0.000  |0.000  |    0.000  |0.000  |0.000
mydata   |172.32.1.105:3000                              |  32|     2|    0.000  | 71.000  | 40.000  | 31.000  |    0.000  |0.000  |0.000  |    0.000  |0.000  |0.000
mydata   |172.32.7.25:3000                               |  32|     2|    0.000  | 60.000  | 33.000  | 27.000  |    0.000  |0.000  |0.000  |    0.000  |0.000  |0.000
mydata   |172.32.8.190:3000                              |  32|     2|    0.000  | 69.000  | 38.000  | 31.000  |    0.000  |0.000  |0.000  |    0.000  |0.000  |0.000
mydata   |172.33.35.102:3000                             |  33|     2|    0.000  | 98.000  | 46.000  | 52.000  |    0.000  |0.000  |0.000  |    0.000  |0.000  |0.000
mydata   |172.33.42.240:3000                             |  33|     2|    0.000  | 88.000  | 35.000  | 53.000  |    0.000  |0.000  |0.000  |    0.000  |0.000  |0.000
mydata   |ip-172-33-46-58.eu-west-3.compute.internal:3000|  33|     2|    0.000  | 85.000  | 44.000  | 41.000  |    0.000  |0.000  |0.000  |    0.000  |0.000  |0.000
mydata   |                                               |    |      |    0.000  |542.000  |271.000  |271.000  |    0.000  |0.000  |0.000  |    0.000  |0.000  |0.000
Number of rows: 7

Go ahead and insert some more data and perhaps try doing some accounting checks like above.

python3.8 app.py
Connected to Server
Timeout reached. Exiting loop.

# 517 master records
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Namespace Object Information (2024-08-21 12:24:13 UTC)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Namespace|                                           Node|Rack|  Repl|Expirations|    Total|~~~~~~~~~~~~Objects~~~~~~~~~~~~|~~~~~~~~~Tombstones~~~~~~~~|~~~~Pending~~~~
         |                                               |  ID|Factor|           |  Records|   Master|    Prole|Non-Replica| Master|  Prole|Non-Replica|~~~~Migrates~~~
         |                                               |    |      |           |         |         |         |           |       |       |           |     Tx|     Rx
mydata   |172.32.0.79:3000                               |  32|     2|    0.000  |134.000  | 68.000  | 66.000  |    0.000  |0.000  |0.000  |    0.000  |0.000  |0.000
mydata   |172.32.1.105:3000                              |  32|     2|    0.000  |137.000  | 78.000  | 59.000  |    0.000  |0.000  |0.000  |    0.000  |0.000  |0.000
mydata   |172.32.7.25:3000                               |  32|     2|    0.000  |110.000  | 63.000  | 47.000  |    0.000  |0.000  |0.000  |    0.000  |0.000  |0.000
mydata   |172.32.8.190:3000                              |  32|     2|    0.000  |136.000  | 77.000  | 59.000  |    0.000  |0.000  |0.000  |    0.000  |0.000  |0.000
mydata   |172.33.35.102:3000                             |  33|     2|    0.000  |192.000  | 79.000  |113.000  |    0.000  |0.000  |0.000  |    0.000  |0.000  |0.000
mydata   |172.33.42.240:3000                             |  33|     2|    0.000  |160.000  | 67.000  | 93.000  |    0.000  |0.000  |0.000  |    0.000  |0.000  |0.000
mydata   |ip-172-33-46-58.eu-west-3.compute.internal:3000|  33|     2|    0.000  |165.000  | 85.000  | 80.000  |    0.000  |0.000  |0.000  |    0.000  |0.000  |0.000
mydata   |                                               |    |      |    0.000  |  1.034 K|517.000  |517.000  |    0.000  |0.000  |0.000  |    0.000  |0.000  |0.000
Number of rows: 7

How many master records did you get in rack 32 and how many prole records did you get in rack 33?

Bin bang! Go ahead and shutdown all 4 nodes in the London region. Your asadm info and pmap commands should resemble below:

Admin+> info
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Namespace Object Information (2024-08-21 12:26:44 UTC)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Namespace|                                           Node|Rack|  Repl|Expirations|    Total|~~~~~~~~~~Objects~~~~~~~~~~|~~~~~~~~~Tombstones~~~~~~~~|~~~~Pending~~~~
         |                                               |  ID|Factor|           |  Records| Master|  Prole|Non-Replica| Master|  Prole|Non-Replica|~~~~Migrates~~~
         |                                               |    |      |           |         |       |       |           |       |       |           |     Tx|     Rx
~~       |172.32.0.79:3000                               |  ~~|    ~~|         ~~|       ~~|     ~~|     ~~|         ~~|     ~~|     ~~|         ~~|     ~~|     ~~
~~       |172.32.1.105:3000                              |  ~~|    ~~|         ~~|       ~~|     ~~|     ~~|         ~~|     ~~|     ~~|         ~~|     ~~|     ~~
~~       |172.32.7.25:3000                               |  ~~|    ~~|         ~~|       ~~|     ~~|     ~~|         ~~|     ~~|     ~~|         ~~|     ~~|     ~~
~~       |172.32.8.190:3000                              |  ~~|    ~~|         ~~|       ~~|     ~~|     ~~|         ~~|     ~~|     ~~|         ~~|     ~~|     ~~
~~       |                                               |    |      |         ~~|       ~~|     ~~|     ~~|         ~~|     ~~|     ~~|         ~~|     ~~|     ~~
mydata   |172.33.35.102:3000                             |  33|     2|    0.000  |192.000  |0.000  |0.000  |  192.000  |0.000  |0.000  |    0.000  |0.000  |0.000
mydata   |172.33.42.240:3000                             |  33|     2|    0.000  |160.000  |0.000  |0.000  |  160.000  |0.000  |0.000  |    0.000  |0.000  |0.000
mydata   |ip-172-33-46-58.eu-west-3.compute.internal:3000|  33|     2|    0.000  |165.000  |0.000  |0.000  |  165.000  |0.000  |0.000  |    0.000  |0.000  |0.000
mydata   |                                               |    |      |    0.000  |517.000  |0.000  |0.000  |  517.000  |0.000  |0.000  |    0.000  |0.000  |0.000
Number of rows: 7

Admin+> show pmap
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Partition Map Analysis (2024-08-21 12:28:14 UTC)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Namespace|                                            Node| Cluster Key|~~~~~~~~~~~~Partitions~~~~~~~~~~~~
         |                                                |            |Primary|Secondary|Unavailable|Dead
~~       |172.32.0.79:3000                                |          ~~|     ~~|       ~~|         ~~|  ~~
~~       |172.32.1.105:3000                               |          ~~|     ~~|       ~~|         ~~|  ~~
~~       |172.32.7.25:3000                                |          ~~|     ~~|       ~~|         ~~|  ~~
~~       |172.32.8.190:3000                               |          ~~|     ~~|       ~~|         ~~|  ~~
~~       |                                                |            |     ~~|       ~~|         ~~|  ~~
mydata   |172.33.35.102:3000                              |39A2BCE70658|      0|        0|       4096|   0
mydata   |172.33.46.58:3000                               |39A2BCE70658|      0|        0|       4096|   0
mydata   |ip-172-33-42-240.eu-west-3.compute.internal:3000|39A2BCE70658|      0|        0|       4096|   0
mydata   |                                                |            |      0|        0|      12288|   0

At this stage your client cannot access any partitions. To resolve this you will need to issue an operator command once sure the system is stable. Essentially, remove all nodes on rack '32' from the roster.

Admin+> manage roster remove nodes A541@32 A445@32 A333@32 A332@32  ns mydata

Node(s) successfully removed from pending-roster.

Run "manage recluster" for your changes to take affect.
Admin+> manage recluster
Successfully started recluster

Admin+> show roster
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Roster (2024-08-21 12:29:03 UTC)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                                            Node|        Node ID|Namespace|         Current Roster|         Pending Roster|         Observed Nodes
172.32.1.105:3000                               |000000000000000|~~       |~~                     |~~                     |~~
172.32.8.190:3000                               |000000000000000|~~       |~~                     |~~                     |~~
172.32.7.25:3000                                |000000000000000|~~       |~~                     |~~                     |~~
172.32.0.79:3000                                |000000000000000|~~       |~~                     |~~                     |~~
172.33.46.58:3000                               |A250           |mydata   |A829@33,A476@33,A250@33|A829@33,A476@33,A250@33|A829@33,A476@33,A250@33
ip-172-33-42-240.eu-west-3.compute.internal:3000|A476           |mydata   |A829@33,A476@33,A250@33|A829@33,A476@33,A250@33|A829@33,A476@33,A250@33
172.33.35.102:3000                              |*A829          |mydata   |A829@33,A476@33,A250@33|A829@33,A476@33,A250@33|A829@33,A476@33,A250@33
Number of rows: 7

Admin+> show pmap
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Partition Map Analysis (2024-08-21 12:29:11 UTC)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Namespace|                                            Node| Cluster Key|~~~~~~~~~~~~Partitions~~~~~~~~~~~~
         |                                                |            |Primary|Secondary|Unavailable|Dead
~~       |172.32.0.79:3000                                |          ~~|     ~~|       ~~|         ~~|  ~~
~~       |172.32.1.105:3000                               |          ~~|     ~~|       ~~|         ~~|  ~~
~~       |172.32.7.25:3000                                |          ~~|     ~~|       ~~|         ~~|  ~~
~~       |172.32.8.190:3000                               |          ~~|     ~~|       ~~|         ~~|  ~~
~~       |                                                |            |     ~~|       ~~|         ~~|  ~~
mydata   |172.33.35.102:3000                              |E6BE8E200C70|   1366|     1365|          0|   0
mydata   |172.33.46.58:3000                               |E6BE8E200C70|   1365|     1365|          0|   0
mydata   |ip-172-33-42-240.eu-west-3.compute.internal:3000|E6BE8E200C70|   1365|     1366|          0|   0
mydata   |                                                |            |   4096|     4096|          0|   0
Number of rows: 7

At this point your client should be able to write data. However, let's validate the overall record count.

# After recluster we have 517 records as before

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Namespace Object Information (2024-08-21 12:29:31 UTC)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Namespace|                                           Node|Rack|  Repl|Expirations|    Total|~~~~~~~~~~~~Objects~~~~~~~~~~~~|~~~~~~~~~Tombstones~~~~~~~~|~~~~Pending~~~~
         |                                               |  ID|Factor|           |  Records|   Master|    Prole|Non-Replica| Master|  Prole|Non-Replica|~~~~Migrates~~~
         |                                               |    |      |           |         |         |         |           |       |       |           |     Tx|     Rx
~~       |172.32.0.79:3000                               |  ~~|    ~~|         ~~|       ~~|       ~~|       ~~|         ~~|     ~~|     ~~|         ~~|     ~~|     ~~
~~       |172.32.1.105:3000                              |  ~~|    ~~|         ~~|       ~~|       ~~|       ~~|         ~~|     ~~|     ~~|         ~~|     ~~|     ~~
~~       |172.32.7.25:3000                               |  ~~|    ~~|         ~~|       ~~|       ~~|       ~~|         ~~|     ~~|     ~~|         ~~|     ~~|     ~~
~~       |172.32.8.190:3000                              |  ~~|    ~~|         ~~|       ~~|       ~~|       ~~|         ~~|     ~~|     ~~|         ~~|     ~~|     ~~
~~       |                                               |    |      |         ~~|       ~~|       ~~|       ~~|         ~~|     ~~|     ~~|         ~~|     ~~|     ~~
mydata   |172.33.35.102:3000                             |  33|     2|    0.000  |364.000  |195.000  |169.000  |    0.000  |0.000  |0.000  |    0.000  |0.000  |0.000
mydata   |172.33.42.240:3000                             |  33|     2|    0.000  |331.000  |158.000  |173.000  |    0.000  |0.000  |0.000  |    0.000  |0.000  |0.000
mydata   |ip-172-33-46-58.eu-west-3.compute.internal:3000|  33|     2|    0.000  |339.000  |164.000  |175.000  |    0.000  |0.000  |0.000  |    0.000  |0.000  |0.000
mydata   |                                               |    |      |    0.000  |  1.034 K|517.000  |517.000  |    0.000  |0.000  |0.000  |    0.000  |0.000  |0.000
Number of rows: 7

Insert some additional test data before reviving rack '32' in London.

python3.8 app.py
Connected to Server
Timeout reached. Exiting loop.

# We now have 778 records
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Namespace Object Information (2024-08-21 12:44:51 UTC)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Namespace|                                           Node|Rack|  Repl|Expirations|    Total|~~~~~~~~~~~~Objects~~~~~~~~~~~~|~~~~~~~~~Tombstones~~~~~~~~|~~~~Pending~~~~
         |                                               |  ID|Factor|           |  Records|   Master|    Prole|Non-Replica| Master|  Prole|Non-Replica|~~~~Migrates~~~
         |                                               |    |      |           |         |         |         |           |       |       |           |     Tx|     Rx
~~       |172.32.0.79:3000                               |  ~~|    ~~|         ~~|       ~~|       ~~|       ~~|         ~~|     ~~|     ~~|         ~~|     ~~|     ~~
~~       |172.32.1.105:3000                              |  ~~|    ~~|         ~~|       ~~|       ~~|       ~~|         ~~|     ~~|     ~~|         ~~|     ~~|     ~~
~~       |172.32.7.25:3000                               |  ~~|    ~~|         ~~|       ~~|       ~~|       ~~|         ~~|     ~~|     ~~|         ~~|     ~~|     ~~
~~       |172.32.8.190:3000                              |  ~~|    ~~|         ~~|       ~~|       ~~|       ~~|         ~~|     ~~|     ~~|         ~~|     ~~|     ~~
~~       |                                               |    |      |         ~~|       ~~|       ~~|       ~~|         ~~|     ~~|     ~~|         ~~|     ~~|     ~~
mydata   |172.33.35.102:3000                             |  33|     2|    0.000  |557.000  |295.000  |262.000  |    0.000  |0.000  |0.000  |    0.000  |0.000  |0.000
mydata   |172.33.42.240:3000                             |  33|     2|    0.000  |498.000  |250.000  |248.000  |    0.000  |0.000  |0.000  |    0.000  |0.000  |0.000
mydata   |ip-172-33-46-58.eu-west-3.compute.internal:3000|  33|     2|    0.000  |501.000  |233.000  |268.000  |    0.000  |0.000  |0.000  |    0.000  |0.000  |0.000
mydata   |                                               |    |      |    0.000  |  1.556 K|778.000  |778.000  |    0.000  |0.000  |0.000  |    0.000  |0.000  |0.000
Number of rows: 7

Now you have been reliably informed the outage has been rectified, bring all the nodes in rack '32' back online.

Admin+> manage roster stage observed A541@32,A445@32,A333@32,A332@32 ns mydata
Pending roster now contains observed nodes.
Run "manage recluster" for your changes to take affect.

Admin+> manage recluster
Successfully started recluster

Admin+> show racks
~Racks (2024-08-21 12:48:07 UTC)~~
Namespace|Rack|              Nodes
         |  ID|
mydata   |  32|A541,A445,A333,A332
mydata   |  33|A829,A476,A250
Number of rows: 2

Admin+> show pmap
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Partition Map Analysis (2024-08-21 12:48:13 UTC)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Namespace|                                            Node| Cluster Key|~~~~~~~~~~~~Partitions~~~~~~~~~~~~
         |                                                |            |Primary|Secondary|Unavailable|Dead
mydata   |172.32.0.79:3000                                |4AC63FB091A3|    109|      915|          0|   0
mydata   |172.32.1.105:3000                               |4AC63FB091A3|    103|      921|          0|   0
mydata   |172.32.7.25:3000                                |4AC63FB091A3|    111|      913|          0|   0
mydata   |172.32.8.190:3000                               |4AC63FB091A3|     99|      925|          0|   0
mydata   |172.33.35.102:3000                              |4AC63FB091A3|   1213|      179|          0|   0
mydata   |172.33.46.58:3000                               |4AC63FB091A3|   1226|      185|          0|   0
mydata   |ip-172-33-42-240.eu-west-3.compute.internal:3000|4AC63FB091A3|   1235|      167|          0|   0
mydata   |                                                |            |   4096|     4205|          0|   0
Number of rows: 7

Admin+> show roster
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Roster (2024-08-21 12:48:19 UTC)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                                            Node| Node|Namespace|                                         Current Roster|                                         Pending Roster|                                         Observed Nodes
                                                |   ID|         |                                                       |                                                       |
172.33.46.58:3000                               |A250 |mydata   |A829@33,A541@32,A476@33,A445@32,A333@32,A332@32,A250@33|A829@33,A541@32,A476@33,A445@32,A333@32,A332@32,A250@33|A829@33,A541@32,A476@33,A445@32,A333@32,A332@32,A250@33
172.32.8.190:3000                               |A332 |mydata   |A829@33,A541@32,A476@33,A445@32,A333@32,A332@32,A250@33|A829@33,A541@32,A476@33,A445@32,A333@32,A332@32,A250@33|A829@33,A541@32,A476@33,A445@32,A333@32,A332@32,A250@33
172.32.1.105:3000                               |A333 |mydata   |A829@33,A541@32,A476@33,A445@32,A333@32,A332@32,A250@33|A829@33,A541@32,A476@33,A445@32,A333@32,A332@32,A250@33|A829@33,A541@32,A476@33,A445@32,A333@32,A332@32,A250@33
172.32.7.25:3000                                |A445 |mydata   |A829@33,A541@32,A476@33,A445@32,A333@32,A332@32,A250@33|A829@33,A541@32,A476@33,A445@32,A333@32,A332@32,A250@33|A829@33,A541@32,A476@33,A445@32,A333@32,A332@32,A250@33
ip-172-33-42-240.eu-west-3.compute.internal:3000|A476 |mydata   |A829@33,A541@32,A476@33,A445@32,A333@32,A332@32,A250@33|A829@33,A541@32,A476@33,A445@32,A333@32,A332@32,A250@33|A829@33,A541@32,A476@33,A445@32,A333@32,A332@32,A250@33
172.32.0.79:3000                                |A541 |mydata   |A829@33,A541@32,A476@33,A445@32,A333@32,A332@32,A250@33|A829@33,A541@32,A476@33,A445@32,A333@32,A332@32,A250@33|A829@33,A541@32,A476@33,A445@32,A333@32,A332@32,A250@33
172.33.35.102:3000                              |*A829|mydata   |A829@33,A541@32,A476@33,A445@32,A333@32,A332@32,A250@33|A829@33,A541@32,A476@33,A445@32,A333@32,A332@32,A250@33|A829@33,A541@32,A476@33,A445@32,A333@32,A332@32,A250@33
Number of rows: 7


# 778 records
Admin+> info
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Namespace Object Information (2024-08-21 12:49:17 UTC)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Namespace|                                           Node|Rack|  Repl|Expirations|    Total|~~~~~~~~~~~~Objects~~~~~~~~~~~~|~~~~~~~~~Tombstones~~~~~~~~|~~~~Pending~~~~
         |                                               |  ID|Factor|           |  Records|   Master|    Prole|Non-Replica| Master|  Prole|Non-Replica|~~~~Migrates~~~
         |                                               |    |      |           |         |         |         |           |       |       |           |     Tx|     Rx
mydata   |172.32.0.79:3000                               |  32|     2|    0.000  |195.000  |100.000  | 95.000  |    0.000  |0.000  |0.000  |    0.000  |0.000  |0.000
mydata   |172.32.1.105:3000                              |  32|     2|    0.000  |209.000  |115.000  | 94.000  |    0.000  |0.000  |0.000  |    0.000  |0.000  |0.000
mydata   |172.32.7.25:3000                               |  32|     2|    0.000  |176.000  |102.000  | 74.000  |    0.000  |0.000  |0.000  |    0.000  |0.000  |0.000
mydata   |172.32.8.190:3000                              |  32|     2|    0.000  |198.000  |108.000  | 90.000  |    0.000  |0.000  |0.000  |    0.000  |0.000  |0.000
mydata   |172.33.35.102:3000                             |  33|     2|    0.000  |291.000  |126.000  |165.000  |    0.000  |0.000  |0.000  |    0.000  |0.000  |0.000
mydata   |172.33.42.240:3000                             |  33|     2|    0.000  |252.000  |104.000  |148.000  |    0.000  |0.000  |0.000  |    0.000  |0.000  |0.000
mydata   |ip-172-33-46-58.eu-west-3.compute.internal:3000|  33|     2|    0.000  |235.000  |123.000  |112.000  |    0.000  |0.000  |0.000  |    0.000  |0.000  |0.000
mydata   |                                               |    |      |    0.000  |  1.556 K|778.000  |778.000  |    0.000  |0.000  |0.000  |    0.000  |0.000  |0.000
Number of rows: 7

Voila! - Looks good to me. We have all 778 records that we expected.

Automating Network Partition Scenarios

Automating these scenarios can save time and reduce the potential for human error. You can use scripts and tools like AWS CLI or Terraform to create and manage these network partitions.

Future: Service Mesh and Kubernetes

In the next article, we will explore creating a similar network partition scenario using a service mesh and stretching a database cluster in Kubernetes with the Aerospike Kubernetes Operator.

Contact and Feedback

If you have any questions or suggestions, feel free to drop me a line at icloud.nkm@gmail.com.

Conclusion

We have discussed various network partition scenarios and their potential impacts on a distributed data system. By understanding and testing these scenarios, you can design more resilient and robust systems.

Hope you have enjoyed reading this article and learned something new.

How to use the Kubernetes Operator to establish connectivity between clusters

Naresh Maharaj — Fri, 09 Feb 2024 16:32:21 +0000

Discover how Aerospike's Cross Datacenter Replication feature (XDR) transfers data between clusters effortlessly.

This post outlines how to establish connectivity between two Aerospike clusters by leveraging Aerospike's Cross Datacenter Replication (XDR) feature to seamlessly transmit data from a source cluster to a destination cluster, providing high availability for critical data systems.

Ensuring network visibility for Aerospike service ports in the remote data center from a source cluster is crucial. However, this can pose challenges, especially in a Kubernetes environment. Fortunately, there is a solution: deploying a proxy server in front of the private Kubernetes destination cluster. To showcase this solution, we recommend installing the Kubernetes Operator to facilitate the creation and scheduling of the source and destination databases. While the demonstration is on setting up replication in one direction, it's worth noting that Aerospike supports active-active replication. This is accompanied by a conflict resolution mechanism to handle update clashes. This XDR proxy also supports this feature as well.

Let's begin the process.

Fig. 1 Aerospike Kubernetes Operator

The following Kubernetes nodes have been created using EKS.

First, display the following private and public IP addresses from listing the nodes with the kubectl command.

kubectl get nodes -o wide

NAME                             STATUS   ROLES    AGE     VERSION                INTERNAL-IP      EXTERNAL-IP      OS-IMAGE         KERNEL-VERSION                 CONTAINER-RUNTIME
ip-192-168-11-132.ec2.internal   Ready    <none>   2m53s   v1.22.15-eks-fb459a0   192.168.11.132   44.201.67.177    Amazon Linux 2   5.4.219-126.411.amzn2.x86_64   docker://20.10.17
ip-192-168-31-131.ec2.internal   Ready    <none>   2m52s   v1.22.15-eks-fb459a0   192.168.31.131   44.192.83.79     Amazon Linux 2   5.4.219-126.411.amzn2.x86_64   docker://20.10.17
ip-192-168-41-140.ec2.internal   Ready    <none>   2m51s   v1.22.15-eks-fb459a0   192.168.41.140   18.208.222.35    Amazon Linux 2   5.4.219-126.411.amzn2.x86_64   docker://20.10.17
ip-192-168-41-63.ec2.internal    Ready    <none>   2m51s   v1.22.15-eks-fb459a0   192.168.41.63    54.173.138.131   Amazon Linux 2   5.4.219-126.411.amzn2.x86_64   docker://20.10.17
ip-192-168-59-220.ec2.internal   Ready    <none>   2m52s   v1.22.15-eks-fb459a0   192.168.59.220   54.227.122.222   Amazon Linux 2   5.4.219-126.411.amzn2.x86_64   docker://20.10.17
ip-192-168-6-124.ec2.internal    Ready    <none>   2m51s   v1.22.15-eks-fb459a0   192.168.6.124    35.174.60.1      Amazon Linux 2   5.4.219-126.411.amzn2.x86_64   docker://20.10.1

Now, get a copy of the Aerospike git repo for the Kubernetes Operator.

git clone \ https://github.com/aerospike/aerospike-kubernetes-operator.git
cp features.conf \ aerospike-kubernetes-operator/config/samples/secrets/.

Setup

Run the following commands in the order specified. Wait for the CSV “Succeeded phase” to appear after running this line. Initially, it might take between 30 seconds and a minute to show up.

kubectl get csv -n operators -w

cd aerospike-kubernetes-operator/
kubectl apply -f config/samples/storage/eks_ssd_storage_class.yaml
curl -sL \
https://github.com/operator-framework/operator-lifecycle-manager/releases/download/v0.22.0/install.sh | bash -s v0.22.0
kubectl create -f \
https://operatorhub.io/install/aerospike-kubernetes-operator.yaml
kubectl get csv -n operators -w
cd..
git clone \ https://github.com/nareshmaharaj-consultant/kubernetes-anything
cd kubernetes-anything
./createNamespace.sh aerospike
cd ../aerospike-kubernetes-operator/
kubectl -n aerospike create secret generic aerospike-secret \
--from-file=config/samples/secrets
kubectl -n aerospike create secret generic auth-secret \
--from-literal=password='admin123'

Destination Cluster

Use the following YAML configuration file for our destination cluster. Save the file and name it ssd1_xdr_dest_6.1_cluster_cr.yaml:

apiVersion: asdb.aerospike.com/v1beta1
kind: AerospikeCluster
metadata:
  name: aerocluster-dest-xdr
  namespace: aerospike

spec:
  size: 1
  image: aerospike/aerospike-server-enterprise:6.1.0.2

  storage:
    filesystemVolumePolicy:
      initMethod: deleteFiles
      cascadeDelete: true
    blockVolumePolicy:
      cascadeDelete: true
    volumes:
      - name: workdir
        aerospike:
          path: /opt/aerospike
        source:
          persistentVolume:
            storageClass: ssd
            volumeMode: Filesystem
            size: 1Gi
      - name: ns
        aerospike:
          path: /opt/aerospike/data/
        source:
          persistentVolume:
            storageClass: ssd
            volumeMode: Filesystem
            size: 1Gi
      - name: aerospike-config-secret
        source:
          secret:
            secretName: aerospike-secret
        aerospike:
          path: /etc/aerospike/secret

  podSpec:
    multiPodPerHost: true

  aerospikeAccessControl:
    roles:
      - name: writer
        privileges:
        - read-write
      - name: reader
        privileges:
        - read
    users:
      - name: admin
        secretName: auth-secret
        roles:
          - sys-admin
          - user-admin
          - read-write
      - name: xdr-writer
        secretName: xdr-user-auth-secret
        roles:
          - writer

  aerospikeConfig:
    service:
      feature-key-file: /etc/aerospike/secret/features.conf
    security: {}
    network:
      service:
        port: 3000
      fabric:
        port: 3001
      heartbeat:
        port: 3002
    namespaces:
      - name: test
        memory-size: 134217728
        replication-factor: 1
        storage-engine:
          type: device
          files:
            - /opt/aerospike/data/test.dat
          filesize: 1073741824
          data-in-memory: true

Next, create the following Kubernetes resources for our Aerospike destination cluster:

XDR destination database user login credentials, as a Kubernetes secret
Destination database cluster using our YAML file named ssd1_xdr_dest_6.1_cluster_cr.yaml

export secret_auth_name=xdr-user-auth-secret
export password_secret=admin123
kubectl -n aerospike create secret generic $secret_auth_name \
--from-literal=password=$password_secret
kubectl create -f ssd1_xdr_dest_6.1_cluster_cr.yaml
kubectl -n aerospike get po -w

You should see the database pods up and running successfully.

kubectl get po -n aerospike -w

NAME                       READY   STATUS     RESTARTS   AGE
aerocluster-dest-xdr-0-0   0/1     Init:0/1   0          13s
aerocluster-dest-xdr-0-0   0/1     Init:0/1   0          18s
aerocluster-dest-xdr-0-0   0/1     PodInitializing   0          19s
aerocluster-dest-xdr-0-0   1/1     Running           0          24s

XDR-Proxy

Next, set up the xdr-proxy. Take a brief look at Fig. 1 Aerospike Kubernetes Operator, and you’ll notice that we are working from the right to the left in that order.

Configuration

Create the following xdr-proxy configuration file. Replace the seed address with a fully qualified domain name (FQDN) for the destination database pod(s) you created earlier. Multiple seed addresses may be added (hint: recommended in production).

cd ..
mkdir -p xdr-cfg/etc/auth
cd xdr-cfg/etc/

cat <<EOF> aerospike-xdr-proxy.yml
# Change the configuration for your use case.
# Naresh Maharaj
# Refer to https://www.aerospike.com/docs/connectors/enterprise/xdr-proxy/configuration/index.html
# for details.

# The connector's listening ports, manage service, TLS, and network interface.
service:
  port: 8901
  # Aerospike Enterprise Server >= 5.0
  manage:
    address: 0.0.0.0
    port: 8902

# The destination aerospike cluster.
aerospike:
  seeds:
    - aerocluster-dest-xdr-0-0.aerospike.svc.cluster.local:
        port: 3000
  credentials:
    username: xdr-writer
    password-file: /etc/aerospike-xdr-proxy/auth/password_DC1.txt
    auth-mode: internal

# The logging config
logging:
  enable-console-logging: true
  file: /var/log/aerospike-xdr-proxy/aerospike-xdr-proxy.log
  levels:
    root: debug
    record-parser: debug
    server: debug
    com.aerospike.connect: debug
  # Ticker log interval in seconds
  ticker-interval: 3600
EOF

sudo tee auth/password_DC1.txt <<EOF
admin123
EOF
cd ..

kubectl -n aerospike create configmap xdr-proxy-cfg --from-file=etc/
kubectl -n aerospike create secret generic xdr-proxy-auth-secret \
--from-file=etc/auth

Deployment

Now that you have the xdr-proxy configuration file created, produce the Kubernetes deployment YAML file for the xdr-proxy. The following YAML file is used to deploy the xdr-proxy pods, ideally in the same data center or location where the destination databases will be hosted.

Tip: Remember that your xdr-proxy configuration differs from the Kubernetes xdr-proxy deployment file.

cat <<EOF> xdr-proxy-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: xdr-proxy
  namespace: aerospike
  labels:
    app: xdr-proxy
spec:
  replicas: 2
  selector:
    matchLabels:
      app: xdr-proxy
  template:
    metadata:
      labels:
        app: xdr-proxy
    spec:
      containers:
      - name: xdr-proxy
        image: aerospike/aerospike-xdr-proxy:2.1.0
        volumeMounts:
        - name: xdr-proxy-dir
          mountPath: "/etc/aerospike-xdr-proxy/"
          readOnly: true
        - name: xdr-auth-dir
          mountPath: "/etc/aerospike-xdr-proxy/auth"
          readOnly: true
        ports:
          - name: xdr-proxy-main
            containerPort: 8901
          - name: xdr-proxy-mng
            containerPort: 8902
      volumes:
      - name: xdr-proxy-dir
        configMap:
          name: xdr-proxy-cfg
          optional: false
      - name: xdr-auth-dir
        secret:
          secretName: xdr-proxy-auth-secret
          optional: false
---
apiVersion: v1
kind: Service
metadata:
  name: xdr-proxy
  namespace: aerospike
spec:
  selector:
    app: xdr-proxy
  ports:
  - name: main
    protocol: TCP
    port: 8901
    targetPort: xdr-proxy-main
  - name: manage
    protocol: TCP
    port: 8902
    targetPort: xdr-proxy-mng
EOF

kubectl create -f xdr-proxy-deployment.yaml
kubectl get po -n aerospike -w

The following command shows the current pods are running successfully. So far, so good!

kubectl get po -n aerospike -w
NAME                         READY   STATUS    RESTARTS   AGE
aerocluster-dest-xdr-0-0     1/1     Running   0          77m
xdr-proxy-7d9fccd6c8-g5mjt   1/1     Running   0          2m26s
xdr-proxy-7d9fccd6c8-mjxp4   1/1     Running   0          2m26s

Source cluster

Create the Aerospike source cluster using the following configuration. We will insert our sample messages here.

cd ../aerospike-kubernetes-operator/

cat <<EOF> ssd1_xdr_src_6.1_cluster_cr.yaml
apiVersion: asdb.aerospike.com/v1beta1
kind: AerospikeCluster
metadata:
  name: aerocluster-source-xdr
  namespace: aerospike

spec:
  size: 1
  image: aerospike/aerospike-server-enterprise:6.1.0.2

  storage:
    filesystemVolumePolicy:
      initMethod: deleteFiles
      cascadeDelete: true
    blockVolumePolicy:
      cascadeDelete: true
    volumes:
      - name: workdir
        aerospike:
          path: /opt/aerospike
        source:
          persistentVolume:
            storageClass: ssd
            volumeMode: Filesystem
            size: 1Gi
      - name: ns
        aerospike:
          path: /opt/aerospike/data/
        source:
          persistentVolume:
            storageClass: ssd
            volumeMode: Filesystem
            size: 1Gi
      - name: aerospike-config-secret
        source:
          secret:
            secretName: aerospike-secret
        aerospike:
          path: /etc/aerospike/secret

  podSpec:
    multiPodPerHost: true

  aerospikeAccessControl:
    roles:
      - name: writer
        privileges:
        - read-write
      - name: reader
        privileges:
        - read
    users:
      - name: admin
        secretName: auth-secret
        roles:
          - sys-admin
          - user-admin
          - read-write

  aerospikeConfig:
    service:
      feature-key-file: /etc/aerospike/secret/features.conf
    security: {}
    network:
      service:
        port: 3000
      fabric:
        port: 3001
      heartbeat:
        port: 3002
    xdr:
      dcs:
        - name: DC2
          connector: true
          node-address-ports:
            - xdr-proxy.aerospike.svc.cluster.local 8901
          namespaces:
            - name: test
    namespaces:
      - name: test
        memory-size: 134217728
        replication-factor: 1
        storage-engine:
          type: device
          files:
            - /opt/aerospike/data/test.dat
          filesize: 1073741824
          data-in-memory: true
EOF

kubectl create -f ssd1_xdr_src_6.1_cluster_cr.yaml
kubectl get po -n aerospike -w

From the source cluster, confirm the XDR component has connected to the xdr-proxy by filtering the Kubernetes log file as shown in the following kubectl command.

kubectl -n aerospike logs aerocluster-source-xdr-0-0 -c aerospike-server | grep xdr | grep conn
Dec 08 2022 13:49:21 GMT: INFO (xdr): (dc.c:581) DC DC2 connected Oct 10 2022 13:57:53 GMT: INFO (xdr): (dc.c:581) DC DC2 connected

Simple message test

Add some sample messages to the source database and confirm they are being received in the destination database cluster. Start by getting the source database service address and connect using Aerospike's command line tool - AQL.

kubectl get svc -n aerospike

NAME                         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
aerocluster-dest-xdr         ClusterIP   None             <none>        3000/TCP            3h38m
aerocluster-dest-xdr-0-0     NodePort    10.100.226.179   <none>        3000:30168/TCP      3h38m
aerocluster-source-xdr       ClusterIP   None             <none>        3000/TCP            33m
aerocluster-source-xdr-0-0   NodePort    10.100.116.173   <none>        3000:31999/TCP      33m
xdr-proxy                    ClusterIP   10.100.44.96     <none>        8901/TCP,8902/TCP   41m

Insert a source record using the following command in AQL

kubectl run -it --rm --restart=Never aerospike-tool -n aerospike --image=aerospike/aerospike-tools:latest -- aql -U admin -P admin123 -h aerocluster-source-xdr-0-0

insert into test.a1 (PK,a,b,c,d) values(1,"A","B","C","D")
OK, 1 record affected.

aql> select * from test
+----+-----+-----+-----+-----+
| PK | a   | b   | c   | d   |
+----+-----+-----+-----+-----+
| 1  | "A" | "B" | "C" | "D" |
+----+-----+-----+-----+-----+
1 row in set (0.023 secs)

Now, run the following select query in the destination cluster.

kubectl run -it --rm --restart=Never aerospike-tool -n aerospike --image=aerospike/aerospike-tools:latest -- aql -U admin -P admin123 -h aerocluster-dest-xdr-0-0

aql> select * from test
+----+-----+-----+-----+-----+
| PK | a   | b   | c   | d   |
+----+-----+-----+-----+-----+
| 1  | "A" | "B" | "C" | "D" |
+----+-----+-----+-----+-----+
1 row in set (0.031 secs)

OK

Interim summary

At this point, it's confirmed that xdr-proxy is doing exactly what It should do.

If if you review the log file for the initial two xdr-proxies scheduled, you should see userKey=1.

kubectl logs xdr-proxy-7d9fccd6c8-5q7gn -n aerospike | grep record-parser
2022-12-08 14:53:50.607 GMT DEBUG record-parser - parsed message fields: key=Key(namespace='test', set=a1, digest=[120, 48, -23, -90, 110, 126, 84, -1, 114, -116, -9, -21, 28, 75, 126, -68, -51, 83, 31, -117], userKey=1, lastUpdateTimeMs=1670511230565, userKeyString=1, digestString='eDDppm5+VP9yjPfrHEt+vM1TH4s=')

kubectl logs xdr-proxy-7d9fccd6c8-f5zkt -n aerospike | grep \
record-parser
(none)

Scaling the XDR-Proxies

Go ahead and scale up the xdr-proxy to six pods by editing the file xdr-proxy-deployment.yaml and then apply the changes.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: xdr-proxy
  namespace: aerospike
  labels:
    app: xdr-proxy
spec:
  replicas: 6
  selector:
    matchLabels:
      app: xdr-proxy
...
kubectl apply -f xdr-proxy-deployment.yaml

You can achieve exactly the same by running the following command.

kubectl scale deploy xdr-proxy -n aerospike --replicas=6

You should now have six instances of the xdr-proxies running.

kubectl get po -n aerospike -w
NAME                         READY   STATUS    RESTARTS   AGE
aerocluster-dest-xdr-0-0     1/1     Running   0          3h50m
aerocluster-source-xdr-0-0   1/1     Running   0          75m
xdr-proxy-7d9fccd6c8-49ttl   1/1     Running   0          7s
xdr-proxy-7d9fccd6c8-5q7gn   1/1     Running   0          83m
xdr-proxy-7d9fccd6c8-c4j7k   1/1     Running   0          7s
xdr-proxy-7d9fccd6c8-f5zkt   1/1     Running   0          83m
xdr-proxy-7d9fccd6c8-lscbg   1/1     Running   0          7s
xdr-proxy-7d9fccd6c8-r56vs   1/1     Running   0          7s

Add some sample messages in the source cluster with primary keys 5, 6, and 7. Note that in Aerospike, a primary key serves as a unique identifier for a record within a specific set in a specific namespace. This key is a unique identifier for addressing the record for any operation.

aql> insert into test (PK,a,b) values (5,"A","B")
OK, 1 record affected.
aql> insert into test (PK,a,b) values (6,"A","B")
OK, 1 record affected.
aql> insert into test (PK,a,b) values (7,"A","B")
OK, 1 record affected.

Notice how userKey=5, userKey=6, and userKey=7 have been shipped across to the newly scaled xdr-proxies. Run the commands below to see the same,

kubectl logs xdr-proxy-7d9fccd6c8-5q7gn -n aerospike | grep \
record-parser
2022-12-08 14:53:50.607 GMT DEBUG record-parser - parsed message fields: key=Key(namespace='test', set=a1, digest=[120, 48, -23, -90, 110, 126, 84, -1, 114, -116, -9, -21, 28, 75, 126, -68, -51, 83, 31, -117], userKey=1, lastUpdateTimeMs=1670511230565, userKeyString=1, digestString='eDDppm5+VP9yjPfrHEt+vM1TH4s=')

kubectl logs xdr-proxy-7d9fccd6c8-f5zkt -n aerospike | grep \
record-parser
(none)

kubectl logs xdr-proxy-7d9fccd6c8-49ttl -n aerospike | grep \
record-parser
(none)

kubectl logs xdr-proxy-7d9fccd6c8-c4j7k -n aerospike | grep \
record-parser
2022-12-08 15:05:37.511 GMT DEBUG record-parser - parsed message fields: key=Key(namespace='test', set=null, digest=[-88, 104, 104, -114, 19, -44, -19, 29, -15, 18, 118, 72, -117, -106, -28, 21, -48, 50, 26, 113], userKey=7, lastUpdateTimeMs=1670511937250, userKeyString=7, digestString='qGhojhPU7R3xEnZIi5bkFdAyGnE=')

kubectl logs xdr-proxy-7d9fccd6c8-lscbg -n aerospike | grep record-parser
2022-12-08 15:05:27.548 GMT DEBUG record-parser - parsed message fields: key=Key(namespace='test', set=null, digest=[68, 4, -94, -44, -75, 112, -102, 73, -120, 41, -101, -120, 33, -111, 15, -114, -85, 46, -2, 80], userKey=5, lastUpdateTimeMs=1670511927465, userKeyString=5, digestString='RASi1LVwmkmIKZuIIZEPjqsu/lA=')
2022-12-08 15:05:32.300 GMT DEBUG record-parser - parsed message fields: key=Key(namespace='test', set=null, digest=[33, -100, 127, 120, 17, 45, -79, 115, -40, 53, -70, -57, 120, 73, 20, -50, -99, -98, -104, 85], userKey=6, lastUpdateTimeMs=1670511932288, userKeyString=6, digestString='IZx/eBEtsXPYNbrHeEkUzp2emFU=')

kubectl logs xdr-proxy-7d9fccd6c8-r56vs -n aerospike | grep record-parser
(none)

However, we need to consider one critical factor. When data is actively flowing between source and destination clusters, the existing cached list of xdr-proxy connections from the source cluster will not be refreshed just because you added a bunch of new xdr-proxies. Consequently, the newly scaled xdr-proxies you have just scheduled will not be utilized immediately.

To demonstrate this, let's add data to the source cluster using Aerospike's benchmark tool. At the same time, we will scale the xdr-proxies on the destination side and observe the results.

Add data to the source cluster

Before you begin, reduce the xdr-proxy server count to one (1) to clarify the observations. In this example, I use my local machine, which has the benchmark tool installed, to send data to the source EC2 instances. To do this, you’ll need to obtain the external IP address of the source cluster’s Kubernetes service. You can download the benchmark tool from https://aerospike.com/docs/tools/install.

kubectl get AerospikeCluster aerocluster-source-xdr -n aerospike  -o yaml

...
pods:
    aerocluster-source-xdr-0-0:
      aerospike:
        accessEndpoints:
        - 192.168.41.63:31999
        alternateAccessEndpoints:
        - 54.173.138.131:31999
        clusterName: aerocluster-source-xdr
        nodeID: 0a0
        tlsAccessEndpoints: []
        tlsAlternateAccessEndpoints: []
        tlsName: ""
      aerospikeConfigHash: 4aacb9809beaa01d99a9f00293c9f7dc141845f8
      hostExternalIP: 54.173.138.131
      hostInternalIP: 192.168.41.63
      image: aerospike/aerospike-server-enterprise:6.1.0.2
      initializedVolumes:
      - workdir
      - ns
      networkPolicyHash: acbbfab3668e1fceeed201139d1173f00095667e
      podIP: 192.168.50.203
      podPort: 3000
      podSpecHash: 972dc2a779fe9ab407212b547d54d3a72ecef259
      servicePort: 31999
...

You may need to add a firewall rule to allow traffic into the Kubernetes service. Connect the asbenchmark tool to start writing traffic using the public IP address for the NodePort Service.

asbenchmark -h 54.173.138.131:31999 -Uadmin -Padmin123 -z 10 -servicesAlternate -w RU,0 -o B256

Scale up the xdr-proxy servers from one to two and check the logs of both proxies to see what messages have been received. In a production environment, you should always disable the unnecessary logging.

Notice how no data has passed through the new xdr-proxy instance xdr-proxy-7d9fccd6c8-s2tzt.

kubectl -n aerospike logs xdr-proxy-7d9fccd6c8-s2tzt | grep \
record-parser
(none)

kubectl -n aerospike logs xdr-proxy-7d9fccd6c8-5q7gn | grep \
record-parser
...
2022-12-08 18:21:36.158 GMT DEBUG record-parser - parsed message fields: key=Key(namespace='test', set=testset, digest=[51, -120, 114, -17, -44, 72, 123, 125, 50, 92, 3, 110, -21, -38, 74, 25, 42, 35, 117, 72], userKey=null, lastUpdateTimeMs=1670523696059, userKeyString=null, digestString='M4hy79RIe30yXANu69pKGSojdUg=')
2022-12-08 18:21:36.158 GMT DEBUG record-parser - parsed message fields: key=Key(namespace='test', set=testset, digest=[124, -6, -70, -23, 44, 41, -19, 40, -11, -16, 126, 120, 81, -113, -112, -79, 66, 77, -99, -6], userKey=null, lastUpdateTimeMs=1670523696059, userKeyString=null, digestString='fPq66Swp7Sj18H54UY+QsUJNnfo=')
2022-12-08 18:21:36.158 GMT DEBUG record-parser - parsed message fields: key=Key(namespace='test', set=testset, digest=[-127, -118, 63, -32, -74, 60, -60, 86, 31, -119, -1, -105, -108, -59, 111, 48, -34, -61, -108, -5], userKey=null, lastUpdateTimeMs=1670523696105, userKeyString=null, digestString='gYo/4LY8xFYfif+XlMVvMN7DlPs=')
2022-12-08 18:21:36.256 GMT DEBUG record-parser - parsed message fields: key=Key(namespace='test', set=testset, digest=[99, 1, 80, 2, 76, -43, 125, 77, 47, 8, 6, 35, 49, 117, -35, 54, 120, -29, 118, -72], userKey=null, lastUpdateTimeMs=1670523696178, userKeyString=null, digestString='YwFQAkzVfU0vCAYjMXXdNnjjdrg=')
2022-12-08 18:21:36.257 GMT DEBUG record-parser - parsed message fields: key=Key(namespace='test', set=testset, digest=[-88, -46, -48, -33, 77, -120, 123, -101, -70, -20, -96, -104, -51, -90, 28, -15, 70, 11, 118, 83], userKey=null, lastUpdateTimeMs=1670523696202, userKeyString=null, digestString='qNLQ302Ie5u67KCYzaYc8UYLdlM=')
2022-12-08 18:21:36.257 GMT DEBUG record-parser - parsed message fields: key=Key(namespace='test', set=testset, digest=[-97, 18, 28, -43, 75, 42, -22, -126, -61, -108, -36, 118, -86, -105, -52, 119, -39, -33, -127, -76], userKey=null, lastUpdateTimeMs=1670523696175, userKeyString=null, digestString='nxIc1Usq6oLDlNx2qpfMd9nfgbQ=')
2022-12-08 18:21:36.257 GMT DEBUG record-parser - parsed message fields: key=Key(namespace='test', set=testset, digest=[-120, 103, -51, 57, -71, -106, 13, -48, 100, 28, 59, -3, -39, -56, -67, -103, 29, 36, 75, 119], userKey=null, lastUpdateTimeMs=1670523696191, userKeyString=null, digestString='iGfNObmWDdBkHDv92ci9mR0kS3c=')
2022-12-08 18:21:36.257 GMT DEBUG record-parser - parsed message fields: key=Key(namespace='test', set=testset, digest=[-47, 88, -13, 13, -35, 77, 24, 22, -40, -61, -118, -115, 82, 13, 127, -125, 53, 66, -22, -8], userKey=null, lastUpdateTimeMs=1670523696233, userKeyString=null, digestString='0VjzDd1NGBbYw4qNUg1/gzVC6vg=')
2022-12-08 18:21:36.258 GMT DEBUG record-parser - parsed message fields: key=Key(namespace='test', set=testset, digest=[-63, -11, 93, -90, 47, 29, -63, 36, 12, 53, -86, 84, 57, -125, 16, 43, -18, 93, 56, 9], userKey=null, lastUpdateTimeMs=1670523696186, userKeyString=null, digestString='wfVdpi8dwSQMNapUOYMQK+5dOAk=')
2022-12-08 18:21:36.258 GMT DEBUG record-parser - parsed message fields: key=Key(namespace='test', set=testset, digest=[-23, 101, -114, -87, -52, 107, 36, 113, 101, 33, -16, 82, -95, 97, 34, -121, 82, -97, 40, 59], userKey=null, lastUpdateTimeMs=1670523696145, userKeyString=null, digestString='6WWOqcxrJHFlIfBSoWEih1KfKDs=')
2022-12-08 18:21:36.258 GMT DEBUG record-parser - parsed message fields: key=Key(namespace='test', set=testset, digest=[-77, -118, 49, 4, -75, 123, 81, 2, -103, -73, 42, -70, -54, 95, 98, 23, 73, 66, -86, 7], userKey=null, lastUpdateTimeMs=1670523696230, userKeyString=null, digestString='s4oxBLV7UQKZtyq6yl9iF0lCqgc=')
2022-12-08 18:21:36.258 GMT DEBUG record-parser - parsed message fields: key=Key(namespace='test', set=testset, digest=[50, -38, -31, -54, -122, -113, -38, 88, 15, 7, 96, 51, -92, -25, 60, -104, 26, 113, -117, -82], userKey=null, lastUpdateTimeMs=1670523696157, userKeyString=null, digestString='MtrhyoaP2lgPB2AzpOc8mBpxi64=')
2022-12-08 18:21:36.258 GMT DEBUG record-parser - parsed message fields: key=Key(namespace='test', set=testset, digest=[-77, 31, 67, -18, -52, -114, 42, -18, 36, -111, 89, 62, 109, 114, -54, 54, -121, -110, -88, -108], userKey=null, lastUpdateTimeMs=1670523696206, userKeyString=null, digestString='sx9D7syOKu4kkVk+bXLKNoeSqJQ=')
2022-12-08 18:21:36.258 GMT DEBUG record-parser - parsed message fields: key=Key(namespace='test', set=testset, digest=[73, 11, 98, -50, 32, 12, 0, -50, 22, -101, -108, 18, 38, 7, -65, 6, -58, 60, -6, -33], userKey=null, lastUpdateTimeMs=1670523696171, userKeyString=null, digestString='SQtiziAMAM4Wm5QSJge/BsY8+t8=')
2022-12-08 18:21:36.258 GMT DEBUG record-parser - parsed message fields: key=Key(namespace='test', set=testset, digest=[109, -82, 24, 53, 35, 89, -72, -117, -22, 79, 119, -89, 56, -5, 0, -103, -54, 51, 25, 126], userKey=null, lastUpdateTimeMs=1670523696146, userKeyString=null, digestString='ba4YNSNZuIvqT3enOPsAmcozGX4=')

Achieve dependable resiliency with Aerospike

Aerospike's XDR feature provides a reliable solution for mitigating the risk of cluster unavailability. By asynchronously replicating data between data centers, users can ensure availability. This step-by-step walkthrough demonstrates how to accomplish this seamlessly in a Kubernetes environment using the xdr-proxy. With the Aerospike Kubernetes Operator, you can effortlessly avoid network complications and achieve optimal performance with minimal effort.

Share your experience! Your feedback is important to us. Join our Aerospike community!

Using AWS IAM for client authentication

Naresh Maharaj — Wed, 20 Dec 2023 20:11:36 +0000

In this blog post, we detail how to create an Amazon Managed Streaming for Apache Kafka (Amazon MSK) resource using AWS Identity and Access Management (AWS IAM) in roles and policies to authenticate user access. In the initial step, we establish an Aerospike Database cluster and insert sample messages into the database. Subsequently, we observe in real time how these messages are streamed to Amazon MSK using Aerospike's Kafka Source Connector. Below we provide a comprehensive, step-by-step guide for users to successfully implement this process.

AWS MKS Kafka

In this section, you will set up a simple three-node Kafka cluster.

Visit the AWS console and select MSK service.

Create a new cluster by selecting Create Cluster → Quick Create.

Select the provisioned cluster and instance type of kafka.t3.small.

Select the EBS storage type per broker of 10 GB.

NOTE: Take note of the VPC, subnets, and security group ID, as you will require these details later in the article.

The next step is the critical step where you will create the AWS IAM policy and roles. This setup ensures that the Aerospike Database authenticates using AWS IAM to write data to MSK.

From the AWS Console, select the AWS IAM service.

To create a new AWS IAM policy, copy the following JSON and paste it in the JSON tab. Replace region:Account-ID with your own region and AWS account ID.

Save the policy and name it msk-tutorial-policy.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "kafka-cluster:Connect",
                "kafka-cluster:AlterCluster",
                "kafka-cluster:DescribeCluster"
            ],
            "Resource": [
                "arn:aws:kafka:region:Account-ID:cluster/MSKTutorialCluster/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "kafka-cluster:*Topic*",
                "kafka-cluster:WriteData",
                "kafka-cluster:ReadData"
            ],
            "Resource": [
                "arn:aws:kafka:region:Account-ID:topic/MSKTutorialCluster/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "kafka-cluster:AlterGroup",
                "kafka-cluster:DescribeGroup"
            ],
            "Resource": [
                "arn:aws:kafka:region:Account-ID:group/MSKTutorialCluster/*"
            ]
        }
    ]
}

Create the IAM role.

Under Common Use Cases, select EC2 and then Next.

Under Permissions, select the policy named msk-tutorial-policy and then Next.

Give the role a name like msk-tutorial-role and click the Create Role button.

Kafka client machine

Next, create a client machine to install the Kafka tools necessary to access our MSK cluster.

Create a new ec2 instance using type t2.micro

Use the default AMI: Amazon Linux 2023

The AMI may be different depending on your region

Create a key-pair if required. I am using an already existing key-pair.

Under Advanced Options.IAM instance profile, select the IAM role created earlier.

Launch the instance!
Under instances launched, choose the instance you just created. Click on the ‘Security’ tab and note the security group associated with this instance.
e.g., sg-0914e6271c97ae4c9 (launch-wizard-1)
Navigate to the VPC section and select Security Groups from the left-hand menu. Locate the security group associated with the MSK cluster, such as sg-e5f51dfb, and choose Edit Inbound Rules.
Create a new rule to allow all traffic from the new ec2 instance.

Kafka topics

After successfully establishing your initial Kafka cluster and Kafka client machine, proceed to conduct testing. Verify the functionality by accessing the MSK cluster, creating a topic, producing and consuming sample messages, and ensuring that everything operates as anticipated.

From the MSK Cluster, note the Kafka version being used. This examples uses 2.8.1.
From the Kafka client machine, install Java 11+.

sudo yum -y install java-11

Download Apache Kafka using wget, then extract the archive using tar.

wget https://archive.apache.org/dist/kafka/2.8.1/kafka_2.12-2.8.1.tgz
tar -xzf kafka_2.12-2.8.1.tgz

To use IAM, you will need the MSK IAM Auth jar file. Download the jar to the Kafka libs folder you just extracted.

cd kafka_2.12-2.8.1/libs/
wget https://github.com/aws/aws-msk-iam-auth/releases/download/v1.1.1/aws-msk-iam-auth-1.1.1-all.jar
cd ../bin/

Create a file called client.properties to use when authenticating to MSK. It will define the SASL mechanism to use and reference the Java class file that will handle your IAM callbacks.

cat <<EOF> client.properties
security.protocol=SASL_SSL
sasl.mechanism=AWS_MSK_IAM
sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;
sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler
EOF

Creating topics

Go to the AWS Console and view the MSK Cluster Client Information. There will be three endpoints to choose from, but you only require one.

Example choose:

B-2.msktutorialcluster.450050.c11.kafka.us-east-1.amazonaws.com:9098

From the kafka/bin folder, run the command to create a topic. Let's call it aerospike-airforce-1.

export BootstrapServerString="b-2.msktutorialcluster.450050.c11.kafka.us-east-1.amazonaws.com:9098"

./kafka-topics.sh --create --bootstrap-server $BootstrapServerString --command-config client.properties --replication-factor 3 --partitions 1 --topic aerospike-airforce-1

Listing topics

To list the topics, use the following command. Notice our latest topic, called aerospike-airforce-1, just showed up.

./kafka-topics.sh --bootstrap-server $BootstrapServerString --command-config client.properties --list

MSKTutorialTopic
__amazon_msk_canary
__consumer_offsets
aerospike
aerospike-airforce-1

Producer and consumer

I agree that this is more of a Kafka-101 rather than a straightforward Hello-World scenario. Nonetheless, it is essential to test our configuration by sending and receiving messages from the designated Kafka topic before proceeding further.

Produce some messages by opening a new window and running the following Kafka producer command. Type three or four messages, hitting the 'Return' key after each message

./kafka-console-producer.sh --broker-list $BootstrapServerString --producer.config client.properties --topic aerospike-airforce-1
>Instrument Check
>Pre flight checks confirmed
>Ready for takeoff
>Full throttle, flaps

You're now ready to start a client consumer application. Open a new window and run the consumer. You should now see the same messages you published earlier.

./kafka-console-consumer.sh --bootstrap-server $BootstrapServerString --consumer.config client.properties --topic aerospike-airforce-1 --from-beginning
Instrument Check
Pre flight checks confirmed
Ready for takeoff
Full throttle, flaps

Database source

Let's review your achievements thus far. You've established a 3-node Kafka cluster in AWS utilizing MSK, incorporating IAM roles and permissions. Additionally, you have successfully created topics and demonstrated the production and consumption of messages using the IAM credentials established during the setup.

The next phase of your journey involves installing the Aerospike Database, inserting messages, and configuring a simple XDR component. XDR is a Cross Datacenter Replication tool and is crucial for transmitting data from the Aerospike Database to the Aerospike Kafka Source Connector allowing us to subsequently forward messages to Amazon MSK.

Create the Aerospike Database

Start by creating a new ec2 instance. For this demo, you can use Linux Centos 8

Rocky 8 AMI: ami-043ceee68871e0bb5 ( us-east-1 )

Select the instance type as t2.medium.

Add the extra volume for the Aerospike data storage layer. EBS volume is all that is required for now.

Launch the instance and connect to the host using ssh. If you have an Aerospike license feature file, upload it to the instance.

Install the Aerospike Database server

Run the following to install the Aerospike Database Server.

export VER="6.1.0.2"
sudo yum install java python3 openssl-devel wget git gcc maven bind-utils sysstat nc -y
wget -O aerospike-tools.tgz 'https://www.aerospike.com/download/tools/latest/artifact/el8'
tar -xvf aerospike-tools.tgz
cd aerospike-tools_*
sudo ./dep-check
sudo ./asinstall
wget -O aerospike.tgz https://enterprise.aerospike.com/enterprise/download/server/$VER/artifact/el8
tar -xvf aerospike.tgz
cd aerospike-server-enterprise-$VER-el8
sudo ./asinstall
sudo mkdir -p /var/log/aerospike/
sudo systemctl enable aerospike

Confirm the storage disk for Aerospike.

lsblk
NAME    MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda    202:0    0  10G  0 disk
└─xvda1 202:1    0  10G  0 part /
xvdb    202:16   0  10G  0 disk   <<----------------- This one!

When its data is available, replace the Aerospike configuration file under /etc/aerospike/aerospike.conf with the configuration file listed below, also replacing the following lines:
- Under heartbeat.address add in your internal 172.x.x.x address
- For xdr.dc.node-address-port enter the {kafka-client-machine-address}:8080

Aerospike Database configuration file for use with `systemd`

service {
  # paxos-single-replica-limit 1 # Number of nodes where the replica count is automatically
  proto-fd-max 15000
  service-threads 10
  feature-key-file /etc/aerospike/features.conf
  node-id A1
  cluster-name CLA
}

logging {
  file /var/log/aerospike/aerospike.log {
    context any info
  }
}

# public and private addresses
network {
  service {
    address any
    port 3000
  }

  heartbeat {
    mode mesh
    address 172.31.94.201
    port 3002 # Heartbeat port for this node.
    interval 150 # controls how often to send a heartbeat packet
    timeout 10 # number of intervals after which a node is considered to be missing
  }

  fabric {
    port 3001
  }

  info {
    port 3003
  }
}

namespace test {
  replication-factor 2
  memory-size 40G
  default-ttl 0
  index-type shmem
  high-water-disk-pct 50
  high-water-memory-pct 60
  stop-writes-pct 90
  nsup-period 0

  storage-engine device {
    device /dev/xvdb
    data-in-memory false
    write-block-size 128K
    min-avail-pct 5
  }
}

xdr {
  # Change notification XDR block that round-robins between two connector nodes
  dc aerospike-kafka-source {
    connector true
    node-address-port 172.31.58.190 8080
    namespace test {
    }
  }
}

Start the Aerospike service

Copy the license feature file to the aerospike configuration directory.

sudo cp features.conf /etc/aerospike/

Start the Aerospike server and check the logs to ensure there are no errors.

sudo systemctl start aerospike

sudo systemctl status aerospike

Aerospike Kafka Source Connector

The seamless flow of data from Aerospike Database Enterprise Edition to Apache Kafka hinges on the utilization of the Aerospike Kafka source (outbound) connector. This connector subscribes to change notifications. Upon receiving these notifications, the connector converts them into messages, which are dispatched to Kafka topics. Going back to the ec2 instance you created earlier with our Kafka client configured, go ahead and install the Aerospike Kafka Source Connector. This is your outbound connector to send data from the Aerospike to MSK.

sudo yum install java #( install 11+ JDK )

wget https://enterprise.aerospike.com/artifacts/enterprise/aerospike-kafka-outbound/5.0.1/aerospike-kafka-outbound-5.0.1-1.noarch.rpm

sudo rpm -i aerospike-kafka-outbound-5.0.0-1.noarch.rpm

Configure the outbound connector

The terms “outbound” and “source connector” are used interchangeably in this article.

Locate the following file on the Kafka client box: /etc/aerospike-kafka-outbound/aerospike-kafka-outbound.yml.
Replace the broker address for one of the node addresses in the MSK Kafka cluster producer-props.bootstrap.servers.
Then add the following contents to the file with the changes that have been outlined.

# Change the configuration for your use case.
#
# Refer to https://www.aerospike.com/docs/connectors/enterprise/kafka/outbound/configuration/index.html
# for details.

# The connector's listening ports, TLS, and network interface.
service:
  port: 8080

# Format of the Kafka destination message.
format:
  mode: flat-json
  metadata-key: metadata

# Aerospike record routing to a Kafka destination.
routing:
  mode: static
  destination: aerospike

# Kafka producer initialization properties.
producer-props:
  bootstrap.servers:
    - b-3.msktutorialcluster.450050.c11.kafka.us-east-1.amazonaws.com:9098
  ssl.truststore.location: /etc/aerospike-kafka-outbound/kafka.client.truststore.jks
  ssl.truststore.password: changeit
  security.protocol: SASL_SSL
  sasl.mechanism: AWS_MSK_IAM
  sasl.jaas.config: software.amazon.msk.auth.iam.IAMLoginModule required awsProfileName=default;
  sasl.client.callback.handler.class: software.amazon.msk.auth.iam.IAMClientCallbackHandler

# The logging properties.
logging:
  file: /var/log/aerospike-kafka-outbound/aerospike-kafka-outbound.log
  enable-console-logging: true
  levels:
    root: debug
    record-parser: debug
    server: debug
    com.aerospike.connect: debug
  ticker-interval: 3600

Create the CA certificate trust store for use in the Kafka Outbound Connector config. You can see the SSL trust store location referenced in the file above as ssl.truststore.location.

sudo cp /usr/lib/jvm/java-11-amazon-corretto/lib/security/cacerts /etc/aerospike-kafka-outbound/kafka.client.truststore.jks

sudo chmod 755 /etc/aerospike-kafka-outbound/kafka.client.truststore.jks

Finally, make the AWS IAM Kafka Auth Jar file available to the Aerospike Outbound Kafka Connector. This is the same jar file that you downloaded and added to the kafka/libs folder.

sudo cp kafka_2.12-2.8.1/libs/aws-msk-iam-auth-1.1.1-all.jar /opt/aerospike-kafka-outbound/lib/aws-msk-iam-auth-1.1.1-all.jar

Start the service.

sudo systemctl enable aerospike-kafka-outbound

sudo systemctl start aerospike-kafka-outbound

Send data from Aerospike to Kafka

Open a separate window so you can list all messages on the Aerospike Kafka topic. Start by adding one of the private endpoint bootstrap servers as an environment variable for ease of use.

export BootstrapServerString="b-3.msktutorialcluster.450050.c11.kafka.us-east-1.amazonaws.com:9098"

Run the consumer client as follows:

./kafka-console-consumer.sh --bootstrap-server $BootstrapServerString --consumer.config client.properties --topic aerospike --from-beginning

In a new window, start AQL, the Aerospike command line client which connects to your Aerospike Database.

aql -U auser -P a-secret-pwd

Insert some data

insert into test (pk, a) values(400, "Your winning lottery ticket awaits you")

Check to see if the message appears in the Kafka consumer window

{"metadata":{"namespace":"test","userKey":400,"digest":"W7eGav2hKfOU00xx7mnOPYa2uCo=","msg":"write","gen":1,"lut":1681488437767,"exp":0},"a":"Your winning lottery ticket awaits you"}

Conclusion

You've just discovered how straightforward it is to transmit data from Aerospike to AWS MSK Kafka while ensuring client authentication through AWS IAM permissions! From establishing an Aerospike Database from scratch to configuring the AWS MSK Kafka cluster and employing the Aerospike Outbound Kafka Connector, you've effortlessly constructed a real-time streaming data pipeline. Congratulations on this accomplishment!

Share your experience! Your feedback is important to us. Join our Aerospike community!

Aerospike Kubernetes - Talking Cluster to Cluster using XDR-Proxy

Naresh Maharaj — Thu, 19 Jan 2023 09:18:49 +0000

In this article we focus on establishing connectivity between 2 Aerospike clusters. The goal is to use Aerospike's Cross Data Center Replication feature ( XDR ) to seamlessly send data from a source cluster to a destination cluster. The source cluster needs network visibility of all Aerospike service ports in the remote cluster, and this can present problems, particularly in a Kubernetes environment. Placing a proxy server in front of the private Kubernetes destination cluster can overcome this problem and achieve the desired goal. To demonstrate the solution we start by installing the Kubernetes Operator that will schedule our source and destination databases. In this example, we set up our replication in one direction. Aerospike is capable of supporting 'master/master' replication and provides a conflict resolution mechanism in the event of update clashes. This too could be supported using the XDR proxy.

Aerospike Kubernetes Operator

The following Kubernetes nodes have been created using EKS. You can display the following private and public IP addresses from listing the nodes with the kubectl command.

kubectl get nodes -o wide
NAME                             STATUS   ROLES    AGE     VERSION                INTERNAL-IP      EXTERNAL-IP      OS-IMAGE         KERNEL-VERSION                 CONTAINER-RUNTIME
ip-192-168-11-132.ec2.internal   Ready    <none>   2m53s   v1.22.15-eks-fb459a0   192.168.11.132   44.201.67.177    Amazon Linux 2   5.4.219-126.411.amzn2.x86_64   docker://20.10.17
ip-192-168-31-131.ec2.internal   Ready    <none>   2m52s   v1.22.15-eks-fb459a0   192.168.31.131   44.192.83.79     Amazon Linux 2   5.4.219-126.411.amzn2.x86_64   docker://20.10.17
ip-192-168-41-140.ec2.internal   Ready    <none>   2m51s   v1.22.15-eks-fb459a0   192.168.41.140   18.208.222.35    Amazon Linux 2   5.4.219-126.411.amzn2.x86_64   docker://20.10.17
ip-192-168-41-63.ec2.internal    Ready    <none>   2m51s   v1.22.15-eks-fb459a0   192.168.41.63    54.173.138.131   Amazon Linux 2   5.4.219-126.411.amzn2.x86_64   docker://20.10.17
ip-192-168-59-220.ec2.internal   Ready    <none>   2m52s   v1.22.15-eks-fb459a0   192.168.59.220   54.227.122.222   Amazon Linux 2   5.4.219-126.411.amzn2.x86_64   docker://20.10.17
ip-192-168-6-124.ec2.internal    Ready    <none>   2m51s   v1.22.15-eks-fb459a0   192.168.6.124    35.174.60.1      Amazon Linux 2   5.4.219-126.411.amzn2.x86_64   docker://20.10.1

Start by getting a copy of the Aerospike git repo for the Kubernetes Operator.

git clone https://github.com/aerospike/aerospike-kubernetes-operator.git
cp features.conf aerospike-kubernetes-operator/config/samples/secrets/.

Setup

Run the following commands in the order specified. Wait for the csv "Succeeded phase" to show up after running this line. Initially it might take between 30 seconds and a minute to show up.
kubectl get csv -n operators -w

cd aerospike-kubernetes-operator/
kubectl apply -f config/samples/storage/eks_ssd_storage_class.yaml
curl -sL https://github.com/operator-framework/operator-lifecycle-manager/releases/download/v0.22.0/install.sh | bash -s v0.22.0
kubectl create -f https://operatorhub.io/install/aerospike-kubernetes-operator.yaml
kubectl get csv -n operators -w
cd..
git clone https://github.com/nareshmaharaj-consultant/kubernetes-anything
cd kubernetes-anything
./createNamespace.sh aerospike
cd ../aerospike-kubernetes-operator/
kubectl -n aerospike create secret generic aerospike-secret --from-file=config/samples/secrets
kubectl -n aerospike create secret generic auth-secret --from-literal=password='admin123'

Destination Cluster

Use the following yaml configuration file for our destination cluster. Save the file and name it ssd1_xdr_dest_6.1_cluster_cr.yaml:

apiVersion: asdb.aerospike.com/v1beta1
kind: AerospikeCluster
metadata:
  name: aerocluster-dest-xdr
  namespace: aerospike

spec:
  size: 1
  image: aerospike/aerospike-server-enterprise:6.1.0.2

  storage:
    filesystemVolumePolicy:
      initMethod: deleteFiles
      cascadeDelete: true
    blockVolumePolicy:
      cascadeDelete: true
    volumes:
      - name: workdir
        aerospike:
          path: /opt/aerospike
        source:
          persistentVolume:
            storageClass: ssd
            volumeMode: Filesystem
            size: 1Gi
      - name: ns
        aerospike:
          path: /opt/aerospike/data/
        source:
          persistentVolume:
            storageClass: ssd
            volumeMode: Filesystem
            size: 1Gi
      - name: aerospike-config-secret
        source:
          secret:
            secretName: aerospike-secret
        aerospike:
          path: /etc/aerospike/secret

  podSpec:
    multiPodPerHost: true

  aerospikeAccessControl:
    roles:
      - name: writer
        privileges:
        - read-write
      - name: reader
        privileges:
        - read
    users:
      - name: admin
        secretName: auth-secret
        roles:
          - sys-admin
          - user-admin
          - read-write
      - name: xdr-writer
        secretName: xdr-user-auth-secret
        roles:
          - writer

  aerospikeConfig:
    service:
      feature-key-file: /etc/aerospike/secret/features.conf
    security: {}
    network:
      service:
        port: 3000
      fabric:
        port: 3001
      heartbeat:
        port: 3002
    namespaces:
      - name: test
        memory-size: 134217728
        replication-factor: 1
        storage-engine:
          type: device
          files:
            - /opt/aerospike/data/test.dat
          filesize: 1073741824
          data-in-memory: true

Create the following Kubernetes resources for our Aerospike destination cluster:

xdr destination database user login credentials, as a Kubernetes secret
destination database cluster using our yaml file named ssd1_xdr_dest_6.1_cluster_cr.yaml

export secret_auth_name=xdr-user-auth-secret
export password_secret=admin123
kubectl -n aerospike create secret generic $secret_auth_name --from-literal=password=$password_secret
kubectl create -f ssd1_xdr_dest_6.1_cluster_cr.yaml
kubectl -n aerospike get po -w

You should see the database pods running successfully.

kubectl get po -n aerospike -w
NAME                       READY   STATUS     RESTARTS   AGE
aerocluster-dest-xdr-0-0   0/1     Init:0/1   0          13s
aerocluster-dest-xdr-0-0   0/1     Init:0/1   0          18s
aerocluster-dest-xdr-0-0   0/1     PodInitializing   0          19s
aerocluster-dest-xdr-0-0   1/1     Running           0          24s

XDR-Proxy

Next we set up the xdr-proxy. If we look back at the main title image above, you will notice that we are working from the RIGHT hand side to the LEFT hand side in that order.

Configuration

Create the following xdr-proxy configuration file. Replace the seeds address with a FQN DNS name for the destination database pod(s) you created earlier. Multiple seed addresses may be added (recommended in production).

cd ..
mkdir -p xdr-cfg/etc/auth
cd xdr-cfg/etc/

cat <<EOF> aerospike-xdr-proxy.yml
# Change the configuration for your use case.
# Naresh Maharaj
# Refer to https://www.aerospike.com/docs/connectors/enterprise/xdr-proxy/configuration/index.html
# for details.

# The connector's listening ports, manage service, TLS and network interface.
service:
  port: 8901
  # Aerospike Enterprise Server >= 5.0
  manage:
    address: 0.0.0.0
    port: 8902

# The destination aerospike cluster.
aerospike:
  seeds:
    - aerocluster-dest-xdr-0-0.aerospike.svc.cluster.local:
        port: 3000
  credentials:
    username: xdr-writer
    password-file: /etc/aerospike-xdr-proxy/auth/password_DC1.txt
    auth-mode: internal

# The logging config
logging:
  enable-console-logging: true
  file: /var/log/aerospike-xdr-proxy/aerospike-xdr-proxy.log
  levels:
    root: debug
    record-parser: debug
    server: debug
    com.aerospike.connect: debug
  # Ticker log interval in seconds
  ticker-interval: 3600
EOF

sudo tee auth/password_DC1.txt <<EOF
admin123
EOF
cd ..

kubectl -n aerospike create configmap xdr-proxy-cfg --from-file=etc/
kubectl -n aerospike create secret generic xdr-proxy-auth-secret --from-file=etc/auth

Deployment

Now that we have our xdr-proxy config file created we can now produce the kubernetes deployment yaml file for the xdr-proxy itself. The following yaml file is used to deploy our xdr-proxy ideally in the same data centre or location where the destination databases will be hosted.

cat <<EOF> xdr-proxy-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: xdr-proxy
  namespace: aerospike
  labels:
    app: xdr-proxy
spec:
  replicas: 2
  selector:
    matchLabels:
      app: xdr-proxy
  template:
    metadata:
      labels:
        app: xdr-proxy
    spec:
      containers:
      - name: xdr-proxy
        image: aerospike/aerospike-xdr-proxy:2.1.0
        volumeMounts:
        - name: xdr-proxy-dir
          mountPath: "/etc/aerospike-xdr-proxy/"
          readOnly: true
        - name: xdr-auth-dir
          mountPath: "/etc/aerospike-xdr-proxy/auth"
          readOnly: true
        ports:
          - name: xdr-proxy-main
            containerPort: 8901
          - name: xdr-proxy-mng
            containerPort: 8902
      volumes:
      - name: xdr-proxy-dir
        configMap:
          name: xdr-proxy-cfg
          optional: false
      - name: xdr-auth-dir
        secret:
          secretName: xdr-proxy-auth-secret
          optional: false
---
apiVersion: v1
kind: Service
metadata:
  name: xdr-proxy
  namespace: aerospike
spec:
  selector:
    app: xdr-proxy
  ports:
  - name: main
    protocol: TCP
    port: 8901
    targetPort: xdr-proxy-main
  - name: manage
    protocol: TCP
    port: 8902
    targetPort: xdr-proxy-mng
EOF

kubectl create -f xdr-proxy-deployment.yaml
kubectl get po -n aerospike -w

The following shows the current scheduled pods. So far, so good.

kubectl get po -n aerospike -w
NAME                         READY   STATUS    RESTARTS   AGE
aerocluster-dest-xdr-0-0     1/1     Running   0          77m
xdr-proxy-7d9fccd6c8-g5mjt   1/1     Running   0          2m26s
xdr-proxy-7d9fccd6c8-mjxp4   1/1     Running   0          2m26s

Source Cluster

Create the source Aerospike cluster using the following configuration

cd ../aerospike-kubernetes-operator/

cat <<EOF> ssd1_xdr_src_6.1_cluster_cr.yaml
apiVersion: asdb.aerospike.com/v1beta1
kind: AerospikeCluster
metadata:
  name: aerocluster-source-xdr
  namespace: aerospike

spec:
  size: 1
  image: aerospike/aerospike-server-enterprise:6.1.0.2

  storage:
    filesystemVolumePolicy:
      initMethod: deleteFiles
      cascadeDelete: true
    blockVolumePolicy:
      cascadeDelete: true
    volumes:
      - name: workdir
        aerospike:
          path: /opt/aerospike
        source:
          persistentVolume:
            storageClass: ssd
            volumeMode: Filesystem
            size: 1Gi
      - name: ns
        aerospike:
          path: /opt/aerospike/data/
        source:
          persistentVolume:
            storageClass: ssd
            volumeMode: Filesystem
            size: 1Gi
      - name: aerospike-config-secret
        source:
          secret:
            secretName: aerospike-secret
        aerospike:
          path: /etc/aerospike/secret

  podSpec:
    multiPodPerHost: true

  aerospikeAccessControl:
    roles:
      - name: writer
        privileges:
        - read-write
      - name: reader
        privileges:
        - read
    users:
      - name: admin
        secretName: auth-secret
        roles:
          - sys-admin
          - user-admin
          - read-write

  aerospikeConfig:
    service:
      feature-key-file: /etc/aerospike/secret/features.conf
    security: {}
    network:
      service:
        port: 3000
      fabric:
        port: 3001
      heartbeat:
        port: 3002
    xdr:
      dcs:
        - name: DC2
          connector: true
          node-address-ports:
            - xdr-proxy.aerospike.svc.cluster.local 8901
          namespaces:
            - name: test
    namespaces:
      - name: test
        memory-size: 134217728
        replication-factor: 1
        storage-engine:
          type: device
          files:
            - /opt/aerospike/data/test.dat
          filesize: 1073741824
          data-in-memory: true
EOF

kubectl create -f ssd1_xdr_src_6.1_cluster_cr.yaml
kubectl get po -n aerospike -w

From the source cluster, confirm the xdr component has made a connection to the xdr-proxy by filtering the Kubernetes log file as shown in the following kubectl command.

kubectl -n aerospike logs aerocluster-source-xdr-0-0 -c aerospike-server | grep xdr | grep conn
Dec 08 2022 13:49:21 GMT: INFO (xdr): (dc.c:581) DC DC2 connectedOct 10 2022 13:57:53 GMT: INFO (xdr): (dc.c:581) DC DC2 connected

Simple Message Test

Add some data to the source database and confirm it is being received in the destination cluster. Start by getting the source database service address and connect using the Aerospike's command line tool called aql.

kubectl get svc -n aerospike
NAME                         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
aerocluster-dest-xdr         ClusterIP   None             <none>        3000/TCP            3h38m
aerocluster-dest-xdr-0-0     NodePort    10.100.226.179   <none>        3000:30168/TCP      3h38m
aerocluster-source-xdr       ClusterIP   None             <none>        3000/TCP            33m
aerocluster-source-xdr-0-0   NodePort    10.100.116.173   <none>        3000:31999/TCP      33m
xdr-proxy                    ClusterIP   10.100.44.96     <none>        8901/TCP,8902/TCP   41m

kubectl run -it --rm --restart=Never aerospike-tool -n aerospike --image=aerospike/aerospike-tools:latest -- aql -U admin -P admin123 -h aerocluster-source-xdr-0-0
Insert a source record using the following command in aql
insert into test.a1 (PK,a,b,c,d) values(1,"A","B","C","D")
OK, 1 record affected.

aql> select * from test
+----+-----+-----+-----+-----+
| PK | a   | b   | c   | d   |
+----+-----+-----+-----+-----+
| 1  | "A" | "B" | "C" | "D" |
+----+-----+-----+-----+-----+
1 row in set (0.023 secs)

Connect to the destination cluster in the same way and confirm data has successfully arrived.

kubectl run -it --rm --restart=Never aerospike-tool -n aerospike --image=aerospike/aerospike-tools:latest -- aql -U admin -P admin123 -h aerocluster-dest-xdr-0-0
Run the following select query in the destination cluster.
aql> select * from test
+----+-----+-----+-----+-----+
| PK | a   | b   | c   | d   |
+----+-----+-----+-----+-----+
| 1  | "A" | "B" | "C" | "D" |
+----+-----+-----+-----+-----+
1 row in set (0.031 secs)

OK

Interim summary

So far, at this point it's confirmed that xdr-proxy is doing exactly what we need it to do.

If you now review the logs for the initial 2 xdr-proxies that were scheduled you should indeed see userKey=1.

kubectl logs xdr-proxy-7d9fccd6c8-5q7gn -n aerospike | grep record-parser
2022-12-08 14:53:50.607 GMT DEBUG record-parser - parsed message fields: key=Key(namespace='test', set=a1, digest=[120, 48, -23, -90, 110, 126, 84, -1, 114, -116, -9, -21, 28, 75, 126, -68, -51, 83, 31, -117], userKey=1, lastUpdateTimeMs=1670511230565, userKeyString=1, digestString='eDDppm5+VP9yjPfrHEt+vM1TH4s=')

kubectl logs xdr-proxy-7d9fccd6c8-f5zkt -n aerospike | grep record-parser
(none)

Scaling the XDR-Proxies

Go ahead and scale up the xdr-proxy to 6 pods by editing the file xdr-proxy-deployment.yaml and then applying the changes.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: xdr-proxy
  namespace: aerospike
  labels:
    app: xdr-proxy
spec:
  replicas: 6
  selector:
    matchLabels:
      app: xdr-proxy
...
kubectl apply -f xdr-proxy-deployment.yaml

You can also achieve the same by running the following command

kubectl scale deploy xdr-proxy -n aerospike --replicas=6

You should now have 6 instances of the xdr-proxies running.

kubectl get po -n aerospike -w
NAME                         READY   STATUS    RESTARTS   AGE
aerocluster-dest-xdr-0-0     1/1     Running   0          3h50m
aerocluster-source-xdr-0-0   1/1     Running   0          75m
xdr-proxy-7d9fccd6c8-49ttl   1/1     Running   0          7s
xdr-proxy-7d9fccd6c8-5q7gn   1/1     Running   0          83m
xdr-proxy-7d9fccd6c8-c4j7k   1/1     Running   0          7s
xdr-proxy-7d9fccd6c8-f5zkt   1/1     Running   0          83m
xdr-proxy-7d9fccd6c8-lscbg   1/1     Running   0          7s
xdr-proxy-7d9fccd6c8-r56vs   1/1     Running   0          7s

Lets add some sample messages from our source xdr cluster with primary keys 5,6,7

aql> insert into test (PK,a,b) values (5,"A","B")
OK, 1 record affected.
aql> insert into test (PK,a,b) values (6,"A","B")
OK, 1 record affected.
aql> insert into test (PK,a,b) values (7,"A","B")
OK, 1 record affected.

Notice how we have userKey=5, userKey=6 and userKey=7 across the newly scaled xdr-proxies.

kubectl logs xdr-proxy-7d9fccd6c8-5q7gn -n aerospike | grep record-parser
2022-12-08 14:53:50.607 GMT DEBUG record-parser - parsed message fields: key=Key(namespace='test', set=a1, digest=[120, 48, -23, -90, 110, 126, 84, -1, 114, -116, -9, -21, 28, 75, 126, -68, -51, 83, 31, -117], userKey=1, lastUpdateTimeMs=1670511230565, userKeyString=1, digestString='eDDppm5+VP9yjPfrHEt+vM1TH4s=')

kubectl logs xdr-proxy-7d9fccd6c8-f5zkt -n aerospike | grep record-parser
(none)

kubectl logs xdr-proxy-7d9fccd6c8-49ttl -n aerospike | grep record-parser
(none)

kubectl logs xdr-proxy-7d9fccd6c8-c4j7k -n aerospike | grep record-parser
2022-12-08 15:05:37.511 GMT DEBUG record-parser - parsed message fields: key=Key(namespace='test', set=null, digest=[-88, 104, 104, -114, 19, -44, -19, 29, -15, 18, 118, 72, -117, -106, -28, 21, -48, 50, 26, 113], userKey=7, lastUpdateTimeMs=1670511937250, userKeyString=7, digestString='qGhojhPU7R3xEnZIi5bkFdAyGnE=')

kubectl logs xdr-proxy-7d9fccd6c8-lscbg -n aerospike | grep record-parser
2022-12-08 15:05:27.548 GMT DEBUG record-parser - parsed message fields: key=Key(namespace='test', set=null, digest=[68, 4, -94, -44, -75, 112, -102, 73, -120, 41, -101, -120, 33, -111, 15, -114, -85, 46, -2, 80], userKey=5, lastUpdateTimeMs=1670511927465, userKeyString=5, digestString='RASi1LVwmkmIKZuIIZEPjqsu/lA=')
2022-12-08 15:05:32.300 GMT DEBUG record-parser - parsed message fields: key=Key(namespace='test', set=null, digest=[33, -100, 127, 120, 17, 45, -79, 115, -40, 53, -70, -57, 120, 73, 20, -50, -99, -98, -104, 85], userKey=6, lastUpdateTimeMs=1670511932288, userKeyString=6, digestString='IZx/eBEtsXPYNbrHeEkUzp2emFU=')

kubectl logs xdr-proxy-7d9fccd6c8-r56vs -n aerospike | grep record-parser
(none)

When data is actively flowing between source and destination clusters, the existing cached list of xdr-proxy connections from the source cluster is not refreshed, so the newly scaled xdr-proxies we have just scheduled will not be utilized immediately.

To demonstrate, let's add data to the source cluster using Aerospike's benchmark tool. At the same time, we will scale the xdr-proxies on the destination side and observe the results. Before we start, we reduce the xdr-proxy server count to 1 to make the observations clear.

I use my own local machine which has the benchmark tool already installed to send data to the source EC2 instances. In order to do this, we need to get the external IP address of the service.

kubectl get AerospikeCluster aerocluster-source-xdr -n aerospike  -o yaml

...
pods:
    aerocluster-source-xdr-0-0:
      aerospike:
        accessEndpoints:
        - 192.168.41.63:31999
        alternateAccessEndpoints:
        - 54.173.138.131:31999
        clusterName: aerocluster-source-xdr
        nodeID: 0a0
        tlsAccessEndpoints: []
        tlsAlternateAccessEndpoints: []
        tlsName: ""
      aerospikeConfigHash: 4aacb9809beaa01d99a9f00293c9f7dc141845f8
      hostExternalIP: 54.173.138.131
      hostInternalIP: 192.168.41.63
      image: aerospike/aerospike-server-enterprise:6.1.0.2
      initializedVolumes:
      - workdir
      - ns
      networkPolicyHash: acbbfab3668e1fceeed201139d1173f00095667e
      podIP: 192.168.50.203
      podPort: 3000
      podSpecHash: 972dc2a779fe9ab407212b547d54d3a72ecef259
      servicePort: 31999
...

Add the AWS firewall rule to allow traffic into the Kubernetes service. Connect the asbenchmark tool to start writing traffic using the public IP address for the NodePort Service.

asbenchmark -h 54.173.138.131:31999 -Uadmin -Padmin123 -z 10 -servicesAlternate -w RU,0 -o B256

Scale up the xdr-proxy servers from 1 to 2, and check the logs of both proxies to see what messages they received. In a production environment, you should disable the logging.

Notice that no data has passed through the new xdr-proxy instance xdr-proxy-7d9fccd6c8-s2tzt.

kubectl -n aerospike logs xdr-proxy-7d9fccd6c8-s2tzt | grep record-parser
(none)

kubectl -n aerospike logs xdr-proxy-7d9fccd6c8-5q7gn | grep record-parser
...
2022-12-08 18:21:36.158 GMT DEBUG record-parser - parsed message fields: key=Key(namespace='test', set=testset, digest=[51, -120, 114, -17, -44, 72, 123, 125, 50, 92, 3, 110, -21, -38, 74, 25, 42, 35, 117, 72], userKey=null, lastUpdateTimeMs=1670523696059, userKeyString=null, digestString='M4hy79RIe30yXANu69pKGSojdUg=')
2022-12-08 18:21:36.158 GMT DEBUG record-parser - parsed message fields: key=Key(namespace='test', set=testset, digest=[124, -6, -70, -23, 44, 41, -19, 40, -11, -16, 126, 120, 81, -113, -112, -79, 66, 77, -99, -6], userKey=null, lastUpdateTimeMs=1670523696059, userKeyString=null, digestString='fPq66Swp7Sj18H54UY+QsUJNnfo=')
2022-12-08 18:21:36.158 GMT DEBUG record-parser - parsed message fields: key=Key(namespace='test', set=testset, digest=[-127, -118, 63, -32, -74, 60, -60, 86, 31, -119, -1, -105, -108, -59, 111, 48, -34, -61, -108, -5], userKey=null, lastUpdateTimeMs=1670523696105, userKeyString=null, digestString='gYo/4LY8xFYfif+XlMVvMN7DlPs=')
2022-12-08 18:21:36.256 GMT DEBUG record-parser - parsed message fields: key=Key(namespace='test', set=testset, digest=[99, 1, 80, 2, 76, -43, 125, 77, 47, 8, 6, 35, 49, 117, -35, 54, 120, -29, 118, -72], userKey=null, lastUpdateTimeMs=1670523696178, userKeyString=null, digestString='YwFQAkzVfU0vCAYjMXXdNnjjdrg=')
2022-12-08 18:21:36.257 GMT DEBUG record-parser - parsed message fields: key=Key(namespace='test', set=testset, digest=[-88, -46, -48, -33, 77, -120, 123, -101, -70, -20, -96, -104, -51, -90, 28, -15, 70, 11, 118, 83], userKey=null, lastUpdateTimeMs=1670523696202, userKeyString=null, digestString='qNLQ302Ie5u67KCYzaYc8UYLdlM=')
2022-12-08 18:21:36.257 GMT DEBUG record-parser - parsed message fields: key=Key(namespace='test', set=testset, digest=[-97, 18, 28, -43, 75, 42, -22, -126, -61, -108, -36, 118, -86, -105, -52, 119, -39, -33, -127, -76], userKey=null, lastUpdateTimeMs=1670523696175, userKeyString=null, digestString='nxIc1Usq6oLDlNx2qpfMd9nfgbQ=')
2022-12-08 18:21:36.257 GMT DEBUG record-parser - parsed message fields: key=Key(namespace='test', set=testset, digest=[-120, 103, -51, 57, -71, -106, 13, -48, 100, 28, 59, -3, -39, -56, -67, -103, 29, 36, 75, 119], userKey=null, lastUpdateTimeMs=1670523696191, userKeyString=null, digestString='iGfNObmWDdBkHDv92ci9mR0kS3c=')
2022-12-08 18:21:36.257 GMT DEBUG record-parser - parsed message fields: key=Key(namespace='test', set=testset, digest=[-47, 88, -13, 13, -35, 77, 24, 22, -40, -61, -118, -115, 82, 13, 127, -125, 53, 66, -22, -8], userKey=null, lastUpdateTimeMs=1670523696233, userKeyString=null, digestString='0VjzDd1NGBbYw4qNUg1/gzVC6vg=')
2022-12-08 18:21:36.258 GMT DEBUG record-parser - parsed message fields: key=Key(namespace='test', set=testset, digest=[-63, -11, 93, -90, 47, 29, -63, 36, 12, 53, -86, 84, 57, -125, 16, 43, -18, 93, 56, 9], userKey=null, lastUpdateTimeMs=1670523696186, userKeyString=null, digestString='wfVdpi8dwSQMNapUOYMQK+5dOAk=')
2022-12-08 18:21:36.258 GMT DEBUG record-parser - parsed message fields: key=Key(namespace='test', set=testset, digest=[-23, 101, -114, -87, -52, 107, 36, 113, 101, 33, -16, 82, -95, 97, 34, -121, 82, -97, 40, 59], userKey=null, lastUpdateTimeMs=1670523696145, userKeyString=null, digestString='6WWOqcxrJHFlIfBSoWEih1KfKDs=')
2022-12-08 18:21:36.258 GMT DEBUG record-parser - parsed message fields: key=Key(namespace='test', set=testset, digest=[-77, -118, 49, 4, -75, 123, 81, 2, -103, -73, 42, -70, -54, 95, 98, 23, 73, 66, -86, 7], userKey=null, lastUpdateTimeMs=1670523696230, userKeyString=null, digestString='s4oxBLV7UQKZtyq6yl9iF0lCqgc=')
2022-12-08 18:21:36.258 GMT DEBUG record-parser - parsed message fields: key=Key(namespace='test', set=testset, digest=[50, -38, -31, -54, -122, -113, -38, 88, 15, 7, 96, 51, -92, -25, 60, -104, 26, 113, -117, -82], userKey=null, lastUpdateTimeMs=1670523696157, userKeyString=null, digestString='MtrhyoaP2lgPB2AzpOc8mBpxi64=')
2022-12-08 18:21:36.258 GMT DEBUG record-parser - parsed message fields: key=Key(namespace='test', set=testset, digest=[-77, 31, 67, -18, -52, -114, 42, -18, 36, -111, 89, 62, 109, 114, -54, 54, -121, -110, -88, -108], userKey=null, lastUpdateTimeMs=1670523696206, userKeyString=null, digestString='sx9D7syOKu4kkVk+bXLKNoeSqJQ=')
2022-12-08 18:21:36.258 GMT DEBUG record-parser - parsed message fields: key=Key(namespace='test', set=testset, digest=[73, 11, 98, -50, 32, 12, 0, -50, 22, -101, -108, 18, 38, 7, -65, 6, -58, 60, -6, -33], userKey=null, lastUpdateTimeMs=1670523696171, userKeyString=null, digestString='SQtiziAMAM4Wm5QSJge/BsY8+t8=')
2022-12-08 18:21:36.258 GMT DEBUG record-parser - parsed message fields: key=Key(namespace='test', set=testset, digest=[109, -82, 24, 53, 35, 89, -72, -117, -22, 79, 119, -89, 56, -5, 0, -103, -54, 51, 25, 126], userKey=null, lastUpdateTimeMs=1670523696146, userKeyString=null, digestString='ba4YNSNZuIvqT3enOPsAmcozGX4=')
...

Conclusion

Dependable resiliency is a core part of the Aerospike value proposition. Aerospike's XDR feature ensures that users can mitigate the risk of a cluster becoming unavailable by replicating data asynchronously from one data center to another. We show here how you can achieve this in a Kubernetes environment by using the xdr-proxy, allowing you to avoid network complications. All this has been achieved with minimum effort thanks to the Aerospike Kubernetes Operator.

Aerospike Kubernetes Talking Cluster to Cluster

Naresh Maharaj — Mon, 16 Jan 2023 20:03:05 +0000

In this article we focus on establishing 2 separate Kubernetes clusters both each hosting the Aerospike low latency database. The goal is to be able to use Aerospike's Cross Data centre Replication tool ( XDR ) to send data from a source cluster to the destination cluster seamlessly. The problem sometimes encountered in exposing a cluster to the outside world is the organisation's security restriction policies. By placing a proxy server in front of the private Kubernetes destination cluster can overcome this problem an achieve the desired goal. To demonstrate the solution we will first start by installing the Kubernetes operator that will schedule our said source and destination databases.

Aerospike Kubernetes Operator

The following Kubernetes nodes have been created using EKS. You can see the following private and public IP addresses from listing the nodes with the kubectl command.

kubectl get nodes -o wide
NAME                             STATUS   ROLES    AGE     VERSION                INTERNAL-IP      EXTERNAL-IP      OS-IMAGE         KERNEL-VERSION                 CONTAINER-RUNTIME
ip-192-168-11-132.ec2.internal   Ready    <none>   2m53s   v1.22.15-eks-fb459a0   192.168.11.132   44.201.67.177    Amazon Linux 2   5.4.219-126.411.amzn2.x86_64   docker://20.10.17
ip-192-168-31-131.ec2.internal   Ready    <none>   2m52s   v1.22.15-eks-fb459a0   192.168.31.131   44.192.83.79     Amazon Linux 2   5.4.219-126.411.amzn2.x86_64   docker://20.10.17
ip-192-168-41-140.ec2.internal   Ready    <none>   2m51s   v1.22.15-eks-fb459a0   192.168.41.140   18.208.222.35    Amazon Linux 2   5.4.219-126.411.amzn2.x86_64   docker://20.10.17
ip-192-168-41-63.ec2.internal    Ready    <none>   2m51s   v1.22.15-eks-fb459a0   192.168.41.63    54.173.138.131   Amazon Linux 2   5.4.219-126.411.amzn2.x86_64   docker://20.10.17
ip-192-168-59-220.ec2.internal   Ready    <none>   2m52s   v1.22.15-eks-fb459a0   192.168.59.220   54.227.122.222   Amazon Linux 2   5.4.219-126.411.amzn2.x86_64   docker://20.10.17
ip-192-168-6-124.ec2.internal    Ready    <none>   2m51s   v1.22.15-eks-fb459a0   192.168.6.124    35.174.60.1      Amazon Linux 2   5.4.219-126.411.amzn2.x86_64   docker://20.10.1

Start by getting a copy of the Aerospike git repo for the k8s operator.

git clone https://github.com/aerospike/aerospike-kubernetes-operator.git
cp features.conf aerospike-kubernetes-operator/config/samples/secrets/.

Install the Aerospike Kubernetes Operator.

cd aerospike-kubernetes-operator/
kubectl apply -f config/samples/storage/eks_ssd_storage_class.yaml
kubectl apply -f config/samples/storage/local_storage_class.yaml
curl -sL https://github.com/operator-framework/operator-lifecycle-manager/releases/download/v0.22.0/install.sh | bash -s v0.22.0
kubectl create -f https://operatorhub.io/install/aerospike-kubernetes-operator.yaml
kubectl get csv -n operators -w
cd..
git clone https://github.com/nareshmaharaj-consultant/kubernetes-anything
cd kubernetes-anything
./createNamespace.sh aerospike
cd ../aerospike-kubernetes-operator/
kubectl -n aerospike create secret generic aerospike-secret --from-file=config/samples/secrets
kubectl -n aerospike create secret generic auth-secret --from-literal=password='admin123'

Install the destination cluster first by using the following configuration

apiVersion: asdb.aerospike.com/v1beta1
kind: AerospikeCluster
metadata:
  name: aerocluster-dest-xdr
  namespace: aerospike

spec:
  size: 1
  image: aerospike/aerospike-server-enterprise:6.1.0.2

  storage:
    filesystemVolumePolicy:
      initMethod: deleteFiles
      cascadeDelete: true
    blockVolumePolicy:
      cascadeDelete: true
    volumes:
      - name: workdir
        aerospike:
          path: /opt/aerospike
        source:
          persistentVolume:
            storageClass: ssd
            volumeMode: Filesystem
            size: 1Gi
      - name: ns
        aerospike:
          path: /opt/aerospike/data/
        source:
          persistentVolume:
            storageClass: ssd
            volumeMode: Filesystem
            size: 1Gi
      - name: aerospike-config-secret
        source:
          secret:
            secretName: aerospike-secret
        aerospike:
          path: /etc/aerospike/secret

  podSpec:
    multiPodPerHost: true

  aerospikeAccessControl:
    roles:
      - name: writer
        privileges:
        - read-write
      - name: reader
        privileges:
        - read
    users:
      - name: admin
        secretName: auth-secret
        roles:
          - sys-admin
          - user-admin
          - read-write
      - name: xdr-writer
        secretName: xdr-user-auth-secret
        roles:
          - writer

  aerospikeConfig:
    service:
      feature-key-file: /etc/aerospike/secret/features.conf
    security: {}
    network:
      service:
        port: 3000
      fabric:
        port: 3001
      heartbeat:
        port: 3002
    namespaces:
      - name: test
        memory-size: 134217728
        replication-factor: 1
        storage-engine:
          type: device
          files:
            - /opt/aerospike/data/test.dat
          filesize: 1073741824
          data-in-memory: true

Create the Aerospike destination

database login credentials using k8s secrets
schedule the destination single node database cluster.

export secret_auth_name=xdr-user-auth-secret
export password_secret=admin123
kubectl -n aerospike create secret generic $secret_auth_name --from-literal=password=$password_secret
kubectl create -f ssd1_xdr_dest_6.1_cluster_cr.yaml
kubectl -n aerospike get po -w

You should see the database pods successfully running.

kubectl get po -n aerospike -w
NAME                       READY   STATUS     RESTARTS   AGE
aerocluster-dest-xdr-0-0   0/1     Init:0/1   0          13s
aerocluster-dest-xdr-0-0   0/1     Init:0/1   0          18s
aerocluster-dest-xdr-0-0   0/1     PodInitializing   0          19s
aerocluster-dest-xdr-0-0   1/1     Running           0          24s

Next we set up the xdr-proxy. If we look back at the main title image above, you will notice that we are working from the RIGHT hand side to the LEFT hand side in that order.

Go ahead and create the following xdr-proxy configuration file. Replace the seed address with a FQN DNS name for the destination database pod(s) you created earlier. If you have more than one seed address now is the time to add them to the yaml list.

cd ..
mkdir -p xdr-cfg/etc/auth
cd xdr-cfg/etc/

cat <<EOF> aerospike-xdr-proxy.yml
# Change the configuration for your use case.
# Naresh Maharaj
# Refer to https://www.aerospike.com/docs/connectors/enterprise/xdr-proxy/configuration/index.html
# for details.

# The connector's listening ports, manage service, TLS and network interface.
service:
  port: 8901
  # Aerospike Enterprise Server >= 5.0
  manage:
    address: 0.0.0.0
    port: 8902

# The destination aerospike cluster.
aerospike:
  seeds:
    - aerocluster-dest-xdr-0-0.aerospike.svc.cluster.local:
        port: 3000
  credentials:
    username: admin
    password-file: /etc/aerospike-xdr-proxy/auth/password_DC1.txt
    auth-mode: internal

# The logging config
logging:
  enable-console-logging: true
  file: /var/log/aerospike-xdr-proxy/aerospike-xdr-proxy.log
  levels:
    root: debug
    record-parser: debug
    server: debug
    com.aerospike.connect: debug
  # Ticker log interval in seconds
  ticker-interval: 3600
EOF

sudo tee auth/password_DC1.txt <<EOF
admin123
EOF

kubectl -n aerospike create configmap xdr-proxy-cfg --from-file=etc/
kubectl -n aerospike create secret generic xdr-proxy-auth-secret --from-file=etc/auth

cat <<EOF> xdr-proxy-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: xdr-proxy
  namespace: aerospike
  labels:
    app: xdr-proxy
spec:
  replicas: 2
  selector:
    matchLabels:
      app: xdr-proxy
  template:
    metadata:
      labels:
        app: xdr-proxy
    spec:
      containers:
      - name: xdr-proxy
        image: aerospike/aerospike-xdr-proxy:2.1.0
        volumeMounts:
        - name: xdr-proxy-dir
          mountPath: "/etc/aerospike-xdr-proxy/"
          readOnly: true
        - name: xdr-auth-dir
          mountPath: "/etc/aerospike-xdr-proxy/auth"
          readOnly: true
        ports:
          - name: xdr-proxy-main
            containerPort: 8901
          - name: xdr-proxy-mng
            containerPort: 8902
      volumes:
      - name: xdr-proxy-dir
        configMap:
          name: xdr-proxy-cfg
          optional: false
      - name: xdr-auth-dir
        secret:
          secretName: xdr-proxy-auth-secret
          optional: false
---
apiVersion: v1
kind: Service
metadata:
  name: xdr-proxy
  namespace: aerospike
spec:
  selector:
    app: xdr-proxy
  ports:
  - name: main
    protocol: TCP
    port: 8901
    targetPort: xdr-proxy-main
  - name: manage
    protocol: TCP
    port: 8902
    targetPort: xdr-proxy-mng
EOF

kubectl create -f xdr-proxy-deployment.yaml
kubectl get po -n aerospike -w

The following shows the current scheduled pods. So far, so good.

kubectl get po -n aerospike -w
NAME                         READY   STATUS    RESTARTS   AGE
aerocluster-dest-xdr-0-0     1/1     Running   0          77m
xdr-proxy-7d9fccd6c8-g5mjt   1/1     Running   0          2m26s
xdr-proxy-7d9fccd6c8-mjxp4   1/1     Running   0          2m26s

Create the source cluster using the following configuration

cd ../aerospike-kubernetes-operator/

cat <<EOF> ssd1_xdr_src_6.1_cluster_cr.yaml
apiVersion: asdb.aerospike.com/v1beta1
kind: AerospikeCluster
metadata:
  name: aerocluster-source-xdr
  namespace: aerospike

spec:
  size: 1
  image: aerospike/aerospike-server-enterprise:6.1.0.2

  storage:
    filesystemVolumePolicy:
      initMethod: deleteFiles
      cascadeDelete: true
    blockVolumePolicy:
      cascadeDelete: true
    volumes:
      - name: workdir
        aerospike:
          path: /opt/aerospike
        source:
          persistentVolume:
            storageClass: ssd
            volumeMode: Filesystem
            size: 1Gi
      - name: ns
        aerospike:
          path: /opt/aerospike/data/
        source:
          persistentVolume:
            storageClass: ssd
            volumeMode: Filesystem
            size: 1Gi
      - name: aerospike-config-secret
        source:
          secret:
            secretName: aerospike-secret
        aerospike:
          path: /etc/aerospike/secret

  podSpec:
    multiPodPerHost: true

  aerospikeAccessControl:
    roles:
      - name: writer
        privileges:
        - read-write
      - name: reader
        privileges:
        - read
    users:
      - name: admin
        secretName: auth-secret
        roles:
          - sys-admin
          - user-admin
          - read-write

  aerospikeConfig:
    service:
      feature-key-file: /etc/aerospike/secret/features.conf
    security: {}
    network:
      service:
        port: 3000
      fabric:
        port: 3001
      heartbeat:
        port: 3002
    xdr:
      dcs:
        - name: DC2
          connector: true
          node-address-ports:
            - xdr-proxy.aerospike.svc.cluster.local 8901
          namespaces:
            - name: test
    namespaces:
      - name: test
        memory-size: 134217728
        replication-factor: 1
        storage-engine:
          type: device
          files:
            - /opt/aerospike/data/test.dat
          filesize: 1073741824
          data-in-memory: true
EOF

kubectl create -f ssd1_xdr_src_6.1_cluster_cr.yaml
kubectl get po -n aerospike -w

From the source cluster, confirm the xdr component has made a connection to the xdr-proxy by filtering the k8s log file as shown in the following kubectl command.

kubectl -n aerospike logs aerocluster-source-xdr-0-0 -c aerospike-server | grep xdr | grep conn
Dec 08 2022 13:49:21 GMT: INFO (xdr): (dc.c:581) DC DC2 connectedOct 10 2022 13:57:53 GMT: INFO (xdr): (dc.c:581) DC DC2 connected

kubectl get svc -n aerospike
NAME                         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
aerocluster-dest-xdr         ClusterIP   None             <none>        3000/TCP            3h38m
aerocluster-dest-xdr-0-0     NodePort    10.100.226.179   <none>        3000:30168/TCP      3h38m
aerocluster-source-xdr       ClusterIP   None             <none>        3000/TCP            33m
aerocluster-source-xdr-0-0   NodePort    10.100.116.173   <none>        3000:31999/TCP      33m
xdr-proxy                    ClusterIP   10.100.44.96     <none>        8901/TCP,8902/TCP   41m

kubectl run -it --rm --restart=Never aerospike-tool -n aerospike --image=aerospike/aerospike-tools:latest -- aql -U admin -P admin123 -h aerocluster-source-xdr-0-0
Insert a source record using the following command in aql
insert into test.a1 (PK,a,b,c,d) values(1,"A","B","C","D")
OK, 1 record affected.

aql> select * from test
+----+-----+-----+-----+-----+
| PK | a   | b   | c   | d   |
+----+-----+-----+-----+-----+
| 1  | "A" | "B" | "C" | "D" |
+----+-----+-----+-----+-----+
1 row in set (0.023 secs)

Connect to the destination cluster in the same way and confirm data has successfully arrived.

kubectl get svc -n aerospike
NAME                         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
aerocluster-dest-xdr         ClusterIP   None             <none>        3000/TCP            3h38m
aerocluster-dest-xdr-0-0     NodePort    10.100.226.179   <none>        3000:30168/TCP      3h38m
aerocluster-source-xdr       ClusterIP   None             <none>        3000/TCP            33m
aerocluster-source-xdr-0-0   NodePort    10.100.116.173   <none>        3000:31999/TCP      33m
xdr-proxy                    ClusterIP   10.100.44.96     <none>        8901/TCP,8902/TCP   41m

kubectl run -it --rm --restart=Never aerospike-tool -n aerospike --image=aerospike/aerospike-tools:latest -- aql -U admin -P admin123 -h aerocluster-dest-xdr-0-0
Run the following select query in the destination cluster.
aql> select * from test
+----+-----+-----+-----+-----+
| PK | a   | b   | c   | d   |
+----+-----+-----+-----+-----+
| 1  | "A" | "B" | "C" | "D" |
+----+-----+-----+-----+-----+
1 row in set (0.031 secs)

OK

Interim summary: So far, at this point it's confirmed that xdr-proxy is doing exactly what we need it to do.

If you now review the logs for the initial 2 xdr-proxies that were scheduled you should indeed see userKey=1.

kubectl logs xdr-proxy-7d9fccd6c8-5q7gn -n aerospike | grep record-parser
2022-12-08 14:53:50.607 GMT DEBUG record-parser - parsed message fields: key=Key(namespace='test', set=a1, digest=[120, 48, -23, -90, 110, 126, 84, -1, 114, -116, -9, -21, 28, 75, 126, -68, -51, 83, 31, -117], userKey=1, lastUpdateTimeMs=1670511230565, userKeyString=1, digestString='eDDppm5+VP9yjPfrHEt+vM1TH4s=')

kubectl logs xdr-proxy-7d9fccd6c8-f5zkt -n aerospike | grep record-parser
(none)

Scale up the xdr-proxy to 6 pods.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: xdr-proxy
  namespace: aerospike
  labels:
    app: xdr-proxy
spec:
  replicas: 6
  selector:
    matchLabels:
      app: xdr-proxy
...
kubectl apply -f xdr-proxy-deployment.yaml

We should now have 6 instances of the xdr-proxies running.

kubectl get po -n aerospike -w
NAME                         READY   STATUS    RESTARTS   AGE
aerocluster-dest-xdr-0-0     1/1     Running   0          3h50m
aerocluster-source-xdr-0-0   1/1     Running   0          75m
xdr-proxy-7d9fccd6c8-49ttl   1/1     Running   0          7s
xdr-proxy-7d9fccd6c8-5q7gn   1/1     Running   0          83m
xdr-proxy-7d9fccd6c8-c4j7k   1/1     Running   0          7s
xdr-proxy-7d9fccd6c8-f5zkt   1/1     Running   0          83m
xdr-proxy-7d9fccd6c8-lscbg   1/1     Running   0          7s
xdr-proxy-7d9fccd6c8-r56vs   1/1     Running   0          7s

Lets add some sample messages from our source xdr cluster with primary keys 5,6,7

aql> insert into test (PK,a,b) values (5,"A","B")
OK, 1 record affected.
aql> insert into test (PK,a,b) values (6,"A","B")
OK, 1 record affected.
aql> insert into test (PK,a,b) values (7,"A","B")
OK, 1 record affected.

Notice how we have userKey=5, userKey=6 and userKey=7 across the new scaled xdr-proxies.

kubectl logs xdr-proxy-7d9fccd6c8-5q7gn -n aerospike | grep record-parser
2022-12-08 14:53:50.607 GMT DEBUG record-parser - parsed message fields: key=Key(namespace='test', set=a1, digest=[120, 48, -23, -90, 110, 126, 84, -1, 114, -116, -9, -21, 28, 75, 126, -68, -51, 83, 31, -117], userKey=1, lastUpdateTimeMs=1670511230565, userKeyString=1, digestString='eDDppm5+VP9yjPfrHEt+vM1TH4s=')

kubectl logs xdr-proxy-7d9fccd6c8-f5zkt -n aerospike | grep record-parser
(none)

kubectl logs xdr-proxy-7d9fccd6c8-49ttl -n aerospike | grep record-parser
(none)

kubectl logs xdr-proxy-7d9fccd6c8-c4j7k -n aerospike | grep record-parser
2022-12-08 15:05:37.511 GMT DEBUG record-parser - parsed message fields: key=Key(namespace='test', set=null, digest=[-88, 104, 104, -114, 19, -44, -19, 29, -15, 18, 118, 72, -117, -106, -28, 21, -48, 50, 26, 113], userKey=7, lastUpdateTimeMs=1670511937250, userKeyString=7, digestString='qGhojhPU7R3xEnZIi5bkFdAyGnE=')

kubectl logs xdr-proxy-7d9fccd6c8-lscbg -n aerospike | grep record-parser
2022-12-08 15:05:27.548 GMT DEBUG record-parser - parsed message fields: key=Key(namespace='test', set=null, digest=[68, 4, -94, -44, -75, 112, -102, 73, -120, 41, -101, -120, 33, -111, 15, -114, -85, 46, -2, 80], userKey=5, lastUpdateTimeMs=1670511927465, userKeyString=5, digestString='RASi1LVwmkmIKZuIIZEPjqsu/lA=')
2022-12-08 15:05:32.300 GMT DEBUG record-parser - parsed message fields: key=Key(namespace='test', set=null, digest=[33, -100, 127, 120, 17, 45, -79, 115, -40, 53, -70, -57, 120, 73, 20, -50, -99, -98, -104, 85], userKey=6, lastUpdateTimeMs=1670511932288, userKeyString=6, digestString='IZx/eBEtsXPYNbrHeEkUzp2emFU=')

kubectl logs xdr-proxy-7d9fccd6c8-r56vs -n aerospike | grep record-parser
(none)

However, one should be aware that when data is actively flowing between source and destination clusters, there is no reset on the source connections, so the newly scaled xdr-proxies will not be utilised.

We can go ahead and prove this out. Lets try adding some data using the Aerospike's benchmark tool which will ingest data into the source cluster. At the same time we will scale the xdr-proxies on the destination side and see the result. Before we start however, it might be prudent to first scale down the xdr-proxy servers to 1 so there is no possibility of misconstrued results.

I am going to use my own local machine which has the benchmark tool already installed to send data to the source ec2 instances. In order to do this we need to get the external IP address of the service.

kubectl get AerospikeCluster aerocluster-source-xdr -n aerospike  -o yaml

...
pods:
    aerocluster-source-xdr-0-0:
      aerospike:
        accessEndpoints:
        - 192.168.41.63:31999
        alternateAccessEndpoints:
        - 54.173.138.131:31999
        clusterName: aerocluster-source-xdr
        nodeID: 0a0
        tlsAccessEndpoints: []
        tlsAlternateAccessEndpoints: []
        tlsName: ""
      aerospikeConfigHash: 4aacb9809beaa01d99a9f00293c9f7dc141845f8
      hostExternalIP: 54.173.138.131
      hostInternalIP: 192.168.41.63
      image: aerospike/aerospike-server-enterprise:6.1.0.2
      initializedVolumes:
      - workdir
      - ns
      networkPolicyHash: acbbfab3668e1fceeed201139d1173f00095667e
      podIP: 192.168.50.203
      podPort: 3000
      podSpecHash: 972dc2a779fe9ab407212b547d54d3a72ecef259
      servicePort: 31999
...

Add the AWS firewall rule to allow traffic into the k8s service. Connect the asbenchmark tool to start writing traffic using the public IP address for the NodePort Service.

asbenchmark -h 54.173.138.131:31999 -Uadmin -Padmin123 -z 10 -servicesAlternate -w RU,0 -o B256

Scale up the xdr-proxy servers from 1 to 2 and check the logs of both proxies to see what messages are being received. In a production environment you should disable the logging.

Notice how the new xdr-proxy instance xdr-proxy-7d9fccd6c8-s2tzt has no data passed through it.

kubectl -n aerospike logs xdr-proxy-7d9fccd6c8-s2tzt | grep record-parser
(none)

kubectl -n aerospike logs xdr-proxy-7d9fccd6c8-5q7gn | grep record-parser
...
2022-12-08 18:21:36.158 GMT DEBUG record-parser - parsed message fields: key=Key(namespace='test', set=testset, digest=[51, -120, 114, -17, -44, 72, 123, 125, 50, 92, 3, 110, -21, -38, 74, 25, 42, 35, 117, 72], userKey=null, lastUpdateTimeMs=1670523696059, userKeyString=null, digestString='M4hy79RIe30yXANu69pKGSojdUg=')
2022-12-08 18:21:36.158 GMT DEBUG record-parser - parsed message fields: key=Key(namespace='test', set=testset, digest=[124, -6, -70, -23, 44, 41, -19, 40, -11, -16, 126, 120, 81, -113, -112, -79, 66, 77, -99, -6], userKey=null, lastUpdateTimeMs=1670523696059, userKeyString=null, digestString='fPq66Swp7Sj18H54UY+QsUJNnfo=')
2022-12-08 18:21:36.158 GMT DEBUG record-parser - parsed message fields: key=Key(namespace='test', set=testset, digest=[-127, -118, 63, -32, -74, 60, -60, 86, 31, -119, -1, -105, -108, -59, 111, 48, -34, -61, -108, -5], userKey=null, lastUpdateTimeMs=1670523696105, userKeyString=null, digestString='gYo/4LY8xFYfif+XlMVvMN7DlPs=')
2022-12-08 18:21:36.256 GMT DEBUG record-parser - parsed message fields: key=Key(namespace='test', set=testset, digest=[99, 1, 80, 2, 76, -43, 125, 77, 47, 8, 6, 35, 49, 117, -35, 54, 120, -29, 118, -72], userKey=null, lastUpdateTimeMs=1670523696178, userKeyString=null, digestString='YwFQAkzVfU0vCAYjMXXdNnjjdrg=')
2022-12-08 18:21:36.257 GMT DEBUG record-parser - parsed message fields: key=Key(namespace='test', set=testset, digest=[-88, -46, -48, -33, 77, -120, 123, -101, -70, -20, -96, -104, -51, -90, 28, -15, 70, 11, 118, 83], userKey=null, lastUpdateTimeMs=1670523696202, userKeyString=null, digestString='qNLQ302Ie5u67KCYzaYc8UYLdlM=')
2022-12-08 18:21:36.257 GMT DEBUG record-parser - parsed message fields: key=Key(namespace='test', set=testset, digest=[-97, 18, 28, -43, 75, 42, -22, -126, -61, -108, -36, 118, -86, -105, -52, 119, -39, -33, -127, -76], userKey=null, lastUpdateTimeMs=1670523696175, userKeyString=null, digestString='nxIc1Usq6oLDlNx2qpfMd9nfgbQ=')
2022-12-08 18:21:36.257 GMT DEBUG record-parser - parsed message fields: key=Key(namespace='test', set=testset, digest=[-120, 103, -51, 57, -71, -106, 13, -48, 100, 28, 59, -3, -39, -56, -67, -103, 29, 36, 75, 119], userKey=null, lastUpdateTimeMs=1670523696191, userKeyString=null, digestString='iGfNObmWDdBkHDv92ci9mR0kS3c=')
2022-12-08 18:21:36.257 GMT DEBUG record-parser - parsed message fields: key=Key(namespace='test', set=testset, digest=[-47, 88, -13, 13, -35, 77, 24, 22, -40, -61, -118, -115, 82, 13, 127, -125, 53, 66, -22, -8], userKey=null, lastUpdateTimeMs=1670523696233, userKeyString=null, digestString='0VjzDd1NGBbYw4qNUg1/gzVC6vg=')
2022-12-08 18:21:36.258 GMT DEBUG record-parser - parsed message fields: key=Key(namespace='test', set=testset, digest=[-63, -11, 93, -90, 47, 29, -63, 36, 12, 53, -86, 84, 57, -125, 16, 43, -18, 93, 56, 9], userKey=null, lastUpdateTimeMs=1670523696186, userKeyString=null, digestString='wfVdpi8dwSQMNapUOYMQK+5dOAk=')
2022-12-08 18:21:36.258 GMT DEBUG record-parser - parsed message fields: key=Key(namespace='test', set=testset, digest=[-23, 101, -114, -87, -52, 107, 36, 113, 101, 33, -16, 82, -95, 97, 34, -121, 82, -97, 40, 59], userKey=null, lastUpdateTimeMs=1670523696145, userKeyString=null, digestString='6WWOqcxrJHFlIfBSoWEih1KfKDs=')
2022-12-08 18:21:36.258 GMT DEBUG record-parser - parsed message fields: key=Key(namespace='test', set=testset, digest=[-77, -118, 49, 4, -75, 123, 81, 2, -103, -73, 42, -70, -54, 95, 98, 23, 73, 66, -86, 7], userKey=null, lastUpdateTimeMs=1670523696230, userKeyString=null, digestString='s4oxBLV7UQKZtyq6yl9iF0lCqgc=')
2022-12-08 18:21:36.258 GMT DEBUG record-parser - parsed message fields: key=Key(namespace='test', set=testset, digest=[50, -38, -31, -54, -122, -113, -38, 88, 15, 7, 96, 51, -92, -25, 60, -104, 26, 113, -117, -82], userKey=null, lastUpdateTimeMs=1670523696157, userKeyString=null, digestString='MtrhyoaP2lgPB2AzpOc8mBpxi64=')
2022-12-08 18:21:36.258 GMT DEBUG record-parser - parsed message fields: key=Key(namespace='test', set=testset, digest=[-77, 31, 67, -18, -52, -114, 42, -18, 36, -111, 89, 62, 109, 114, -54, 54, -121, -110, -88, -108], userKey=null, lastUpdateTimeMs=1670523696206, userKeyString=null, digestString='sx9D7syOKu4kkVk+bXLKNoeSqJQ=')
2022-12-08 18:21:36.258 GMT DEBUG record-parser - parsed message fields: key=Key(namespace='test', set=testset, digest=[73, 11, 98, -50, 32, 12, 0, -50, 22, -101, -108, 18, 38, 7, -65, 6, -58, 60, -6, -33], userKey=null, lastUpdateTimeMs=1670523696171, userKeyString=null, digestString='SQtiziAMAM4Wm5QSJge/BsY8+t8=')
2022-12-08 18:21:36.258 GMT DEBUG record-parser - parsed message fields: key=Key(namespace='test', set=testset, digest=[109, -82, 24, 53, 35, 89, -72, -117, -22, 79, 119, -89, 56, -5, 0, -103, -54, 51, 25, 126], userKey=null, lastUpdateTimeMs=1670523696146, userKeyString=null, digestString='ba4YNSNZuIvqT3enOPsAmcozGX4=')
...

Conclusion

Aerospike renowned for its core resiliency has an industry established uptime of 5 x 9s - that's basically ( 0.99999 ). So adding to the data resiliency matrix, XDR now means its possible to have an additional copy of the data distributed asynchronously from one data-centre to another with minimal effort. Taking this one step further, we have shown how this can be achieved using the xdr-proxy to access k8s clusters in a private network. All this has been achieved with minimum effort thanks to the Aerospike Kubernetes Operator.

Block and Filesystem side-by-side with K8s and Aerospike

Naresh Maharaj — Wed, 30 Nov 2022 18:34:25 +0000

In this article we focus on side-by-side block and filesystem requests using Kubernetes. The driver for this is it will allow us to deploy Aerospike using Aerospike's all flash mode.

Storing database values on a raw block device with index information on file can bring significant cost reductions, especially when considering use cases for Aerospike’s All-Flash. To support such a workload, you can configure Aerospike to use NVMe Flash drives as the primary index store.

How does this operation work in practice?
Suppose a devOps engineer has a host machine with one or more raw block devices and wants to use a filesystem formatted to Ext4 or XFS for storage. devOps could log into each host, create partitions, format the Filesystem, and mount the device in the traditional way but this seems less than pragmatic.
That's where side-by-side block and filesystem requests using Kubernetes comes in.

This article describes the following process:

Spin up an AWS EKS Cluster with eksctl
Clarify our understanding of block storage devices
Introduce the Local Static Provisioner
Create a Persistent Volume (PV) using a raw device
- 1. intended for block storage
- 2. intended for storing files
Install httpd and review
- 1. scheduling Pods to a host with a filesystem
- 2. use of a PersistentVolumeClaim (PVC)
- 3. node affinity with a PersistentVolume
Install Aerospike using the Kubernetes Operator
Insert some test data
Use Kubernetes Taints and Tolerations
- 1. to reserve Hosts with filesystems
- 2. schedule Pods to Hosts with filesystems
Clean everything up.

Prerequisites

Before we zoom into action, we need to have the following:

lots of energy and some basic knowledge of Kubernetes
jq ( a cool json command line parser )
eksctl ( cli tool for working with EKS clusters )
kubectl ( cli tool to run commands against k8s clusters )
parted ( cli tool for creating and deleting partitions)

Create an EKS Cluster

Setup

Create a file called my-cluster.yaml with the following contents. This specifies the configuration for the Kubernetes cluster itself. The ssh configuration is optional.

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: my1-eks-cluster
  region: us-east-1
nodeGroups:
  - name: ng-1
    labels: { role: database }
    instanceType: m5d.large
    desiredCapacity: 3
    volumeSize: 13
    ssh:
      allow: true # will use ~/.ssh/id_rsa.pub as the default ssh key

Use the AWS eksctl tool to create our Kubernetes cluster. You may need to install eksctl if this is not present on your machine.
Using eksctl ...

eksctl create cluster -f my-cluster.yaml

The result on completion will be a 3 node Kubernetes cluster. Confirm successful creation as follows :

kubectl get nodes -L role
NAME                             STATUS   ROLES    AGE     VERSION                ROLE
ip-192-168-22-209.ec2.internal   Ready    <none>   2m36s   v1.22.12-eks-ba74326   database
ip-192-168-28-68.ec2.internal    Ready    <none>   2m38s   v1.22.12-eks-ba74326   database
ip-192-168-50-205.ec2.internal   Ready    <none>   2m33s   v1.22.12-eks-ba74326   database

Raw Block Storage

Block storage stores a sequence of bytes in a fixed size block (page) on a storage device. Each block has a unique hash that references the address location of the specified block. Unlike a filesystem, block storage doesn't have the associated metadata such as format-type, owner, date, etc. Also, block storage doesn’t use the conventional storage paths to access data like a filesystem file. This reduction in overhead contributes to improved overall access speeds when using raw block devices. The ability to store bytes in blocks allows applications the flexibility to decide how these blocks are accessed and managed, making block storage an ideal choice for low latency databases such as Aerospike. From a developer's perspective, a block device is simply a large array of bytes, usually with some minimum granularity for reads and writes. In Aerospike this granularity is configured and referred to as the write-block-size. The Aerospike Kubernetes Operator uses the storage infrastructure software inside of Kubernetes and the need for data platforms to use raw block storage becomes ever more important.

Local Static Provisioner

The local volume static provisioner manages the PersistentVolume (PV) lifecycle for pre-allocated disks by detecting and creating PVs for each local disk on the host, and cleaning up the disks when released. It doesn't support dynamic provisioning where storage volumes can be created on-demand. Without dynamic provisioning a manual step is required to create new storage volumes which are then mapped to PersistentVolume objects.

Why do I need it?

As a captain of a ship, your time is best spent at the helm and not constantly going down below deck. Similarly, the LocalStaticProvisioner maintains the low level details like mappings between paths of local volumes and PV objects and keeps these stable across reboots and device changes, so you can focus on orchestration efforts.

The PersistentVolume, often seen as a remote storage resource, decouples the volumes from the Pods. However, a distributed application with fast access patterns and replication logic can benefit from the disks being accessed locally. Local disk usage will strongly couple your application data to a specific node, so if the Pod fails and is successfully rescheduled, the data on that node will still be available to the container. If the node is no longer accessible, data could be inaccessible but applications with built in replication logic will ensure data is copied ( we like to say replicated ) onto other nodes and this means applications can survive a node level outage. Without using the LocalStaticProvisioner the operator would be responsible for partitioning, formatting and mounting devices before they could be used as a Kubernetes PersistentVolume. The operator would also need to be responsible for cleaning up the volume and preparing it for reuse, something which is now the responsibility of the LocalStaticProvisioner when the PersistentVolumeClaim is released.

Persistent Volume Block Store

To use a device as raw block storage, you can edit the following yaml file as required default_example_provisioner_generated.yaml.

The following section is the key part to this file. For the majority of the time you will not need to change it.

apiVersion: v1
kind: ConfigMap
metadata:
  name: local-provisioner-config 
  namespace: default 
data:
  storageClassMap: |     
    fast-disks:
       hostDir: /mnt/fast-disks
       mountDir:  /mnt/fast-disks 
       blockCleanerCommand:
         - "/scripts/shred.sh"
         - "2"
       volumeMode: Filesystem
       fsType: ext4
       namePattern: "*"

To use a device as block storage for Aerospike, we need to replace the section above with the following. A pre-configured file is available here for your convenience.

In a moment we will deploy this file so there is no need to do anything just right now.

apiVersion: v1
kind: ConfigMap
metadata:
  name: local-provisioner-config
  namespace: aerospike
data:
  useNodeNameOnly: "true"
  storageClassMap: |
    local-ssd:
       hostDir: /mnt/disks
       mountDir:  /mnt/disks
       blockCleanerCommand:
         - "/scripts/shred.sh"
         - "2"
       volumeMode: Block

To learn more about how to configure and Operate the LocalStaticProvisioner see the Operation and Configuration guide.

Once your Kubernetes cluster is up and running, create the soft links to your local storage device(s) on each node.

kubectl get nodes -o wide
NAME                             STATUS   ROLES    AGE   VERSION
ip-192-168-10-182.ec2.internal   Ready    <none>   20m   v1.22.12-eks-ba74326
ip-192-168-15-174.ec2.internal   Ready    <none>   20m   v1.22.12-eks-ba74326
ip-192-168-45-233.ec2.internal   Ready    <none>   21m   v1.22.12-eks-ba74326

For each node, add a permanent soft link to the block device. In our case, the device name is nvme1n1 as below. You will need to ssh into each of the Kubernetes worker nodes to do this.

lsblk
NAME          MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
nvme1n1       259:0    0 69.9G  0 disk
nvme0n1       259:1    0   13G  0 disk
├─nvme0n1p1   259:2    0   13G  0 part /
└─nvme0n1p128 259:3    0    1M  0 part

sudo mkdir /mnt/disks -p
cd /mnt/disks/
sudo ln -sf /dev/nvme1n1 /mnt/disks/
ls -lrt /mnt/disks/

Now apply the local static provisioner yaml file.

kubectl create ns aerospike
kubectl create -f https://raw.githubusercontent.com/aerospike/aerospike-kubernetes-operator/2.2.1/config/samples/storage/aerospike_local_volume_provisioner.yaml
kubectl create -f https://raw.githubusercontent.com/aerospike/aerospike-kubernetes-operator/2.2.1/config/samples/storage/local_storage_class.yaml

You should now see the following resources created.

kubectl get sc,pv -n aerospike
NAME                                        PROVISIONER                    RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
storageclass.storage.k8s.io/gp2 (default)   kubernetes.io/aws-ebs          Delete          WaitForFirstConsumer   false                  39m
storageclass.storage.k8s.io/local-ssd       kubernetes.io/no-provisioner   Delete          WaitForFirstConsumer   false                  73s

NAME                                 CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   REASON   AGE
persistentvolume/local-pv-45df0f3d   69Gi       RWO            Delete           Available           local-ssd               6m42s
persistentvolume/local-pv-72d40f98   69Gi       RWO            Delete           Available           local-ssd               6m42s
persistentvolume/local-pv-7637ecb    69Gi       RWO            Delete           Available           local-ssd               6m42s

Remove the resources for now.

kubectl delete -f https://raw.githubusercontent.com/aerospike/aerospike-kubernetes-operator/2.2.1/config/samples/storage/aerospike_local_volume_provisioner.yaml
kubectl delete -f https://raw.githubusercontent.com/aerospike/aerospike-kubernetes-operator/2.2.1/config/samples/storage/local_storage_class.yaml
kubectl get pv -o=json|jq .items[].metadata.name -r | grep ^local | while read -r line; do kubectl delete pv $line;done

Persistent Volume Filesystem

This brings us to the core part of this article which is, how do we create an XFS filesystem from raw storage devices? In this section we partition the disk so we can have a block and a filesystem on a single device, but we do this only on 2 out of the 3 nodes. So, of the 3 nodes only 2 will have a filesystem.

Host Addresses

We start by getting a list of the nodes and IP addresses:

kubectl get no -o wide
NAME                             STATUS   ROLES    AGE   VERSION                INTERNAL-IP      EXTERNAL-IP    OS-IMAGE         KERNEL-VERSION                 CONTAINER-RUNTIME
ip-192-168-23-238.ec2.internal   Ready    <none>   63m   v1.22.12-eks-ba74326   192.168.23.238   44.212.74.12   Amazon Linux 2   5.4.209-116.367.amzn2.x86_64   docker://20.10.17
ip-192-168-63-152.ec2.internal   Ready    <none>   63m   v1.22.12-eks-ba74326   192.168.63.152   3.234.239.82   Amazon Linux 2   5.4.209-116.367.amzn2.x86_64   docker://20.10.17
ip-192-168-7-181.ec2.internal    Ready    <none>   63m   v1.22.12-eks-ba74326   192.168.7.181    3.87.204.50    Amazon Linux 2   5.4.209-116.367.amzn2.x86_64   docker://20.10.17

Create Partitions

We are going to use the 'parted' application to create our disk partitions. If this is not installed then use the following command to install it.

sudo yum install parted -y

Next, we partition our nvme1n1 disk to give us partitions nvme1n1p1 and nvme1n1p2:

sudo parted -a opt --script /dev/nvme1n1 mklabel gpt mkpart primary 0% 10% mkpart primary 10% 100%
lsblk

NAME          MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
nvme1n1       259:0    0 69.9G  0 disk
├─nvme1n1p1   259:4    0    7G  0 part
└─nvme1n1p2   259:5    0 62.9G  0 part
nvme0n1       259:1    0   13G  0 disk
├─nvme0n1p1   259:2    0   13G  0 part /
└─nvme0n1p128 259:3    0    1M  0 part

Discovery Directories

Lets create the soft links in our discovery directory on 2 of our chosen hosts which will have a 7GB filesystem.

sudo mkdir /mnt/fs-disks -p
sudo mkdir /mnt/fast-disks -p
sudo ln -sf /dev/nvme1n1p1 /mnt/fs-disks/
sudo ln -sf /dev/nvme1n1p2 /mnt/fast-disks/
ls -lrt /mnt/fs-disks/ /mnt/fast-disks/

On the remaining 1 x host which does not have a filesystem do the following. In a real production system all disks would be the identical.

sudo mkdir /mnt/fast-disks -p
sudo ln -sf /dev/nvme1n1 /mnt/fast-disks/

Storage Classes & Local Static Provisioner

Download the local static provisioner & the storage class files, make a copy so we can edit them as required.

wget https://raw.githubusercontent.com/aerospike/aerospike-kubernetes-operator/2.2.1/config/samples/storage/local_storage_class.yaml
cp local_storage_class.yaml local_storage_class-fast-ssd.yaml
cp local_storage_class.yaml local_storage_class-fs-ssd.yaml

Storage Class

To create the storage classes [fast-ssd] and [fs-ssd] we need to change the metadata.name in each of our the edited storage_class files respectively and then deploy them as below. Make one [fast-ssd] the other [fs-ssd].

kubectl create -f local_storage_class-fast-ssd.yaml
kubectl create -f local_storage_class-fs-ssd.yaml
kubectl get sc

NAME            PROVISIONER                    RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
fast-ssd        kubernetes.io/no-provisioner   Delete          WaitForFirstConsumer   false                  9s
fs-ssd          kubernetes.io/no-provisioner   Delete          WaitForFirstConsumer   false                  3m23s

Local Static Provisioner

We now want to ensure that the block storage and the filesystem storage can work side by side on the same host. In order to do this we have edited both files

aerospike_local_volume_provisioner-fast-ssd.yaml
aerospike_local_volume_provisioner-fs-ssd.yaml

This ensures that the names of the following Kubernetes resources do not clash.

ServiceAccount
ClusterRole
ClusterRoleBinding
ConfigMap
DaemonSet

Following are direct links to the edited files for your convenience.

Create an aerospike namespace and deploy the storage yaml files.

kubectl create ns aerospike
kubectl create -f https://raw.githubusercontent.com/nareshmaharaj-consultant/localstaticprovisioner-k8s-config-files/main/aerospike_local_volume_provisioner-fs-ssd.yaml
kubectl create -f https://raw.githubusercontent.com/nareshmaharaj-consultant/localstaticprovisioner-k8s-config-files/main/aerospike_local_volume_provisioner-fast-ssd.yaml

Here is the result of deploying our changes. Notice the 7GB filesystem using storage class [fs-ssd].

kubectl get sc,pv
NAME                                        PROVISIONER                    RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
storageclass.storage.k8s.io/fast-ssd        kubernetes.io/no-provisioner   Delete          WaitForFirstConsumer   false                  17m
storageclass.storage.k8s.io/fs-ssd          kubernetes.io/no-provisioner   Delete          WaitForFirstConsumer   false                  17m
storageclass.storage.k8s.io/gp2 (default)   kubernetes.io/aws-ebs          Delete          WaitForFirstConsumer   false                  126m

NAME                                 CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   REASON   AGE
persistentvolume/local-pv-42dc4d3f   62Gi       RWO            Delete           Available           fast-ssd                22s
persistentvolume/local-pv-ba8bc9e    69Gi       RWO            Delete           Available           fast-ssd                22s
persistentvolume/local-pv-d7501418   69Gi       RWO            Delete           Available           fast-ssd                22s
persistentvolume/local-pv-f5282eeb   7152Mi     RWO            Delete           Available           fs-ssd                  2s

Using the Filesystem

One misconception that might arise is how do we know which node has the XFS filesystem - so our Pod can be scheduled to the correct node? To answer this question - we start by testing our new filesystem and see where it takes us. First we create a simple PersistentVolumeClaim referencing our filesystem [fs-ssd] storage class. Go ahead and create a file called pvc-fs.yaml with the following content.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: fs-1g-pvc
spec:
  accessModes:
    - ReadWriteOnce
  volumeMode: Filesystem
  resources:
    requests:
      storage: 1Gi
  storageClassName: fs-ssd

Create a Pod with the httpd server image and reference the PersistentVolumeClaim. Call your file pod-fs.yaml

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    app: fs-httpd
  name: fs-httpd
spec:
  containers:
  - image: httpd
    name: fs-httpd
    resources: {}
    volumeMounts:
      - mountPath: "/datafiles/"
        name: my-fs
  dnsPolicy: ClusterFirst
  restartPolicy: Always
  volumes:
    - name: my-fs
      persistentVolumeClaim:
        claimName: fs-1g-pvc
status: {}

Deploy both the PVC and the Pod

kubectl create -f pvc-fs.yaml
kubectl create -f pod-fs.yaml

Run the following command to see where this pod was scheduled.

kubectl describe po fs-http

Output shows that the Pod was scheduled to the correct Node with the XFS filesystem ip-192-168-28-160.ec2.internal/192.168.28.160. (Your Node IP address will be different)

This happened because our StorageClass [fs-ssd] was used in the PersistentVolumeClaim. The PersistentVolumeClaim was used in our Pod and the scheduler knew exactly where the Pod needed to be scheduled.

We can go a bit further and prove this by adding a Taint to all Nodes with a filesystem. Delete the pod and then add the following taint. Then recreate the Pod once more. ( learn more about Kubernetes taints & tolerations )

(Note: If you are keeping the taint and later redeploy the LocalStaticProvisioner be sure to uncomment the tolerations section in each LocalStaticProvisioner yaml file.)

# chg node name here!! 
kubectl delete -f pod-fs.yaml
kubectl taint node ip-192-168-28-160.ec2.internal storageType=filesystem:NoSchedule
kubectl taint node ip-192-168-63-152.ec2.internal storageType=filesystem:NoSchedule
kubectl create -f pod-fs.yaml

If we look at the events in the Pod, we see something very interesting.

"0/3 nodes are available: 2 node(s) had taint {storageType: filesystem}, that the pod didn't tolerate, 1 node(s) had volume node affinity conflict."

kubectl describe po fs-httpd
Name:             fs-httpd
Namespace:        default
Priority:         0
Service Account:  default
Node:             <none>
Labels:           app=fs-httpd
Annotations:      kubernetes.io/psp: eks.privileged
Status:           Pending
...
Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  35s   default-scheduler  0/3 nodes are available: 2 node(s) had taint {storageType: filesystem}, that the pod didn't tolerate, 1 node(s) had volume node affinity conflict.

This means that 2 of our Nodes rejected the Pod as it did not tolerate the Taint, but it did match the PersistentVolume on the Node. Of the remaining 1 x Node, it failed to meet the Node Affinity of the PersistentVolume attached to the PersistentVolumeClaim. To be clear, each of our PersistentVolume(s) are coupled to Nodes using the Node Affinity rules. If you describe a volume you will see the following:

kubectl describe pv local-pv-9dbcb710
Name:              local-pv-9dbcb710
StorageClass:      fs-ssd
Node Affinity:
  Required Terms:
    Term 0:        kubernetes.io/hostname in [ip-192-168-28-160.ec2.internal]

Now add a toleration to the Pod or delete the taint and we should get a Pod scheduled to the correct Node. Below is our new Pod configuration with the tolerations added

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    app: fs-httpd
  name: fs-httpd
spec:
  containers:
  - image: httpd
    name: fs-httpd
    resources: {}
    volumeMounts:
      - mountPath: "/datafiles/"
        name: my-fs
  dnsPolicy: ClusterFirst
  restartPolicy: Always
  volumes:
    - name: my-fs
      persistentVolumeClaim:
        claimName: fs-1g-pcv
  tolerations:
    - key: storageType
      operator: "Equal"
      value: filesystem
      effect: NoSchedule
status: {}

If we describe our Pod we can see what events occured. I've only included the relevant information.

kubectl describe po fs-httpd
Name:             fs-httpd
Node:             ip-192-168-28-160.ec2.internal/192.168.28.160
Volumes:
    ClaimName:  fs-1g-pcv
Tolerations:                storageType=filesystem:NoSchedule
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  91s   default-scheduler  Successfully assigned default/fs-httpd to ip-192-168-28-160.ec2.internal
  Normal  Created    90s   kubelet            Created container fs-httpd
  Normal  Started    90s   kubelet            Started container fs-httpd

One last interesting fact is to see the mounted XFS filesystem /dev/nvme1n1p1 in the container.

kubectl exec -it fs-httpd -- bash
root@fs-httpd:/usr/local/apache2# df -Th
Filesystem     Type     Size  Used Avail Use% Mounted on
overlay        overlay   13G  2.9G   11G  22% /
tmpfs          tmpfs     64M     0   64M   0% /dev
tmpfs          tmpfs    3.8G     0  3.8G   0% /sys/fs/cgroup
/dev/nvme1n1p1 xfs      7.0G   40M  7.0G   1% /datafiles
/dev/nvme0n1p1 xfs       13G  2.9G   11G  22% /etc/hosts
shm            tmpfs     64M     0   64M   0% /dev/shm
tmpfs          tmpfs    6.9G   12K  6.9G   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs          tmpfs    3.8G     0  3.8G   0% /proc/acpi
tmpfs          tmpfs    3.8G     0  3.8G   0% /sys/firmware

Installing Aerospike - Block & Filesystem Storage

Aerospike Kubernetes Operator

Install the AKO (Aerospike Kubernetes Operator) and set up the database cluster. Refer to the following documentation for details on how to do this.

You can confirm that your installation is successful by checking the pods in the operators namespace.

kubectl get pods -n operators
NAME                                                     READY   STATUS    RESTARTS   AGE
aerospike-operator-controller-manager-7946df5dd9-lmqvt   2/2     Running   0          2m44s
aerospike-operator-controller-manager-7946df5dd9-tg4sh   2/2     Running   0          2m44s

Create the Aerospike Database Cluster

In this section we create an Aerospike database cluster. For full details, refer to the following documentation.

The basic steps are as follows.

Get the code

Clone the Aerospike Kubernetes Git repo and be sure to copy your feature/licence key file, if you have one, to the following directory - config/samples/secrets.

git clone https://github.com/aerospike/aerospike-kubernetes-operator.git
cd aerospike-kubernetes-operator/

Initialize Storage

kubectl apply -f config/samples/storage/eks_ssd_storage_class.yaml

As of version 6.0, for each block storage device, erase the disk to avoid critical errors:

Oct 26 2022 14:05:53 GMT: CRITICAL (drv_ssd): (drv_ssd.c:2216) /test/dev/xvdf: not an Aerospike device but not erased - check config or erase device

Use the following command to erase the device on each host. Remember on 2 of the Nodes the device will be nvme1n1p2 (demo only)

export diskPart=/dev/nvme1n1
sudo blkdiscard ${diskPart} && sudo blkdiscard -z --length 8MiB ${diskPart} &

Add Secrets

Secrets, for those not familiar with the terminology, allow data to be introduced into the Kubernetes environment, while ensuring that it cannot be read. Examples might include private PKI keys or passwords.

kubectl  -n aerospike create secret generic aerospike-secret --from-file=config/samples/secrets
kubectl  -n aerospike create secret generic auth-secret --from-literal=password='admin123'

Create Aerospike Cluster

Before we create the cluster, delete the httpd Pod and PersistentVolumeClaim we created earlier:

kubectl delete -f pod-fs.yaml
kubectl delete pvc fs-1g-pvc

Make sure all the PersistentVolume(s) have a status of 'Available':

kubectl get pv -n aerospike
NAME                CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   REASON   AGE
local-pv-684418fc   62Gi       RWO            Delete           Available           fast-ssd                4h13m
local-pv-9dbcb710   7152Mi     RWO            Delete           Available           fs-ssd                  5m42s
local-pv-aa49ff2c   69Gi       RWO            Delete           Available           fast-ssd                4h13m
local-pv-ad278bd7   69Gi       RWO            Delete           Available           fast-ssd                4h13m

If some of the PVs are not in the 'Available' state run the following patch to release the PersistentVolumeClaim (PVC).

kubectl get pv -o=json|jq .items[].metadata.name -r | grep ^local | while read -r line; do kubectl patch pv $line -p '{"spec":{"claimRef": null}}';done

Use the file all_flash_cluster_cr.yaml to configure the Aerospike Database to use the All Flash recipe so we get to use the Block and Filesystem storage as a single unit of work. Note where we have used the different Storage classes and Tolerations.

storageClass: ssd ( AWS gps ext4 Filesystem Dynamic Storage )
storageClass: fast-ssd ( LocalStaticProvisioner Block Storage )
storageClass: fs-ssd ( LocalStaticProvisioner XFS filesystem )

Deploy the Aerospike Database

kubectl create -f all_flash_cluster_cr.yaml
kubectl get po -n aerospike -w

Verify Aerospike Database Running

The get pods command should show you two active Aerospike pods.

kubectl get po -n aerospike
NAME                                        READY   STATUS    RESTARTS   AGE
aerocluster-0-0                             1/1     Running   0          8m32s
aerocluster-0-1                             1/1     Running   0          8m32s

Insert test data with the benchmark tool

The Aerospike Benchmark tool measures the performance of an Aerospike cluster. It can mimic real-world workloads with configurable record structures, various access patterns, and UDF calls. Use the benchmark tool to insert some data easily.

kubectl exec -it aerocluster-0-0 -n aerospike -- bash

Add some data using the simplest of commands

asbench -Uadmin -Padmin123

To verify we successfully added some data use aql tool, login and run a query

aql -Uadmin -Padmin123
select * from test

Cleaning up

WARNING: If this step is bypassed and the EKS cluster is deleted, all persistent volumes created will remain.

Delete Database Cluster

This is the important step and should always be run when using cloud storage classes.

kubectl delete -f all_flash_cluster_cr.yaml

Delete EKS Cluster

We can now delete the EKS cluster:

eksctl delete cluster -f my-cluster.yaml

Conclusion

We have seen that with little effort we can easily spin up a deployment with local disks for block storage, combined with filesystem storage. We also confirmed the Pods were being scheduled automatically to the correct nodes. We also learned that by using Kubernetes taints, we could effectively reserve nodes for specific usage. You had a sneak preview into Aerospike - All Flash use cases too. And one final point, we only looked at the Kubernetes Special Interest Group (SIG) version of the LocalStaticProvisioner. You may also wish to consider Rancher's version which is listed below.

Review Rancher
https://github.com/rancher/local-path-provisioner

Aerospike k8s Volume Cleanup

Naresh Maharaj — Thu, 13 Oct 2022 21:33:16 +0000

When using the Aerospike Kubernetes Operator, the complexity of configuring a high performance distributed database is abstracted away, making the task of instantiating an Aerospike database incredibly easy. However, even though Kubernetes leads us towards expecting equivalent results regardless of platform, we need to be mindful of the peculiarities of individual frameworks, particularly if we are repeatedly iterating processes.

This article focuses on AWS EKS provisioned storage which is dynamically created when using the Aerospike Kubernetes Operator. Ensuring that storage has been fully deleted and other redundant resources removed is a necessary housekeeping step if you are to avoid unwelcome AWS charges.

In this article we review

How to create a simple AWS EKS cluster
Install the Aerospike Kubernetes Operator
How AWS Provisioned EBS Storage volumes are created
The results of repeated Aerospike database cluster creation - particularly the resulting storage utilization
Database cluster termination, with a focus on how the storage assets are managed
What to watch out for

Create an EKS Cluster

In this section we create an EKS cluster, on which we will install Aerospike.

Setup

Create a file called my-cluster.yaml with the following contents. This specifies the configuration for the Kubernetes cluster itself. The ssh configuration is optional.

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: my1-eks-cluster
  region: us-east-1

nodeGroups:
  - name: ng-1
    labels: { role: database }
    instanceType: t3.medium
    desiredCapacity: 3
    volumeSize: 13
    ssh:
      allow: true # will use ~/.ssh/id_rsa.pub as the default ssh key

We use the AWS eksctl tool to create our Kubernetes cluster. For more information on using eksctl see the official documentation.

Using eksctl ...

eksctl create cluster -f my-cluster.yaml

The result on completion will be a 3 node Kubernetes cluster. Confirm successful creation as follows :

kubectl get nodes -L role
NAME                             STATUS   ROLES    AGE   VERSION                ROLE
ip-192-168-13-47.ec2.internal    Ready    <none>   10m   v1.22.12-eks-ba74326   database
ip-192-168-23-112.ec2.internal   Ready    <none>   10m   v1.22.12-eks-ba74326   database
ip-192-168-40-59.ec2.internal    Ready    <none>   10m   v1.22.12-eks-ba74326   database

We can check what volumes are created. In my-cluster.yaml volume size is 13Gb, allowing these to be easily located when using the AWS EC2 console.

Install Aerospike

Aerospike Kubernetes Operator

In this section we install the AKO ( Aerospike Kubernetes Operator ) and set up our database cluster.

Please refer to the following documentation for details as to how to do this.

AKO installation can be confirmed by checking the pods in the operators namespace.

kubectl get pods -n operators
NAME                                                     READY   STATUS    RESTARTS   AGE
aerospike-operator-controller-manager-7946df5dd9-lmqvt   2/2     Running   0          2m44s
aerospike-operator-controller-manager-7946df5dd9-tg4sh   2/2     Running   0          2m44s

Create the Aerospike Database Cluster

In this section we create an Aerospike cluster. For full detail please refer to the documentation below.

The basic steps are as follows.

Get the code

Clone the Aerospike Kubernetes Git repo and be sure to copy your feature/licence key file, if you have one, to the following directory - config/samples/secrets.

git clone https://github.com/aerospike/aerospike-kubernetes-operator.git
cd aerospike-kubernetes-operator/

Initialise Storage

kubectl apply -f config/samples/storage/eks_ssd_storage_class.yaml
kubectl apply -f config/samples/storage/local_storage_class.yaml

Add Secrets

Secrets, for those not familiar with the terminology, allow data to be introduced into the Kubernetes environment, while ensuring that it canot be read. Examples might include private PKI keys or passwords.

kubectl  -n aerospike create secret generic aerospike-secret --from-file=config/samples/secrets
kubectl  -n aerospike create secret generic auth-secret --from-literal=password='admin123'

Create Aerospike Cluster

We set up Aerospike using the ssd storage recipe.

kubectl create -f config/samples/ssd_storage_cluster_cr.yaml

Verify Aerospike Database Running

The get pods command should show you two active Aerospike pods.

kubectl get pods -n aerospike
NAME              READY   STATUS    RESTARTS   AGE
aerocluster-0-0   1/1     Running   0          36s
aerocluster-0-1   1/1     Running   0          36s

Persistent Volumes and Claims (PV,PVC)

If we run the following command, we can see what persistent volume storage has been claimed.

kubectl get pv,pvc -n aerospike -o json | jq .'items[].spec.capacity.storage' | egrep -v -e null
"1Gi"
"5Gi"
"5Gi"
"1Gi"

Note we see the same persistent volumes when using the AWS Console as expected - included for completeness.

Cleaning up

WARNING: If this step is bypassed and the EKS cluster is deleted, all persistent volumes created will remain.

Delete Aerospke Database Cluster

This is the important step and should always be run when using cloud storage classes.

kubectl delete -f ssd_storage_cluster_cr.yaml 
aerospikecluster.asdb.aerospike.com "aerocluster" deleted

Check Persistent Volume Claims

If we now check our persistent volumes we see there are no persistent volumes remaining.

kubectl get pv,pvc -n aerospike -o json | jq .'items[].spec.capacity.storage' | egrep -v -e null
(nothing returned...)

Similarly, if visiting the AWS Console, we can see only the volumes for the k8s nodes we initially created using the eksctl command.

Delete EKS Cluster

We can now go ahead and delete the EKS cluster

eksctl delete cluster -f my-cluster.yaml

Verify Resources Deleted

Final check of our EBS volumes and we should find everything has been removed.

How not to clean up

Assume we have a new EKS cluster, having installed the Aerospike Operator and a new Aerospike database. We then see the resources below.

Suppose we only we excecute the eksctl delete cluster command.

eksctl delete cluster -f my-cluster.yaml

In that case, the persistent volumes associated with Aerospike will NOT be deleted. In this example the volumes are small, but at $0.10/GB/month these costs can accumulate (100TB = $10k / month!). Numerous test iterations can result in capacity accumulating, generating large unwelcome AWS bills for.

Note that we can go ahead and manually delete these volumes via the AWS Console - but only because we know they are there.

Conclusion

Automation tools are fantastic and this level of deployment agility is an astonishing convenience. It is always wise however to check on final results and not assume the tools take care of everything. Another moral is that following instructions in full may be the wise initial path - the author probably detailed those steps for good reason.

Authorize Atlas to Access your AWS Account

Naresh Maharaj — Tue, 15 Jun 2021 10:55:41 +0000

AWS IAM Role Access

In MongoDB Atlas you may at some stage wish to use a cloud provider to either store files to S3, encrypt at rest and so on.

MongoDB Atlas allows you to configure the Cloud Provider Role and authenticate that role to use later as a service.

In this article we use terraform to

1) Create the Cloud Service in MongoDB Atlas
2) Get the external ref from Atlas and provide this as a trusted source to a Role on your own AWS Account.
3) Authenticate the role as an Assumed Role to Atlas.

https://github.com/nareshmaharaj-consultant/atlas_cloud_provider_terraform

Use the following

variables.tf

variable "mongodb_atlas_api_pub_key" {
 default = "qvesfrig"
}

variable "mongodb_atlas_api_pri_key" {
 default = "enter your private key here"
}

variable "mongodb_atlas_project_id" {
 default = "60ab6ed5fb4a1f43c4950e71"
}

variable "atlas_project_name" {
  type        = string
  description = "Name of the Atlas project the role is associated with"
  default     = "my-atlas"
}

variable "account_name" {
  type        = string
  description = "Name of the AWS account.  Used as a name prefix"
  default     =  "naresh.maharaj"
}

variable "tags" {
  type        = map(string)
  description = "Key/value pairs of additional information attached to resources"
  default     = {}
}

variable "atlas_aws_root_account_id" {
  type        = number
  description = "Atlas AWS root account ARN IAM account id"
  default     = "536727724300"
}

variable "aws_root_account_id" {
  type        = number
  description = "Atlas AWS root account ARN IAM account id"
  default     = "521195893806"
}

variable "atlas_external_ids" {
  type        = list(any)
  description = "List of unique external IDs (per-Atlas project)"
  default     = []
}

main.tf

terraform {
  required_providers {
    mongodbatlas = {
      source = "mongodb/mongodbatlas"
      version = "0.9.1"
    }
  }
}

provider "mongodbatlas" {
  # Configuration options
  public_key  = var.mongodb_atlas_api_pub_key
  private_key = var.mongodb_atlas_api_pri_key
}

resource "mongodbatlas_cloud_provider_access" "test_role" {
  project_id    = "${var.mongodb_atlas_project_id}"
  provider_name = "AWS"
}

data "aws_iam_policy_document" "atlas-assume-role-policy" {
  statement {
    sid     = "rolepolicy"
    actions = ["sts:AssumeRole"]

    condition {
      test     = "StringEquals"
      variable = "sts:ExternalId"
      values   = ["${mongodbatlas_cloud_provider_access.test_role.atlas_assumed_role_external_id}"]
    }

    principals {
      type        = "AWS"
      identifiers = ["arn:aws:iam::${var.atlas_aws_root_account_id}:root"]
    }

    principals {
      type        = "Service"
      identifiers = ["ec2.amazonaws.com"]
    }
  }
}

resource "aws_iam_role" "atlas-cmk-access-role" {
  name               = "${var.account_name}-atlas-cmk-${var.atlas_project_name}-role"
  tags               = merge({ "Name" = "${var.account_name}-atlas-cmk-role" }, var.tags)
  assume_role_policy = data.aws_iam_policy_document.atlas-assume-role-policy.json
}

resource "mongodbatlas_cloud_provider_access_authorization" "auth_role" {
   project_id =  mongodbatlas_cloud_provider_access.test_role.project_id
   role_id    =  mongodbatlas_cloud_provider_access.test_role.role_id

   aws = {
      iam_assumed_role_arn = "arn:aws:iam::${var.aws_root_account_id}:role/${var.account_name}-atlas-cmk-${var.atlas_project_name}-role"
   }
}


output "atlas_assumed_role_external_id" {
  value = mongodbatlas_cloud_provider_access.test_role.atlas_assumed_role_external_id
}

Run

terraform init terraform plan ( check everything meets your expectations ) terraform apply

DEV Community: Naresh Maharaj

How to Copy GCP Storage Buckets Between Accounts Without Using the Internet

Prerequisites

Step 1: Create a Source Bucket in Account#1

Step 2: Set Up a Transfer Job in Account #2

Step 3: Grant Necessary Permissions

Step 4: Complete the Transfer

Step 5: Verify the Data

Conclusion

Implementing RBAC in Kubernetes: A Real-World Example

Implementing RBAC in Kubernetes: A Real-World Example

What is RBAC?

What is a Role?

What is a RoleBinding?

Real-World Example: A Python Server and Client in Kubernetes

Prerequisites

Clone the Git Repository to Follow Along

Server Code

Create the Docker Image for the Server

Client Code

Create the Docker Image for the Client

Running the Docker Containers

Testing the Setup with netcat

Kubernetes Environment using Kind

Deploying RBAC to Kubernetes

Server

Client

Testing the Client and Server

Using RBAC in Kubernetes

Step 1: Create a Role for Access

Step 2: Create a RoleBinding

Testing in Kubernetes

Python Kubernetes API Client

Conclusion

MongoDB Sharding - Single Node Macbook - Quick Setup

MongoDB Sharding - Single Node Macbook - Quick Setup

Overview

Replica Set

Sharded Cluster

VPC Peering, Split Brain with Distributed Cross Region NoSQL-DB

Network Partitioning - confidence and knowledge read.

Motivation

Overview

Part 1: Talking Cross Region

VPC in Region London 🏴󠁧󠁢󠁥󠁮󠁧󠁿 󠁧󠁢󠁥󠁮󠁧󠁿󠁧󠁢󠁥

EC2 Host in Region London 🏴󠁧󠁢󠁥󠁮󠁧󠁿

VPC in Region Paris 🇫🇷

EC2 Host in Region Paris 🇫🇷

Network Peering - Stretched Network

Messaging 'chat' (using nc Netcat)

Part 2: Aerospike NoSQL DB Stretch Cluster

Create Database Hosts

EC2 Database Hosts Region London 🏴󠁧󠁢󠁥󠁮󠁧󠁿

EC2 Database Hosts Region Paris 🇫🇷

Installing Aerospike NoSQL DB

ACL User Authentication

Strong Consistency

Edit RACK-IDs

Part 3: Insert some records

Part 4: Split Brain

Understanding Strong Consistency in Aerospike

Key Features of Strong Consistency

Evaluating Strong Consistency in Split-Brain Scenarios

How to create the Split Brain

Restoring Network Partition Configuration

Part 5: Understanding Partition Management in Strong Consistency

Key Points About Aerospike’s Partition Management

Finally: What's next

Exploring Different Network Partition Scenarios

Automating Network Partition Scenarios

Future: Service Mesh and Kubernetes

Contact and Feedback

Conclusion

How to use the Kubernetes Operator to establish connectivity between clusters

Setup

Destination Cluster

XDR-Proxy

Configuration

Deployment

Source cluster

Testing the Setup with `netcat`

Messaging 'chat' (using `nc` Netcat)

Aerospike Database configuration file for use with `systemd`