DEV Community: Aditya Kanekar

Installing SciSpacy on Apple M1 Laptops

Aditya Kanekar — Mon, 11 Jul 2022 03:09:39 +0000

Background

I was working on developing a API based on SciSpacy to search for medical terms. Looking at the slow performance of loading the model and entity linker on the Intel i5 chip on my Lenovo laptop, I was curious to check how the Apple M1 chip would perform since I own a MacBook Air with M1 chip with following specifications,

RAM - 16GB
Storage - 512GB SSD
Apple M1 chip
- 8-core CPU with 4 performance cores and 4 efficiency cores
- 8-core GPU
- 16-core Neural Engine

The Process

Following are the dependencies from requirement.txt file for SciSpacy and the en_core_sci_lg model,

scispacy==0.5.0
https://s3-us-west-2.amazonaws.com/ai2-s2-scispacy/releases/v0.5.0/en_core_sci_lg-0.5.0.tar.gz

The ** pip install -r requirement.txt ** failed with lot of errors while installing a dependency ** nmslib ** with an error ** clang compiler does not support '-march=native' **. Since I already had the Xcode installed on my Mac the version of C was 11 but somehow the mentioned command was failing, since the said argument was not supported on the C version. With a lot of digging around I finally found a solution mentioned on SciSpacy GitHub page.

Install the nmslib using following command,

CFLAGS="-mavx -DWARN(a)=(a)" pip install nmslib

After running this command all the packages got installed as expected and I was able to load and run the queries on SciSpacy model.

Next I will be coming up with the performance comparison on Apple M1 vs Intel i5 chip. I understand the comparison isn't fair but it just shows how much improved the M1 Chip is over the Intel processors.

Tips for CKAD Exam

Aditya Kanekar — Sun, 15 Aug 2021 18:22:27 +0000

In the last post I talked about How to prepare for CKAD exam. In this post we will talk about tips and tricks for passing the CKAD exam.

Setup your Kubernetes environment

Before starting the exam, spend few mins to setup the environment which will help you save time later.

Setup autocompletion

Ref - https://kubernetes.io/docs/reference/kubectl/cheatsheet/

Run following statements on the shell to setup autocompletion for kubectl commands,

source <(kubectl completion bash) 

# Only if you want to add this to your bashrc profile
echo "source <(kubectl completion bash)" >> ~/.bashrc

alias k=kubectl
complete -F __start_kubectl k

Now you can execute commands on shell using alias k instead of kubectl e.g.

k get pods

Also you can now press tab after writing first 2-3 letters of your sub commands to complete the command. This will save you sometime in the exam, remember the time is limited and saving time typing each command will add up to a significant saving in time.

Setup alias for frequently used commands

Setup alias or environment variables for frequently used commands, I personally used following alias's and environment variables.

alias kg='k get po -o wide --show-labels'

Environment variables

dr='--dry-run=client -o yaml'
fd='--force --grace-period=0'

The first command can be used with k run command to generate yaml definition like,

k run nginx --image=nginx $dr > nginx.yaml

The second command can be used with k delete po to delete pod's quickly. As part of the exams, there will be lot of questions which will require to create/delete pods. To delete pod quickly you can use the second environment variable like,

k delete po nginx $fd

Use imperative commands

It's difficult to create the YAML definition files required for deploying resources like PODs, Deployments, ReplicaSets etc. by creating YAML definition files manually. Copying the files manually from the Kubernetes documentation is one way, but its time consuming and exam time is limited.

There is a better way for creating these complex YAML definition files using kubectl imperative commands. I am listing down few commands for creating resources like POD, Deployments, Service etc.

Create POD

Ref - https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#run
Use the below command to generate the Pod definition file,

k run <pod-name> --image=<image-name> --dry-run=client -o yaml > <file-name>.yaml

#or if you setup the environment variable simply run

k run <pod-name> --image=<image-name> $dr > <file-name>.yaml

Following command demonstrates how you can use different option to quickly create a Pod or YAML definition file without creating the YAML file at all,

k run nginx --image=nginx --labels tier=webserver --env="key=value" --port=8080 --requests="cpu=200m,memory=256Mi" --limits="cpu=250m,memory=512Mi" --command -- echo "test" 

# If you want to save the definition to a file then use
k run nginx --image=nginx --dry-run=client -o yaml --labels tier=webserver --env="key=value" --port=8080 --requests="cpu=200m,memory=256Mi" --limits="cpu=250m,memory=512Mi" --command -- echo "test" > nginx.yaml

k apply -f nginx.yaml

Make sure to use '--dry-run=client -o yaml' before command, also you can use the environment variable $dr with this command. Also if you want to expose the pod as a service you can suffix --expose to expose the Pod as a service.

Create Deployment

To create deployment definition or deployment run command below,

k create deploy <deployment-name> --image=<image> --replicas=4  $dr > <filename>.yaml

# Nginx deployment sample
k create deploy nginx --image=nginx --replicas=4 $dr > nginx.yaml

k apply -f nginx.yaml

Exposing Service

To expose the deployment as a service run below command,

k expose deployment nginx --image=nginx --replicas=4 --port=8080 --target-port=8080 --name=nginx-service --type=ClusterIP $dr > service.yaml

# for pod
k expose pod nginx --image=nginx --replicas=4 --port=8080 --target-port=8080 --name=nginx-service --type=ClusterIP $dr > service.yaml

To create a NodePort service however you will have to run the above command and add nodePort in the YAML file.

Small but significant things

Testing your deployments

Always make sure to test the deployments, to be sure to not miss on any requirements. To test the deployments you can run temporary pods by running below command,

k run temp --image=nginx:alpine --restart=Never --rm -i -- curl -m 5 nginx-service:8080

If the service is in a different namespace you can specify namespace like,

k run temp --image=nginx:alpine --restart=Never --rm -i -- curl -m 5 nginx-service:8080 -n 
<namespace>

Or specify the name as . like,

k run temp --image=nginx:alpine --restart=Never --rm -i -- curl -m 5 nginx-service.<namespace>:8080

Get CPU/Memory usage for node/pod

To get the CPU/Memory usage by node/pod run,

k top node

#for pods run
k top pod

Using Init Containers

Init containers run un before the app containers are started. You can learn more about Init Containers here

Multi-Container Pod or Sidecar

Multi-container POD is simply a mode with more than one containers. Following YAML shows example of a multi-container POD also called as Sidecar,
Ref - https://kubernetes.io/docs/tasks/access-application-cluster/communicate-containers-same-pod-shared-volume/

apiVersion: v1
kind: Pod
metadata:
  name: two-containers
spec:

  restartPolicy: Never

  volumes:
  - name: shared-data
    emptyDir: {}

  containers:

  - name: nginx-container
    image: nginx
    volumeMounts:
    - name: shared-data
      mountPath: /usr/share/nginx/html

  - name: debian-container
    image: debian
    volumeMounts:
    - name: shared-data
      mountPath: /pod-data
    command: ["/bin/sh"]
    args: ["-c", "echo Hello from the debian container > /pod-data/index.html"]

Reading logs from Pod

To read logs from POD run,

k logs <pod_name>

To see running logs run,

k logs -f <pod_name>

To read logs from multi-container Pod,

k logs -f <pod_name> -c <container_name>

Making changes to the existing deployments

You can make changes to the existing deployments by simply editing them like,

k edit pod <pod_name>

This will open the Pod definition (YAML) in Vim editor. Some changes can be done without recreating the pod but changes like changing the image name for example, requires you to delete and recreate the Pod. The trick to make such changes are to edit the Pod and try to save the changes, this will prompt you to create a new temporary file, just hit ':wq' to save the changes to the file. You can then delete the old pod and use this temporary file to apply the changes. The trick here is to save as much time as possible to cover all the questions.

So thats it for this post, I will keep updating this post with more information. Till then, happy practicing!

Preparing for CKAD exam

Aditya Kanekar — Tue, 03 Aug 2021 20:31:09 +0000

Background

I was planning for my next certification and I was thinking of either taking cloud certification like AWS or GCP, but then I recollected my discussion with one of my colleague who just passed CKAD exam and he also suggested me to go for it. Since, we were already working on Kubernetes, it would be a little easy to get certified. Also, Kubernetes has no dependency on one specific cloud platform. Every cloud platform has its own version of managed Kubernetes platform e.g. Azure offers AKS, AWS offers EKS and GCP offers GKE. So the certification is independent of the cloud providers, that means this certification will still hold value against the majority of cloud platforms.

I recently passed my CKAD certification and I am sharing my experience in this post.

The certification has been developed by the Cloud Native Computing Foundation (CNCF), in collaboration with The Linux Foundation.

Prepare for the exam

In this section we will look at what it takes to prepare for the exam.

Get trained for the certification

If you have basic understanding of Kubernetes look no further start with Mumshad Mannambeth's course (KodeKloud) on Udemy,

Certified Kubernetes Application Developer - By KodeKloud

This exam curriculum includes these general domains and their weights on the exam:

13% – Core Concepts
18% – Configuration
10% – Multi-Container Pods
18% – Observability
20% – Pod Design
13% – Services & Networking
8% – State Persistence

This course covers all the topics required by the exam in detail with hands-on lab, lightning labs and mock exams. You will get access to labs on Kode Kloud after you purchase the course and you can practice as many times as you like for the exam.

However if you are at beginner level and want to get comfortable with Kubernetes you can start with,

Kubernetes for the Absolute Beginners - Hands-on

Practice Exams/Labs

KodeKloud provides you with hands-on labs, lightning labs and 2 mock exams (2 parts with 10 questions each). You can take the labs as many times as you like. The labs simulates all real life scenarios you will face while working with Kubernetes and that's what the exam is all about.

In addition to the mock exams you should practice CKAD-Excercises. This one is pretty extensive and covers all the topic in details. The good thing about this lab is that it provides you alternate approaches to implement the same problem. The lab also covers how to validate the answers (implementation) which is one important aspect of the exam. You should be able to test what you implemented.

Along with this you can also practice on killer.sh its a paid lab and it will cost you around 29.99 €, you do normally get 10% discount after you register with them (Register and wait for a day :p). However I will advice you not to buy this exam separately as it comes complementary as a exam simulator with CKAD Certification Exam. Killer.sh will provide you 2 sessions, each session will be valid for 36 hours once you start the session. The session gives you a mock exam with 20 questions with a timer of 120 mins to pass the mock exam. You can restart your session as many times as you like in the 36 hours, at the end of the exam you will get answers with explanations.

Each of the practice labs I suggested above will help you learn smart ways of implementing and validating your answers.

Exam Tips

Practice, practice and practice

This is a hands-on exam, you have to solve 19 questions in 120 minutes. The time just flies when you are giving the exam, so there is no time to relax. The only way to get around this exam is to practice every day and make sure you complete the mock exams from Kode Kloud in half time, once you reach there you are ready for the exam.

Don't rely on the Kubernetes documentation

This is a open book exam. You are allowed to open one extra tabs with following sites,

https://kubernetes.io/docs/ and their subdomains
https://github.com/kubernetes/ and their subdomains
https://kubernetes.io/blog/ and their subdomains

If he documentation is allowed then why I am saying don't rely on it? No, the documentation is not wrong, the time is short. You can't be searching for documentation on Kubernetes documentation site while in the exam. Its too time consuming and you will loose time quickly.

So what's the best way?
The easiest thing to do is to use imperative commands as much as possible. However resources like Persistent Volumes, Network policies or Ingress is not available as imperative commands. So, its best to rely on documentation in this case.

Always check the weightage for the question

Some of the questions might carry lower weightage and can have multiple things to do like create persistent volume, create volume claim and mount it to a pod. Asses the complexity of the question against weightage, if it takes more time to execute flag it and move to next question and come back to it later. The trick here is to cover maximum questions in less time and then come back to the flagged questions.

I had a similar problem where one question had a weightage of 2% but I was not sure how to answer it. So I flagged it and came back to it later, searched for the documentation and answered the same.

Don't spend too much time on one question

Remember this you should not spend too much time on one question, if you are stuck on a question and you are not sure don't panic just move to the next question. Yes, I know everybody wants high score but you need to pass the certification as well, so just move ahead flag the question, move ahead and then come back to the flagged questions.

Bookmark important links

When you are practicing make sure you bookmark the links you are referring for solving the question. The best place to bookmark is on the bookmark bar create a folder and start bookmarking all the documentation links there. This makes it easy to access during the exam.

You can import bookmarks from Kubernetes. However you should also create your own version of the bookmarks, it helps you in locating the required documents faster.

Also, bookmark documentation which has all things you require to complete a solution in one place e.g. Create persistent volume, volume claim, mount in pod.

https://kubernetes.io/docs/tasks/configure-pod-container/configure-persistent-volume-storage/

This link has all the things required to solve such scenario in one place.

Use one YAML file with multiple definitions

Try to use one YAML files separated by --- to store different definitions related to the solution e.g.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: task-pv-volume
  labels:
    type: local
spec:
  storageClassName: manual
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/mnt/data"
---
apiVersion: v1
kind: Pod
metadata:
  name: task-pv-pod
spec:
  volumes:
    - name: task-pv-storage
      persistentVolumeClaim:
        claimName: task-pv-claim
  containers:
    - name: task-pv-container
      image: nginx
      ports:
        - containerPort: 80
          name: "http-server"
      volumeMounts:
        - mountPath: "/usr/share/nginx/html"
          name: task-pv-storage
---
apiVersion: v1
kind: Pod
metadata:
  name: task-pv-pod
spec:
  volumes:
    - name: task-pv-storage
      persistentVolumeClaim:
        claimName: task-pv-claim
  containers:
    - name: task-pv-container
      image: nginx
      ports:
        - containerPort: 80
          name: "http-server"
      volumeMounts:
        - mountPath: "/usr/share/nginx/html"
          name: task-pv-storage

Practice your Vim skills

Vim is the default editor for editing Kubernetes definitions. Make sure you learn the shortcuts to implement the solution faster. You can refer Vim Crash Course to learn more about the shortcuts.

This is a pretty long post, read my next post for Tips for CKAD exam. Till then, happy learning!

Look ma, no Zookeeper

Aditya Kanekar — Mon, 03 May 2021 20:08:13 +0000

Background

Kafka always had dependency on Kafka since its inception for managing topics, managing ACLs, partitions, leader elections etc. While this is not a big issue you still need to maintain a quorum of Zookeeper (3 servers - recommended) for making sure Kafka runs smoothly in a clustered environment. That means maintaining a Zookeeper quorum which translates into running 3 instances of Zookeeper which adds to infrastructure cost as well as one more thing to maintain for DevOps.

Also Zookeeper had its share of limitations w.r.t. how much data you can store on each ZNode which is important if you are using ACLs on Kafka (details coming soon on this issue). Ever since KIP-500 was added to Kafka backlog, everyone was waiting for Kafka version which gets rid of Zookeeper. Kafka team had started moving in this direction from quite sometime now, with the new Kafka Admin API introduced in 2.1.0 version, it featured features like topic crating, adding/removing ACLs, deleting topics through Kafka without connecting to Zookeeper.

The day has now come with the introduction of Kafka 2.8 which gives us preview of Kafka without Zookeeper. *Please be aware that this feature is not intended to be used in production, Kafka team is still working on putting some missing pieces of the puzzle.

Getting started

Before getting started download Kafka binaries,

wget https://downloads.apache.org/kafka/2.8.0/kafka_2.13-2.8.0.tgz
tar -xzvf kafka_2.13-2.8.0.tgz
mv kafka_2.13-2.8.0 kafka
cd kafka

Step 1 - Generate cluster id

To get started for running Kafka without Zookeeper, a new cluster uuid needs to be generated.

./bin/kafka-storage.sh random-uuid

Step 2 - Format storage directories

The next step is to format storage directories, you will need to run this command on every broker needless to say you will run this command only on one broker if you are running in single mode. This step is new if you have worked with previous version of Kafka.

./bin/kafka-storage.sh format -t <uuid> -c ./config/kraft/server.properties

Replace with the id generated in Step bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic test-topic1.

Step 3 - Start Kafka Server

./bin/kafka-server-start.sh ./config/kraft/server.properties

There you go, if you have followed all the steps you should see Kafka running. Lets create a topic and send/receive messages to/from the topic.

Create Topic,

./bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic test-topic --partitions 1 --replication-factor 1

Run Kafka producer and send some messages,

./bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic test-topic

Run Kafka consumer and verify if the messages are flowing through,

./bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test-topic --from-beginning

There you go! If all worked well, you should see messages on the test-topic.

Make your home smart, the easy way!

Aditya Kanekar — Tue, 06 Apr 2021 08:13:22 +0000

For so many days I was playing with Raspberry Pi, ESP8266 and relay board to control the devices at home remotely through mobile app and digital assistants like Alexa/Google Assistant. I was able to achieve the same using Raspberry Pi and Relay control and using Alexa skill to remotely turn the devices on/off using little bit of coding in Python.

However I am not a expert in electronics and I wanted something like plug and play device which can work with existing board. Now you might be thinking that I can easily do it with Raspberry Pi/ESP8266/Arduino and relay board. Yes I agree that its not that complicated, however there are some practical issues for a guy like me who has not played with electric boards and circuits.

Issues

Power supply The controller unit (Raspberry Pi/ESP8266/Arduino) will require continuous power supply to power itself. This can be resolved by designing a board with the power supply built-in which will work with direct power supply from the board or may be a mobile charger can power the board. However for me designing this circuit will be a tedious process, also packaging mobile charger inside the board is not practical.
Packaging Fitting the whole assembly Raspberry Pi/Esp8266 with Relay inside the switch board with all those floating wires connecting to the pins is messy.
Additional coding Additional coding is required for this to work with Alexa/Google Assistant. However this is trivial for me as I had done the integration with Alexa.

So then what are the options. So while watching a video on YouTube by TechBurner came across device which you can fit inside your switch board with existing manual switches. Wow, how cool is that? That's what I wanted. So then I started looking for the gadgets on Amazon and I came across two three options and based on the review I choose

Image Ref - tinxy.in

Tinxy Smart Switch. It comes in various specifications I wanted to control 2 fans and 2 lights in our living room so I chose 4-Node Smart Switch. Lets look at how to connect the device in the switch board, shall we?

Installation Diagram

As shown in the diagram above, the installation is very simple. This can be done in 4 steps. But before that make sure you have spare red and black wires for doing the connection.

Steps

Remove the phase wire (Wire going to your device - fan/light etc.) from the switch. This is usually located at the top of the switch.
Attach this phase wire to the marking denoted as D1,D2,D3,D4 where D is Device 1
From the other side of Tinxy board (Colored black) by using a piece of wire - preferably black to the switch in the same place which was removed in step 1
Powering the device - from the wires take connection to Live and Neutral wire and connect it to the board to L - Live and N - Neutral terminal

You will see the Smart switch will switch on green light will start blinking that means the device is not connected to WiFi. At this point you need to download Tinxy app to your mobile and follow the instructions in the app to complete registraions, set WiFi credentials in the smart switch and configure the connected devices. Try switching devices on/off. Also you will notice that if the device is manually switched off it will show in the app. The other cool thing about this Smart Switch is, it converts your normal switch to two-way switch. So the position of the switch doesn't matter. How cool is that?

Connecting with Alexa

Now assuming you have completed the above steps, this step is pretty straight forward. In your Alexa app, go to Skills and search for Tinxy. Locate the Tinxy skill and proceed with the sign-in process and that's it you are good to go. You will now see all the devices which you configured in the Tinxy App, as a next step you can change the settings of the devices to rename them and add them to a group for e.g. Living room. This makes it simple for Alexa to find the right devices when you give command like 'Alexa Switch on living room light 1'.

Word of caution

Before touching the board make sure you have a tester, to check the live/neutral wires
Before doing the actual connections switch of the main supply
Make sure you don't expose wires accidently as it might cause short circuit.

In the next article we will see how I connected my Smart AC to Alexa. Till then happy coding.

Connecting to Kafka cluster using SSL with Python

Aditya Kanekar — Tue, 19 Jan 2021 14:47:20 +0000

This article specifically talks about how to write producer and consumer for Kafka cluster secured with SSL using Python. I won't be getting into how to generate client certificates in this article, that's the topic reserved for another article :).

Pre-Requisites

Kafka Cluster with SSL
Client certificate (KeyStore) in JKS format
Linux environment with keytool and openssl installed
Python 3.6

Step 1 - Converting JKS to PEM file

Why I need this step?

Unlike Java, Python and C# uses .pem files to connect to Kafka. For this purpose we will have to convert the JKS files to PEM with the help of keytool and openssl commands. If you are working on Windows 10 you can refer to my article on how to run WSL on Windows here.

To make your life easy I have created a shell script to quickly convert JKS to PEM.

#!/bin/bash
srcFolder=$1
keyStore=$1/$2
password=$3
alias=$4
outputFolder=$5

echo $keyStore
echo "Generating certificate.pem"
keytool -exportcert -alias $alias -keystore $keyStore -rfc -file $outputFolder/certificate.pem -storepass $password

echo "Generating key.pem"
keytool -v -importkeystore -srckeystore $keyStore -srcalias $alias -destkeystore $outputFolder/cert_and_key.p12 -deststoretype PKCS12 -storepass $password -srcstorepass $password
openssl pkcs12 -in $outputFolder/cert_and_key.p12 -nodes -nocerts -out $outputFolder/key.pem -passin pass:$password

echo "Generating CARoot.pem"
keytool -exportcert -alias $alias -keystore $keyStore -rfc -file $outputFolder/CARoot.pem -storepass $password

The script generates following files from the keystore file,

key.pem
certificate.pem
CARoot.pem

How to run this script?

Save the script in a file e.g. jkstopem.sh and give execute permissions like below,

chmod +x jkstopem.sh

To generate the PEM files. Run the shell script as shown in the below example,

./jkstopem.sh <source_path_to_jks> <keystore_file_name> <keystore_password> <alias> <output_folder>

How to find Alias?

If you are not aware of what alias your certificate has. Run following command in the folder where you have the keystore file.

keytool -list -v -keystore kafka.client.keystore.jks

You will be prompted to enter the password. Enter the keystore password, this will list the contents of the keystore file. You will be able to see *Alias name.

Following is the example to run the shell script,

./jkstopem.sh ~/client-cert kafka.client.keystore.jks welcome123 client-alias ~/client-cert/pem

Now you should be able to see following files in the output folder,

key.pem
certificate.pem
CARoot.pem

Now as we have all the PEM files, lets get cracking.

Step 2 - Writing Kafka Producer in Python

We will be using 'kafka-python' package to connect to Kafka. You can install it using pip,

pip install kafka-python

Now, lets write our producer.

#Producer.py
from kafka import KafkaProducer

kafkaBrokers='kafka1.xyz.com:443,kafka2.xyz.com:443,kafka3.xyz.com:443'
caRootLocation='CARoot.pem'
certLocation='certificate.pem'
keyLocation='key.pem'
topic='test-topic'
password='welcome123'

producer = KafkaProducer(bootstrap_servers=kafkaBrokers,
                          security_protocol='SSL',
                          ssl_check_hostname=True,
                          ssl_cafile=caRootLocation,
                          ssl_certfile=certLocation,
                          ssl_keyfile=keyLocation,
                          ssl_password=password)

producer.send(topic, bytes('Hello Kafka!','utf-8'))

# Send to a particular partition
producer.send(topic, bytes('Hello Kafka!','utf-8'),partition=0)
producer.flush()

In the above example we are using the pem files we generated in the last step with the password to read the pem file.

kafkaBrokers='kafka1.xyz.com:443,kafka2.xyz.com:443,kafka3.xyz.com:443'
caRootLocation='CARoote.pem'
certLocation='certificate.pem'
keyLocation='key.pem'
password='welcome123'
producer = KafkaProducer(bootstrap_servers=kafkaBrokers,
                          security_protocol='SSL',
                          ssl_check_hostname=True,
                          ssl_cafile=caRootLocation,
                          ssl_certfile=certLocation,
                          ssl_keyfile=keyLocation,
                          ssl_password=password)

Sending data to random topic partition

Below code snippet will send data to random partition decided by Kafka.

producer.send(topic, bytes('Hello Kafka!','utf-8'))
producer.flush()

Sending data to specific topic partition

To send data to a specific partition, you just need to specify the partition as shown in below code snippet,

producer.send(topic, bytes('Hello Kafka - Partition 0!','utf-8'),partition=0)
producer.flush()

So we have built our Python producer for Kafka. In the next part we will write consumer to consume the message from the topic.

Creating local Redis cluster in minutes on WSL2

Aditya Kanekar — Sun, 22 Nov 2020 12:37:00 +0000

This article focuses on setting up Redis cluster quickly on local environment using WSL2 (If you have not set it up on your Windows 10 machine, you can refer my post). However this is completely optional and the same should work on other Ubuntu Distros.

Install Redis on Ubuntu

To install Redis on Ubuntu we will launch Ubuntu 18.04 WSL instance on the Windows 10 machine. This step is completely optional and you can also use the same steps on any Ubuntu 18.04 instance.

Before installing Redis we need to install few dependencies. Run following command in the bash to install the dependencies required for running Redis.

sudo apt-get update
sudo apt install make gcc libc6-dev tcl build-essential

Now lets download Redis from the official Redis channel,

cd ~
wget http://download.redis.io/redis-stable.tar.gz
tar xvzf redis-stable.tar.gz

Install Redis build dependencies

cd redis-stable/deps
make hiredis lua jemalloc linenoise

This will take some time, once the above step finishes run make command in the redis-stable directory as below,

cd ~/redis-stable
make

Make command will also take some time to complete, once its complete you can run 'make test' command to see if everything is setup properly. This is a optional step, if the make is completed successfully you can skip it.

Create cluster

There are lot of articles available for setting up the cluster manually. However Redis has a utility to create a cluster quickly which will get you started in minutes. To create cluster run following command,

cd ~/redis-stable/utils/create-cluster
./create-cluster start

The start command will create the cluster with port ranging from 30001 to 30006. The output should look like below,

Starting 30001
Starting 30002
Starting 30003
Starting 30004
Starting 30005
Starting 30006

This will start 6 instances of Redis, now to create a cluster run,

./create-cluster create

This command will create the cluster with 3 master and 3 slaves configuration. It will also display you hash slots for all 3 masters and the slave instance mapping with master instance. Type yes in the prompt to continue.

>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 127.0.0.1:30005 to 127.0.0.1:30001
Adding replica 127.0.0.1:30006 to 127.0.0.1:30002
Adding replica 127.0.0.1:30004 to 127.0.0.1:30003
>>> Trying to optimize slaves allocation for anti-affinity
[WARNING] Some slaves are in the same host as their master
M: 8106f2118a007250a1b003c01e3d84dff87e7117 127.0.0.1:30001
   slots:[0-5460] (5461 slots) master
M: 7f0c05048e8864334de8a24de1318327c11a1b8d 127.0.0.1:30002
   slots:[5461-10922] (5462 slots) master
M: f5d904e68e316d0c310cb7acb4201e5ad9539202 127.0.0.1:30003
   slots:[10923-16383] (5461 slots) master
S: e5ccc0ac06c1921f70530fa01c62ba2b6de441a3 127.0.0.1:30004
   replicates f5d904e68e316d0c310cb7acb4201e5ad9539202
S: a140fe7519762b1f2c35e06f8e5c46487bad1f6c 127.0.0.1:30005
   replicates 8106f2118a007250a1b003c01e3d84dff87e7117
S: 63d7a31d2820e346eec839470fc3ab5cafd25cc7 127.0.0.1:30006
   replicates 7f0c05048e8864334de8a24de1318327c11a1b8d
Can I set the above configuration? (type 'yes' to accept):

This will create the cluster and you should be able to see output as below,

>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
.
>>> Performing Cluster Check (using node 127.0.0.1:30001)
M: 8106f2118a007250a1b003c01e3d84dff87e7117 127.0.0.1:30001
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
M: 7f0c05048e8864334de8a24de1318327c11a1b8d 127.0.0.1:30002
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
M: f5d904e68e316d0c310cb7acb4201e5ad9539202 127.0.0.1:30003
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
S: 63d7a31d2820e346eec839470fc3ab5cafd25cc7 127.0.0.1:30006
   slots: (0 slots) slave
   replicates 7f0c05048e8864334de8a24de1318327c11a1b8d
S: e5ccc0ac06c1921f70530fa01c62ba2b6de441a3 127.0.0.1:30004
   slots: (0 slots) slave
   replicates f5d904e68e316d0c310cb7acb4201e5ad9539202
S: a140fe7519762b1f2c35e06f8e5c46487bad1f6c 127.0.0.1:30005
   slots: (0 slots) slave
   replicates 8106f2118a007250a1b003c01e3d84dff87e7117
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

Connecting to Redis Cluster

To connect to Redis cluster using redis-cli, open another bash window and cd into the redis-stable directory.

cd ~/redis-stable/src
./redis-cli -c -p 30001

You should now be connected to the Redis cluster. The output will be similar to,

127.0.0.1:30001>

Testing the cluster

To set key run,

set foo bar

To retrieve the value of the key run,

get foo

The get command will show output as "bar".

So your Redis cluster is now setup. If you want to stop your cluster run,

./create-cluster stop

You will notice all the nodes listening from 30001-30006 have stopped.

Stopping 30001
Stopping 30002
Stopping 30003
Stopping 30004
Stopping 30005
Stopping 30006

To delete the cluster run,

./create-cluster stop
./create-cluster clean

Now if you run ls you will notice that the folder contains only two files,

README
create-cluster

Now to run a new cluster you can run the start and then create command as demonstrated earlier in this article.

Happy coding!

Creating base docker image for hosting your Java 8 application

Aditya Kanekar — Sun, 22 Nov 2020 12:25:34 +0000

Creating clean image is very important from trusted sources. This article demonstrates how you can create a docker image for hosting your Java 8 application using Alpine Linux distro.

Dockerfile for Java8

FROM alpine

RUN apk update && \
    apk upgrade 
RUN apk add openjdk8=8.252.09-r0

ARG JAR_FILE=build/libs/*.jar
COPY ${JAR_FILE} app.jar
EXPOSE 8080
ENTRYPOINT ["java","-Djava.security.egd=file:/dev/./urandom","-jar","/app.jar"]

The above Dockerfile creates image for hosting Java8 application using alpine distro. Lets go through each instruction one by one,

FROM alpine -
FROM instruction gets the baseimage for your application which in this case is alpine. You can also specify tags if you want a specific image of alpine e.g. alpine:3.9.

RUN apk update && apk upgrade
Runs update and upgrade instructions to update the sources and install any upgrade available.

RUN apk add openjdk8=8.252.09-r0
Installs specific version of JDK 8 on alpine.

ARG JAR_FILE=build/libs/*.jar
Argument for getting all jar files

COPY ${JAR_FILE} app.jar
Copy the jar file in the docker image

EXPOSE 8080
Exposes port 8080 on the docker image, if your application should be accessible on this port. (This is only applicable if your application should be accessible on certain port e.g. REST API)

ENTRYPOINT ["java","-Djava.security.egd=file:/dev/./urandom","-jar","/app.jar"]

This instruction is responsible for running your application. So that's how you can create a lightweight clean image for your Java8 application. We have not touched on the safety aspect of this image, we will see how to secure this image in our next article.

Happy coding!

Upgrading from WSL1 to WSL2

Aditya Kanekar — Thu, 19 Nov 2020 17:45:49 +0000

Background

In my last post I talked about how to get started with developing applications on Linux on Windows 10 using WSL 2 and VS Code. Some of you might have installed the previous version of WSL1 on your Windows 10 machines. This article guides on how to upgrade from WSL to WSL2.

Install WSL2 Linux Kernel

You need at least 1903 version of Windows 10. You can check the minimum requirement for upgrading from WSL1 here To check windows version, click Windows + R Key and type,



winver

You will be able to see the build version of your windows e.g. 1903.

Download the installer for the WSL2 kernel from,
https://docs.microsoft.com/en-us/windows/wsl/wsl2-kernel

Run the installer to install WSL2 Kernel for Linux. Once the installation is successful. Open PowerShell or command prompt and run,



wsl --list

This will list all the distros and the version of WSL in the version column. The list will look somewhat similar to this,

Since you were running WSL1 previously on your machine the distro is still using WSL1. No magic tricks here, none of your distros will be migrated to WSL2 by default.

Upgrading from WSL1 to WSL2

To upgrade your current distro(s) from WSL1 to WSL2. Run following command in PowerShell or command prompt,



wsl --set-version <distro-name> 2

Replace the distro-name with the Name displayed in the last section. So in our case the command will look like,



wsl --set-version Ubuntu-20.04 2

This will take some time to upgrade, in my experience all your data will be retained as it was previously. Once the upgrade is complete you can run the list distros command to verify whether the version was successfully upgraded.



wsl -l -v

You should be able to see output similar to,

Now you can launch Ubuntu-20.04 from Start menu or from the command prompt run,



wsl -d Ubuntu-18.04

This will take you to the Ubuntu-18.04 shell. The * denotes the default distribution. What does that mean? To check run following command,

wsl

This will take you to Ubuntu-20.04 shell without mentioning the distro. If you want to change the default distro run,



wsl -s Ubuntu-18.04
wsl -l -v

You will notice the default distribution changed to Ubuntu-18.04.

NAME STATE VERSION

Ubuntu-18.04 Stopped 2 Ubuntu-20.04 Stopped 2

Now if you run,

wsl

Ubuntu-18.04 shell will launch. To verify if the right version is launched. Run following command,



lsb_release -a

You should get output similar to,

No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.5 LTS
Release: 18.04
Codename: bionic

Now open another PowerShell or command prompt window and run,



wsl -l -v

You will notice Ubuntu-18.04 is in Running state. IF you close the wsl window or run exit the state will change to Stopped.

Hopefully this will get you started on WSL2. In my next articles we will use WSL for running tools supported on Linux environment.

Till then, happy coding!

Developing for Linux in Windows 10 with WSL 2 - Part 2

Aditya Kanekar — Sun, 15 Nov 2020 18:40:27 +0000

In the Part 1 we installed WSL 2 with Ubuntu 20.04 on our machine. To complete our development environment setup lets now install Dotnet Core on the newly installed VM. Launch Ubuntu 20.04 App from Start menu and follow along this article. We are going to install .Net Core 3.1.

wget https://packages.microsoft.com/config/ubuntu/20.04/packages-microsoft-prod.deb -O packages-microsoft-prod.deb
sudo dpkg -i packages-microsoft-prod.deb

sudo apt-get update; \
  sudo apt-get install -y apt-transport-https && \
  sudo apt-get update && \
  sudo apt-get install -y dotnet-sdk-3.1

Reference link for .Net Core installation,

Once installation is complete, run following command,

dotnet --version

You should be able to see installed .Net Core version 3.1.xxx.

Launching VS Code from Ubuntu Shell

mkdir HelloWorld
cd HelloWorld

dotnet new console

This will generate project for the console application. List the file in the directory by using ls command at the shell,

ls

You should be able to see two files in the folder,

HelloWorld.csproj -> Project file for console application
Program.cs

To check if your application is running properly, run following command,

dotnet run

This will print Hello World!

Congratulations, you have completed the first step. Now lets move to the interesting part, launching VS Code from Ubuntu app by entering following command,

code .

You should be able to see that VS Code is downloading itself and in a few minutes you should see VS Code launched in Windows 10 with the project we just created. Wow that's what I called magic!

If you see a prompt to install C# extensions, click Install to complete the installation. Now click Program.cs from the Explorer menu,

You might get a prompt install tools for debugging and building .Net project, click Install. Now right click the Program.cs file from Explorer menu and click Reveal in Explorer. This will open your folder in Explorer. You will notice that the path is from the Ubuntu app, it will be something like,

\\wsl$\Ubuntu-20.04\home\xxxx\HelloWorld

Running application in VS Code

To run your application Press F5, you will see Hello World! printed in the Debug Console window. The application is running in Windows 10 but on Ubuntu.

Debugging application in VS Code

To add a breakpoint, Press F9 inside the main method or alternatively you can click on the empty gutter on the left side. You should be able to see the familiar red-dot indicating the breakpoint is set. Now press F5 to launch the application, your application should pause at the breakpoint. You can press F10 to Step Over or F11 to Step Into the function.

So, there you have it you have successfully built a .Net Core application running on Ubuntu without leaving Windows 10.

Developing for Linux in Windows 10 with WSL 2 - Part 1

Aditya Kanekar — Sun, 15 Nov 2020 17:25:18 +0000

Background

Most of the open source software's are developed for Linux distros and this makes it difficult to run on Windows. Of course you can run a Ubuntu VM on Hyper-V or Virtual Box and install your development environment on the machine as you would on a regular machine.

However to run the VM's on your machine your machine should have enough juice to run the VM's smoothly and its not seamless either if you want to transfer files between your Windows host and your VM. You can also setup dual boot with Windows 10 and Linux, however maintaining different operating systems on a machine is a pain.

With WSL, installing Linux distros in Windows 10 is as easy as installing a app. You can share files easily between Windows and your Linux distro and you can also develop for Linux right inside Windows 10.

But I already have WSL1, can I upgrade?

Sure you can upgrade from WSL1 to WSL2, you can refer my post to upgrade from WSL1 to WSL2.

Windows Subsystem for Linux (WSL)

Most of you might have heard of Windows Sub-system for Linux AKA WSL. What it enables you to do is install Linux distros on your Windows 10 machine like a app from Windows 10. Yes, you heard it right install distros as app. However things are not that simpler if you have Windows 10 Home Edition since WSL uses Hyper-V to run a VM behind the scenes. The other problem is Docker or MicroK8S are not supported on WSL and this was a bit of a downer. However it was a step in the right direction from Microsoft.

WSL 2

Microsoft has resolved most of the issues which were bugging WSL with WSL 2. WSL 2 does uses VM behind the scenes but its managed, also it uses full Linux Kernel built by Microsoft unlike WSL. You can learn more about WSL here,

https://docs.microsoft.com/en-us/windows/wsl/compare-versions

Pre-Requisites

Installing / Upgrading WSL 2

You can refer the instructions for installing WSL 2 on the official Microsoft site.

https://docs.microsoft.com/en-us/windows/wsl/install-win10

VS Code

I will be using VS Code for this article and its my choice of IDE for development (I am obsessed with VS Code :p)

https://code.visualstudio.com/download

Install following plugins,

Remote WSL

Windows Terminal Preview - Optional

You can install Windows Terminal from here. This step is not necessary, however I will recommend you to try it.

https://www.microsoft.com/en-in/p/windows-terminal-preview/9n8g5rfz9xk3?rtc=1&activetab=pivot:overviewtab

Install Linux Distro from Windows store

For this article I have install Ubuntu 20.04LTS from Windows Store. Assuming you have completed above steps for installing WSL 2 on your machine. This step should be straightforward. Jut got to your Windows Store and search for Ubuntu. Install Ubuntu 20.04LTS.

Once you install, search for Ubuntu in your Windows Search bar and click it.

Since the app is being launched for the first time, just follow the installation steps. After installation you will be asked to create username and password, give username and password of your choice and you are all set.

Launching Ubuntu

Assuming you have completed all the steps, just search for Ubuntu or if you have installed Windows Terminal, select the shell you want to open as shown below,

Listing installed distros

Run following command in Powershell or command prompt to display all installed distros,

wsl -l -v

You should be able to see similar output as shown below,

 NAME                   STATE           VERSION
* docker-desktop         Running         2
  Ubuntu-20.04           Stopped         2
  docker-desktop-data    Stopped         2

In the next part we will create a sample dotnet core console app and debug it in Windows 10 while developing on Ubuntu-20.04.

Till then, Happy Coding !

How to pass arguments to Flink App

Aditya Kanekar — Wed, 21 Oct 2020 17:31:41 +0000

At first when I started writing Flink application I thought of it as any other Java console application and started writing the same using Spring Boot Framework. However I soon realized that running the application in Flink is a bit different than executing a Java application on say command line using

java -jar <jar_path>

My application was connecting to Kafka for reading stream from the topic and performing operation on the streams. Our Kafka cluster is secured using SSL and ACL that means I need certificate to read from the topic and publish the processed output on another topic and the certificates will change per environment. In the past I had written a Java consumer app which works on the similar principle but it was a containerized app and passing parameters to Docker container is simple enough using,

-e <env_variable>=<value>

However this was a Flink application and things are quite different since the Flink manager has all the controls to run the application. After breaking my head for several hour(s) I figured out that its like running any other Java app. Now the beauty of Java app is you can pass the properties you are reading from application.properties from the arguments without writing a single line of code as,

java -jar <jar_path> --<property_name>=<value>

So I thought why not try the same thing with Flink application after all its a Java app. So I ran below command,

~/flink/bin/flink run app.jar --brokers=Broker1:9093 --topic=some-topic

And to my surprise this worked and I could verify that by checking the logs on the Flink dashboard. Since this option worked I thought what if I pass the property and their values in a properties file, why not give it a try by running below command,

~/flink/bin/flink run app.jar -Dspring.config.name=<full-path>/application.properties

This also worked since I had implemented the application using Spring boot, I could pass the properties file with all the required values.

You can choose either of the method according to your preference and it should work seamlessly.

Happy Coding :)