DEV Community: Keyvan Soleimani

PySpark & Jupyter Notebooks Deployed On Kubernetes

Keyvan Soleimani — Mon, 05 May 2025 12:15:24 +0000

PySpark is an interface for Apache Spark in Python. It not only allows you to write Spark applications using Python APIs, but also provides the PySpark shell for interactively analyzing your data in a distributed environment.

Jupyter Notebook is used to create interactive notebook documents that can contain live code, equations, visualizations, media and other computational outputs. Jupyter Notebook is often used by programmers, data scientists and students to document and demonstrate coding workflows or simply experiment with code.

Kubernetes is an open-source container orchestration system for automating software deployment, scaling, and management. Originally designed by Google.

The kubernetes control-plane & worker nodes addresses are :

192.168.56.115
192.168.56.116
192.168.56.117

Kubernetes cluster nodes :

you can install helm via the link helm :

Install Spark On Kubernetes via bitnami helm chart

The Steps :

you can install helm chart via the link helm chart :

Important: the spark version of helm chart must be the same as the PySpark version of jupyter

Install spark via helm chart (bitnami) :

$ helm repo add bitnami https://charts.bitnami.com/bitnami
$ helm search repo bitnami
$ helm install kayvan-release bitnami/spark --version 8.7.2

Deploy Jupyter workloads :

jupyter.yaml :

apiVersion: apps/v1
kind: Deployment
metadata:
   name: jupiter-spark
   namespace: default
spec:
   replicas: 1
   selector:
      matchLabels:
         app: spark
   template:
      metadata:
         labels:
            app: spark
      spec:
         containers:
            - name: jupiter-spark-container
              image: docker.arvancloud.ir/jupyter/all-spark-notebook
              imagePullPolicy: IfNotPresent
              ports:
              - containerPort: 8888
              env: 
              - name: JUPYTER_ENABLE_LAB
                value: "yes"
---
apiVersion: v1
kind: Service
metadata:
   name: jupiter-spark-svc
   namespace: default
spec:
   type: NodePort
   selector:
      app: spark
   ports:
      - port: 8888
        targetPort: 8888
        nodePort: 30001
---
apiVersion: v1
kind: Service
metadata:
  name: jupiter-spark-driver-headless
spec:
  clusterIP: None
  selector:
    app: spark

kubectl apply -f jupyter.yaml

the installed pods:

and Services (headless for statefull) :

Note: spark master url address is :

spark://kayvan-release-spark-master-0.kayvan-release-spark-headless.default.svc.cluster.local:7077

Open jupyter notebook and write some python codes based on pyspark and press shift + enter keys on each block to execute:

#import os

#os.environ['PYSPARK_SUBMIT_ARGS']='pyspark-shell'
#os.environ['PYSPARK_PYTHON']='/opt/bitnami/python/bin/python'
#os.environ['PYSPARK_DRIVER_PYTHON']='/opt/bitnami/python/bin/python'

from pyspark.sql import SparkSession

spark = SparkSession.builder.master("spark://kayvan-release-spark-master-0.kayvan-release-spark-headless.default.svc.cluster.local:7077")\
            .appName("Mahla").config('spark.driver.host', socket.gethostbyname(socket.gethostname()))\
            .getOrCreate()

note:

socket.gethostbyname(socket.gethostname()) ---> returns the jupyter pod's ip address

enjoy from sending python codes to spark cluster on kubernetes via jupyter.

Note: of course you can work with pyspark single node installed on jupyter without kubernetes and when you will be sure that the code is correct, then send it via spark-submit or like above code to spark cluster on kubernetes.

Deploy on the Docker Desktop :

docker-compose.yml :

version: '3.6'

services:

  spark-master:
    container_name: spark
    image: docker.arvancloud.ir/bitnami/spark:3.5.0
    environment:
      - SPARK_MODE=master
      - SPARK_RPC_AUTHENTICATION_ENABLED=no
      - SPARK_RPC_ENCRYPTION_ENABLED=no
      - SPARK_LOCAL_STORAGE_ENCRYPTION_ENABLED=no
      - SPARK_SSL_ENABLED=no
      - SPARK_USER=root   
      - PYSPARK_PYTHON=/opt/bitnami/python/bin/python3
    ports:
      - 127.0.0.1:8081:8080
      - 127.0.0.1:7077:7077
    networks:
      - spark-network

  spark-worker:
    image: docker.arvancloud.ir/bitnami/spark:3.5.0
    environment:
      - SPARK_MODE=worker
      - SPARK_MASTER_URL=spark://spark:7077
      - SPARK_WORKER_MEMORY=2G
      - SPARK_WORKER_CORES=2
      - SPARK_RPC_AUTHENTICATION_ENABLED=no
      - SPARK_RPC_ENCRYPTION_ENABLED=no
      - SPARK_LOCAL_STORAGE_ENCRYPTION_ENABLED=no
      - SPARK_SSL_ENABLED=no
      - SPARK_USER=root
      - PYSPARK_PYTHON=/opt/bitnami/python/bin/python3
    networks:
      - spark-network


  jupyter:
    image:  docker.arvancloud.ir/jupyter/all-spark-notebook:latest
    container_name: jupyter
    ports:
      - "8888:8888"
    environment:
      - JUPYTER_ENABLE_LAB=yes
    networks:
      - spark-network
    depends_on:
      - spark-master


networks:
  spark-network:

run in cmd :

docker-compose up --scale spark-worker=2

Copy csv file to inside spark worker container :

docker cp file.csv spark-worker-1:/opt/file
docker cp file.csv spark-worker-2:/opt/file

Open jupyter notebook and write some python codes based on pyspark and press shift + enter keys on each block to execute:

from pyspark.sql import SparkSession

# Create a Spark session
spark = SparkSession.builder.appName("YourAppName")\
            .master("spark://8fa1bd982ade:7077").getOrCreate()

data = spark.read.csv("/opt/file/file.csv", header=True)
data.limit(3).show()

spark.stop()

Note again: you can work with pyspark single node installed on jupyter without spark cluster and when you will be sure that the code is correct, then send it via spark-submit or like above code to spark cluster on docker desktop.

Execute on the pyspark that installed on jupyter, no on spark cluster

Copy csv file to inside jupyter container :

docker cp file.csv jupyter:/opt/file

from pyspark.sql import SparkSession

# Create a Spark session
spark = SparkSession.builder.appName("YourAppName").getOrCreate()

data = spark.read.csv("/opt/file/file.csv", header=True)
data.limit(3).show()

spark.stop()

and also you can practice on single node pyspark in jupyter :

Congratulation 🍹

Spark On Kubernetes

Keyvan Soleimani — Mon, 05 May 2025 11:45:11 +0000

Spark On Kubernetes via helm chart

This article is aimed at introducing the feature of Apache Spark on Kubernetes, which is officially announced as production-ready and Generally Available with the Apache Spark 3.1 release in March 2021.

To gain hands-on experience with the feature, we go through running Apache Spark on a local Kubernetes cluster with Workload Identity to explore the nuance of environment set up and arguments when submitting jobs.

The control-plane & worker nodes addresses are :

192.168.56.115
192.168.56.116
192.168.56.117

Kubernetes cluster nodes :

you can install helm via the link helm

Installing Apache Spark

The Steps :

1) Install spark via helm chart (bitnami) :

$ helm repo add bitnami https://charts.bitnami.com/bitnami
$ helm search repo bitnami
$ helm install kayvan-release oci://registry-1.docker.io/bitnamicharts/spark
$ helm upgrade kayvan-release bitnami/spark - set worker.replicaCount=5

the installed 6 pods :

and Services (headless for statefull) :

and the spark master ui is :

2) type the below commands on kubernetes kube-apiserver :

kubectl exec -it kayvan-release-spark-master-0 -- ./bin/spark-submit \
 --class org.apache.spark.examples.SparkPi \
 --master spark://kayvan-release-spark-master-0.kayvan-release-spark-headless.default.svc.cluster.local:7077 \
 ./examples/jars/spark-examples_2.12–3.4.1.jar 1000

kubectl exec -it kayvan-release-spark-master-0 -- /bin/bash


./bin/spark-submit \
 --class org.apache.spark.examples.SparkPi \
 --master spark://kayvan-release-spark-master-0.kayvan-release-spark-headless.default.svc.cluster.local:7077 \
 ./examples/jars/spark-examples_2.12–3.4.1.jar 1000


./bin/spark-submit \
 --class org.apache.spark.examples.SparkPi \
 --master spark://kayvan-release-spark-master-0.kayvan-release-spark-headless.default.svc.cluster.local:7077 \
 ./examples/src/main/python/pi.py 1000


./bin/spark-submit \
 --class org.apache.spark.examples.SparkPi \
 --master spark://kayvan-release-spark-master-0.kayvan-release-spark-headless.default.svc.cluster.local:7077 \
 ./examples/src/main/python/wordcount.py //filepath

the exact scala & python code of spark-examples_2.12–3.4.1.jar , pi.py & wordcount.py :

SparkPi.scala

pi.py

wordcount.py

3) The final result is 🍹 :

for scala :

for python :

The other python Programm :

1) Copy People.csv (large file) inside spark worker pods :

kubectl cp people.csv kayvan-release-spark-worker-{x}:/opt/bitnami/spark

Notes:

you can download the file from link
you can also use a nfs share folder for read large csv file from it instead of copying it inside pods.

2) Write some python codes inside readcsv.py please :

from pyspark.sql import SparkSession
#from pyspark.sql.functions import sum
from pyspark.context import SparkContext

spark = SparkSession\
 .builder\
 .appName("Mahla")\
 .getOrCreate()


sc = spark.sparkContext

path = "people.csv"

df = spark.read.options(delimiter=",", header=True).csv(path)

df.show()

#df.groupBy("Job Title").sum().show()

df.createOrReplaceTempView("Peopletable")
df2 = spark.sql("select Sex, count(1) countsex, sum(Index) sex_sum " \
                     "from peopletable group by Sex")

df2.show()

#df.select(sum(df.Index)).show()

3) copy readcsv.py file inside spark master pod :

kubectl cp readcsv.py kayvan-release-spark-master-0:/opt/bitnami/spark

4) run the code :

kubectl exec -it kayvan-release-spark-master-0 -- ./bin/spark-submit 
   --class org.apache.spark.examples.SparkPi
   --master spark://kayvan-release-spark-master-0.kayvan-release-spark-headless.default.svc.cluster.local:7077 
       readcsv.py

5) showing some data :

6) the next result data:

7) the time consuming for processing :

The other python program on Docker Desktop :

docker-compose.yml :

version: '3.6'

services:

  spark:
    container_name: spark
    image: bitnami/spark:latest
    environment:
      - SPARK_MODE=master
      - SPARK_RPC_AUTHENTICATION_ENABLED=no
      - SPARK_RPC_ENCRYPTION_ENABLED=no
      - SPARK_LOCAL_STORAGE_ENCRYPTION_ENABLED=no
      - SPARK_SSL_ENABLED=no
      - SPARK_USER=root   
      - PYSPARK_PYTHON=/opt/bitnami/python/bin/python3
    ports:
      - 127.0.0.1:8081:8080

  spark-worker:
    image: bitnami/spark:latest
    environment:
      - SPARK_MODE=worker
      - SPARK_MASTER_URL=spark://spark:7077
      - SPARK_WORKER_MEMORY=2G
      - SPARK_WORKER_CORES=2
      - SPARK_RPC_AUTHENTICATION_ENABLED=no
      - SPARK_RPC_ENCRYPTION_ENABLED=no
      - SPARK_LOCAL_STORAGE_ENCRYPTION_ENABLED=no
      - SPARK_SSL_ENABLED=no
      - SPARK_USER=root
      - PYSPARK_PYTHON=/opt/bitnami/python/bin/python3

docker-compose up --scale spark-worker=2

copy required files to containers :

for e.g.

docker cp file.csv spark-worker-1:/opt/bitnami/spark

python code on master :

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("Writingjson").getOrCreate()

df = spark.read.option("header", True).csv("csv/file.csv").coalesce(2)

df.show()

df.write.partitionBy('name').mode('overwrite').format('json').save('file_name.json')

run the code on spark master docker container :

./bin/spark-submit --master spark://4f28330ce077:7077 csv/ctp.py

showing some data on master :

and the seperated json files based on name partitioning on worker 2 :

data for name=kayvan :

Congratulation 🍹

Nginx Ingress Controller in Kubernetes

Keyvan Soleimani — Mon, 05 May 2025 11:05:46 +0000

Setting up Ingress controller NGINX (as a router) along with HAproxy for Microservice deployed inside Kubernetes cluster (Bare-metal servers)

The NGINX Ingress Controller is an Ingress Controller implementation for NGINX and NGINX Plus that can load balance Websocket, gRPC, TCP and UDP applications. It supports standard Ingress features such as content-based routing and TLS/SSL termination. Several NGINX and NGINX Plus features are available as extensions to Ingress resources through Annotations and the ConfigMap resource.

The NGINX Ingress Controller supports the VirtualServer and VirtualServerRoute resources as alternatives to Ingress, enabling traffic splitting and advanced content-based routing. It also supports TCP, UDP and TLS Passthrough load balancing using TransportServer resources.

General Design (big picture) :

The control-plane & worker nodes addresses are :

192.168.56.115
192.168.56.116
192.168.56.117

and HAProxy as a Load Balancer :

192.168.56.118

Kubernetes cluster nodes :

Note: All manifests needed for below steps is present at my github

The Steps :

1) Install NGINX Ingress Controller from nginx-ingress :

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.3/deploy/static/provider/baremetal/deploy.yaml
kubectl get all - namespace=ingress-nginx

Ingress-Nginx workloads (ingress port in our case is 30798) :

2) On the nodes, where the PODs will be located (node1 and node2 in our case) :

DIRNAME="vol1"
mkdir -p /mnt/disk/$DIRNAME 
chcon -Rt svirt_sandbox_file_t /mnt/disk/$DIRNAME
chmod 777 /mnt/disk/$DIRNAME

3) Deploy the Storage Class & PV & PVC :

kubectl apply -f storageClass.yaml
kubectl apply -f persistentVolume.yaml
kubectl apply -f persistentVolume1.yaml
kubectl apply -f persistentVolumeClaim.yaml
kubectl apply -f persistentVolumeClaim1.yaml

4) Deploy the web apps :

kubectl apply -f http-pod.yaml
kubectl apply -f http-pod1.yaml

Get pod ip & curl the related web app :

POD_IP=$(kubectl get pod www2-c5644ff98-trk4d -o yaml | grep podIP | awk '{print $2}'); echo $POD_IP
curl $POD_IP

5) Deploy the Services :

kubectl apply -f IngressService.yaml
kubectl apply -f IngressService1.yaml

6) Deploy the Ingress resource :

kubectl apply -f Ingress.yaml

Note: you can also Secure NGINX-ingress via this [Securing NGINX-ingress]

7) HAProxy config as a Load Balancer (On 192.168.56.118) :

sudo nano /etc/haproxy/haproxy.cfg

8) DNS Record (On DNS Server) :

The final results are 🍹 :

Deploying Kubernetes Cluster On On-premises

Keyvan Soleimani — Sun, 04 May 2025 13:08:45 +0000

Kubernetes assembles one or more computers, either virtual machines or bare metal, into a cluster which can run workloads in containers. It works with various container runtimes, such as Docker.

The Kubernetes master node handles the Kubernetes control plane of the cluster, managing its workload and directing communication across the system, etcd is a persistent, lightweight, distributed, key-value data store (originally developed for Container Linux). It reliably stores the configuration data of the cluster, representing the overall state of the cluster at any given point of time.

The API server serves the Kubernetes API using JSON over HTTP, which provides both the internal and external interface to Kubernetes. The API server processes, validates REST requests, and updates the state of the API objects in etcd, thereby allowing clients to configure workloads and containers across worker nodes.

The scheduler is an extensible component that selects the node that an unscheduled pod (the basic unit of workloads to be scheduled) runs, based on resource availability and other constraints. The scheduler tracks resource allocation on each node to ensure that workload is not scheduled in excess of available resources.

A controller is a reconciliation loop that drives the actual cluster state toward the desired state, communicating with the API server to create, update, and delete the resources it manages.

A node, also known as a worker or a minion, is a machine where containers (workloads) are deployed. Every node in the cluster must run a container runtime, as well as the below-mentioned components, for communication with the primary network configuration of these containers.

kubelet is responsible for the running state of each node, ensuring that all containers on the node are healthy. It takes care of starting, stopping, and maintaining application containers organized into pods as directed by the control plane

A container runtime is responsible for the lifecycle of containers, including launching, reconciling and killing of containers. kubelet interacts with container runtimes via the Container Runtime Interface (CRI), which decouples the maintenance of core Kubernetes from the actual CRI implementation.

kube-proxy is an implementation of a network proxy and a load balancer, and it supports the service abstraction along with the other networking operations. It is responsible for routing traffic to the appropriate container based on IP and port number of the incoming request.

This article is for deploying kubernetes on-permises cluster (bare-metal servers).

Note : The kubernetes folder contains the files for preparing server to install kubernetes cluster or join to the cluster.

Please run the following script on all servers :

sudo ansible servers -m ping -i inventory.ini -u root

sudo ansible-playbook -i inventory.ini Kubernetes/ServerPrepare.yml -u root

Kubernetes Cluster :

The control-plane nodes addresses are :

192.168.56.120
192.168.56.121
192.168.56.122

The worker nodes addresses are :

192.168.56.123
192.168.56.124

The vms hosted on virtualbox are like the below schema :

HAProxy server (Load Balancer for kube apiserver) address is :

192.168.56.118

haproxy.cfg :

stats enable
(frontend bind to 192.168.56.118:6443)
(backend bind to 192.168.56.120:6443 192.168.56.121:6443 192.168.56.122:6443)

For starting a Kubernetes cluster, follow the below lines :

Run below scripts only on 192.168.56.120 :

sudo kubeadm init  --control-plane-endpoint="192.168.56.118:6443"   
      --upload-certs  --apiserver-advertise-address=192.168.56.120
      --pod-network-cidr=192.168.0.0/16  
      --cri-socket=unix:///var/run/cri-dockerd.sock  
      --ignore-preflight-errors=all

And below code for all nodes :

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Install Calico network policy for on-premises deployments, 50 nodes or less :

curl https://raw.githubusercontent.com/projectcalico/calico/v3.27.2/manifests/calico.yaml -O

kubectl apply -f calico.yaml

and run the below on other servers to join to the cluster :

On control-plane (e.g. 192.168.56.122) :

kubeadm join 192.168.56.118:6443 --token c4c6wt.2rzubblajmxx7wf1 \
     --discovery-token-ca-cert-hash sha256:91877d933445148c650e5fa11acca05d455fe1e9e53cd33f8497ad06a2126142 \
     --control-plane --certificate-key 2e8c3d0a1f2d4aec3e4ccb09a0dd6f43756344269c0b414cdd83c0ef02c0293d \
     --apiserver-advertise-address=192.168.56.122 
     --cri-socket=unix:///var/run/cri-dockerd.sock 
     --ignore-preflight-errors=all

On worker nodes :

kubeadm join 192.168.56.118:6443 --token  c4c6wt.2rzubblajmxx7wf1 \
     --discovery-token-ca-cert-hash sha256:91877d933445148c650e5fa11acca05d455fe1e9e53cd33f8497ad06a2126142 \
     --cri-socket=unix:///var/run/cri-dockerd.sock 
     --ignore-preflight-errors=all

and at final step enjoy from your cluster :

kubectl get nodes -o wide

kubectl get pod -A

HAProxy Stats :

haproxy.cfg :

Congratulation 🍹