🌟📝 Basic commands to work with Spark in Notebooks like a Standalone cluster
🔗Related content
You can find post related in:
You can find video related in:
You can find repo related in:
🐱🏍GitHub
You can connect with me in:
Resume 🧾
I will install Spark program and will use a library of Python to write a job that answer the question, how many row exists by each rating?
Before start we setup environment to run Spark Standalone Cluster.
1st - Mount Google Drive 🚠
We will mount Google Drive to can use it files.
I use following script:
from google.colab import drive
drive.mount('/content/gdrive')
2nd - Install Spark 🎇
Later got a Colab notebook up, to get Spark running you have to run the following script (I apologize for how ugly it is).
I use following script:
%%bash
apt-get install openjdk-8-jdk-headless -qq > /dev/null
if [[ ! -d spark-3.3.1-bin-hadoop3.tgz ]]; then
echo "Spark hasn't been installed, Downloading and installing!"
wget -q https://downloads.apache.org/spark/spark-3.3.1/spark-3.3.1-bin-hadoop3.tgz
tar xf spark-3.3.1-bin-hadoop3.tgz
rm -f spark-3.3.1-bin-hadoop3.tgz
fi
pip install -q findspark
You would can get other version if you need in: https://downloads.apache.org/spark/ and later replace it in the before command.
3rd - Setting environment variables 🌐
I use following command:
import os
os.environ["JAVA_HOME"] = "/usr/lib/jvm/java-8-openjdk-amd64"
os.environ["SPARK_HOME"] = "/content/spark-3.3.1-bin-hadoop3"
import findspark
findspark.init()
4th - Configuring a SparkSession 🚪
I use following command:
from pyspark.sql import SparkSession
spark = SparkSession.builder \
.master("local[*]") \ # set up like master using all(*) threads
.appName("BLOG_XLMRIOSX") \ # a generic name
.getOrCreate()
sc = spark.sparkContext
sc
Later of execute this we can use many data strcuture to manage data, like RDD and Dataframe(Spark and Pandas).
Exists one more but only exists in Scala and it is Dataset.
5th - Getting a dataset to anlyze with Spark 💾
I use a dataset from grouplens. You can get other in:
http://files.grouplens.org/datasets/
This time I use movieslens and you can download it using:
!wget http://files.grouplens.org/datasets/movielens/ml-100k.zip
To use data extract files. I extract files in path later of -d in command:
!unzip "/content/ml-100k.zip" -d "/content/ml-100k_folder"
6th - Configuring data to analyze 💿
We create a variable called data where put path where data is like:
data = '/content/ml-100k_folder/ml-100k/u.data'
Later define a variable called df_spark where put information of data:
df_spark = spark.read.csv(data, inferSchema=True, header=True)
We can inspect type of variable df_spark like:
print(type(df_spark))
We can inspect data frame of variable df_spark like:
df_spark.show()
We can see format is incorrect so we will fix that where we configure format of data by following way:
df_spark = spark.read.csv(data, inferSchema=True, header=False, sep="\t")
9th - Making a query 🙈
To make this we need know format of data, so I infer the following structure:
- First column reference to userID.
- Second column reference to movieID.
- Third column reference to rating.
- Fourth column reference to timestamp.
I will answer the question, how many movies by each rating exists...
9th - Making a query with SQL syntax ❔🛢
- First, create a table with dataframe.
df_spark.createOrReplaceTempView("table")
- Second, can make query to answer question.
sql = spark.sql("SELECT _c2, COUNT(*) FROM table GROUP BY _c2")
To see results:
sql.show()
9th - Making a query with Dataframe ❔📄
- It's so easy make this query with dataframe.
dataframe.groupBy("_c2").count().show()
9th - Making a query with RDD ❔🧊
- First, we transform dataframe to rdd type.
rdd = df_spark.rdd
- Second, make query with RDD functions.
rdd\
.groupBy(lambda x: x[2])\
.mapValues(lambda values: len(set(values)))\
.collect()
Top comments (0)