DEV Community: Fady GA 😎

How I've implemented the Medallion architecture using Apache Spark and Apache Hdoop

Fady GA 😎 — Mon, 17 Jun 2024 13:35:05 +0000

What is this project all about
System architecture
- The Retail app
- Apache Spark Cluster
- The Development Jupyter server
- The Orchestrator
- Spark incremental database loading
- Hadoop cluster
- The dashboard app
Spark Operations
- Bronze stage
- Silver stage
- Gold stage
Exposed ports and services:
Instructions
Stopping the project
Things to consider if you want to make this into real-world project
Key takeaways
References

What is this project all about

In this project, I'm exploring the Medallion Architecture which is a data design pattern that organizes data into different layers based on structure and/or quality. I'm creating a fictional scenario where a large enterprise that has several branches across the country. Each branch receives purchase orders from an app and deliver the goods to their customers. The enterprise wants to identify the branch that receives the most purchase requests and the branch that has the minimum average delivery time.

To achieve that, I've used Apache Spark as a distributed compute engine and Apache Hadoop, in particular HDFS, as my data storage layer. Apache Spark ingest, processes, and stores the app's data on HDFS to be served to a custom dashboard app. You can find all about it, in this Github repo

System architecture

The Retail app

I've simulated the retail app with a python script that randomly generates purchases and handle their lifecycle as NEW -> INTRANSIT -> DELIVERED. The app then stores that in a Postgresql database named retail that has two tables, orders and ordershistory. Separate users with appropriate privileges are created to the app to write the data to the database and to Spark, to read the data. The database is initiated with postgresql/init.sql script.

Apache Spark Cluster

A standalone cluster with one master none (spark-master) and two worker nodes (spark-worker-1 and spark-worker-2). The REST API endpoint and history server are enabled. I've build its docker image and didn't use any image from Docker Hub.

The Development Jupyter server

A jupyter server on the Master Node of the spark cluster running on port 8888 that is exposed to localhost in the Docker compose file. This is how I developed the pipeline's PySpark scripts. This allowed rapid and convenient development for the Spark jobs. The server is configures to have a password pass that can be changed from spark/jupyter/jupyter_notebook_config.py or via JUPYTER_NOTEBOOK_PASSWORD if passed to the Docker compose file.

The Orchestrator

A python script that communicates with Apache Spark via its REST API endpoint to submit PySpark scripts stored on HDFS in hdfs:<namenode>/jobs/* and "orchestrate" the pipeline Ingestion -> Transformation -> Serving via Spark's drivers status API. All the PySpark files are in spark/pipeline/.

Spark incremental database loading

The ingestion.py script implements a bookmarking mechanism which allows Spark to check for and sets bookmarks during the database data ingestion so it only loads new data and not to strain the app's databases with long running queries each time the pipeline is executed. The bookmark is stored in HDFS in hdfs:<namenode>/data/bronze/bookmark as JSON.

Hadoop cluster

The official image on Docker Hub with minimal configuration changes on hadoop/config where WebHDFS is enabled.

The dashboard app

A Flask app the read the csv file stored in the Gold folder in HDFS and outputs the resulting data as graphs. It meant to be the data consumer for this pipeline where business users can get their metrics.

Spark Operations

Bronze stage

Spark is incrementally ingesting the orders and ordershistory table partitions by date in parquet format in the Bronze.

Silver stage

Spark joins the two table from the Bronze stage and filter out "INTRANSIT" records and stores them in the Silver directory in parquet format.

Gold stage

Spark pivot the joined table to get one record per order that contains the order date and delivery date, subtracts then and get the delivery time in hours.
Then aggregates the result to get the orders count and average delivery time per branch. Finally, it stores them as CSV files in the Gold directory.

Exposed ports and services:

I've configured the the majority of the the services in the Docker compose file to be accessible from localhost:

Dashboard App: http://localhost:5000
Spark master node UI: http://localhost:8080
Spark worker node 1 UI: http://localhost:8081
Spark worker node 2 UI: http://localhost:8082
Spark history server: http://localhost:18080
Spark running driver UI: http://localhost:4040
Jupyter server running on Spark cluster: http://localhost:8888
Hadoop namenode UI: http://localhost:9870

Instructions

I assume that you have Python and Docker installed in your system.

Go to the project's root directory.
Create a Python virtual env and install the requirements in ./requirements.txt
Build the custom Spark docker image (this could take a while) ```bash

docker build spark:homemade ./spark/

- Build the Dashboard app docker image
  ```bash


  docker build dashboard:homemade ./dashboard/

Spin up the system ```bash

docker compose up -d

- Activate the virtual env and run the retail app. You will see simulated transactions in your terminal.
  ```bash


  python ./retailapp/retailapp.py

Open a new terminal and initiate HDFS directories and copy the pipeline PySpark scripts to HDFS ```bash

docker compose exec hadoop-namenode bash -c "/opt/hadoop/init.sh"

- Go to the dashboard's app url (http://localhost:5000) and validate that there aren't any data yet.
- Wait a while for the retail app's database to populate then run the Spark pipeline withe the orchestrator
  ```bash


  python ./orchestrator/scheduler.py

(Optional) You can check the progress from Spark's master node UI.
After the pipeline finishes successfully (all stages are ended with FINISHED), visit the dashboard's app url again and refresh. You will get the processed data now.
(Optional) You can rerun the pipeline and notice the dashboard's data changes.

Stopping the project

I didn't configure any Named volumes on Docker, so once you stop the project, all the data is lost.
To stop the project just execute docker compose down

Things to consider if you want to make this into real-world project

Instead of the custom orchestrator I used, a proper orchestration tool should replace it like Apache Airflow, Dagster, ..., etc.
Also, instead of the custom Dashboard app, a proper BI tool like Power BI, Tableau, Apache Superset, ..., etc. will be more powerful and flexible.
Bookmarking should be in the Silver stage as well.
In a multi-tenant environment, controlling access to the Spark and Hadoop cluster is a must. Tool like Apache Ranger are usually used to serve this purpose.
This architecture isn't limited to ingesting from one source, multiple sources can be ingested too and enriched from other sources in the Silver stage.
The Spark operation in this project are rather simple. More complex operations can be done leveraging Spark's powerful analytics engine.

Key takeaways

You can control almost everything in a spark cluster from the spark-defaults.conf file.
Spark has a REST endpoint to submit jobs and manage the cluster but it isn't enabled by default.
Spark history server isn't enabled by default.
Don't use the latest Java version for Spark even if the docs says it's supported. (I had compatibility issue that appeared down the road as I was using Java 17).
Developing Spark driver apps in a jupyter notebook will make your life way easier.
Hadoop also has an http endpoint called WebHDFS which isn't enabled by default too.
docker-entrypoint-initdb.d is very useful for Postgresql initiation.
Never use plaintext passwords in your code, instead you env vars or cloud secrets services.

References

All the used tool official docs.

Groovy 🎷 Cheat Sheet - 01 Say "Hello" from Groovy

Fady GA 😎 — Thu, 07 Mar 2024 10:43:03 +0000

"Java is great, but I don't like it!"

This was my first impression when I started to learn the language. Maybe if my first language I learnt was C++ instead of Python, I'd have another opinion, but who knows 🤷‍♂️.

Recently I had to revisit the "JVM languages universe" again. Yes, language(s), plural! Java isn't the only language that uses the JVM. I previously used Scala, which is a JVM language, to use Apache Spark for Data Engineering workloads, but this is for another post 😉.

This time, I visited Groovy. Given my biased first impression about Java and the JVM, I wasn't too exited! But I couldn't have been more wrong! 😁

When I started to pick up Groovy, I instantly noticed the similarities between it and Python despite being a JVM language. I could see that Groovy was built to supplement Java with "scripting" capabilities. When I started to dig deeper into the learning resources, I found out that Groovy wasn't meant to replace Java! But to supercharge your Java codebase and to make the "Ceremonious boiler plate code" redundant (a phrase I stole from Dan Vega 😁) while still using the JVM. I also read that "Groovy is Python for Java" 😉

To demonstrate this, it took me around 2 minutes to create a very simple class that just print "Hello" using java with this code. Now imagine creating a whole app like this! We will see how it is significantly faster (and shorter) with Groovy:



public class SayHello {
    public static void main(String args[]) {
        System.out.println("Hello, from JAVA");
    }
}

I'm a bit of a slow typer and other super devs out there might do the same in a heartbeat, but you get the idea 😉

But that said, - and again I might be a bit biased - Groovy is too slow for me! I compared it to Rust in this LinkedIn post and it was waaaaay slow. Keep in mind that subjectively comparing programming languages might be a tricky business. But at the end, it will be up to your use case/project to prefer a language over the other.

Now, a trick I picked up earlier, is that

If you need to learn something, teach it!

The logic behind that, is whenever you need to teach anything, you are either an expert in that topic or not. When you aren't an expert, you will need to excel at that topic in order to offer credible and high value knowledge to the ones you teach.

So, I want to learn Groovy, then I'll teach it! I'll post a short series (3-4 posts at most) about Groovy. It won't be an in-depth series, it will only give you what you need to get up and running with Groovy. A Cheat Sheet if you will...

In this one, I'll start with the basics, enjoy 😎

Setting up your "Groovy" environment

As with any programming language, your development environment is basically installing the language itself and an IDE. In the following section we will see how to setup our environment and the different options that we have to do so.

Installation

As Groovy is a JVM language, you will need 2 things to start coding with it:

The JDK (Java Development Kit)
The Groovy jar files.

The manual setup process contains:

Download and install you favorite JDK. (I use OpenJDK 17. You can use the 21 stable version, it won't really matter at this stage which JDK provider you choose (yes, there are many) and the version as long as it's relatively new 😁)
Depending on your OS, make sure that there is a JAVA_HOME env var pointing to the bin directory insider your JDK installation directory.
Download and install the latest Groovy version either using the installers or the binary zip files. It all falls down to your personal preference.
Like with the JDK, make sure that you have a GROOVY_HOME env var pointing to the bin directory in your Groovy installation directory.

If you survive all of this, you should be more or less good to go! 😂

Alternatively, you can use sdkman. A great tool to install your Software Development Kit. The downside is that it only works on *nix systems. So for Widnows users, you will have to use WSL or Cygwin as the official page suggests.
It is really simple to use sdkman. after a successful installation, just type those commands into your *nix shell:



sdk install java



sdk install groovy

And that's it. You are good to go 😃

For a complete usage guide, check its official page.

You can check if you have a correct environment istallation by typing those commands into your system's shell:



java --version



groovy --version

Groovy tools

Groovy installation comes with 3 main cli-ish tools:

groovy: The Groovy command line processor

groovyc: The Groovy compiler (I rarely use that, at least for now)
groovyConsole: The Groovy REPL (Read Evaluate Print Loop). In other words, a minimalist IDE. While learning, the groovyConsole will be all what you need. It's very basic and can't work within WSL.

IDEs buffet

While the groovyConsole will satisfy all of our learning needs, if you ever wanted to go the next "groovy" level, you will most probably need an IDE. IDEs comes with all the bells and whistles that should make you more productive and grooving 😁.

I personally prefer VS Code. But for Groovy, it didn't offer the right extensions.

Next, I've tried Apache Netbeans. It felt like it's built for Java rather than Groovy. I couldn't make its basic "IntelliSense" (code auto-completion) feature correctly work for Groovy so I ended up abandoning it.

IntelliJ (look for the community edition) on the other hand offered great out-of-box Groovy support including IntelliSence, building, and running features. So, I sticked with it 😊

Groovy "Hello, World"

Now this won't be an introductory programming language post without the classic "Hello, World" example, right? 😉

I'll use the most basic commands using bash:



echo "println 'Hello, World'" > helloWorld.groovy
groovy helloWorld

And another "Hello, World" from the groovyConsole:

Yup, that's it! Compare that to the Java class I wrote earlier!

[BONUS] Running Groovy in a Jupyter Notebook

Jupyter Notebooks is very popular among data people specially Python users. So, I tried to find a way to run the Groovy kernel inside a Jupyter Notebook, and to my surprise, I found a way, BeakerX!

The easiest way to try it is to use its Docker container as follows (you must have docker installed in your system):



docker run -p 8888:8888 beakerx/beakerx
# Image pulling will take sometime if it's the first time you run it.

The open up the url that you will be presented with and you will be good to go. (you will have to replace the funky server name with localhost for it to work, or at least this is how it worked for me)

This is another "Hello, World" example using Jupyter with a Groovy kernel:

Conclusion

Groovy is very close to Python that runs in the JVM universe. It's not meant to replace Java, but to supplement it. Groovy already comes with all the tools you need to get started. In this one, I didn't go into Groovy language specifics. I'll do that in later posts. See you then!

Meet Bob 🙂 and his Databases exploration journey 🗺️! [Relational Databases - Basics - Theory]

Fady GA 😎 — Tue, 24 Oct 2023 14:08:42 +0000

In this article, we will meet Bob, the proud owner of a promising retail store, and see his journey that led him to discover the Relational Databases 😯! Let's begin.

This article is part of my Data Analytics Made Simpler series. On its website, I'm giving more details about what exactly this series is, and what's my motivation to do something like that, and how I plan to release its content!

Bob has taken responsibility for his family business, a retail store in his home town! He is very creative and business-oriented; he managed to expand his customer base threefold over the last year alone! In order to retain his current customers and attract new ones, he had the idea of implementing a "Loyality Program", where he awards his "loyal" customers with discounts based on their number of store visits and the amount they spent.
The problem is, he doesn't have any database knowledge, and he thought that he could pull this off using a spreadsheet application that he is very proficient with.

Bob's first attempt:

As Bob is a spreadsheet guru, he created one with the following columns:

Date
Customer Name
Customer Phone
Customer Email
First Purchase
Item Description
Quantity
Unit Price
Total Price

From his point of view, this spreadsheet captures everything he needs! The date of the purchase, the customer data, the purchased items, and the amount spent by the customers.
It worked fine for a couple of weeks until he was faced with some challenges that made it very hard to continue this way:

It was very daunting to enter the same date and the customer's data with each item he/she purchases, knowing that one customer is more likely to purchase several items at once.
Customers started complaining as it takes a long time for their entire purchase to be registered, especially if there are a lot of purchased items.
He noticed that the data for the same customer might be entered differently for separate purchases on different days (or even on the same day!). It may be due to typos; the data is entirely different or completely missing.
Similar to the customer data, the Items data might suffer from similar inconsistencies, like description changes or price changes, which make something like tracking the performance of a single item hard to track.
As he kept entering the same customer and item data over and over again, the spreadsheet file's size exploded very quickly!

How Bob improved his data model:

Bob has realized that his current data model isn't scalable! Like when the data has accumulated for a couple of weeks, it becomes difficult to maintain and get accurate insights from it.
The first thing he really wanted to change was entering the same customer data with each purchased item. As this is redundant and makes modifying any piece of data, like the customer's phone, very challenging, as he must check all the previous purchases for this particular customer and change the phone number in each of his/her related records!
He thought of separating the Customers in their own "worksheet" (table) and renaming the original worksheet to Purchases and removing all the customer data columns from it, leaving only the "Customer Name". He first sketched it out as follows before he performed the actual change to his current file:

Bob has unknowingly created his own version of an "Entity Relationship Diagram" (ERD) which still missing some key ERD features but is close enough!
What he did is that he has created two Entities, "Customers" and "Purchases", and he has used the "Customer Name" as the Link AKA "relationship" between the two entities.

Now his model has become more maintainable and far less in size as if he needed to do the customer phone number modification we discussed earlier, he only needed to do it once in one record in the "Customers" table. Also, as he now doesn't include the full customer's data at each purchase record and only include the Customer Name, the size of his workbook has become substantially smaller.

But soon he encountered another problem!

He noticed, that some customers might share the same name! So this will become problimatic in his "Loyality Program" implementation as for such cases, he wouldn't be able to tell wich is the correct customer to award the discounts to!

The introduction of keys:

So, he realized that seting the "Customer Name" as the "Key" to identify the relationship between the "Customers" and "Purchases" entities might not be ideal as it might not be Unique. He inspected the columns in the Customers table to identify another column that is 100% unique for each customer, he found some good candidates like phone number or email address. He thought that those should be unique for every customer! As one phone number can be owned by one customer and the same goes for the email address, right?

Then he discoverd that if he used those columns, he might introduce further problems down the road! Customers are allowed to change their phone numbers and emails. So, if he had used the phone number for example as the identifying field for the customers and later customers changed their numbers, he then would have different purchase records with diffirent phone numbers that could be expired and not refering to any customers in his "Customers" table!

He decided to introduce an new column to his "Customers" table to act as the unique ID for every customer he has. He though that this column should be unique and can't be empty. He wanted it to be simple, so he chose it to be an incrementing number starting at 1. Then instead of the "Customer Name" to link the two table, he would use this newly introduced "Key" column. He updated his model diagram adding a "Purchase ID" key column as he felt he could use it later:

Again without knowing, Bob has created something called a "Primary Key" by introducing the "Customer ID" column in the "Customers" table. The main function of a "Primary Key" in a table is to be able to uniquely identify each record in this table. Also, by using the same column in the "Purchases" table to define the relationship between the "Customers" table, he used it this time as a "Foreign Key" (foreign to the table). In contrast to the "Primary Keys" that must be unique in a table, "Foreign Keys" can be non-unique in the same table.

In a real ERD, "Primary Keys" (PK) and "Foreign Keys" (FK) should be annotated on the diagram and the "relationship" should be named, conventionally with a "verb". Also, something called relationship "cardinality" must be annotated too on the line related to the relationship.

This way, Bob managed to enhanced his data model a lot. All he has to do to register a new purchase, is to lookup the customer's record, retreive its PK (Primary Key), and write it in front of each purchased item.
But he felt that there is more to be done, so he invited his friend Ben, a database designer, for checking what Bob has done so far with his model.

Relationships Cardinality:

Ben has really admired what Bob has done so far without any prior database knowledge! He saw the diagram that Bob has made and told him that it looked pretty much like a real ERD and explained how he could make it so. He also explained the concept of "Primary Keys" and "Foreing Keys" that Bob has used unknowingly.
He felt that he needed to explain to Bob, the concept of "cardinality" of the relationships between the entities as he saw room for improvements for the current data model.

Relationship Cardinalty is a fairly simple concept. In general, it defines how the records from one table are related to the records from the other table at both ends of a relationship. It can be "One-to-One", "One-to-Many", or "Many-to-Many".

For "One-to-One", it means that one record for the first table is linked to only one record from the second table. For example, if we split the "Customers" table into two table, with "Name" and "Customer ID" in one table and "Customer ID" and the rest of the columns in the other table. In that case, each record from any table will be linked to one and only one record from the other table using the "Customer ID". "One-to-One" has its application in databases but it isn't commonly used.

Similary for "One-to-Many", it indicates that one record for the "One" side of the relationship can be linked to many records from the "Many" side. The relationship between the "Customers" and the "Purchases" tables is "One-to-many" as "One" customer can perform "Many" purchases but one purchase can't be done by many customers.

"Many-to-Many" relationships might need some time to grasp but they aren't hard to understand! It means that "Many" records from the first table can be linked with many records from the second table. Think of "Students" and "Classes" tables. One Student can register to "Many" classes and one class can contain "Many" students.

Ben has recommended to Bob that he could seperate the "Items" entity similar to what he has done to the "Customers" entity (table). He also identified the cardinality between "Customers" and "Items" as "Many-to-Many" as one customer can purchase "Many" items and one Item can be purchased by "Many" customers.

Finally, he modified Bob's diagram to show PKs, FKs, and relationships cardinalities. He separated "Items" into their own table and used the "Purchases" table as an intermediate table to implement the "Many-to-Many" relationship between "Customers" and "Items". The final ERD became as follows:

Again, this is a real ERD but in its simpleset form. ERDs aren't in the scope of this post and I might discuss them in later posts.

Enter: RDBMS

Bob has thanked Ben for his advice! And indeed acknowledged that the overall size of the workbook file has become smaller and the whole model has become more maintainable than before those changes. But he rememebered that he still uses spreadsheets app to manage his data! He realized that there is a lot of manual work to be done like looking up the IDs of customers and items, ensuring that the IDs in the "Customers" and "Items" tables are unique and present, the "quantity" in the "Purchases" table is positive and present, and so much more "rules" that he must enforce at data entry time! Also, he noticed that the "Total Price" column is not straight forward to calculate as before he splitted the "Items" from "Purchases" and the "Loyality Program" analysis now requires a lot more work!

He told his friend Ben all of his concerns. Ben smiled and told Bob, "Why don't you use a Relational Database Management System (RDBMS)?"

Ben continued to list all the benifits of using a dedicated RDBMS software like MySQL, Postgresql, MariaDB, or many others over the use of spreadsheets app:

With RDBMSs, you won't manage any files. The RDBMS will do it for you.
The "rules" that you want to enforce are called in the RDBMS world, "Constraints" and the RDBMS can enforce them for you. For example, for a "Primary Key" column you can enforce the UNIQUE and NOT NULL constaints. "Foreign Keys" are a from of constraint too. RDBMSs check the existance of the key used as a foreing key in a table as a primary keyin the other table in a "One-to-Many" relationship.
Some RDBMS has the "Calculated Columns" feature and you can calculate the "Total Price" with relative ease like in a spreadsheet application.
You won't have to "join" the tables manualy like in a spreadsheet appplication as RDBMSs uses SQL (Structured Query Language) and "joining" all the tables can be joined with a statement as simple as:

SELECT 
    p.Date, 
    c.Name, 
    c.Phone, 
    c.Email, 
    c.FirstPurchase, 
    i.Description, 
    i.UnitPrice, 
    p.TotalPrice
FROM Purchases AS p
JOIN Customers AS c
ON p.CustomerID = c.CustomerID
JOIN Items AS i
ON i.ItemID = p.ItemID

This may look intemidating if you don't know SQL but trust me it is very simple and easy to understand! 😉

Similar to joining tables together, you can have your "Loyality Program" analysis using a stored SQL Query with just a click of a button!
You can host your RDBMS on a dedicated server and build an application on top of it leveraging SQL for more user friendly environment while entring the purchases data.

That and many other benefits!

Conclusion:

Bob was really happy with his discovery of Relational Databases and RDBMSs! He even hired a developer to build his purchases application and integrating his inventory systems with it!
He managed to keep his current customers happy and win more customers! He expanded his familly business into other districts and his store became a well known franchise! 😎

The bottom line, Relational Database makes handling data a very easy job! SQL is used by RDBMSs and it is a very powerfull tool to query and analyse your data. The model we have explored today is call Online Transactional Processing (OLTP) as it is optimized for "writes". Meaning, you can write data very quickly but reading it will take time as you will probably need to join several table to get a meaningfull analysis. And it is called "Transctional" as it supports transactions that can be one or more operation that are done together or not done at all. More on OLTP and ACID compliance (that describes transactions) in later posts. See you then 👋

Learning Rust 🦀: 18 - Rust Collections: HashMaps, accessing values with keys instead of indices

Fady GA 😎 — Fri, 20 Oct 2023 14:48:37 +0000

I'll continue Rust's collections discusion that started two articles ago with Vectors and Strings as a special kind of Vectors. In this last dedicated article about collections, I'll bring out the HashMaps! Let's begin.

⚠️ Remember!

You can find all the code snippets for this series in its accompanying repo

If you don't want to install Rust locally, you can play with all the code of this series in the official Rust Playground that can be found on its official page.

⚠️⚠️ The articles in this series are loosely following the contents of "The Rust Programming Language, 2nd Edition" by Steve Klabnik and Carol Nichols in a way that reflects my understanding from a Python developer's perspective.

⭐ I try to publish a new article every week (maybe more if the Rust gods 🙌 are generous 😁) so stay tuned 😉. I'll be posting "new articles updates" on my LinkedIn and X (fromarly Twitter).

Hash Maps aren't something native to Rust. Other programming languages has similar structures but with different names, like Dictionaries in Python, Objects in Javascrip, Hash Tables in Powershell (althoug not a programming language per-sey), and many more.
They work more or less the same way as you would expect! You retreive a Value with a Key instead of and index as in Vectors.
Rust's HashMap is pretty similar to Python's Dictionary except that in Rust, the HashMap must have the keys of the same type and the values also must be of the same type but the keys and values can be of similar of different types than eachother.

Hash Maps is written in Rust as HashMap<K, V> where K is the keys' type and V is the values' type which can be the same.

I'll demonstrate here the basic API of Rust's Hash Maps but as always, the documentations contains a lot of other metods that can be very useful.

Creating a Hash Map:

Appearently Hash Maps isn't used much in Rust that they aren't loaded in the prelude like Vectors and Strings. So, we have to "import" them first wiht the use keyword then use the new method to create a new one. In the Example below, I'me creating a map HashMap with the keys of type String and the values of type u8. It should represent the scores that students have obtained after taking and exam. I then inserted the scores of two students, Bob and Ben:

use std::collections::HashMap;
fn main() {
    // Create a Hash Map
    let mut map: HashMap<String, u8> = HashMap::new();

    map.insert(String::from("Bob"), 30);
    map.insert(String::from("Ben"), 25);
    println!("{map:?}");
}

Here, I've used the insert method to insert the student names (keys) and there exam scores (values). The println output will be:

{"Bob": 30, "Ben": 25}

Accessing values in a Hash Map:

To access a certain value in a Hash Map, we can use the get method wich takes the key as its parameter. The get method retun an Option<&V> which will return Some<&V> in case if the key exists or None if it doesn't. For Example, to get Bob's score we can use the follwing code:

let bob_score = map.get("Bob").copied().unwrap_or(0);
println!("Bob scored {bob_score}");

OK, this looks a little strange. Let's break it down.
As I've mentioned, the get method returns the Option<&V> enum of a reference for the values type (here it is &u8). The copied method will return Option<V> instead of Option<&V> i.e. Option<u8> as I don't want to return a reference of the value. Then the unwrap_or method will return V if the Option enum is Some or a default value that it takes as a parameter. So, bob_score will hold eventually the value of the "Bob" key with u8 type and it will be printed as follows:

Bob scored 30

If we haven't used copied here, we would write something strange as the paramtere of unwrap_or like unwrap_or(&0) and bob_scrore will be a refrence of the retrieved value.

Another way to access values inside a Hash Map is to use a for loop as follows:

for (key, value) in &map {
    println!("{}: {}", key, value);
}

This will print the following as expected:

Ben: 25
Bob: 30

One more thing before we can leave this section is that Mash Maps takes ownership over the passed keys and values. Look at the follwing example:

let jane = String::from("Jane");
let jane_score = 29;
map.insert(jane, jane_score);

println!("{jane} scored {jane_score}"); // Error: borrow of moved value: `jane`

Here we have declared jane as String and her score jane_score as i32. As we have used the insert method, jane has "moved" into the Hash Map as it's a String while jane_score is "copied" as it's a simple type. The println will cause a compilation error as jane isn't valid anymore after the usage of insert.

Updating a Hash Map:

We usualy update a Hash Map in one of the following scenarios:

Overwite the old values
Only update if the key isn't present
Update a value based on its old value Let's see how we can perform each.

Overwrite the values:

This is probably the simplest implementation. We just have to use insert using a pre-existing key.

map.insert(String::from("Jane", 29);
let jane_score = map.get("Jane").copied().unwrap();
println!("Jane score before the update {jane_score}");

map.insert(String::from("Jane"), 30);
let jane_score = map.get("Jane").copied().unwrap();
println!("Jane score after the update {jane_score}");

Here, we have updated Jane's score from 29 to 30

Jane score before the update 29
Jane score after the update 30

Only update if the key isn't present:

In this use case, we use the entry method that takes a key as a parameter and returns Entry enum representing an entry that could exist or not. Then we use a similarly useful method, or_insert that insert the passed value if the entry doesn't exist. Let's look at this example:

map.entry(String::from("Jane")).or_insert(25);
map.entry(String::from("Windy")).or_insert(25);
println!("{map:?}");

Here, we were retrieving the Jane and Windy entries and inserting 25 as their exam score if they arent present. As Jane entry already exist from the last section, it will remain unchanged with the value of 30. As for Windy, because it's a new key, it will be inserted with the value of 25 and the println will print the following:

{"Ben": 25, "Windy": 25, "Bob": 30, "Jane": 30}

Update a value based on its old value:

Let's say that we will give all students a 5 points bonus. So, we will have a new value of a certain key that is the result of the addition of its old value and the 5 points bonus. One way to do this is as follows:

for (_, score) in map.iter_mut() {
    *score += 5;
}
println!("{map:?}");

Let's explain what's going on here.
First, I've used the iter_mut method that returns "mutable" references of the keys and values of the Hash Map because I want to change the values. And as I don't care about the keys, I've used the underscore in its place in the for expression. Now, score is a &mut u8 or a mutable reference of the values of u8 type. Next, I've used the "de-referencing" operator "*" on score. We will have a look at de-referencing later, but let's just say it's a way to get the value that the reference is "referencing".

If I didn't use the "*" here, the compilier will complain about not being able to add {integer} refering to the 5 and &mut u8 (the score) as it considers them of different types.

Then I've added the 5 points bonus to the old values.
The result of this will be all the values have increase by 5 from their old values (compare them to the last println output):

{"Ben": 30, "Windy": 30, "Bob": 35, "Jane": 35}

We have now looked at Hash Maps and saw how they are almost similar to Python's Dictionaries except for the types restrictions.
Next we will look at Rust's Error Handling. See you then 👋

Data Engineering Lifecycle - Basics - Theory

Fady GA 😎 — Sun, 15 Oct 2023 15:34:51 +0000

This is the first article in my Data Analytics Made Simpler series. To know more about it, please visit it's website where I'm giving more details about what's exactly this series and what's my motivation to do something like that and how I plan to release its content!

I wanted to start my series by drawing a mental image of what it will cover. In this article, I'll explain what I mean by "data analytics" in the series title and will give more details on what exactly is Data Engineering and will explain its general components through the Data Engineering Lifecycle framework.

For the same article but in Arabic, please click here

Table of Content:

What I mean by Data Analytics
What is Data Engineering
The Data Engineering Lifecycle
What a Data Engineer is not
Conclusion

What I mean by Data Analytics:

The term Data Analytics is very broad and can have different meanings for different people! So, I want to be clear about it.

When I talk about Data Analytics, I almost visualize the following diagram in my head.

In my head, a huge chunk of Data Analytics is Data Engineering that I'll be explaining shortly, regardless of the data volume, variety, or velocity (understanding those terms will become easier over time, trust me 😉).
The remaining chunk is Data Visualization. To me it's the most natural thing to do with data after that we have done stuff to it in the Data Engineering phase, but it isn't the only thing we can do.
By all means, this is not an official definition for Data Analytics but this is how I'll be tackling it in my series.

What is Data Engineering?

The overly simplified version of the Data Engineering definition is as follows:

Data Engineering is the data field concerned by moving data from source to destination systems while applying transformations to the data according to the business requirements.

For example, imagine an enterprise with multiple applications. Due to different Development teams, each application is generating its logs with its own format different from the other applications but it still contains the same info. Now, the management wants a dashboard to show users access trends over the past 3 months aggregated for all the applications.
The operation of collecting the logs from each application, unify their format and extract the users access data then perform aggregation to it, and finally loading the aggregated data to a data warehouse (more on that later on in the series) that the dashboard will eventually use, is called, you've guessed it, Data Engineering!

Now that you have some idea on what Data Engineering is, I should add to the previous definition that Data Engineering isn't just the data movement and the applied transformations, it's the design, implementation, and maintenance of the whole automated system (called, pipeline) that performs the whole thing!

You may now get the impression that Data Engineering is an enterprise thing! You won't be wrong! But you won't be entirely correct either 😁. Think about it, if you have a bunch of Excel workbooks laying around your laptop of say, monthly sales for the past 15 years with each month is in its own workbook and you've created a Python script to open those up, get the monthly sales total, and finally load all of those in an aggregated report. According to the mentioned definition, you've kind of did Data Engineering!

The Data Engineering Lifecycle:

I'll start this section by showing it to you then I'll explain:

So, if we talked about Data Engineering leaving out the technical details and tools selection, we will get a "framework" that tries to explain how it works, i.e. The Data Engineering Lifecycle!

If you take a closer look at the previous figure, you will be able to verify the earlier Data Engineering definition! You will see that we are extracting or "ingesting" data from sources that "generate" it, applying "transformations" to it, then finally load or "serve" it to destination systems that will generally be either analytics or machine learning workloads.
You may have noticed that there is a "storage" block that spans over the "ingestion", "transformation", and "serving" blocks. This is because we can actually store data at any stage of the lifecycle. For example, a common Datalake architecture (we will know about Datalakes later in the series) is to store raw data before processing. Then, do some filtering and cleaning transformations and store the resulting data in a dedicated location. And finally, do some aggregations to the processed data and store those as well in a dedicated location.

This is called the Medallion Architecture as the raw area is called the Bronze stage, the processed area is called the Silver stage, and the aggregated area is called the Gold stage.

One other thing to know about the Data Engineering Lifecycle is that it isn't a sequential flow! Meaning that you can (and often will) do blocks out of the illustrated order and/or more than once. For example, we can serve the ingested data directly to some "data consumer" systems before any transformation then perform the transformations and serve the final output to other systems.

What a Data Engineer is NOT:

There are a lot of Data related roles out there with blurry boundaries between them. In my opinion, this could be due to the rapid evolution of how we (or the systems) produce data and consume it, which adds scope to existing data roles or creates new roles entirely.
I just want to clear the confusion about what a data engineer is not to better understand what a data engineer actually is!

A Data Engineer should be familiar with statistics if their pipeline requires statistical transformations but it's a "Data Analyst's" role to make sense of the data and use statistical analysis to better understand it and uncover hidden insights!
A Data Engineer might be familiar with statistical modelling and machine learning algorithms, but it is the job of a "Data Scientist" to create models from data for machine learning use cases.
A Data Engineer should know how to handle production workloads, but it is the job of a "Machine Learning Engineer" to deploy production Machine learning systems.
A Data Engineer works a lot with databases, but it's the role of a "Database Administrator" to handle the administration part of a database like users' privileges, DB maintenance, backups, ..., etc.
A Data Engineer could know a bit of Software Engineering (in fact, it will be very beneficial if they do!) but it is the job of a "Software Engineer" to create apps even if data related.

Like we've seen in the Data Engineering definition, A Data Engineer should focus on the design, creating, and maintaining their Data Pipeline that works within the Data Engineering Lifecycle framework working with some data role like DB Admins or Software engineers to serve other Data roles like the Data Analysts and/or Scientists!

Conclusion:

Data Engineering is a relatively new Data role and is still evolving in my opinion. I just wanted here to give you a taste of what we will be handling in the Data Analytics Made Simpler series and to help you identify the scope of a Data Engineer.

References:

I've used the same approach explaining Data Engineer as the "Fundamentals of Data Engineering" book by Joe Reis and Matt Housley and explained the Data Engineering Lifecycle as they did in their book. A highly recommended read that mostly doesn't require prior knowledge of Data Engineering.

دورة حياة هندسة البيانات - الأساسيات - نظري

Fady GA 😎 — Sun, 15 Oct 2023 15:34:31 +0000

هذه هي المقالة الأولى في سلسلة تحليلات البيانات الخاصة بي. لمعرفة المزيد عنها، يرجى زيارة موقعها على الإنترنت حيث أقدم المزيد من التفاصيل حول ما هي بالضبط هذه السلسلة وما هو دافعي للقيام بشيء من هذا القبيل وكيف أخطط لإصدار محتواها!

أردت أن أبدأ سلسلتي برسم صورة ذهنية لما ستغطيه. في هذه المقالة ، سأشرح ما أعنيه ب "تحليلات البيانات" الموجود في عنوان السلسلة وسأقدم مزيدا من التفاصيل حول ماهية هندسة البيانات بالضبط وسأشرح مكوناتها العامة من خلال إطار دورة حياة هندسة البيانات.

لقراءة نفس المقالة باللغة الانجليزية، اذهب هنا

جدول المحتويات :

ما أعنيه بتحليلات البيانات
ما هي هندسة البيانات
دورة حياة هندسة البيانات
ما لا يفعله مهندس البيانات
الخلاصة

ما أعنيه بتحليلات البيانات:

مصطلح تحليلات البيانات واسع جدا ويمكن أن يكون له معاني مختلفة لأشخاص مختلفين! لذا ، أريد أن أكون واضحا بشأن ذلك.

عندما أتحدث عن تحليلات البيانات ، أتخيل تقريبا الرسم التالي في رأسي.

في رأسي ، جزء كبير من تحليلات البيانات هو هندسة البيانات التي سأشرحها قريبا ، بغض النظر عن حجم البيانات أو تنوعها أو سرعتها (سيصبح فهم هذه المصطلحات أسهل بمرور الوقت ، ثق بي 😉).
الجزء المتبقي هو تصور البيانات. بالنسبة لي ، هذا هو الشيء الطبيعي الذي يجب القيام به مع البيانات بعد أن قمنا بمعالجتها في مرحلة هندسة البيانات ، لكنه ليس الشيء الوحيد الذي يمكننا القيام به.
هذا ليس تعريفا رسميا لتحليلات البيانات ولكن هذه هي الطريقة التي سأتناولها بها في سلسلتي.

ما هي هندسة البيانات؟

النسخة المبسطة من تعريف هندسة البيانات هي كما يلي:

هندسة البيانات هي المجال المعني بنقل البيانات من أنظمة المصدر إلى أنظم الوجهة مع تطبيق التحولات عليها وفقا لمتطلبات العمل.

على سبيل المثال ، تخيل مؤسسة بها عدة تطبيقات خاصة بها. نظرا لاختلاف فرق التطوير ، يقوم كل تطبيق بإنشاء سجلاته بتنسيق خاص به مختلف عن التطبيقات الأخرى ولكنه لا يزال يحتوي على نفس المعلومات. الآن ، تريد الإدارة لوحة معلومات لإظهار انماط استخدام المستخدمين على مدار الأشهر ال 3 الماضية مجمعة لجميع التطبيقات.

عملية جمع السجلات من كل تطبيق ، وتوحيد تنسيقها واستخراج بيانات وصول المستخدمين ثم تجميعها ، وأخيرا تحميل البيانات المجمعة إلى مستودع بيانات (المزيد حول ذلك لاحقا في السلسلة) التي ستستخدمها لوحة المعلومات في النهاية ، تسمى ، لقد خمنت ذلك ، هندسة البيانات!

الآن بعد أن أصبح لديك فكرة عن ماهية هندسة البيانات ، يجب أن أضيف إلى التعريف السابق أن هندسة البيانات ليست مجرد حركة البيانات والتحولات المطبقة عليها فقط، إنها تصميم وتنفيذ وصيانة النظام الآلي بأكمله (يسمى ، خط الأنابيب) الذي يؤدي كل شيء!

قد تحصل الآن على انطباع بأن هندسة البيانات هي شيء له علاقة بالمؤسسات الكبيرة فقط! لن تكون مخطئا! لكنك لن تكون صحيحا تماما أيضا 😁. فكر في الأمر ، إذا كان لديك مجموعة من ملفات اكسل موجودة في الكمبيوتر المحمول الخاص بك ، على سبيل المثال ، تحتوي على المبيعات الشهرية على مدار ال 15 عاما الماضية مع كل شهر موجود في الملف الخاص به وقمت بإنشاء برنامج بايثون لفتح الملفات ، والحصول على إجمالي المبيعات الشهرية ، وأخيرا تحميل كل الارقام في تقرير مجمع. وفقا للتعريف المذكور ، لقد قمت نوعا ما بممارسة هندسة البيانات!

دورة حياة هندسة البيانات:

سأبدأ هذا القسم بعرض الرسم الذي يوضح ماهية دورة حياة هندسة البيانات ثم سأشرح بالتفضيل لاحقاً:

إذا تحدثنا عن هندسة البيانات مع ترك التفاصيل الفنية واختيار الأدوات المستخدمة، فسنحصل على "إطار" يحاول شرح كيف تعمل هندسة البيانات ، أي دورة حياة هندسة البيانات!

إذا ألقيت نظرة فاحصة على الشكل السابق ، فستتمكن من التحقق من تعريف هندسة البيانات السابق ذكره! سترى أننا نستخرج أو "نستوعب" البيانات من المصادر التي "تولدها" ، ونطبق "التحولات" عليها ، ثم نقوم أخيرا بتحميلها أو "تقديمها" إلى أنظمة الوجهة التي ستقوم بشكل عام إما بتحليلها أو تغذية انظمة تعلم الالة.

ربما لاحظت أن هناك كلمة "تخزين" تمتد تحت "الاستخراج" و "التحويل" و "التقديم". هذا لأنه يمكننا بالفعل تخزين البيانات في أي مرحلة من مراحل دورة الحياة. على سبيل المثال ، هناك بنية شائعة عند تصميم "برك البيانات" (سنعرف عن "برك البيانات" لاحقا في السلسلة) هي تخزين البيانات الأولية قبل المعالجة. بعد ذلك ، نقوم ببعض تحويلات التصفية والتنظيف ثم تخزين البيانات الناتجة في مكان مخصص. وأخيرا ، نقوم ببعض التجميعات للبيانات المعالجة ونقوم بتخزينها أيضا في مكان مخصص.

وهذا ما يسمى بنية الميدالية حيث تسمى المنطقة الخام المرحلة البرونزية ، وتسمى المنطقة المعالجة المرحلة الفضية ، وتسمى المنطقة المجمعة المرحلة الذهبية.

شيء آخر يجب معرفته عن دورة حياة هندسة البيانات هو أنها ليست تدفقا متسلسلا! بمعنى أنه يمكنك (وغالبا ما سوف) تقوم بتنفيذ المراحل المذكورة خارج الترتيب الموضح و / أو تنفيذها أكثر من مرة. على سبيل المثال ، يمكننا تقديم البيانات التي تم استخراجها مباشرة إلى بعض أنظمة "مستهلكين البيانات" قبل أي تحويل ثم إجراء التحولات وتحميل الناتج النهائي إلى أنظمة أخرى.

ما لا يفعله مهندس البيانات

هناك الكثير من الأدوار المتعلقة بالبيانات ولكن الحدود ضبابية بينها. في رأيي ، قد يكون هذا بسبب التطور السريع لكيفية ننتج (أو تنتج الأنظمة) البيانات واستهلاكها ، مما قد يضيف مسؤوليات جديدة لأدوار البيانات الحالية أو يخلق أدوارا جديدة تماما.

أريد فقط إزالة الالتباس حول ما لا يفترض ان يقوم به مهندس البيانات لفهم ما الذي يجب ان يقوم به مهندس البيانات!

يجب أن يكون مهندس البيانات على دراية بعلوم الاحصاء حيث من الممكن ان يقوم بتحويلات احصائية على البيانات ، لكن هذا دور "محلل البيانات"! هو الذي يفهم البيانات باستخدام التحليل الإحصائي لفهمها بشكل أفضل والكشف عن الدلائل الخفية!

قد يكون مهندس البيانات على دراية بالنماذج الإحصائية وخوارزميات تعلم الآلة ، ولكنها مهمة "عالم البيانات" إنشاء نماذج من البيانات لحالات استخدام تعلم الآلة.

يجب أن يعرف مهندس البيانات كيفية التعامل مع الانظمة التي تعمل في بيئة الانتاج ، ولكنا مهمة "مهندس اتعلم الآلة" نشر أنظمة تعلم الآلة للإنتاج.

يعمل مهندس البيانات كثيرا مع قواعد البيانات ، ولكنه دور "مسؤول قاعدة البيانات" هو التعامل مع الجزء الإداري من قاعدة البيانات مثل امتيازات المستخدمين ، وصيانة قاعدة البيانات ، والنسخ الاحتياطية ، ...

يمكن لمهندس البيانات معرفة القليل من هندسة البرمجيات (في الواقع ، سيكون من المفيد جدا إذا فعل ذلك!) ولكنها مهمة "مهندس البرمجيات" هي إنشاء تطبيقات حتى لو كانت متعلقة بالبيانات.

كما رأينا في تعريف هندسة البيانات ، يجب أن يركز مهندس البيانات على تصميم وإنشاء وصيانة "خط أنابيب" البيانات الخاص به الذي يعمل ضمن إطار دورة حياة هندسة البيانات حيث يعمل مع بعض أدوار البيانات مثل مسؤولي قواعد البيانات أو مهندسي البرمجيات لخدمة أدوار البيانات الأخرى مثل محللي البيانات و / أو علماء البيانات!

الخلاصة:

هندسة البيانات هي مجال جديد نسبيا ولا تزال في طور النمو في
رأيي. أردت فقط هنا أن أقدم لك لمحة عما سنتعامل معه في سلسلة "Data Analytics Made Simpler"
ولمساعدتك في تحديد هوية مهندس البيانات.

المصادر

لقد استخدمت كتاب
"Fundamentals of Data Engineering" Joe Reis and Matt Housley
في اعداد هذه المقالة حيث يعتبر هذا الكتاب من المصادر الجيدة جدا للتعرف على مجال هندسة البيانات بدون الحاجة لخبرة مسبقة

Learning Rust 🦀: 17 - Rust Collections: Strings - How complex can it be?!

Fady GA 😎 — Fri, 13 Oct 2023 15:26:54 +0000

The second common Rust collection we will visit is the String! Rust has taken a ... "different" approach when working with the String type than other programming languages but seriously, how complex can it be?! Let's find out!

⚠️ Remember!

You can find all the code snippets for this series in its accompanying repo

If you don't want to install Rust locally, you can play with all the code of this series in the official Rust Playground that can be found on its official page.

⚠️⚠️ The articles in this series are loosely following the contents of "The Rust Programming Language, 2nd Edition" by Steve Klabnik and Carol Nichols in a way that reflects my understanding from a Python developer's perspective.

⭐ I try to publish a new article every week (maybe more if the Rust gods 🙌 are generous 😁) so stay tuned 😉. I'll be posting "new articles updates" on my LinkedIn and Twitter.

That's easy, because in Rust is a "collection" of bytes 😁!
Actually, the String type in Rust is a wrapper of a u8 Vector (Vec<u8>) as it contains all the methods available for the Vector type plus other methods that makes working with strings easier. It holds the bytes of a UTF-8 encoded string which might become problematic as we will see later.

You may recall that String type is located in the Heap and it's of variable length while string slice str (usually used as a reference &str) is located in the stack.

Creating a new String:

We can create new String as follows:

let my_string: String = String::new();

This will create a new and empty "immutable" String in my_string.

Sometimes, we would want to create a String from a seed text instead of an empty one. To do that, we can either use the from method of the String type or use the convenient to_string method that is avaible to some of Rust's core types.

let my_string: String = String::from("Hello");
println!("from: {my_string}");
let word: &str = "Hello";
let my_string: String = word.to_string();
println!("to_string: {my_string}");

The use of from or to_string is just a matter of style and readability!

Updating a String:

There are more than one way to update a "mutable" String, we can use the push_str method that the String type implements to append the new string passed to it to the original String. As the String type is a wrapper for the Vector type, it to has a push method that - similar to push_str - pushes a Char type to the original String. Let's see both of those in action:

let mut new_str = String::from("Hello, ");
new_str.push_str("There");
println!("Greeting: {new_str}");
new_str.push('!');
println!("New greeting: {new_str}");

This will output:

Greeting: Hello, There
New greeting: Hello, There!

This code does what you expect it to do. We've used push_str to append "There" to the original String ("Hello, "). Next, we've used push to append "!" at the end of the original String now containing "Hello, There".

Notice how the push takes a Char type since the "!" is passed to it with single quotes denoting the Char type.

Another way to update a string is the "+" operator, let's check it out:

let s1 = String::from("Hello,");
let s2 = String::from("There!");
let s3 = s1 + &s2;

In short, after executing this code s3 will hold "Hello, There!" but there is a lot that is going on here!
You may notice that s1 is "moved" into s3 i.e. we can't use s1 anymore in our app after the s3 line and we are "borrowing" s2. This is not a random thing; this has to do with the "+" operator as it uses the add method with has the signature:

fn add(self, s: &str) -> String

As you can see, the add method takes ownership of self (the s1 in our example) and takes s as a string literal which is supposed to represent s2.

You may wonder how this code works as s2 is a String not a &str in our example. At this point, let's just say that the Rust compiler can coerce (convert) the &String type into &str and we will revisit that later.

Now, consider the example below:

let s1 = String::from("Hello");
let s2 = String::from("There");
let s3 = String::from("!");
let s4 = s1 + ", " + &s2 + &s3;
println!("s4 is {s4}");

This will output:

Hello, There!

But as you can see, it may become unnecessarily complicated to use the "+" operator when try to modify a String with a relatively big number of string literals. This is a perfect use case for yet another new Rust macro, format!. Now, let's see how we can use it in the previous example:

let s1 = String::from("Hello");
let s2 = String::from("There");
let s3 = String::from("!");
let s4 = format!("{s1}, {s2}{s3}");
println!("s4 is {s4}");

This will produce the exact same result as before but notice that s1 doesn't "move" into s4 which can be convenient if we were planning to use it later in the app.

Indexing into Strings:

Up until now, everything more or less looks the same as other programming languages but here is where the Rust fun starts

In Python, doing something like this is perfectly fine:

s = "Hello"
H = s[0]

Now given that the String type is in fact a wrap for a Vector and the later can have its elements accessed by using zero-based indices, so the following should work, right?

let my_string = String::from("Greetings 😉 !");
let G = my_string[0];

Wrong! This will produce the following error:

the type `String` cannot be indexed by `{integer}`

For reasons that will become clear in just seconds, Rust only permits you to "slice" the String type! So, something like the following code will compile and work:

let my_string = String::from("Greetings 😉 !");
let Gree = &my_string[..4];
println!("{Gree}");

This will work and will print Gree in the terminal.
Now, let's try this again and this time we will try to extract the "wink" emoji that is at position 10 with the preceding and trailing spaces:

let my_string = String::from("Greetings 😉 !");
let wink = &my_string[9..12];
println!("{wink}");

Here, we are starting our slice at the first space at position 9 then end it at the trailing space at position 11 (the end isn't inclusive here). The app compiles but it will panic!

The reason behind that is the String stores - like I've mentioned - UTF-8 encoded "bytes" and note the visual character that we see! It turned out that "😉" is four bytes long, and although we've correctly set the slice's start at position 9 as all the character up to this point are using only one byte, the slice's end happens to be the second byte of the "😉" which isn't a printable character and that why the app panicked! Although, String slicing is perfectly legal in Rust, it can be dangerous at runtime!

Now, why doesn't Rust allow indexing?
It's because of the UTF-8 encoding thing to avoid cases when you could return unprintable characters and introducing bugs that might not be discoverable immediately, Rust doesn't compile that code!

So why use slicing as it also can return unprintable characters?
Slicing is Rust's way to tell you to be more specific to return a range of bytes if you know what you are doing!

The String is not simple!

String in Rust isn't that complex but isn't that simple either!
Generally, you can represent a String by either its "bytes" scalars or its "characters" (char type). For example, the string used in the previous example "Greetings 😉 !" has the following scalar bytes representation:

And the following "characters" representations:

G
r
e
e
t
i
n
g
s

😉

!

Notice how there are 13 characters and 16 bytes representing the same string!

This distinction in strings representation can be abstracted away in some programming languages such as Python and the developer doesn't think much of how the strings are represented (unless he needs to, say when sending strings over the network). But in Rust, it by design forces the developers to think about how strings are represented, namely the UTF-8 encoding, to avoid bugs early on in the development.

Iterating over a String:

Finally, the String type has some useful methods to iterate over its elements.
For example, if we want to iterate over the string's characters representation, we can use the following:

for c in my_string.chars() {
    println!("{c}")
}

Similarly, if we want to iterate over its bytes, we use:

for b in my_string.as_bytes() {
    println!("{b}")
}

At the end, Rust's String isn't that complex! It's a collection of the UTF-8 encoded bytes of a string and Rust wants you to think of that from the start. In the next one we will explore another common Rust collection, the Hash map. See you then 👋

Learning Rust 🦀: 16 - Rust Collections: Vectors

Fady GA 😎 — Thu, 05 Oct 2023 13:12:11 +0000

Now we will have a look at some of Rust's most common Collections, The Vector, String (yes! String type is a collection of characters 😎), and Hash Map types. We will start first with Cyborg mmm... I mean, Vector 😁

⚠️ Remember!

You can find all the code snippets for this series in its accompanying repo

If you don't want to install Rust locally, you can play with all the code of this series in the official Rust Playground that can be found on its official page.

⚠️⚠️ The articles in this series are loosely following the contents of "The Rust Programming Language, 2nd Edition" by Steve Klabnik and Carol Nichols in a way that reflects my understanding from a Python developer's perspective.

⭐ I try to publish a new article every week (maybe more if the Rust gods 🙌 are generous 😁) so stay tuned 😉. I'll be posting "new articles updates" on my LinkedIn and Twitter.

Before I show you how to create a new Vector, there is something you need to know. Unlike Arrays or Tuples that can hold a list of items themselves, Vectors are created in the Heap. Meaning, that they don't have to be of known length at compile time! But similar to Arrays and Tuples, they must contain elements of the same type.

We can create an empty Vector as follows:

let v: Vec<i32> = Vec::new();

Because the compiler doesn't know the type of the elements of the newly created Vector, there type must be explicitly stated using Vec<T> (Yes, generics which we will talk about them later). So here, we are creating an empty Vector that will hold element of type i32.

Also, we can create prefilled Vectors like this:

let v = vec![1, 2, 3];

By using the vec! macro, we are creating also a Vec<i32> Vector as i32 is the default integer type.

Vectors - like any other Rust type - are immutable by default meaning that you can't add or remove items from them. To make them mutable, we use the mut keyword.

Updating Vectors:

The Vector type has a lot of useful methods that can update its elements. push and pop are very common in such operations. Consider the following example where we create an empty Vector and add 3 elements to it then remove one:

fn main() {
    let mut v: Vec<i32> = Vec::new();
    v.push(1);
    v.push(2);
    v.push(3);
    println!("Length of v is: {}", v.len());
    v.pop();
    println!("New length of v is: {}", v.len());
}

The first print will give you Length of v is: 3 and the second one will print New length of v is: 2. push adds elements to the Vector. I like to imagine the Vector as a "stack" and push adds a new element each time at the top of the stack which means that the very first item that was added using push will be at the bottom of the stack. pop on the other hand, remove the top item from this stack.

len is used to get the Vector "length" which in this case is the elements count.

Reading Vectors elements:

You can access the Vector's elements using the square brackets "[ ]" similar to arrays (or Python's Lists and Tuples). We can also use the get method as follows:

fn main(){
    let v = vec![1, 2, 3];

    let first_element = &v[0];
    println!("First element is {first_element}");

    let second_element: Option<&i32> = v.get(1); 
    match second_element {
        Some(second) => println!("Second element is {second}"),
        None => println!("There isn't a second element!"),
    }
}

Vectors index start at 0 so &v[0] will return a reference to the first element in the Vector which in this case will be 1. get returns an Option type (remember the Option type?) as the index passed to get can be outside of the Vector boundaries, then it will return the None variant of the Option type. Otherwise, it will return Some(T). Here, get(1) should return a Some(i32) variant as the element at index 1 exists.

Now let's see what will happen if we requested an "out of bound" element:

let v2 = vec![1, 2, 3];
let val = &v2[100];

This will compile just fine but will make the application "panic"! This behavior can be used if you want to crash the program if an out-of-bound value is entered. Now let's look at get:

let v2 = vec![1, 2, 3];
let val = v2.get(100); // Returns None
    if let None = val {
        println!("None is returned!")
    }

The application won't "panic" in this case and None is returned! will be printed out.

if let is another way to do pattern matching that I mentioned here

One last thing before we leave this point, do you remember that we can't have both mutable and immutable references for a type in the same scope? This applies for Vector elements as well! Let's examine the following:

let mut v = vec![1, 2, 3];
let first = &v[0];
v.push(4);
println!("{first}");

This should compile just fine, right? We are using an immutable reference for the first item in the Vector and then adding a new item at the end of the vector, those should be unrelated, right? No, they are related and the code won't be compiled!

Do you recall that Vectors are stored in the Heap? Which means that they can expand as they store their elements next to each other. And if the allocated memory to the Vector isn't large enough to hold all of its new elements, a new memory location will be assigned to it. Meaning that the let first = &v[0]; will be referring to an invalid location in memory if the v.push(4); expression caused the Vector to be reallocated. Therefore, prinln!("{first}"); will cause a compilation error!

Reading Vectors elements:

Similar to Arrays, we can iterate over a Vector like this:

let v = vec![1, 2, 3];
for i in &v {
    println!("{i}");
}

This will result in:

1
2
3

We can also change the values of the Vector's elements while iterating:

let mut v2 = vec![20, 40, 60];
for i in &mut v2 {
    *i -= 20;
}
println!("Modified vector:");
for i in &v2 {
    println!("{i}")
}

This will print:

0
20
40

The "*" before i in the first for is called "dereference" operator which we will talk about in future articles.

Using Enum to make a Vector contain multiple types:

Vectors are lists of elements of the same type but we can encapsulate different types with Enum in a custom type and use it with Victors! Let's see that in action:

#[derive(Debug)]
enum Item {
    Apple(u8),
    Banana(u8),
    Tape(f32),
    Book(String),
}

let shopping_cart = vec![
    Item::Apple(10),
    Item::Banana(5),
    Item::Book(String::from("Moby-Dick")),
    Item::Tape(5.5),
];
println!("Items in the shopping cart:");
for item in &shopping_cart {
    println!("{item:?}");
}

When running this code, we will have the following output:

Items in the shopping cart:
Apple(10)
Banana(5)
Book("Moby-Dick")
Tape(5.5)

Here we are building a shopping_cart Vector. But since Vectors can only have elements of the same type, we have created the Item Enum that has only 4 variants, Apple that takes their count, Banna that also takes their count, Tape that takes its length, and finally Book that takes its title. Then, we filled our shopping_cart Vector with variants of the Item enum which its variants encapsulate other data types and are technically of the same type. This technique will work if we know all the variants that we can use. But if we don't, the Enum won't work and we will have to use Traits which we will discuss later.

Vectors are the closest thing Rust has to Python's List's. They are relatively straight forward to create and use. Next, we will revisit another Rust Collection, The String! See you then 👋

Learning Rust 🦀: 15 - How you can organize your Rust code with "Modules"

Fady GA 😎 — Sun, 01 Oct 2023 13:59:08 +0000

In this article we will see how the Rust's Module system works and how we can group related code together to enhance our project's maintainability. Let's jump in!

⚠️ Remember!

You can find all the code snippets for this series in its accompanying repo

If you don't want to install Rust locally, you can play with all the code of this series in the official Rust Playground that can be found on its official page. That said, the playground may not be suitable for this topic.

⚠️⚠️ The articles in this series are loosely following the contents of "The Rust Programming Language, 2nd Edition" by Steve Klabnik and Carol Nichols in a way that reflects my understanding from a Python developer's perspective.

⭐ I try to publish a new article every week (maybe more if the Rust gods 🙌 are generous 😁) so stay tuned 😉. I'll be posting "new articles updates" on my LinkedIn and Twitter.

But before I show you how to define a module in Rust, I just want to demonstrate how a Rust project is structured using cargo. When we type cargo new <package name> into our terminal, cargo makes a "package" structure. A package contains "crates" which are in the src directory. Crates can be either a library crate or a binary crate. The difference between the two is that a library crate doesnt contain a main function and it just provides some functionality to the binary crate(s) in the same package or when used as an external crate.

A package can contain only one library crate and one or more binary crates.

So far, all the packages we have created contain only one binary crate.

In the src directory, the binary crate root is the main.rs and the library crate is the lib.rs. The root crate will have the same name as the package's. If our package contains binary crates other than the one in main.rs, we put them in src/bin/ directory and they will be compiled into executables alongside our default binary crate.

If we typed cargo new building, it will create a "building" package and the root binary crate (written in scr/main.rs) will be named "building" too. If we wrote a library crate (in src/lib.rs), it will also have the same name - building - as the package.

Now that we've got this introduction out of our way, let's see how we can define a module:

mod my_module {
    // module contents
}

That's it! 🙂

To better demonstrate Rust's module system, let's assume that we are building a digital "hotel" in Rust and we will use the modules system to organize similar code together.

Type cargo new hotel and create a lib.rs in the src directory.

We can type cargo new hotel --lib and this will create a package hotel with only one library crate.

Next, put the following module in our recently created lib.rs file.

mod reception {
    mod booking {
        fn book_room() {
            println!("Booking room!")
        }
        fn cancel_booking() {}
    }

    mod guests_management {
        fn guest_checkin() {
            println!("Checking in!")
        }
        fn guest_checkout() {}
        fn receive_guest_request() {}
    }
}

mod facilities {
    mod house_keeping {
        fn clean_room() {}
        fn deliver_meal() {}
    }

    mod maintenance {
        fn pool_maintenance() {}
        fn electrical_maintenance() {}
        fn building_maintenance() {}
    }

    mod restaurants {
        fn prepare_tables() {
            println!("Preparing table")
        }
        fn make_meal() {
            println!("Making meal")
        }
    }
}

A module in Rust can contain other items such as functions, structs, enums, constants, traits, and other modules. Here we have two main modules, reception and facilities both containing "child" modules that contain function.

Using Paths:

To access an item inside a module we use a "file system-like" path which can be either absolute or relative. To demonstrate that, write the following function in src/lib.rs outside of any module.

pub fn go_on_vacation() {
    crate::reception::booking::book_room();         // Absolute path
    reception::guests_management::guest_checkin();  // Relative path
}

This function calls two other function from the reception module. It will be a lot easier to understand the module paths if we imagined our lib.rs as the following file system:

crate
    ├── reception
    |       ├── booking
    |       |       ├── book_room
    |       |       └── cancel_booking
    |       └── guests_management
    |               ├── guest_checkin
    |               └── guest_checkout
    └── facilities
            ├── house_keeping
            |       ├── clean_room
            |       └── deliver_meal
            ├── maintenance
            |       ├── pool_maintenance
            |       ├── electrical_maintenance
            |       └── building_maintenance
            └── restaurants
                    ├── prepare_tables
                    └── make_meal

Where the crate keyword represents the root crate or the "/" (root) in a Linux file system.

In the first call, we used an absolute path where we started with the root crate (crate keyword) then all the way to the book_room function. In the second call, we use a relative path as both the go_on_vacation function and the reception module are on the same level (children of the root crate), we can refer to the reception module from the go_on_vacation function like we did.

Unfortunately, if we tried to run this code, it wouldn't compile! The compilier will complain about booking and guests_management modules being private!

The `pub` keyword:

In order to solve the problem we saw in the previous section, we can use the pub (for public) keyword. Using pub before mod will make the module public. Armed with our new found knowledge, let's add the pub keyword as follows:

mod reception {
    pub mod booking {
        // snip
    }

    pub mod guests_management {
        // snip
    }
}

If we run the code now, the compiler will complain, again!

This time, it will complain about book_rook and guest_checkin functions being private. This highlights an important rule about the pub keyword, making a module public won't make its children public! and you will have to explicitly choose which components for public access as they are private by default. Let's do the same for those functions as we did with the modules:

mod reception {
    pub mod booking {
        pub fn book_room() {
            println!("Booking room!")
        }
        // snip
    }

    pub mod guests_management {
        pub fn guest_checkin() {
            println!("Checking in!")
        }
        // snip
    }
}

Now, it will be compiled! But wait! we didn't put pub before reception module, how we could access it?

As the reception module and the go_on_vacation function are both on the same level in the "modules system" (the root crate), they can see each other.

Items that are on the same level are called "siblings" while items that are one level down are called "children" and the items that are one level up are called "parents".

Structs and Enums inside a module:

As I've mentioned before, modules can contain Structs and Enums and both of them (like functions) can accept the pub keyword but with a subtle difference! First, let's add a Room struct to the reception module.

mod reception {
    // snip
    pub mod guests_management {
        // snip
        pub fn get_room_number() -> i8 {
            10
        }
    }
    #[derive(Debug)]
    pub struct Room {
        pub view: String,
        pub beds_count: i8,
        number: i8,
    }
    impl Room {
        pub fn new(view: &str, beds: i8) -> Room {
            Room {
                view: String::from(view),
                beds_count: beds,
                number: guests_management::get_room_number(),
            }
        }
    }
}

And add the following in src/main.rs:

use hotel::guests;
fn main() {
    hotel::go_on_vacation();
}

Okay! There is a lot going on here, so let's break it down! 😉
Let's start first with the Struct definition. One thing to notice is that both the struct and some of its elements have the pub keyword. For structs if you are going to make the public, in addition to the pub before the struct definition, you have to explicitly set which elements are public as the default behavior for them is to be private. Here we have created a Room struct that has two public elements, view and beds_count, and one private element, number. But this presents a problem! As the number property is private, how can it be set??

We solve that by creating an implementation for this struct including a new function that acts as a factory function for new Rooms. This function sets the view and beds_number according to the guest's request and sets the number using a public function, get_room_number, inside the sibling guests_management module.

The implementation of the get_room_number function is really silly! It just returns the number 10 for every request! A better function would check the rooms in a database for example and return a vacant one. But this one here will do the trick.

Add the following code to the go_on_vacation function and run it:

pub fn go_on_vacation() {
    // snip

    let my_room = reception::Room::new("Sea view", 2);

    println!(
        "My room's vew is {} and my room number is {}. My room: {:?}",
        my_room.view, my_room.number, my_room
    );
}

As expected, it will produce a compilation error as follows:

error[E0616]: field `number` of struct `Room` is private
   --> src/lib.rs:100:31
    |
100 |         my_room.view, my_room.number, my_room
    |                               ^^^^^^ private field

We are not allowed to access "private" fields of a struct even if the struct itself is "public". If we ommited the erroneous code and replaced it with:

    println!("My room's vew is {}. My room: {:?}", my_room.view, my_room);

It will be compiled and we will be able to see that the number field of the Room struct is set to 10

My room's vew is Sea view. My room: Room { view: "Sea view", beds_count: 2, number: 10 }

For Enums, however, if we set it to public, all of its variants will be public too without the need to explicitly set their access to public which makes sense as all the variants of an enum are checked exhaustively so we can't choose a subset to make them public. Add the following enumb to the restaurant module inside the facilities module:

mod facilities {
    // snip
    pub mod restaurants {
       // snip
        #[derive(Debug)]
        pub enum Meal {
            Breakfast,
            Brunch,
            Lunch,
            Diner,
        }
    }
}

To test it, let's add the following new module to src/lib.rs:

pub mod guests {
    fn book_a_room() {}
    fn go_to_beach() {}
    fn go_to_pool() {}
    pub fn eat_meal() {
        let my_meal = crate::facilities::restaurants::Meal::Diner;
        println!("Eating {my_meal:?}");
    }
    fn end_vacation() {
        println!("Bye bye!");
    }
}

And add the following to the main function in src/main.rs:

fn main() {
    // snip
    hotel::guests::eat_meal();
}

If we try to run it, we will get the following result as expected

Eating Diner

The `super` and `self` keywords:

super and self are very useful keywords when used in Rust's module system paths. super refers to the direct parent of the module that is calling it and self refers to the same module that is calling it.

The use of self will become evidence in the "nested paths" section.

To demonstrate their use, add this code to the book_a_room function in the guest module:

    fn book_a_room() {
        super::reception::booking::book_room();
        self::go_to_beach();
    }

Here, the super expression is referring to the root crate as its the guest module's direct parent. Then it accesses the reception which is accessible from the root level all the way down to the book_room function.

In the expression with the self, it calls the go_to_beach function from the same guest module.

Scope management with the `use` keyword:

As we saw, we can use the module system paths to call functions and use elements inside modules provided that they are public. But sometimes the path can be very long! One way to overcome this is by using the use keyword as it brings to scope the element at its end and we can use it directly afterwards. Let's try it by making the following changes to the guests module:

pub mod guests {
    use super::reception::guests_management::guest_checkout;
    use crate::facilities::restaurants;

   // snip

    pub fn eat_meal() {
        restaurants::prepare_tables();
        restaurants::make_meal();
        // snip
    }
    fn end_vacation() {
        guest_checkout();
        println!("Bye bye!");
    }
}

Here we are bringing both the restaurants module and the guest_checkout function to scope using the use keyword. Notice that in the eat_meal function, we are writing restaurants directly without its parents as we already brought it into scope. And in the end_vacation function, we are also using the guest_checkout directly as -similar to the restaurants module- we brought it into scope.

There is a convention to use the path of the parent module if we are bringing in functions. That way, our code will be more readable as we know the source module of the function used. On the other hand, if we are bringing in structs or enums, we should use the full path.

Using external packages:

Up until this moment, we've used our own crates. To use external crates, we use the cargo.toml file in our project's root like we did here in our guessing game. We have to specify the crate name and version under the dependencies in the toml file:

rand = "0.8.5"

Then we can use the use keyword of the path to use the functionality provided by this crate.

Nested paths and globing:

Nested paths is a way to reduce the clutter when importing several elements from the same module. Consider this example when we try to import the make_meal and prepare_tables functions from the restaurants module:

use hotel::facilities::restaurants::make_meal;
use hotel::facilities::restaurants::prepare_tables;

The parents for both functions are the same, so we can use nested paths to reduce the preceding into this:

use hotel::facilities::restaurants::{make_meal, prepare_tables};

And what about the following situation where we want to import the module and one of its elements?:

use hotel::facilities::restaurants;
use hotel::facilities::restaurants::prepare_tables;

We simply use the self keyword which we talked about earlier:

use hotel::facilities::restaurants::{self, prepare_tables};

And we can use "globing" using "*" to bring "everything" (that is public) into scope from a module:

use hotel::facilities::restaurants::*

This will bring the prepare_tables and make_meal functions alongside with the Meal enum into scope.

Separate your modules in different files:

Up to this moment, our src/lib.rs contains all of our modules which is acceptable with a small code base. But what happens when our project grows bigger? One file containing all of our modules sounds like a bad idea and will decrease the code maintainability!

Luckily, we can split our modules into separate files. The rules are simple:

Write the module contents in a separate file inside src directory.
Use mod <module name> in its parent module (or binary crate) where is the same as its file name.
If the module contains other submodules, create a directory at the same module file level and follow point 1 and 2.

Let's demonstrate that with an example. Create a new package typing cargo new hotel_different_files. Next, create src/lib.rs and add pub mod reception; to it. After that, create a new file scr/reception.rs and the following to it:

pub mod booking;
pub mod guests_management;

#[derive(Debug)]
pub struct Room {
    view: String,
    beds_count: i8,
    number: i8,
}
impl Room {
    pub fn get_room(view: &str, beds: i8) -> Room {
        Room {
            view: String::from(view),
            beds_count: beds,
            number: guests_management::get_room_number(),
        }
    }
}

Here, we have referred to the booking and guests_management submodules and kept the Room struct and its implementation the same as before.

As the reception module contains submodules, we will create the src/reception/ directory which contains both submodules as src/reception/booking.rs and src/reception/guests_management.rs which will contain the following respectively:

// booking.rs
pub fn book_room() {
    println!("Booking room!")
}
pub fn cancel_booking() {}

// guests_management.rs
pub fn guest_checkin() {
    println!("Checking in!")
}
pub fn guest_checkout() {}
fn receive_guest_request() {}

pub fn get_room_number() -> i8 {
    10
}

And finally, to test everything out, add the following to src/main.rs:

use hotel_different_files::reception;

fn main() {
    let my_room = reception::Room::get_room("pool view", 1);
    println!("My room is {my_room:?}");
}

And it will print the Room struct exactly as our "one-file" implementation!

Few! 😮‍💨, this was a long but interesting one! Buy now, we should have a basic understanding of Rust's Module system. In the next article, We will start exploring Rust's common collections! See you then 👋

Learning Rust 🦀: 14 - Option Enum: An Enum and Pattern matching use case

Fady GA 😎 — Fri, 15 Sep 2023 10:46:23 +0000

We will finish off our Enum and pattern matching discussion that was started here by a case study on the Option standard enum. Let's begin.

⚠️ Remember!

You can find all the code snippets for this series in its accompanying repo

If you don't want to install Rust locally, you can play with all the code of this series in the official Rust Playground that can be found on its official page.

⚠️⚠️ The articles in this series are loosely following the contents of "The Rust Programming Language, 2nd Edition" by Steve Klabnik and Carol Nichols in a way that reflects my understanding from a Python developer's perspective.

⭐ I try to publish a new article every week (maybe more if the Rust gods 🙌 are generous 😁) so stay tuned 😉. I'll be posting "new articles updates" on my LinkedIn and Twitter.

But you have to remember to do that for every value (or data structure) that may become - for some reason 🤷‍♂️ - absent or invalid (i.e. Null) which might not be very scalable in a large code base.

Rust fixes that! by ... removing the Null 😁! Yes, you have read it write, there is no intrinsic Null type in Rust.

But ...

There is a standard Enum called "Option" (Option) that can be used in this use case.

The Option<T> Enum:

As I've mentioned, Rust has a standard Enum called "Option" which is loaded in the "prelude" (you don't have to load it with the use keyword) to deal with the Null situation that we have here. It has two variants:

Some(T)
None

And it is defined like this:

enum Option<T> {
    Some(T),
    None,
}

The first thing to notice is that weird "<T>" notation. This is a generic type notation (T for Type) and we will encounter it later on. What's important now is that you imagine "Option" as a "wrapper" for Any Type that you use in your program that might be either "Some" value or "None" (Null) and as we have learned in the last article, that pattern matching for Enums must be "Exhaustive", when using the Option Enum the compiler will force the Null value handling sparing you from remembering to do that yourself.

Using the Option Enum:

The Option Enum is load into the Rust program prelude by default so as its variants Some(T) and None and you don't have to use namespacing to use them.

Something like Option<i32>::Some(5) isn't necessary. Just use Some(5).

fn main() {
    // Using the Option Enum without importing it.
    let some_number = Some(5);
    let name = Some("My Name");
    let none: Option<i32> = None;
}

Here, we are directly using Some and None. Note that for Some, it can take any Type and the compiler can infer that type. But for None, we have to set its type.

The Option/ Enum has a set of useful methods like is_some() or is_none(). See the docs for a complete listing:

fn main() {
    // Using the Option Enum without importing it.
    let some_number = Some(5);
    let none: Option<i32> = None;

    // Some Option Enum methods
    assert_eq!(some_number.is_some(), true);
    assert_eq!(none.is_none(), false);  // Will cause assertion error
}

One important thing to know, is that Some(5) isn't 5! Let's try it out:

fn main() {
    // Using the Option Enum without importing it.
    let some_number = Some(5);
    let number = 5;

    let addition = some_number + number;
}

If we run this code, we will get the following compilation error:

error[E0369]: cannot add `{integer}` to `Option<{integer}>`
  --> src/main.rs:13:32
   |
13 |     let addition = some_number + number;
   |                    ----------- ^ ------ {integer}
   |                    |
   |                    Option<{integer}>

Basically, what this error message is telling us is that Rust don't know how to add a 5 that is wrapped in Option Enum with a normal i32 5 as they are now two different types.

So how could we do that??🤔

This is when Pattern Matching comes in and the purpose of the Option Enum becomes clear.

We will demo the use of bound data by using a function that doubles what's inside Some:

fn main() {
    // Using the data bound to Some.
    let five = Some(5);

    let mut result = double(five);
    println!("Double of 5 is: {result:?}");

    result = double(none);
    println!("Double of None is: {result:?}");
}

fn double(num: Option<i32>) -> Option<i32> {
    match num {
        Some(n) => Some(n * 2),
        None => None,
    }
}

The function is called double and it takes an Option Enum with i32 data type and outputs the same type. Inside the function, there is a match expression that has to matching arms for the two Option variants, Some and None. And hence pattern matching must be exhaustive, the compiler checks if all variants are mentioned and therefore enforces a "Null" handling arm.

If we run this code we will get:

Double of 5 is: Some(10)
Double of None is: None

So, when should we use the Option Enum you say?

We should use it whenever our values could be a Null because as long as the value isn't wrapped in it, Rust assumes that it will be always valid through the program execution.

The Option Enum is yet another way where Rust enforces its safety guard rails and we should use that for our advantage.

If Let: alternate way to pattern matching:

One more thing before we leave pattern matching (for now 😉), is the if let expression. Image that you are receiving a sensor readings that may be null due to network problems, corrupt data, or for whatever reason and you are interested in printing the value when it's 5 only. One way to do it is as follows:

fn main(){
    let reading: Option<u8> = Some(5);

    match reading {
        Some(5) => println!("Reading of 5 is received [match]"),
        _ => (),
    }
}

Remember the "_" placeholder from here? Here, we are only interested when we receive Some(5). Any other value including Null, will produce the Unit tuple () (basically, do nothing). But this is a bit verbose.

Another way to write the same code is by using if let:

fn main(){
    let reading: Option<u8> = Some(5);

    if let Some(5) = reading {
        println!("Reading of 5 is received [if let]")
    }
}

If we run this code, you will get the same result as before but less verbose. The if let syntax is a bit confusing but once you use it often, it will make sense:

if let <pattern> = <value> {
   <expression(s) to be executed if matched>
} else {
   <expression(s) to be executed if not matched>
}

So, when to use if let? There is no advantage of using if let over match for "one" pattern matching except it's less verbose. So, using one over the other depends on what you are trying to do...

If you want to take advantage of the exhaustiveness of the match, use it. Otherwise, if you are only interested in one match and you can safely ignore all the others, use if let.

As show, if let has and else clause that that be used as a "catch-all" for all other not matched patterns

Next, we will explore more about how we can use Packages, Crates and Modules in Rust. I may break it into several articles too. See you then 👋.

Learning Rust 🦀: 13 - Pattern matching basics

Fady GA 😎 — Fri, 08 Sep 2023 14:20:16 +0000

We will continue our Enum mini-series. This time we will discuss the pattern matching basics using Enums. Let's begin.

⚠️ Remember!

You can find all the code snippets for this series in its accompanying repo

If you don't want to install Rust locally, you can play with all the code of this series in the official Rust Playground that can be found on its official page.

⚠️⚠️ The articles in this series are loosely following the contents of "The Rust Programming Language, 2nd Edition" by Steve Klabnik and Carol Nichols in a way that reflects my understanding from a Python developer's perspective.

⭐ I try to publish a new article every week (maybe more if the Rust gods 🙌 are generous 😁) so stay tuned 😉. I'll be posting "new articles updates" on my LinkedIn and Twitter.

fn main(){
    let die_roll = 4;

    if die_roll == 1 {
        println!("one");
    else if die_roll == 2 {
        println!("two");
    else if die_roll == 3 {
        println!("three");
    else if die_roll == 4 {
        println!("four");
    else if die_roll == 5 {
        println!("five");
    else {
        println!("six");
}

In this example, we are throwing a die and print the number that we got in letters. Here, we are controlling the program flow based on the value of die_roll and the if expression should evaluate to either true or false. This is as far as we can go with if. We can re-write the previous example using match as follows:

fn main(){
    let die_roll = 4;

    match die_roll {
        1 => println!("one"),
        2 => println!("two"),
        3 => println!("three"),
        4 => println!("four"),
        5 => println!("five"),
        6 => println!("six"),
    }
}

This code snippet will work exactly the same as the previous one but with less typing 😉. But match doesn't only provide less typing benefits over if, it can match based on types either and not only values. Take a look at the following example:

#[derive(Debug)]
enum Superhero {
    Batman,
    Superman,
    WonderWoman,
    Flash,
    GreenLantern,
    CatWoman,
}

fn main() {
    // Basic matching.
    let hero = Superhero::Batman;
    let mut is_batman = false;
    let super_power = match hero {
        Superhero::Batman => {
            is_batman = true;
            "Rich"
        }
        Superhero::Superman => "Flying, Bulletproof, Heat vision, Cold breath",
        Superhero::WonderWoman => "Flying, Strength, Divine weapons",
        Superhero::CatWoman => "Stealth, Agility",
        Superhero::Flash => "Speed, Connection with the speed-force",
        Superhero::GreenLantern => "Flying, Instant light constructs, Lantern Corps",
    };
    println!("{hero:?}: {super_power} -- is batman {is_batman}");
}

If you run this code, you will get:

Batman: Rich -- is batman true

Let's break it down. As you can see, we've created our Superhero Enum containing my favorite superheroes 😁.

Note the #[derive(Debug)], remember this one 😉?

Then, we have created hero variable that holds the Superhero variant BATMAAAAN 🦇 and an is_batman boolean flag that is initially set to false. Then, we've created super_power variable that gets its value form a match expression depending on the passed Superhero variant in hero. And Finally, we print what we get.

Now let's inspect the match block. It is constructed by using the match keyword then the "Scrutinee" (as described by the docs) or in simpler terms, the variable that we want to match against. Following that in the curly brackets, we have the "match arm" which is the value - or more generally the pattern - we are matching followed by the => operation then the "expression" we want to execute (or return). Usually, you don't want to use curly bracket in the "expression" block if the expression is short but if your expression block contains more than one expression/statement then their use is mandatory.

Check out the "BATMAAAN" expression where we have changed the value of is_batman to true in addition to returning its superpower as &str (string literal).

There are two things to consider when using match for pattern matching:
1. The match patterns must be exhaustive i.e. using all the variants of the checked variable.
2. The match arms must return the same type i.e. one match arm can't return i32 and another one returns &str.

Notice that in this example we have listed all the superheroes in our Superhero Enum and all the match arms are returning string literals.

Think of the match flow as a "Coins sorting machine" that executes the first match arm that matches the tested variable.

Matching with patterns bind to values:

In the last article, we saw that we can bind data to Enums and using match is a way to use this bound data. Have a look at this example:

#[derive(Debug)]
enum SuperheroWithWeapon {
    Batman(FavoriteWeapon),
    Superman,
    WonderWoman(FavoriteWeapon),
    Flash,
    GreenLantern(FavoriteWeapon),
    CatWoman,
}

#[derive(Debug)]
enum FavoriteWeapon {
    LassoOfTruth,
    GreenLanternRing,
    Batarang,
}

fn main() {
    // Matching with patterns that bind to values with a catch-all.
    let hero_with_weapon_1 = SuperheroWithWeapon::WonderWoman(FavoriteWeapon::LassoOfTruth);
    match hero_with_weapon_1 {
        SuperheroWithWeapon::Batman(weapon) => {
            println!("Batman: {weapon:?}")
        }
        SuperheroWithWeapon::WonderWoman(weapon) => {
            println!("Wonder Woman: {weapon:?}")
        }
        SuperheroWithWeapon::GreenLantern(weapon) => {
            println!("Green Lantern: {weapon:?}")
        }
        other => println!("{other:?} doesn't usually use weapons."),
    };
}

If you run this code, you will get:

Wonder Woman: LassoOfTruth

Here, we've created two Enums, the SuperheroWithWeapon that accepts Superheroes with favorite weapons or without, and the FavoriteWeapon Enums that is passed to the previous one if the superhero uses a favorite weapon.

Notice the #[derive(Debug)] again.

This time, we've created a hero_with_weapon_1 variable that hold a WonderWoman variant of the SuperheroWithWeapon Enum and she uses the LassoOfTruth as her favorite weapon 🪢. In order to use the data of the favorite weapon that is bound to Wonder Woman, we have "named" it weapon in the match arm when we were defining the pattern to match against then we can use it normally in the match arm expression as in

SuperheroWithWeapon::WonderWoman(weapon) => {
            println!("Wonder Woman: {weapon:?}")
        }

You have noticed that we didn't list all the Enum's variants this time and we only listed Batman, WonderWoman, GreenLantern, and a mysterious other placeholder?? Actually, this placeholder serves as a "catch all" match arm (or the else in the if expression). Therefore, our patterns in this example fulfil the match requirement of being exhaustive. Also, notice that all the match arms return the same unit tuple "()" (println! return value).

Using the "_" placeholder:

Another way to use a "catch all" match arm, is using the "_" place holder. But this time, we won't use any data in the match arm (similar to Python's "_" in for loops or tuples expansion).

#[derive(Debug)]
enum SuperheroWithWeapon {
    Batman(FavoriteWeapon),
    Superman,
    WonderWoman(FavoriteWeapon),
    Flash,
    GreenLantern(FavoriteWeapon),
    CatWoman,
}

#[derive(Debug)]
enum FavoriteWeapon {
    LassoOfTruth,
    GreenLanternRing,
    Batarang,
}

fn main() {
    // Matching with patterns that bind to values with _ placeholder.
    let hero_with_weapon_2 = SuperheroWithWeapon::Superman;
    match hero_with_weapon_2 {
        SuperheroWithWeapon::Batman(weapon) => {
            println!("Batman: {weapon:?}")
        }
        SuperheroWithWeapon::WonderWoman(weapon) => {
            println!("Wonder Woman: {weapon:?}")
        }
        SuperheroWithWeapon::GreenLantern(weapon) => {
            println!("Green Lantern: {weapon:?}")
        }
        _ => println!("Doesn't usually use weapons."),
    };
}

If you run this code, you will get:

Doesn't usually use weapons.

This time we have created hero_with_weapon_2 as a Superman variant which doesn't use any weapons. The match in this case will go to the "catch all" arm represented by the "_" placeholder which prints a generic message and doesn't use any data.

We will revisit "pattern matching" again later. In the next one, we will check an important Enum, the "Option" Enum which protects us from Null values bugs 🫢. See you then 👋.

Learning Rust 🦀: 12 - Basic enum

Fady GA 😎 — Sat, 02 Sep 2023 15:58:04 +0000

This is the first article in about 3 or 4 articles that will discuss enum and pattern matching. In this one, I'll show you how to define and use a basic enum.

⚠️ Remember!

You can find all the code snippets for this series in its accompanying repo

If you don't want to install Rust locally, you can play with all the code of this series in the official Rust Playground that can be found on its official page.

⚠️⚠️ The articles in this series are loosely following the contents of "The Rust Programming Language, 2nd Edition" by Steve Klabnik and Carol Nichols in a way that reflects my understanding from a Python developer's perspective.

⭐ I try to publish a new article every week (maybe more if the Rust gods 🙌 are generous 😁) so stay tuned 😉. I'll be posting "new articles updates" on my LinkedIn and Twitter.

In the Struct article, we saw how a Struct can be used to group relevant data together as in the width and height of a Rectangle. The enum is similar to the struct in a way that it "enumerates" all the "variants" of the custom type we are creating in one place. Think of our earlier Rectangle struct, with enum, we can make a Shape type that contains a Rectangle, a Circle, and a Square. Each with its own "properties" and "functionality". To me, I couldn't find a similar concept in Python so this is a new one for me.

Defining enum:

We use the enum keyword to define enums. The definition can be as simple as the following:

enum SimpleAnswer {
    True,
    False,
}

Here we are defining a SimpleAnswer custom type and its variants could be either True or False (needed for an automatic exam grading system for example).

We can very easily use them and pass them to functions as follows:

enum SimpleAnswer {
    True,
    False,
}

fn main() {
    // Enum values
    let correct_answer = SimpleAnswer::True;
    let wrong_answer = SimpleAnswer::False;

    // Enums can be passed to functions
    evaluate_answer(correct_answer);
    evaluate_answer(wrong_answer);
}

fn evaluate_answer(user_answer: SimpleAnswer) {}

Here we are creating two answers that use our SimpleAnswer type, one is correct and the other is wrong. And we can pass any of them to the evaluate_answer function for evaluation (that does nothing for now😁).

Struct and enums:

Right now, our enum doesn't contain any data! Just the variants names. One could be tempted to use his Struct knowledge and write something like this to include data, say the value of the answer:

enum SimpleAnswer {
    True,
    False,
}

struct UserAnswer {
    value: String,
    evaluation: SimpleAnswer,
}

fn main() {
    let true_with_value = UserAnswer {
        value: String::from("The sky is blue"),
        evaluation: SimpleAnswer::True,
    };

    let false_with_value = UserAnswer {
        value: String::from("The sky is brown"),
        evaluation: SimpleAnswer::False,
    };

}

This could work but it's too much typing 😁 also there is a much simpler way to include data in enums. We can define an enum with as many variants as we want and with as many data bound to those variants as we want, take the follow as an example:

enum Superhero {
    Batman, // No data, he is ... Batman
    Superman(u32), // Current altitude as u32
    WonderWoman { lasso_of_truth: bool, sword: bool }, // Struct-like data
    Flash(u32, u32, u32), // Speedforce alarm levels as u32
    GreenLantern(LanternCorps), // LanternCorps is a Struct!
}

Each variant can take its data in a different way, this is cool 😎! So, getting back to our "Answers" enum example, we could integrate the answer value as follows:

enum Answer {
    True(String),
    False(String),
}

fn main() {

    // Enums with values
    let true_with_value = Answer::True(String::from("The sky is blue"));
    let false_with_value = Answer::False(String::from("The sky is brown"));

}

Implementing enum methods:

Similar to structs, we can bind enums with methods with the impl keyword and use the &self shorthand to pass the enum to the methods. For demonstrating this one, take a look at the following example implementing a Shape type similar to what I've mentioned at the beginning of this article:

enum Shape {
    Rectangle { width: f32, height: f32 },
    Circle(f32),
    Square(f32),
}

impl Shape {
    fn area(&self) -> f32 {
        match self {
            self::Shape::Rectangle {
                width: w,
                height: h,
            } => w * h,
            self::Shape::Circle(radius) => radius * radius * 3.14,
            self::Shape::Square(side) => side * side,
        }
    }

    fn get_type(&self) -> &str {
        match self {
            self::Shape::Rectangle {
                width: _,
                height: _,
            } => "Rectangle",
            self::Shape::Circle(_) => "Circle",
            self::Shape::Square(_) => "Square",
        }
    }
}

fn main() {
    let r = Shape::Rectangle {
        width: 10.0,
        height: 5.0,
    };
    let c = Shape::Circle(5.0);
    let s = Shape::Square(4.0);

    let shapes = [r, c, s];

    for shape in shapes.iter() {
        println!("{}: {}", shape.get_type(), shape.area())
    }
}

There is a lot going on in this example so let's break it down 😉. First, we defined our Shape enum that has 3 variants, Rectangle, Circle, and Square. For the Rectangle, it has a struct-like data structure that accepts the width and height properties as floats (f32). For the Circle and Square, they only take one float value which is the radius and the side for each shape respectively.

Then we have implemented two methond bound to this Shape enum, the area and get_type. They use something called Pattern Matching with the match keyword that we will discuss in detail in the next article. For now, just know that area calculates the area of the shape and get_type returns the shape type as a string literal.

Lastly, we have created 3 variables, r, c, and s which are Rectangle, Circle, and Square variants of the Shape type respectively. Then we crammed all of them in an array called shapes and we looped over this array printing the type and the area of each shape as follows:

Finished dev [unoptimized + debuginfo] target(s) in 14.11s
     Running `target/debug/structs_basics`
Rectangle: 50
Circle: 78.5
Square: 16

That's it for now for enums. In the next one, we will dig deeper into the Pattern Matching that we glimpsed in this article. See you then 👋.

DEV Community: Fady GA 😎

How I've implemented the Medallion architecture using Apache Spark and Apache Hdoop

Table of contents

What is this project all about

System architecture

The Retail app

Apache Spark Cluster

The Development Jupyter server

The Orchestrator

Spark incremental database loading

Hadoop cluster

The dashboard app

Spark Operations

Bronze stage

Silver stage

Gold stage

Exposed ports and services:

Instructions

Stopping the project

Things to consider if you want to make this into real-world project

Key takeaways

References

Groovy 🎷 Cheat Sheet - 01 Say "Hello" from Groovy

Table of Contents:

My first impressions about Groovy

Setting up your "Groovy" environment

Installation

Groovy tools

IDEs buffet

Groovy "Hello, World"

[BONUS] Running Groovy in a Jupyter Notebook

Conclusion

Meet Bob 🙂 and his Databases exploration journey 🗺️! [Relational Databases - Basics - Theory]

Table of Contents:

Meet Bob! Our store owner:

Bob's first attempt:

How Bob improved his data model:

The introduction of keys:

Relationships Cardinality:

Enter: RDBMS

Conclusion:

Learning Rust 🦀: 18 - Rust Collections: HashMaps, accessing values with keys instead of indices

⚠️ Remember!

Table of Contents:

What is a Hash Map:

Creating a Hash Map:

Accessing values in a Hash Map:

Updating a Hash Map:

Overwrite the values:

Only update if the key isn't present:

Update a value based on its old value:

Data Engineering Lifecycle - Basics - Theory

Table of Content:

What I mean by Data Analytics:

What is Data Engineering?

The Data Engineering Lifecycle:

What a Data Engineer is NOT:

Conclusion:

References:

دورة حياة هندسة البيانات - الأساسيات - نظري

جدول المحتويات :

ما أعنيه بتحليلات البيانات:

ما هي هندسة البيانات؟

دورة حياة هندسة البيانات:

ما لا يفعله مهندس البيانات

الخلاصة:

المصادر

Learning Rust 🦀: 17 - Rust Collections: Strings - How complex can it be?!

⚠️ Remember!

Table of Contents:

Why the String type is considered a collection?

Creating a new String:

Updating a String:

Indexing into Strings:

The String is not simple!

Iterating over a String:

Learning Rust 🦀: 16 - Rust Collections: Vectors

⚠️ Remember!

Table of Contents:

Creating new Vectors:

The `pub` keyword:

The `super` and `self` keywords:

Scope management with the `use` keyword: