DEV Community: Rahul Kumawat

The Crucial Role of Sets, Relations, and Functions in Data Science

Rahul Kumawat — Tue, 06 Feb 2024 11:05:17 +0000

Is it really THAT important to learn these concepts?

In the realm of data science, where the extraction of meaningful insights from vast amounts of data is paramount, the understanding of fundamental mathematical concepts such as sets, relations, and functions sure is indispensable.

These concepts serve as the bedrock upon which various data manipulation, analysis, and modelling techniques are built, making them essential tools in the data scientist’s toolkit.

Sets —

Sets form the basis of data organization and representation.

Sets allow us to perform operations like union, intersection, and complement. The union of two sets could be used to combine datasets, while the intersection might help identify common elements between two datasets.

Practical Application of Sets —
Meteorological data consists of temperature (minimum and maximum), wind speed, wind direction, visibility, sea level pressure, humidity, geographical location, humidity, precipitation, and many more.

Meteorologist uses this data to forecast the weather of any particular region, but it is more complex than it looks. They first pre-process the data, i.e.,

Classifies the given dataset into categorical and numerical datasets
Joining different variables (union & intersection) to find the correlation between the variables
Split the datasets into two different subsets for training and testing data.

Thus it is important to learn sets, types of sets, subsets, the cardinality of the set, the union and intersection of sets.

But are Relations important?

YES! Even I used to underestimate the use of learning Relations but it provides a formal framework for establishing connections or associations between different data entities.

Let’s say we have a dataset containing information about students and their grades. We can represent this dataset using a relation, where each row represents a student and their corresponding grade.
The relation allows us to query the data efficiently, such as finding all students who scored above a certain grade threshold.

But wait, where’s the use of “Reflective, Symmetric and Transitive relations” that we learn?

While the practical application of relations in data science often involves querying datasets and extracting specific information, understanding the reflective, symmetric, and transitive forms of relations remains crucial for several reasons:

Conceptual Understanding : Learning about different forms of relations deepens our understanding of the underlying principles of relational algebra and set theory and also enhances our ability to reason about relationships and structures within data.
Modelling Complex Systems : In some real-world scenarios, data relationships may exhibit reflective, symmetric, or transitive properties.
For example, in social networks, friendships often exhibit symmetry (if A is friends with B, then B is friends with A) and transitivity (if A is friends with B, and B is friends with C, then A is indirectly connected to C). Understanding these properties allows for more accurate modelling and analysis of complex systems.
Algorithm Design: Reflective, symmetric, and transitive properties play a crucial role in algorithm design and optimization. Many algorithms in graph theory, network analysis, and machine learning rely on these properties for efficient computation.
For instance, algorithms for finding shortest paths in graphs often exploit transitivity to reduce computational complexity.
Advanced Analysis Techniques: In advanced data analysis scenarios, such as network analysis or semantic reasoning, knowledge of reflective, symmetric, and transitive properties becomes indispensable. These properties underpin various analysis techniques used to uncover patterns, clusters, or anomalies within interconnected datasets.

The direct application of reflective, symmetric, and transitive forms of relations in everyday data science tasks may not always be apparent, but their conceptual importance and broader implications for understanding and analysing data structures cannot be overstated.

Role of Functions —

Functions play a pivotal role in data transformation, modelling, predictions, Signal processing, Statistical analysis, Machine Learning and Data Mining.

Data Transformation: Suppose we have a dataset containing temperatures in Celsius, and we want to convert them to Fahrenheit.

We can define a function to perform the conversion, where ‘C’ represents the temperature in Celsius. This function allows us to map input data (Celsius temperatures) to output data (Fahrenheit temperatures), enabling us to perform the necessary data transformation.

Data Modelling: Functions are also extensively used in modelling relationships between variables in mathematics, physics, engineering, economics, and other fields.
For example, in physics, functions are used to model the motion of objects, the behaviour of physical systems, or the propagation of waves. Similarly, in economics, functions are employed to model supply and demand relationships, production functions, and utility functions.

Data Predictions: In fields such as weather forecasting, epidemiology, and financial modelling, functions are employed to simulate the behaviour of dynamic systems and make predictions about future states or events.

Signal Processing: In signal processing, functions are used to analyse, manipulate, and extract information from signals.
Functions such as Fourier transforms, wavelet transforms, and filters are applied to signals to perform tasks such as noise reduction, signal compression, and feature extraction. Well these are the complex bits that we don’t need to focus on now as what matters in understanding the applications of our learnings.

Statistical Analysis: Functions are integral to statistical analysis, where they are used to model probability distributions, estimate parameters of statistical models, and compute various statistical measures such as mean, median, variance, and correlation. Probability density functions, cumulative distribution functions, and likelihood functions are examples of functions commonly used in statistical analysis.

Thus, sets, relations, and functions serve as the building blocks for more advanced mathematical concepts and techniques used in data science, such as graph theory, optimization, and statistical inference.
By understanding and leveraging these concepts effectively, we can unlock the full potential of our data, uncovering valuable insights and driving informed decision-making in diverse domains ranging from business and finance to healthcare and beyond.

Quiz App with NodeJS for beginners.

Rahul Kumawat — Sun, 15 Aug 2021 09:09:57 +0000

How to create a quiz app in nodeJS as a beginner?

NodeJS is a backend platform to run Javascript code in server side and very trendy one too. Today we will build a quiz app where the user can answer question and get their score.

What will you learn?

Functions
Readline Sync
Arrays
For loop

How to start?

To start with this quiz app, you don’t need to install any npm package in your device. The only thing required is internet. Open Repl.it on your device and create your account in it. After that, go to the Create section and find “NodeJS” in the language section and then create repl.

So far, so good. Now you have a terminal where you can write code but you don't know the language yet. So let's start with it.

Now to our question, to start with creating the web app, we need to first understand what are we creating. A quote by John Johnson for it,

“First solve the problem, then write the code.”

Starting on, to create a quiz app on any topic, there are three features -

Score of the user
Questions of the quiz
Answers to the quiz

We’ll build these step by step by using functions.

Start with building function.

First, lets import a package called “readline-sync”. I know, you’ll think what’s readline sync?

What is Readline Sync?

Readline sync is an inbuilt function provided by nodeJS that tries to let your script have a conversation with the user via a console.
Just like our mobile phones have inbuilt apps, even programming languages have inbuilt functions to ease our development.

So, to import the function from the web, all we have to type is,

var readlineSync = require('readline-sync');

To read more about readline-sync, go here.

Now, you have stored the inbuilt function as a variable. You have the liberty to use this variable anywhere now. Without doing this, you cannot create the quiz app.

Since NodeJS is a backend langauge, anything we want to print on the screen of user, will be printed by using console.log() function. Just like Readline-Sync, it's an inbuilt function.

So, to start, let's do a simple small exercise-

A program that asks the user their name and then displays a welcome message with their name.

var readlineSync = require("readline-sync");

console.log("welcome to MARVEL quiz");

var username = readlineSync.question("Your Name? ");

console.log("Welcome to THE QUIZ, " + username + "!");

So, using the .question after readline-sync, asks the user a question, which he has to answer. This eases your work.

Good job after completing the starting exercise.

Now, the real coding part. Let's start by building a question and answer function which later through loop will ask questions automatically.

So first we need a database which will contain questions and their answers.

var questions = [
  {question:"how many iron man suits did tony stark create in the cinematic universe? ",
  answer:"85"},
  {question:"Name of Thor's original hammer",
  answer:"mjolnir"},
  {question:"Where was Mjolnir forged?",
  answer:"Nidavellir"},
  {question:"Father of the God of Thunder",
  answer:"Odin"},
  {question:"Who is the god of mischief?",
  answer:"Loki"},
  {question:"Did you like the quiz?",
  answer:"yes"}

The above program is a dictionary with a key: value pair.
So the question and answer are the keys, whereas the content in the double-quotes("") is the value of the key.

The square-brackets([]) declare the following content as an array, which makes it easy for the program to run through the content. Your database is in form of an Array. A array is simply a way to represent data in a collection.

The basic syntax is,

var name_of_the_variable = [{key: value}, {key: value}]

The comma(,) seperates the two elements.

Tip : You can name the variable anything but it's a better practice to name it to somethinng relevant to the content inside the var.

Now the loop of the questions. How are you gonna continuously show the user your questions?

A little loop for that is-

for(var i=0; i<questions.length; i++){
  var currentq = questions[i];
  quiz(currentq.question, currentq.answer);
}

This loop automatically displays the next question after the user has answered the previous one.

Don't get overwhelmed. Let me explain the loop to you.

for(var i=0; i<questions.length; i++)

The for here, says that the loop is a 'for' loop. There are different types of loops.

You can read the detailed version about them here.

The content inside the parentheses are the condition in which the loop has to be executed/start.

One of the confusions to avoid while writing a program is understanding the difference between parameters in functions and global variables. I'll soon write a blog about it.

So, back to the conditions, the (i=0; i < questions.length; i++). 'i' here is an initial value passed in the loop. So when we say i=0, we are declaring the value of i is 0.

i < questions.length means the value of i is less than the length of questions.

You can read more about the array functions here.

and finally, i++ means that the questions have to increase after the user has answered.

i++ simply means i = i + 1, i.e. , the value of i is increased by 1.

Now, lets move onto the {} part of the loop-

for(var i=0; i<questions.length; i++){
  var currentq = questions[i];
  quiz(currentq.question, currentq.answer);
}

Here we declare a variable called currentq qhich displays the current question that the user is implementing. The value assigned to the variable is used from the loop.

Warning! quiz here is not an inbuilt function. We are about to develop it -

var score = 0;
function quiz(question, answer){
  var userAnswer = readlineSync.question(question);
  if(userAnswer == answer){
    console.log("correct! ");
    score++;
  }
  else{
    console.log("wrong! ");
    score--;
  }
  console.log("your score is ",score);

// For neatness
  console.log("-------------------");
}
}

This is the most important part of the program. Here, all the implemetation of the program we have written so far.

We start by declaring a variable score which will be the score of the user after he answers the question.

The function quiz takes in 2 parameters, question and an answer.

Since there are only two outputs for the a question. We will use the conditional statement if()...else.

So, if the userAnswer is equal to the correct answer, then we will show the output as "Correct!" And increment the score of the user.

If the userAnswer is not Equal to the correct answer, the output will be a "Wrong!" and the score will decrease.

In both conditions, after the user has answered the question, his score will be shown.

Your Final Program should look like this-

var readlineSync = require("readline-sync");

console.log("welcome to F.R.I.E.N.D.S. quiz");

//this is line breaker, like it leaves a line.
console.log("\n");


var username = readlineSync.question("Your Name? ");

console.log("\n");

console.log("Welcome to THE QUIZ, " + username + "!");

var score = 0;
function quiz(question, answer){
  var userAnswer = readlineSync.question(question);
  if(userAnswer == answer){
    console.log("correct! ");
    score++;
  }
  else{
    console.log("wrong! ");
    score--;
  }
  console.log("your score is ",score);
  console.log("-------------------");
}

//questions
var questions = [
  {question:"how many iron man suits did tony stark create in the cinematic universe? ",
  answer:"85"},
  {question:"Name of Thor's original hammer",
  answer:"mjolnir"},
  {question:"Where was Mjolnir forged?",
  answer:"Nidavellir"},
  {question:"Father of the God of Thunder",
  answer:"Odin"},
  {question:"Who is the god of mischief?",
  answer:"Loki"},
  {question:"Did you like the quiz?",
  answer:"yes"}
];

//LOOP
for(var i=0; i<questions.length; i++){
  var currentq = questions[i];
  quiz(currentq.question, currentq.answer);
}

//To display the final score.
console.log("YOUR FINAL SCORE IS: " + score+"/10")

Explore the world of NodeJS through reading documentations from MDN docs. and other sources.

If you think there are some errors or a typo in my explanation please write it in the discussion. Happy to help!