DEV Community: stevephary

Understanding the Whys of Dealing with Missing Values in Your Dataset

stevephary — Tue, 01 Aug 2023 19:34:12 +0000

Dealing with missing values is a crucial step in the data cleaning process. Most of us know what techniques to apply when handling missing values, but it's essential to understand why we choose specific techniques for our dataset.

In this article, we will explore the reasons behind using
deletion or imputation techniques.

Deletion: Removing rows or columns with missing values.
Imputation: Filling in missing values with estimates such as mean, median or mode.

Types of Missing Values:

Before diving into the techniques, let's briefly review the three types of missing values: consider the task of predicting performance of a student.

| Hours of Study | Previous Test Score | Attendance Rate | Final Exam Score |
|----------------|---------------------|-----------------|------------------|
|      5         |         70          |       90%       |        80        |
|      2         |         45          |       75%       |        60        |
|      7         |         85          |       95%       |        90        |
|      4         |         60          |       80%       |        70        |
|      6         |         75          |       85%       |        85        |
|      3         |         50          |       70%       |        65        |
|      8         |         90          |       100%      |        95        |
|      4         |         65          |       80%       |        75        |
|      6         |         80          |       90%       |        85        |
|      2         |         40          |       60%       |        55        |
|      9         |         95          |       95%       |        100       |
|      5         |         70          |       85%       |        80        |
|      3         |         55          |       75%       |        70        |
|      7         |         80          |       90%       |        90        |
|      4         |         65          |       80%       |        75        |

Missing Completely At Random (MCAR):
In MCAR, missing values occur completely at random and are unrelated to any other variables in the dataset. The absence of data happens purely by chance, and there is no systematic reason for the missingness. If some rows were missing randomly, irrespective of any pattern or relationship with other variables, it would be considered MCAR.

Missing At Random (MAR):
In MAR, the missing values' occurrence is related to other observed variables in the dataset, but not the missing values themselves. The missingness is conditional on other variables but not on the missing values. For example, if the "Attendance Rate" is missing for students who scored poorly in the previous test, it could be considered MAR.

Missing Not At Random (MNAR):
In MNAR, the missing values' occurrence is related to the missing values themselves. The missingness is related to information that is missing and not captured by other variables in the dataset. For example, if students with high "Hours of Study" tend to have missing values for "Final Exam Score," it would be considered MNAR.

When to Apply Deletion:

Missing Completely At Random (MCAR):
If the missing values occur completely at random, deleting the rows or columns with missing data is a valid option. It does not introduce bias, and the remaining data can still be representative of the overall dataset.

Small Amount of Missing Data:
If the dataset has a small proportion of missing values (e.g., less than 5%), deletion may be reasonable, especially if the missingness is random. In such cases, the impact of removing the missing data might not significantly affect the analysis.

When to Apply Imputation:

Missing At Random (MAR) or Missing Not At Random (MNAR):
When the missing data is not completely random and has some patterns related to other observed variables, imputation methods can be used to estimate missing values. Imputation tries to preserve relationships between variables and can help retain valuable information.

Significant Amount of Missing Data:
If a large portion of the dataset contains missing values, outright deletion may lead to a loss of valuable information and potentially biased results. Imputation, in this case, allows you to keep the entire dataset and work with the complete information.

Drawback of Imputation:
While imputation techniques can be helpful in estimating missing values, they do make assumptions about the data. The quality of imputed values depends on the accuracy of these assumptions. Therefore, it is crucial to validate the imputation techniques used and assess their impact on the final results.

In conclusion, understanding the types of missing values in your dataset is essential for making informed decisions about how to handle them. If the missingness is random, deletion might be a suitable option, but if there are patterns in the missing data, imputation techniques can help retain valuable information. Always remember to validate the chosen techniques to ensure the integrity of your analysis.

TEMPORARY FUNCTION IN SQL

stevephary — Tue, 30 May 2023 16:42:05 +0000

Why use temporary functions?

You might ask “Do we really need functions in SQL?”,
absolutely yes. Temporary functions are important for several reasons:

• Organization -- As program grow in complexity, having all the queries in a big chunk increase difficult in understanding the program. Temporary function is almost like a mini-program that we can write separately from our main program, this allows us to reduce a complicated program into smaller, more manageable chunks, which reduces the overall complexity of our program.
• Reusability -- Once a temporary function is written, it can be called multiple times to the current session. This avoids duplicated queries (“Don’t Repeat Yourself”) and minimizes the probability of copy/paste errors.

Example of temporary function

Assume you are a data analyst of a wholesale company, having a database containing tables (Sales and Products). The “Sales” table contains information about sales transactions, including the product ID, quantity, and total price. The “Products” table holds product information, including the product ID, name, price, and discount. You want to know the price of products after discount.

CREATE OR REPLACE FUNCTION get_discounted_price(product_id INT)
RETURNS NUMERIC(10, 2)
AS
$$
DECLARE
  original_price NUMERIC(10, 2); 
  discount_rate NUMERIC(5, 2);
  discounted_price NUMERIC(10, 2);
BEGIN
  SELECT price, discount INTO original_price, discount_rate
  FROM Products
  WHERE product_id = product_id;

  discounted_price := original_price * (1 - discount_rate);

  RETURN discounted_price;
END;
$$
LANGUAGE plpgsql;

In the above code we create a temporary function named "get_discounted_price" that takes a product ID as a parameter and returns the price after applying the discount for that product.

We utilize our temporary function in our query as follows

SELECT product_id, product_name, get_discounted_price(product_id) AS discounted_price
FROM Products;

REMEMBER: Temporary functions are specific to the current session and will be automatically dropped when the session ends.

ARTIFICIAL NEURAL NETWORK: UNDERSTANDING BIOLOGICAL INSPIRATION

stevephary — Sun, 21 May 2023 13:59:22 +0000

Introduction

**
Artificial neural network can be defined as a model of reasoning based on the human brain. The brain consists of a densely interconnected set of nerve cells, or basic information-processing units, called neurons. The human brain incorporates nearly 10 billion neurons and 60 trillion connections(synapses).
Although each neuron has a very simple structure, an army of such elements constitutes a tremendous processing power. A neuron consists of a cell body called soma, a number of fibers called dendrites, and a single long fiber called the axon. While dendrites branch into a network around the soma, the axon stretches out to the dendrites and somas of other neurons.

Biological neural network

Dynamic Connections of Neurons and Artificial Neural Networks

**
Signals travel between neurons through complicated chemical reactions. When chemical substances are released from synapses, they change the electricity in the cell body. Once the electricity reaches a certain point, it sends an electrical pulse called an action potential down the axon. This pulse spreads and reaches synapses, which can either increase or decrease their electricity. The most interesting thing is that the brain can change and form new connections between neurons, even entire groups of neurons can move from one place to another. These changes are what allow us to learn and remember things.
The most interesting finding is that neural network exhibits plasticity. In response to the stimulation pattern, neurons also can form new connections with other neurons. Even entire collections of neurons may sometimes migrate from one place to another. These mechanisms form the basis for learning in the brain.
Based on the characteristic of plasticity, connections between neurons leading to the ‘right answer’ are strengthened while those leading to the ‘wrong answer’ weaken. As a result, neural networks have the ability to learn through experience.
Although a present-day artificial neural network (ANN) resembles the human brain much as a paper plane resembles a supersonic jet, it is a big step forward. ANNs are capable of ‘learning’, that is, they use experience to improve their performance. When exposed to a sufficient number of samples, ANNs can generalize to others they have not yet encountered.

How artificial neural networks model the brain

**
An artificial neural network consists of a number of very simple and highly interconnected processors, also called neurons, which are analogous to the biological neurons in the brain. The neurons are connected by weighted links passing signals from one neuron to another. Each neuron receives a number of input signals through its connections; however, it never produces more than a single output signal. The output signal is transmitted through the neuron’s outgoing connection (corresponding to the biological axon). The outgoing connection, in turn, splits into a number of branches that transmit the same signal (the signal is not divided among these branches in any way). The outgoing branches terminate at the incoming connections of other neurons in the network.

Structure of artificial neural networks

The following table shows analogy between biological and artificial neural networks