DEV Community: Audrine Marion

Neural Networks: Inspired by the Brain, Built for Data

Audrine Marion — Tue, 08 Sep 2026 07:37:16 +0000

When I first started learning about neural networks, I found the idea both fascinating and slightly intimidating.

The phrase “neural network” makes it sound like something extremely complex, and honestly, once you start looking at layers, weights, activation functions, forward propagation, backpropagation, and optimization, it can become overwhelming pretty quickly.

But beneath all of that mathematics and code is a surprisingly simple idea:

A neural network learns patterns from data by adjusting itself based on its mistakes.

That idea is what made neural networks interesting to me.

As I continue exploring data science and machine learning, understanding neural networks has helped me see machine learning from a different perspective. Instead of simply telling a computer what rules to follow, we can give it data and allow it to learn patterns that may be difficult for us to define manually.

In this article, I'll break down where the inspiration for neural networks came from, what they are, and the major components that make them work.

Where Did Neural Networks Come From?

The inspiration behind artificial neural networks comes from something we already have:

the human brain.

Our brains contain billions of neurons that communicate with one another. A biological neuron receives signals, processes them, and passes a signal forward when certain conditions are met.

Artificial neural networks don't replicate the human brain perfectly. Instead, they borrow a simplified idea from how biological neurons work.

A biological neuron can be thought of roughly as:

receive signals → process them → produce a response

An artificial neuron follows a similar conceptual process:

receive inputs → calculate a weighted sum → apply an activation function → produce an output

This was one of the things I found interesting when learning about neural networks.

The goal isn't to recreate the brain inside a computer.

The goal is to take a useful idea from biology and turn it into a mathematical model that can learn from data.

So, What Exactly Is a Neural Network?

A neural network is a machine learning model made up of interconnected nodes, commonly called neurons.

These neurons are organized into layers.

A basic neural network usually contains:

Input layer
One or more hidden layers
Output layer

A simple representation looks like this:

Input Layer        Hidden Layer        Output Layer

   x₁  ─────────►    ○
                    │
   x₂  ─────────►    ○ ─────────►     ŷ
                    │
   x₃  ─────────►    ○

The input layer receives the data.

The hidden layers transform the data and learn patterns.

The output layer produces the final prediction.

The interesting part is that we don't manually program the network to recognize every possible pattern.

Instead, the network learns the parameters needed to recognize those patterns from examples.

1. The Input Layer

Everything starts with the input.

Suppose we want to predict whether a student is likely to pass an exam.

Our dataset might contain:

Hours studied
Attendance
Previous test scores
Assignment completion

These become our input features.

For example:

Hours Studied     = 6
Attendance        = 85%
Previous Score    = 72
Assignments       = 9

The neural network receives these values through the input layer.

One important thing I learned is that neural networks generally work best when the input data has been appropriately prepared.

That can involve:

Handling missing values
Encoding categorical variables
Scaling numerical features
Removing irrelevant features
Splitting data into training and testing sets

So even though neural networks are powerful, data preparation still matters.

Garbage in can still mean garbage out.

2. Neurons

The basic building block of a neural network is the artificial neuron.

A neuron takes several inputs, assigns different importance to them using weights, adds a bias, and then passes the result through an activation function.

Mathematically, we can represent this as:

$$
z = w_1x_1 + w_2x_2 + ... + w_nx_n + b
$$

Where:

(x) = input
(w) = weight
(b) = bias
(z) = weighted sum

The neuron then applies an activation function:

$$
a = f(z)
$$

The result becomes the neuron's output.

This might look complicated initially, but the basic idea is straightforward:

The neuron combines information from its inputs and decides how strongly to activate.

3. Weights

Weights determine how important different inputs are to a neuron.

Imagine we're predicting whether a student will pass.

Perhaps hours studied is more predictive than the number of assignments completed.

The network can learn this by assigning different weights to those features.

For example:

Hours studied       → weight = 0.8
Attendance          → weight = 0.5
Previous score      → weight = 0.9
Assignments         → weight = 0.3

These numbers are only examples.

During training, the neural network continuously adjusts its weights to improve its predictions.

This is one of the key ideas behind learning in neural networks.

The model isn't explicitly told what the correct weights are. It learns them from data.

4. Bias

Another component is the bias.

Bias gives the neuron some flexibility in determining when it should activate.

Without going too deep into the mathematics, you can think of the bias as an additional parameter that allows the activation function to shift.

The basic equation becomes:

$$
z = wx + b
$$

The weights control the influence of the inputs, while the bias helps adjust the overall output.

Both weights and biases are learned during training.

5. Activation Functions

This was one of the concepts that really helped me understand why neural networks can learn complex patterns.

After calculating the weighted sum, a neuron applies an activation function.

Why?

Because without activation functions, stacking multiple layers of neurons would still result in a fundamentally linear transformation.

Activation functions introduce non-linearity.

And real-world data is rarely perfectly linear.

Some commonly used activation functions include:

Sigmoid

The sigmoid function produces values between 0 and 1.

$$
\sigma(x) = \frac{1}{1 + e^{-x}}
$$

This makes it useful in certain binary classification problems, particularly as an output activation.

For example:

0.91 → high probability of class 1
0.12 → low probability of class 1

ReLU

ReLU stands for Rectified Linear Unit.

It is defined as:

$$
ReLU(x) = \max(0,x)
$$

So:

x = -3 → 0
x =  2 → 2
x =  7 → 7

ReLU became particularly important in deep learning because it is simple and computationally efficient and helps neural networks model nonlinear relationships.

Softmax

Softmax is commonly used in the output layer for multi-class classification.

Suppose we're classifying an image as:

Cat
Dog
Bird

Softmax can convert the model's outputs into probabilities that sum to 1.

For example:

Cat  → 0.10
Dog  → 0.82
Bird → 0.08

The model would predict Dog.

6. Hidden Layers

Now we get to one of the most important parts of a neural network.

The hidden layers.

These layers sit between the input and output layers.

A network with only a few layers might look like:

Input → Hidden → Output

A deeper network could look like:

Input
  ↓
Hidden Layer 1
  ↓
Hidden Layer 2
  ↓
Hidden Layer 3
  ↓
Hidden Layer 4
  ↓
Output

Each layer can learn different representations of the data.

This is where the term deep learning comes from.

A neural network with many hidden layers is generally described as a deep neural network.

The interesting part is that the network can learn increasingly complex representations as information moves through the layers.

For example, in an image recognition task:

Pixels
  ↓
Edges
  ↓
Shapes
  ↓
Patterns
  ↓
Objects

The network isn't necessarily programmed with these exact rules.

It learns useful representations from the training data.

7. Forward Propagation

So how does a neural network actually make a prediction?

This happens through forward propagation.

The input data moves through the network from the input layer toward the output layer.

At each neuron:

Inputs are multiplied by weights.
The results are added together.
A bias is added.
An activation function is applied.
The result is passed to the next layer.

Eventually, the network produces a prediction.

Conceptually:

Input
  ↓
Weighted Sum
  ↓
Activation
  ↓
Next Layer
  ↓
Weighted Sum
  ↓
Activation
  ↓
Prediction

This happens extremely quickly when implemented computationally.

8. Loss Functions

The network makes a prediction.

But how does it know whether the prediction is good or bad?

This is where the loss function comes in.

A loss function measures the difference between the model's prediction and the actual target.

For example:

Actual value      = 1
Predicted value   = 0.3

There is a significant difference.

The loss function quantifies that error.

Different problems use different loss functions.

Some common examples include:

Mean Squared Error (MSE)
Binary Cross-Entropy
Categorical Cross-Entropy

The general goal during training is simple:

Minimize the loss.

9. Backpropagation

This is probably one of the most important concepts in neural networks.

After making a prediction and calculating the loss, the network needs to figure out:

Which weights contributed to the error, and how should they change?

This is where backpropagation comes in.

The error is propagated backward through the network.

Using calculus—particularly the chain rule—the network calculates gradients that indicate how changes in the parameters would affect the loss.

Conceptually:

Prediction
    ↓
Calculate Loss
    ↓
Propagate Error Backward
    ↓
Calculate Gradients
    ↓
Update Weights

Then the process repeats.

Again.

And again.

This is how the neural network gradually improves.

10. Optimizers

Backpropagation tells us how the parameters should change.

The optimizer determines how those changes are actually made.

One of the most well-known optimization algorithms is Gradient Descent.

The basic idea is to move the model's parameters in a direction that reduces the loss.

A simplified update rule looks like:

$$
w_{new} = w_{old} - \eta \frac{\partial L}{\partial w}
$$

Where:

(w) = weight
(\eta) = learning rate
(L) = loss

The learning rate controls how large each update should be.

If the learning rate is too large, the model may overshoot useful solutions.

If it is too small, training may take a very long time.

Other optimizers include:

Adam
RMSprop
Adagrad
SGD with momentum

Adam is particularly common in practical deep learning applications.

11. Epochs and Batches

Neural networks don't normally learn from the entire dataset in a single step.

Instead, training data is often divided into smaller groups called batches.

For example, if we have:

10,000 training samples

we might use:

Batch size = 32

The network processes 32 samples at a time.

Once it has processed the entire training dataset, we have completed one epoch.

So:

Dataset
   ↓
Batch 1
Batch 2
Batch 3
...
Batch N
   ↓
One Epoch

The model typically trains for multiple epochs.

12. Learning Rate

The learning rate determines how aggressively the model updates its parameters.

Think of it as the size of the steps the model takes while trying to minimize its loss.

A very large learning rate might look like:

Too large
    ↓
Overshoot
    ↓
Unstable training

A very small learning rate might look like:

Too small
    ↓
Tiny updates
    ↓
Very slow learning

Finding an appropriate learning rate can make a significant difference in training performance.

13. Training, Validation, and Testing

Another important concept is understanding how we evaluate a neural network.

We generally don't want to train and evaluate the model on exactly the same data.

Instead, data can be divided into:

Dataset
   │
   ├── Training Data
   │
   ├── Validation Data
   │
   └── Test Data

Training data

Used to learn the model's parameters.

Validation data

Used to monitor performance and help with choices such as hyperparameters and model architecture.

Test data

Used to evaluate how well the final model performs on unseen data.

This is important because a model can perform extremely well on its training data but poorly on new data.

That brings us to one of the biggest challenges in machine learning.

14. Overfitting

A neural network can become too good at memorizing the training data.

This is known as overfitting.

Imagine studying for an exam by memorizing the answers to last year's exact questions.

You might perform extremely well if the same questions appear.

But if the questions change, your performance may drop.

A similar thing can happen with machine learning models.

The model learns the training examples extremely well but fails to generalize to unseen data.

Some techniques used to reduce overfitting include:

Dropout
Regularization
Early stopping
Data augmentation
More training data
Simpler model architectures

Why Are Neural Networks So Powerful?

What makes neural networks particularly interesting is their ability to learn nonlinear relationships and complex representations.

Traditional programming often looks like:

Rules + Data → Output

Machine learning changes the approach:

Data + Expected Outputs → Learned Rules

And neural networks take this further by learning multiple levels of representation.

This is why they have become so useful in areas such as:

Computer vision
Natural language processing
Speech recognition
Recommendation systems
Time-series forecasting
Fraud detection
Medical imaging
Generative AI

From Neural Networks to Deep Learning

A neural network with many layers is generally referred to as a deep neural network.

Deep learning is essentially about using these deeper architectures to learn increasingly complex representations from data.

This has led to architectures designed for different kinds of problems.

For example:

Convolutional Neural Networks (CNNs)

Often associated with image and computer vision tasks.

Recurrent Neural Networks (RNNs)

Designed to work with sequential data, although many modern sequence tasks now use transformer-based architectures instead.

Transformers

Transformers have become extremely important in modern AI, particularly for language and other sequence-based problems.

They power many of the systems behind today's generative AI applications.

What I Take Away From Learning Neural Networks

One thing I appreciate about neural networks is that they changed how I think about machine learning.

At first, the equations can make the subject feel intimidating.

You see:

$$
z = wx+b
$$

then activation functions, gradients, derivatives, optimization algorithms, layers, epochs, and suddenly it feels like you're drowning in mathematics.

But when you break the process down, the basic idea becomes much easier to understand.

A neural network:

takes data → makes a prediction → measures its error → adjusts itself → tries again.

That cycle is at the heart of learning.

And I think that's the part that makes neural networks so fascinating.

Final Thoughts

I'm still learning neural networks, and I don't think understanding them means memorizing every equation or being able to build a state-of-the-art architecture from scratch.

For me, understanding the fundamentals is more important.

I want to know:

What is happening to the data?
Why do we need weights?
What does a bias actually do?
Why do we need activation functions?
How does the model know that it made a mistake?
How does backpropagation help?
What does an optimizer actually change?
How do we know whether the model is learning or simply memorizing?

Once these pieces start fitting together, neural networks stop looking like a mysterious black box.

They become a collection of understandable ideas working together.

And that's probably the most important lesson I've taken from studying them so far:

Complex models become much less intimidating when you understand the simple ideas underneath them.

I'm still on that learning journey and neural networks are definitely one of the areas I'm excited to explore more deeply.

Using Scikit-Learn Pipelines: A Cleaner Way to Build Machine Learning Models

Audrine Marion — Fri, 28 Aug 2026 17:49:57 +0000

If you've spent some time building machine learning models with Python, you've probably had a notebook that looked something like this:

X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)

model.fit(X_train, y_train)

predictions = model.predict(X_test)

And then a few cells later, you realize you also need to encode categorical variables.

Then there's imputation.

Then feature selection.

Then maybe PCA.

Before you know it, your notebook has a collection of preprocessing steps that depend on being executed in exactly the right order.

I've been there.

One of the things that made my machine learning workflow much cleaner was learning how to use Scikit-Learn Pipelines.

In this article, I'll walk through what pipelines are, why they matter, and how I use them to make machine learning workflows more reliable and easier to maintain.

What is a Scikit-Learn Pipeline?

A pipeline is essentially a way of connecting multiple machine learning steps together so that they can be treated as one workflow.

For example, suppose we have a dataset where we need to:

Handle missing values
Scale numerical features
Train a machine learning model

Instead of doing everything separately:

X_train = imputer.fit_transform(X_train)
X_train = scaler.fit_transform(X_train)

model.fit(X_train, y_train)

We can put everything into a pipeline:

from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

pipeline = Pipeline([
    ("imputer", SimpleImputer(strategy="median")),
    ("scaler", StandardScaler()),
    ("model", LogisticRegression())
])

pipeline.fit(X_train, y_train)

Now the entire process is represented by one object.

And that's the part I really like about pipelines.

Instead of thinking:

"First I need to do this, then this, then this..."

you can think:

"This is my machine learning workflow."

Why Should We Use Pipelines?

At first, pipelines can feel like extra syntax.

Why not just preprocess the data manually?

You absolutely can.

But pipelines solve several important problems.

1. They reduce data leakage

This is probably the biggest reason to use them.

Imagine you're scaling your dataset before splitting it:

scaler = StandardScaler()

X_scaled = scaler.fit_transform(X)

X_train, X_test, y_train, y_test = train_test_split(
    X_scaled, y, test_size=0.2, random_state=42
)

It looks harmless.

But there's a problem.

The scaler has seen the entire dataset, including the test set.

That means information from your test data has influenced the preprocessing step.

This is a form of data leakage.

The model hasn't technically seen the test labels, but information about the distribution of the test features has already entered the training process.

A pipeline helps prevent this when used correctly:

pipeline = Pipeline([
    ("scaler", StandardScaler()),
    ("model", LogisticRegression())
])

pipeline.fit(X_train, y_train)

During fit(), the scaler learns its parameters only from X_train.

When we evaluate:

pipeline.predict(X_test)

the same fitted scaler transforms X_test without learning anything new from it.

That separation is extremely important.

2. Pipelines Keep Preprocessing and Modeling Together

Without a pipeline, you might end up with something like:

imputer = SimpleImputer(strategy="median")
scaler = StandardScaler()

X_train = imputer.fit_transform(X_train)
X_train = scaler.fit_transform(X_train)

model = LogisticRegression()
model.fit(X_train, y_train)

Then, when making predictions:

X_test = imputer.transform(X_test)
X_test = scaler.transform(X_test)

predictions = model.predict(X_test)

Notice the problem?

You have to remember the exact preprocessing sequence.

With a pipeline:

pipeline = Pipeline([
    ("imputer", SimpleImputer(strategy="median")),
    ("scaler", StandardScaler()),
    ("model", LogisticRegression())
])

you simply do:

pipeline.fit(X_train, y_train)

predictions = pipeline.predict(X_test)

Much cleaner.

3. Pipelines Make Cross-Validation Safer

This is another major advantage.

Suppose we want to evaluate different models using cross-validation.

We could create a pipeline:

pipeline = Pipeline([
    ("scaler", StandardScaler()),
    ("model", LogisticRegression())
])

Then:

from sklearn.model_selection import cross_val_score

scores = cross_val_score(
    pipeline,
    X,
    y,
    cv=5,
    scoring="accuracy"
)

print(scores)
print(scores.mean())

Each fold gets its own preprocessing fitted only on that fold's training data.

This is exactly what we want.

Without a pipeline, it is easy to accidentally preprocess the entire dataset before cross-validation and introduce leakage.

Building a Simple Pipeline

Let's create a slightly more realistic example.

Imagine we're building a model to predict whether a customer belongs to a particular class.

Our workflow might be:

Missing values → Scaling → Logistic Regression

from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

pipeline = Pipeline([
    ("imputer", SimpleImputer(strategy="median")),
    ("scaler", StandardScaler()),
    ("model", LogisticRegression())
])

The names we give each step, such as "imputer" and "scaler", are useful because they allow us to access those individual components later.

Training becomes:

pipeline.fit(X_train, y_train)

Prediction:

y_pred = pipeline.predict(X_test)

And evaluation:

from sklearn.metrics import classification_report

print(classification_report(y_test, y_pred))

That's the whole workflow.

What About Categorical and Numerical Features?

This is where pipelines become even more useful.

Real-world datasets rarely contain only numerical variables.

You might have:

Age          → numerical
Income       → numerical
Education    → categorical
Gender       → categorical

We shouldn't necessarily preprocess all of these columns in the same way.

For example:

Numerical columns → imputation + scaling
Categorical columns → imputation + one-hot encoding

Scikit-Learn gives us ColumnTransformer for exactly this situation.

from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.linear_model import LogisticRegression

Let's define our columns:

numerical_features = [
    "age",
    "income"
]

categorical_features = [
    "education",
    "gender"
]

Now we create separate preprocessing pipelines.

Numerical pipeline

numeric_pipeline = Pipeline([
    ("imputer", SimpleImputer(strategy="median")),
    ("scaler", StandardScaler())
])

Categorical pipeline

categorical_pipeline = Pipeline([
    ("imputer", SimpleImputer(strategy="most_frequent")),
    ("encoder", OneHotEncoder(handle_unknown="ignore"))
])

Now we combine them using ColumnTransformer:

preprocessor = ColumnTransformer([
    ("num", numeric_pipeline, numerical_features),
    ("cat", categorical_pipeline, categorical_features)
])

Finally, we connect preprocessing to our model:

model_pipeline = Pipeline([
    ("preprocessor", preprocessor),
    ("model", LogisticRegression())
])

Now we have one complete workflow:

Raw Data
   ↓
Numerical preprocessing ──┐
                          ├──→ Model
Categorical preprocessing ┘

And training is still just:

model_pipeline.fit(X_train, y_train)

Prediction:

y_pred = model_pipeline.predict(X_test)

This is much closer to how machine learning workflows look in real projects.

Pipelines and Hyperparameter Tuning

Here's where things get even more interesting.

We can use pipelines with GridSearchCV to tune both preprocessing and model parameters.

For example:

from sklearn.model_selection import GridSearchCV

param_grid = {
    "model__C": [0.01, 0.1, 1, 10],
    "model__max_iter": [100, 200, 500]
}

grid_search = GridSearchCV(
    model_pipeline,
    param_grid,
    cv=5,
    scoring="accuracy"
)

grid_search.fit(X_train, y_train)

Notice this:

"model__C"

The double underscore allows us to access parameters inside pipeline steps.

If our pipeline contains:

("model", LogisticRegression())

then:

model__C

means:

"The C parameter belonging to the model step."

We can then check the best parameters:

print(grid_search.best_params_)

And use the best estimator:

best_model = grid_search.best_estimator_

predictions = best_model.predict(X_test)

This makes experimentation much more systematic.

Pipelines Are Not Just for Classification

Pipelines work across many Scikit-Learn workflows.

For example, regression:

from sklearn.ensemble import RandomForestRegressor

regression_pipeline = Pipeline([
    ("imputer", SimpleImputer(strategy="median")),
    ("model", RandomForestRegressor(random_state=42))
])

Or clustering:

from sklearn.cluster import KMeans

clustering_pipeline = Pipeline([
    ("scaler", StandardScaler()),
    ("model", KMeans(n_clusters=3, random_state=42))
])

The same basic idea applies:

Preprocessing
     ↓
Transformation
     ↓
Model / Algorithm

One Important Thing I Learned

One mistake that's easy to make when learning pipelines is assuming that every model needs every preprocessing step.

It doesn't.

For example, scaling is generally important for algorithms that are sensitive to feature magnitude, such as:

Logistic Regression
KNN
SVM
K-Means
PCA

But tree-based models such as:

Decision Trees
Random Forest
Gradient Boosting

generally don't require feature scaling.

So don't build pipelines mechanically.

Think about what your algorithm actually needs.

Pipelines Make Your Code More Reproducible

There's another benefit that isn't always obvious when you're first learning machine learning.

A pipeline makes your workflow easier for someone else to reproduce.

Imagine handing someone this:

pipeline.fit(X_train, y_train)

Compared with giving them five different preprocessing scripts and telling them:

"Run this first, then this, then remember to use the same scaler when predicting."

The pipeline communicates the workflow much more clearly.

This becomes especially useful when working on:

Team projects
Research projects
Production ML systems
GitHub projects
Machine learning competitions

It also makes your notebooks less cluttered.

And honestly, once you start working with several models, that becomes a big deal.

A Complete Example

Here's a compact example putting everything together:

import pandas as pd

from sklearn.model_selection import train_test_split
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report


#### Load data
df = pd.read_csv("customer_data.csv")

# Separate features and target
X = df.drop("target", axis=1)
y = df["target"]

#### Define feature types
numeric_features = ["age", "income"]
categorical_features = ["gender", "education"]

#### Numerical preprocessing
numeric_pipeline = Pipeline([
    ("imputer", SimpleImputer(strategy="median")),
    ("scaler", StandardScaler())
])

#### Categorical preprocessing
categorical_pipeline = Pipeline([
    ("imputer", SimpleImputer(strategy="most_frequent")),
    ("encoder", OneHotEncoder(handle_unknown="ignore"))
])

#### Combine preprocessing
preprocessor = ColumnTransformer([
    ("num", numeric_pipeline, numeric_features),
    ("cat", categorical_pipeline, categorical_features)
])

#### Full pipeline
pipeline = Pipeline([
    ("preprocessor", preprocessor),
    ("model", LogisticRegression(max_iter=1000))
])

#### Split data
X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.2,
    random_state=42,
    stratify=y
)

### Train
pipeline.fit(X_train, y_train)

### Predict
y_pred = pipeline.predict(X_test)

### Evaluate
print(classification_report(y_test, y_pred))

What I like about this structure is that the code tells a story.

You can almost read it from top to bottom:

Load → Separate → Preprocess → Build → Split → Train → Predict → Evaluate

That's what good ML code should do.

When Should You Use Pipelines?

I'd recommend getting comfortable with pipelines as soon as you're moving beyond very simple machine learning exercises.

They're particularly useful when you have:

Multiple preprocessing steps
Missing values
Categorical variables
Feature scaling
Feature selection
Dimensionality reduction
Cross-validation
Hyperparameter tuning
Multiple models to compare

Even when a pipeline isn't strictly necessary, using one can make your workflow cleaner.

Final Thoughts

When I first started working with machine learning workflows, preprocessing felt like a collection of separate tasks.

Clean the data.

Encode it.

Scale it.

Split it.

Train the model.

Then somehow remember to apply the exact same transformations to the test data.

Scikit-Learn Pipelines changed the way I think about that process.

A pipeline isn't just a convenient way to shorten your code. It's a way of defining the entire machine learning workflow as one reproducible object.

And perhaps the biggest lesson is this:

Your model is only one part of a machine learning system.

The preprocessing that happens before the model matters just as much.

Once you start using pipelines, you'll find yourself writing ML code that is cleaner, safer, and much easier to maintain.

And when you eventually move from Jupyter notebooks to real-world machine learning projects, that habit becomes incredibly valuable.

Key Takeaways

Pipeline connects preprocessing and modeling into one workflow.
It helps reduce the risk of data leakage.
It makes cross-validation safer.
ColumnTransformer allows different preprocessing for different feature types.
Pipelines work with GridSearchCV and hyperparameter tuning.
They improve reproducibility and maintainability.
Not every algorithm needs the same preprocessing, so build your pipeline based on the model you're using.

If you're learning Scikit-Learn right now, pipelines are one of those concepts I'd strongly recommend getting comfortable with early. They might seem like extra structure at first, but once your projects become more complicated, you'll be very glad you have them.

Unsupervised Learning: Teaching Machines to Find the Patterns We Didn't Know Were There

Audrine Marion — Fri, 21 Aug 2026 18:43:10 +0000

When I first started learning machine learning, I naturally gravitated toward supervised learning.

It made sense.

Give the model some data, tell it what the correct answer is, train it to recognize the relationship, and then use it to make predictions.

But eventually I came across a different question:

What happens when we don't have the answers?

What if our dataset doesn't have a target column telling us what each observation is?

What if, instead of asking a model to predict something, we want it to help us discover what is already hiding inside the data?

That is where unsupervised learning becomes interesting.

So, what exactly is unsupervised learning?

Unsupervised learning is a branch of machine learning where algorithms work with data without predefined target labels.

Instead of learning:

"These are the correct answers. Learn to predict them."

we essentially say:

"Here is the data. Find something interesting."

The algorithm looks for patterns, similarities, structures, relationships, or unusual observations within the data.

This can be incredibly useful because real-world datasets don't always come neatly labeled.

Imagine having thousands of customers but no column saying:

Customer type A
Customer type B
Customer type C

Instead, you might have information about their income, spending habits, transaction frequency, savings, age, and other characteristics.

You can use unsupervised learning to investigate whether naturally occurring groups exist.

And this is one of the things I find most interesting about it:

The model isn't necessarily confirming what we already know. It can help us discover what we didn't know to look for.

The Main Families of Unsupervised Learning

Unsupervised learning isn't just one algorithm.

It is more useful to think of it as a collection of approaches designed to answer different questions.

Some of the major categories include:

Clustering
Dimensionality reduction
Density estimation
Anomaly/outlier detection
Mixture models
Matrix factorization and decomposition

Let's break these down.

1. Clustering

Clustering is probably the first thing most people encounter when learning unsupervised learning.

The goal is simple:

Group observations that are similar to each other.

Suppose I have customer data:

Customer    Income    Spending    Savings
A           30,000    8,000       5,000
B           32,000    7,500       4,800
C           90,000    25,000      30,000
D           95,000    28,000      35,000

There is no target telling us which customers belong together.

A clustering algorithm might discover that A and B are quite similar, while C and D form another group.

That's the basic idea.

There are several clustering algorithms, and they don't all define "similarity" in the same way.

K-Means

K-Means is probably the most recognizable clustering algorithm.

The basic idea is to divide observations into K clusters, with each observation assigned to the cluster whose center is closest according to the algorithm's objective.

One of the first things I learned with K-Means was that choosing K isn't as simple as saying:

K = 3

and moving on.

You need to investigate whether that number of clusters actually makes sense.

Two common techniques are the:

Elbow Method
Silhouette Score

The Elbow Method looks at how the within-cluster sum of squares changes as K increases.

The silhouette score evaluates how well observations fit within their assigned cluster compared with neighboring clusters.

This is an important lesson with clustering:

The algorithm can create clusters, but that doesn't automatically mean the clusters are meaningful.

K-Means works particularly well when clusters have relatively simple, compact geometry, but it can struggle with irregularly shaped clusters. (GitHub)

Hierarchical Clustering

Another approach I find particularly interesting is hierarchical clustering.

Instead of simply producing a flat set of clusters, hierarchical clustering builds a hierarchy of relationships between observations.

In agglomerative hierarchical clustering, every observation starts as its own cluster. The algorithm progressively merges similar clusters until everything is eventually part of one large cluster. (scikit-learn)

The result can be visualized using a dendrogram.

The dendrogram allows us to look at the different levels at which observations or groups merge and decide where we might "cut" the hierarchy.

This makes hierarchical clustering particularly useful when I want to understand not just which observations are grouped together, but also how those groups relate to one another.

DBSCAN

Not every dataset naturally forms neat circular or compact groups.

This is where algorithms such as DBSCAN become useful.

DBSCAN is a density-based clustering algorithm.

Instead of asking:

"How many clusters should I create?"

it looks for areas where observations are densely packed together.

One advantage is that it can identify clusters with more irregular shapes and can also identify observations that don't belong to dense regions.

That makes it particularly interesting for datasets where unusual observations are part of the problem rather than something we simply want to remove.

2. Dimensionality Reduction

Now let's say my dataset has 100 features.

Trying to visualize or work with all 100 dimensions isn't exactly convenient.

This is where dimensionality reduction comes in.

The goal is to represent high-dimensional data using fewer dimensions while attempting to preserve useful information or structure.

One of the most widely used approaches is Principal Component Analysis (PCA).

PCA

PCA transforms the original features into a smaller number of new variables called principal components.

The first component captures as much of the variance in the data as possible, followed by subsequent components capturing additional variance subject to the method's constraints. (Scikit-learn)

For example:

from sklearn.decomposition import PCA

pca = PCA(n_components=2)

X_pca = pca.fit_transform(X_scaled)

Now a dataset with many features can be represented using two principal components.

Why is this useful?

One reason is visualization.

I can plot:

Principal Component 1
        vs.
Principal Component 2

and potentially see structures that were difficult to observe in the original feature space.

PCA can also be useful before other machine-learning techniques when a dataset has many features.

But there is an important caveat:

PCA doesn't magically make information disappear without consequences.

Reducing dimensions means making a trade-off between simplicity and information retention.

t-SNE

Another dimensionality-reduction technique you'll often encounter is t-SNE — t-distributed Stochastic Neighbor Embedding.

Unlike PCA, which focuses on preserving variance through linear transformations, t-SNE is primarily useful for visualizing complex high-dimensional relationships.

It is particularly popular for creating 2D or 3D visualizations of high-dimensional data.

However, I wouldn't treat a t-SNE plot as proof that a particular number of clusters exists.

It's a visualization tool, and its output can be influenced by its parameters and the structure of the data.

3. Anomaly Detection

Sometimes we're not interested in finding groups.

We're interested in finding the observations that don't fit.

This is anomaly detection.

Consider:

fraudulent transactions
unusual network activity
abnormal sensor readings
unusual customer behavior
manufacturing defects

In these situations, the unusual observation may actually be the most important one.

Algorithms such as Isolation Forest, Local Outlier Factor (LOF), and One-Class SVM can be used for different forms of anomaly or novelty detection.

This is one of the areas where unsupervised learning becomes particularly practical.

Instead of asking:

"What class does this transaction belong to?"

we might ask:

"Does this transaction look unusual compared with the rest?"

Scikit-learn treats novelty and outlier detection as a distinct part of its unsupervised-learning toolkit. (Scikit-learn)

4. Gaussian Mixture Models

Another approach is the Gaussian Mixture Model (GMM).

At a high level, GMM assumes that the data can be represented as a mixture of several underlying probability distributions.

This is different from simply assigning every observation to one cluster.

A GMM can provide probabilities representing how strongly an observation belongs to different components.

For example:

Customer A

Cluster 1: 0.85
Cluster 2: 0.10
Cluster 3: 0.05

This can be useful when boundaries between groups aren't perfectly clear.

Scikit-learn includes both Gaussian Mixture and Variational Bayesian Gaussian Mixture models within its unsupervised-learning tools. (Scikit-learn)

5. Matrix Factorization and Decomposition

Another family of unsupervised methods focuses on breaking complex data into underlying components.

This includes techniques such as:

PCA
Independent Component Analysis (ICA)
Non-negative Matrix Factorization (NMF)
Latent Dirichlet Allocation (LDA)

These approaches can be useful when we believe that complicated observations may be explained by a smaller number of underlying factors or components.

For example, in text analysis, LDA can be used for topic modeling — helping identify groups of words that tend to occur together and may represent underlying topics.

So Which Algorithm Should I Use?

This is probably one of the most important questions.

There isn't one "best" unsupervised learning algorithm.

The right choice depends on the question you're asking and the structure of your data.

Problem	Possible approach
Find natural groups	K-Means
Understand hierarchical relationships	Hierarchical Clustering
Find irregularly shaped dense groups	DBSCAN
Reduce many features	PCA
Visualize complex high-dimensional data	t-SNE
Find unusual observations	Isolation Forest / LOF
Model probabilistic groups	Gaussian Mixture Models
Discover latent components/topics	NMF / LDA / other decomposition methods

Scikit-learn's own comparison of clustering methods emphasizes that different algorithms make different assumptions about cluster geometry, scalability, and the type of structure they are designed to capture. (GitHub)

What I Have Learned About Unsupervised Learning

One thing that has stood out to me while working through clustering is that unsupervised learning requires a different mindset.

In supervised learning, evaluation can feel more straightforward.

You have:

Actual value
      ↓
Prediction
      ↓
Compare them
      ↓
Measure performance

With unsupervised learning, the question becomes much more open-ended.

If K-Means gives me three clusters, how do I know those clusters are actually useful?

If PCA reduces my features to two components, what information did I lose?

If an anomaly detection algorithm identifies an observation as unusual, is it actually an error — or is it simply an important but rare case?

These questions require more than just running .fit() and looking at the output.

They require domain knowledge, exploration, visualization, and critical thinking.

My Practical Unsupervised Learning Workflow

When working with a real dataset, I wouldn't immediately jump into K-Means.

I'd start with understanding the data.

My workflow would look something like:

Load dataset
      ↓
Basic checks
      ↓
Missing values
      ↓
Duplicates
      ↓
Data types
      ↓
Outlier analysis
      ↓
Correlation analysis
      ↓
Feature selection
      ↓
Scaling
      ↓
Choose an unsupervised technique
      ↓
Train the model
      ↓
Evaluate the structure
      ↓
Visualize
      ↓
Interpret the results

And preprocessing matters.

For distance-based algorithms such as K-Means and many hierarchical clustering approaches, features with very different scales can distort the distance calculations. That's why scaling is an important part of the workflow. Scikit-learn also highlights feature scaling as relevant when methods depend on relationships between features. (Scikit-learn)

The Bigger Picture

For me, unsupervised learning is interesting because it changes the role of the data scientist.

Sometimes we're not trying to build a model that predicts an answer.

Sometimes we're trying to ask better questions.

Maybe the dataset contains customer segments we hadn't considered.

Maybe there are unusual transactions hiding among millions of normal ones.

Maybe dozens of variables can be represented by a much smaller number of meaningful components.

Maybe the groups we assumed existed don't actually appear in the data.

And that last possibility is important.

Unsupervised learning doesn't guarantee that the pattern we hoped to find exists.

Sometimes the most valuable result is discovering that our assumptions were wrong.

Final Thoughts

I'm still learning my way through machine learning, but unsupervised learning has made me appreciate something I think is easy to overlook:

Data doesn't always need to give us the answer before we can learn something from it.

Sometimes the most interesting insights come from allowing the data to reveal its own structure.

K-Means, hierarchical clustering, DBSCAN, PCA, t-SNE, Gaussian Mixture Models, anomaly detection, and decomposition methods all approach that idea differently.

The real skill isn't memorizing every algorithm.

It's learning to ask:

What am I trying to discover, what assumptions does this algorithm make, and does the structure it finds actually make sense?

That's where unsupervised learning becomes more than just another machine-learning technique.

It becomes a way of exploring data.

References

Machine Learning: The Technology You Use Every Day Without Realizing It

Audrine Marion — Thu, 06 Aug 2026 10:41:17 +0000

When most people hear the words machine learning, they immediately think of robots, self-driving cars, or science fiction movies. I used to think the same.

But the truth is much simpler and much more interesting.

Machine learning is already part of our daily lives. Every time Netflix recommends your next favorite show, Spotify creates a playlist you'll probably love, or Google Maps finds the fastest route home, you're interacting with machine learning.

The surprising part? Most of us use it every day without even noticing.

In this article, we'll explore what machine learning actually is, how it works, and the real-world applications that are quietly changing the way we live and work.

So, What Exactly Is Machine Learning?

Machine learning (ML) is a branch of artificial intelligence (AI) that enables computers to learn from data instead of being explicitly programmed for every single task.

Think about teaching a child to recognize cats.

You don't give them a long list of mathematical rules describing cats. Instead, you show them hundreds of pictures. Over time, they begin to recognize patterns - pointy ears, whiskers, fur, and a tail.

Machine learning works in a similar way.

Instead of writing thousands of rules, we provide a model with data. The model identifies patterns and uses those patterns to make predictions or decisions when it encounters new information.

For example, rather than programming every possible spam email, we train a machine learning model using thousands of examples of both spam and legitimate emails. Eventually, it becomes capable of identifying spam on its own.

How Does Machine Learning Work?

Although machine learning can become mathematically complex, the overall process is surprisingly straightforward.

1. Collect Data

Everything starts with data.

The quality of a machine learning model depends heavily on the quality of the data it's trained on. This data could include:

Customer purchases
Medical records
Images
Text documents
Sensor readings
Financial transactions

The more relevant and accurate the data, the better the model can learn.

2. Prepare the Data

Raw data is rarely perfect.

It often contains missing values, duplicate records, inconsistent formatting, or irrelevant information.

Before training a model, data scientists clean and prepare the data by:

Removing duplicates
Handling missing values
Correcting errors
Encoding categorical variables
Scaling numerical features

This step may not be glamorous, but it often determines whether a project succeeds or fails.

3. Train the Model

Once the data is ready, it's fed into a machine learning algorithm.

The algorithm studies the relationships between different variables and gradually learns patterns hidden within the data.

For example, a model trained on housing data might learn that:

Larger houses generally cost more.
Houses near schools tend to have higher prices.
Location often matters more than the age of the house.

4. Test the Model

Learning patterns is one thing.

Applying them to new data is another.

That's why we evaluate the model using data it has never seen before. This helps determine whether it truly understands the patterns or has simply memorized the training data.

A good model performs well on new, unseen examples.

5. Make Predictions

Once the model performs well, it can be deployed into real applications.

Examples include:

Predicting loan defaults
Detecting fraudulent transactions
Recommending products
Forecasting sales
Diagnosing diseases

Types of Machine Learning

Not all machine learning problems are the same. Depending on the type of data and the desired outcome, different learning approaches are used.

Supervised Learning

In supervised learning, the model learns using labeled data.

This means every training example already contains the correct answer.

For example:

Input	Output
House features	House price
Email	Spam or Not Spam
Medical symptoms	Disease diagnosis

Common algorithms include:

Linear Regression
Logistic Regression
Decision Trees
Random Forest
Support Vector Machines (SVM)
K-Nearest Neighbors (KNN)

Unsupervised Learning

Here, the data has no labels.

The model tries to discover hidden patterns or group similar data together.

Applications include:

Customer segmentation
Market basket analysis
Document clustering
Anomaly detection

Popular algorithms include:

K-Means Clustering
DBSCAN
Hierarchical Clustering
Principal Component Analysis (PCA)

Reinforcement Learning

In reinforcement learning, an agent learns by interacting with an environment.

It receives rewards for good decisions and penalties for poor ones, gradually improving through trial and error.

Applications include:

Robotics
Game AI
Autonomous vehicles
Recommendation optimization

Real-World Applications of Machine Learning

Machine learning isn't confined to research labs. It's solving real problems across almost every industry.

1. Healthcare

Healthcare has experienced significant improvements thanks to machine learning.

Doctors and researchers use ML to:

Detect diseases earlier
Analyze X-rays and MRI scans
Predict patient outcomes
Discover new medicines
Personalize treatments

For instance, machine learning models can identify early signs of diseases such as diabetic retinopathy from eye scans, helping patients receive treatment before their vision deteriorates.

2. Finance

Banks process millions of transactions every day.

Machine learning helps by:

Detecting fraudulent transactions
Assessing credit risk
Predicting loan defaults
Automating investment decisions
Identifying suspicious financial activity

Whenever your bank sends an alert about unusual activity on your card, there's a good chance machine learning played a role.

3. E-commerce

Online retailers use machine learning to create more personalized shopping experiences.

Examples include:

Product recommendations
Personalized discounts
Demand forecasting
Inventory optimization
Dynamic pricing

Ever wondered how Amazon seems to know exactly what you might want to buy next? That's machine learning analyzing shopping behavior and identifying patterns.

4. Entertainment

Streaming platforms rely heavily on recommendation systems.

Netflix recommends movies.

Spotify builds personalized playlists.

YouTube suggests videos you might enjoy.

These recommendations are generated by machine learning models that learn from your viewing or listening habits and compare them with millions of other users.

5. Transportation

Navigation apps do much more than display maps.

Machine learning helps estimate:

Traffic conditions
Travel times
Accident risks
Optimal delivery routes

Ride-hailing services also use ML to predict demand, estimate fares, and match drivers with passengers efficiently.

6. Agriculture

Farmers are increasingly adopting machine learning to improve productivity.

Applications include:

Crop disease detection
Yield prediction
Weather forecasting
Precision irrigation
Soil monitoring

These technologies help farmers make informed decisions while reducing waste and improving food production.

7. Cybersecurity

As cyber threats become more sophisticated, machine learning has become an essential defense tool.

ML systems can:

Detect malware
Identify unusual network activity
Prevent phishing attacks
Recognize suspicious login attempts
Respond to threats more quickly than manual monitoring

Instead of relying only on predefined rules, these systems continuously learn from new attack patterns.

8. Environmental Conservation

Machine learning is also making a difference in protecting our planet.

Researchers use it to:

Monitor deforestation
Predict floods and droughts
Track wildlife populations
Analyze satellite imagery
Forecast climate trends

By processing massive environmental datasets, ML helps governments and conservation organizations make better decisions about protecting natural resources.

Challenges of Machine Learning

Despite its impressive capabilities, machine learning is not without limitations.

Some common challenges include:

Poor-quality data leading to inaccurate predictions
Bias in training data producing unfair outcomes
High computational costs for large models
Privacy concerns surrounding sensitive data
Difficulty explaining complex model decisions

Building an accurate model is only part of the challenge. Ensuring it is fair, reliable, and transparent is equally important.

Why Learning Machine Learning Matters

As industries continue to generate enormous amounts of data, the demand for professionals who can analyze that data and build intelligent systems is growing rapidly.

The good news is that you don't need a PhD to get started.

A strong foundation in Python, statistics, SQL, and data analysis is often enough to begin exploring machine learning. From there, you can gradually learn algorithms, model evaluation, feature engineering, and deployment.

The field rewards curiosity and consistent practice more than memorizing complex formulas.

Final Thoughts

Machine learning isn't some distant technology reserved for tech giants. It's already woven into everyday life from the recommendations you see online to the fraud alerts that protect your bank account and the navigation apps that save you time.

What makes machine learning so powerful isn't that it replaces human intelligence; it's that it helps us make better decisions by uncovering patterns hidden within data.

As the amount of data we generate continues to grow, machine learning will play an even bigger role in healthcare, finance, education, environmental conservation, and countless other fields.

Whether you're a student, an aspiring data scientist, or simply curious about technology, understanding machine learning is becoming less of a niche skill and more of a valuable form of digital literacy.

And who knows? The next machine learning application that changes people's lives might be the one you build.

Why Hypothesis Testing is the Backbone of Data Science

Audrine Marion — Mon, 29 Jun 2026 11:37:00 +0000

From A/B testing and machine learning to business intelligence and scientific research, hypothesis testing helps data scientists separate signal from noise.

Introduction

Imagine you're analyzing customer data and discover that users who receive promotional emails spend 15% more than those who don't.

Sounds like a breakthrough, right?

But before presenting your findings to stakeholders, you need to answer an important question:

Is this difference real, or could it have happened by random chance?

This is where hypothesis testing comes in.

Hypothesis testing is one of the most important statistical tools in data science. It provides a systematic way to evaluate assumptions, validate findings, and make data-driven decisions with confidence.

Whether you're building machine learning models, conducting market research, optimizing products, or running experiments, hypothesis tests help ensure that your conclusions are supported by evidence rather than coincidence.

In this article, we'll explore the most common hypothesis tests used in data science and why each one matters.

What is Hypothesis Testing?

Hypothesis testing is a statistical method used to determine whether there is enough evidence in a dataset to support a particular claim.

Every hypothesis test starts with two competing statements:

Null Hypothesis (H₀)

The assumption that there is no effect, no difference, or no relationship.

Alternative Hypothesis (H₁)

The assumption that an effect, difference, or relationship exists.

The goal is to collect data and determine whether the evidence is strong enough to reject the null hypothesis.

1. Chi-Square Test

What It Does

The Chi-Square Test is used with categorical data to determine whether variables are related or whether observed frequencies match expected frequencies.

Common Types

Chi-Square Test of Independence
Chi-Square Goodness-of-Fit Test

Why It Matters in Data Science

Customer Behavior Analysis

Suppose an online retailer wants to know whether gender influences product preference.

Gender	Product A	Product B
Male	120	80
Female	90	110

A Chi-Square Test can determine whether the differences are statistically significant.

Market Research

Businesses use Chi-Square tests to answer questions such as:

Does age influence brand choice?
Does region affect purchasing behavior?
Does device type affect website engagement?

Fraud Detection

Banks compare expected and observed transaction patterns to identify unusual behavior that may indicate fraud.

Key Question It Answers

Are categorical variables associated with each other?

2. T-Test

What It Does

A T-Test compares the means of two groups.

Common Types

One-Sample T-Test
Independent T-Test
Paired T-Test

Why It Matters in Data Science

A/B Testing

Imagine testing two versions of a website:

Version A conversion rate: 4.1%
Version B conversion rate: 5.0%

A T-Test helps determine whether Version B truly performs better or if the difference occurred by chance.

Product Improvement

Companies frequently compare:

New versus old interfaces
New versus old recommendation systems
Marketing campaigns

Healthcare Analytics

Researchers compare treatment outcomes, recovery times, and drug effectiveness.

Key Question It Answers

Are the means of two groups significantly different?

3. ANOVA (Analysis of Variance)

What It Does

ANOVA compares the means of three or more groups simultaneously.

Without ANOVA, analysts would need multiple T-Tests, increasing the risk of incorrect conclusions.

Why It Matters in Data Science

Marketing Performance Analysis

Suppose a company advertises on:

Google
Facebook
Instagram
LinkedIn

ANOVA determines whether at least one platform produces significantly different results.

Customer Segmentation

Businesses often classify customers into segments and compare spending behavior across groups.

Machine Learning Evaluation

Researchers may compare the performance of multiple algorithms and use ANOVA to determine whether observed differences are meaningful.

Key Question It Answers

Is there a significant difference among multiple group means?

4. Correlation Testing

What It Does

Correlation measures the strength and direction of a relationship between variables.

Common methods include:

Pearson Correlation
Spearman Correlation

Why It Matters in Data Science

Feature Selection

When building predictive models, data scientists need to identify variables that influence outcomes.

For example:

House size vs house price
Advertising spend vs revenue
Study hours vs exam scores

Strong correlations often indicate useful predictive features.

Business Intelligence

Organizations use correlation analysis to uncover hidden relationships in data.

Key Question It Answers

Are two variables related?

5. Regression Significance Testing

What It Does

Regression models use hypothesis testing to determine whether predictor variables contribute significantly to predictions.

Why It Matters in Data Science

Predictive Modeling

Suppose you're predicting sales using:

Marketing spend
Product price
Website traffic

Regression significance tests help identify which variables genuinely impact sales.

Explainable AI

Organizations increasingly require transparency in machine learning models.

Regression testing helps answer:

Which factors matter most?
Which variables can be removed?
What drives business outcomes?

Key Question It Answers

Which variables significantly influence the target variable?

6. Z-Test

What It Does

A Z-Test compares sample statistics with population parameters, typically when sample sizes are large.

Why It Matters in Data Science

Large-Scale Experiments

Technology companies often analyze millions of users.

When datasets become very large, Z-tests provide efficient statistical evaluation.

Quality Control

Manufacturers use Z-tests to verify whether products meet required standards.

Key Question It Answers

Is a sample significantly different from a population expectation?

7. Non-Parametric Tests

Not all datasets follow a normal distribution.

When traditional assumptions are violated, non-parametric tests become extremely valuable.

Mann-Whitney U Test

Alternative to the Independent T-Test.

Applications

Customer satisfaction surveys
Product ratings
User review analysis

Key Question

Are two independent groups different when normality assumptions are not met?

Wilcoxon Signed-Rank Test

Alternative to the Paired T-Test.

Applications

Before-and-after studies
UX improvement evaluations
Employee performance assessments

Key Question

Has a measurable change occurred between paired observations?

Kruskal-Wallis Test

Alternative to ANOVA.

Applications

Comparing multiple customer segments
Ranking-based datasets
Non-normal distributions

Key Question

Are multiple groups significantly different without assuming normality?

8. Proportion Tests

What They Do

Proportion tests compare percentages between groups.

Why They Matter in Data Science

Conversion Rate Optimization

Suppose an online store tests two landing pages:

Version	Conversions	Visitors
A	400	10,000
B	520	10,000

A proportion test determines whether the higher conversion rate is statistically significant.

Public Health Analytics

Researchers compare:

Vaccination rates
Disease prevalence
Adoption rates

Key Question It Answers

Are differences in proportions statistically significant?

Why Every Data Scientist Should Understand Hypothesis Testing

Hypothesis testing is more than a statistics topic taught in university courses.

It directly impacts real-world decision-making.

It Prevents Costly Mistakes

Without statistical testing, organizations may invest resources based on random fluctuations.

It Enables Data-Driven Decisions

Businesses rely on evidence rather than assumptions.

It Improves Machine Learning Models

Hypothesis tests help validate relationships between variables and target outcomes.

It Powers Experimentation

Modern companies continuously run experiments involving:

Product design
Pricing strategies
Marketing campaigns
User experience improvements

Hypothesis testing determines whether those experiments produced meaningful results.

It Builds Trust

Stakeholders are more likely to trust conclusions backed by statistical evidence than intuition alone.

Final Thoughts

Data science is often described as the intersection of statistics, programming, and domain expertise. Among these pillars, hypothesis testing serves as the mechanism that transforms raw observations into credible knowledge.

Each test serves a unique purpose:

Test	Primary Purpose
Chi-Square	Analyze relationships between categorical variables
T-Test	Compare two means
ANOVA	Compare multiple means
Correlation Test	Measure relationships between variables
Regression Test	Evaluate predictor significance
Z-Test	Compare sample and population statistics
Mann-Whitney U	Compare two non-normal groups
Wilcoxon Test	Compare paired observations
Kruskal-Wallis	Compare multiple non-normal groups
Proportion Test	Compare percentages between groups

As data scientists, our goal is not simply to find patterns in data but to determine whether those patterns are meaningful. Hypothesis testing provides the statistical foundation that makes this possible.

The next time you discover an interesting trend in your dataset, don't just ask "What does the data show?"

Ask:

"Is the evidence strong enough to support this conclusion?"

That's the question hypothesis testing was designed to answer.

Understanding Statistical Distributions and Their Impact on Data Science

Audrine Marion — Mon, 22 Jun 2026 10:25:18 +0000

Data is at the heart of every data science project. Whether you're predicting customer churn, detecting fraud, forecasting sales, or building recommendation systems, understanding how data is distributed can significantly improve your analysis and model performance.

Yet, distributions are often overlooked by beginners who jump straight into machine learning algorithms. In reality, understanding data distributions is one of the most important statistical foundations in data science.

In this article, we'll explore what statistical distributions are, the most common distributions used in data science, and how they influence data analysis and machine learning outcomes.

What Is a Statistical Distribution?

A statistical distribution describes how values in a dataset are spread across different ranges.

Think of a distribution as a map that shows:

Which values occur most frequently
Which values are rare
The overall shape of the data
The likelihood of observing specific values

For example, if you collect exam scores from 1,000 students, a distribution can show whether most students scored around 70%, whether scores are evenly spread, or whether there are extreme outliers.

Understanding this pattern helps data scientists make informed decisions about preprocessing, modeling, and interpretation.

Why Distributions Matter in Data Science

Distributions influence almost every stage of the data science workflow.

1. Better Understanding of Data

Before building any model, analysts need to understand the characteristics of their data.

Questions such as:

Is the data normally distributed?
Are there outliers?
Is the data skewed?
Are there multiple peaks?

can only be answered by examining distributions.

Visualizations such as histograms, density plots, and box plots help reveal these characteristics.

2. Improved Feature Engineering

Many machine learning algorithms perform better when features follow certain distribution patterns.

For example:

Linear Regression assumes normally distributed residuals.
Logistic Regression performs better with appropriately scaled variables.
Neural Networks often benefit from normalized inputs.

Understanding distributions helps determine whether transformations such as logarithmic scaling, standardization, or normalization are necessary.

3. Better Model Selection

Different statistical models are designed for different data distributions.

Examples include:

Poisson Regression for count data
Gaussian Models for normally distributed data
Exponential Models for waiting-time events

Choosing the wrong model for a dataset can lead to poor predictions and unreliable insights.

4. Outlier Detection

Outliers often indicate:

Data entry errors
Fraudulent activities
Rare but important events

Distribution analysis helps identify these unusual observations before they negatively affect model performance.

Common Statistical Distributions in Data Science

1. Normal Distribution

The Normal Distribution, also known as the Gaussian Distribution, is the most widely used distribution in statistics.

Characteristics:

Bell-shaped curve
Symmetrical around the mean
Mean, median, and mode are equal

Examples:

Human heights
IQ scores
Measurement errors

Why it matters:

Many statistical techniques assume normality. Understanding whether data approximates a normal distribution can influence the choice of algorithms and evaluation methods.

           *
         *   *
       *       *
     *           *
   *               *

2. Uniform Distribution

In a Uniform Distribution, every value has an equal probability of occurring.

Examples:

Rolling a fair die
Random number generation

Why it matters:

Uniform distributions are commonly used in simulations, random sampling, and initialization procedures in machine learning.

3. Poisson Distribution

The Poisson Distribution models the number of times an event occurs within a fixed interval.

Examples:

Number of website visits per minute
Number of customer calls per hour
Number of accidents at a junction per month

Why it matters:

Many real-world business problems involve counting events, making the Poisson Distribution highly relevant for predictive analytics.

4. Binomial Distribution

The Binomial Distribution describes the number of successes in a fixed number of independent trials.

Examples:

Email opened or not opened
Customer purchased or did not purchase
Coin toss outcomes

Why it matters:

Classification problems often involve concepts rooted in binomial probability.

5. Exponential Distribution

The Exponential Distribution models the time between independent events.

Examples:

Time until equipment failure
Time between customer arrivals
Waiting time before receiving a call

Why it matters:

It is commonly used in reliability analysis, operations research, and queueing systems.

Understanding Skewness

Not all datasets follow a perfect normal distribution.

Right-Skewed Distribution

Most values are concentrated on the left, with a long tail extending to the right.

Examples:

Income distributions
Property prices
Online transaction values

Left-Skewed Distribution

Most values are concentrated on the right, with a long tail extending to the left.

Examples:

Difficult exam scores where most students perform well

Why it matters:

Skewed data can affect statistical calculations and machine learning model performance.

Common solutions include:

Log transformations
Square root transformations
Box-Cox transformations

The Impact of Distributions on Machine Learning

Linear Regression

Linear Regression assumes:

Normally distributed residuals
Constant variance
Independence of observations

Violating these assumptions may reduce model reliability.

Decision Trees

Decision Trees are generally less sensitive to distributions.

This makes them useful when data contains skewness, outliers, or non-linear relationships.

Neural Networks

Neural Networks often perform better when features are normalized or standardized.

Poorly distributed inputs can slow down learning and reduce accuracy.

Clustering Algorithms

Algorithms such as K-Means rely heavily on distance calculations.

Highly skewed distributions can distort cluster formation and produce misleading results.

Practical Example

Imagine you're analyzing monthly customer spending in an e-commerce business.

A histogram reveals a heavily right-skewed distribution:

Most customers spend less than $100.
A few customers spend thousands of dollars.

If you use the raw data:

The average spending value becomes inflated.
Models may become biased toward high spenders.

A log transformation can make the distribution more balanced, resulting in:

Better visualizations
More accurate predictions
Improved model stability

This simple adjustment demonstrates how understanding distributions can directly improve business outcomes.

Tools for Analyzing Distributions

Python provides several libraries for distribution analysis:

Matplotlib

import matplotlib.pyplot as plt

plt.hist(data, bins=30)
plt.show()

Seaborn

import seaborn as sns

sns.histplot(data, kde=True)

SciPy

from scipy import stats

stats.normaltest(data)

Pandas

data.skew()
data.kurtosis()

These tools help data scientists quickly evaluate distribution characteristics before modeling.

Final Thoughts

Statistical distributions are more than just theoretical concepts taught in statistics classes. They form the foundation of data science and machine learning.

By understanding distributions, data scientists can:

Explore data more effectively
Detect anomalies and outliers
Select appropriate models
Improve feature engineering
Increase prediction accuracy

Before building your next machine learning model, spend time understanding how your data is distributed. The insights gained from distribution analysis can often be more valuable than trying a new algorithm.

Remember: great models begin with a deep understanding of the data behind them.

Python for Data Analytics: A Beginner’s Complete Guide

Audrine Marion — Sun, 10 May 2026 11:43:32 +0000

In today’s digital world, data has become one of the most valuable resources. Every time we shop online, use social media, stream music, make a mobile payment, or search the internet, data is being created. Businesses and organizations use this information to understand customers, improve services, increase profits, and make smarter decisions.

However, raw data alone is not useful unless it can be analyzed properly. This is where data analytics comes in. Data analytics helps transform raw information into meaningful insights that organizations can use to solve problems and make informed decisions.

Among all the tools used in data analytics, Python has become one of the most popular and powerful programming languages in the world. From beginners learning their first programming language to professional data scientists working at major companies, Python is now considered an essential skill in the data industry.

When I first started learning about data analytics, Python seemed intimidating. I assumed programming was only for software engineers and highly technical professionals. However, after exploring small projects and learning the basics, I realized why so many analysts rely on Python every day. It simplifies complex tasks, automates repetitive work, and makes analyzing data far more efficient.

This article explains what Python is, why it is widely used in data analytics, the major Python libraries analysts use, how Python helps clean and visualize data, and why beginners should strongly consider learning it.

What Is Python?

Python is a high-level programming language created by Guido van Rossum and officially released in 1991. It was designed with simplicity and readability in mind, making it easier to understand compared to many older programming languages.

Unlike some languages that use complicated syntax and lengthy code structures, Python focuses on writing clean and readable code. This makes it especially beginner-friendly.

For example, a simple Python program can look like this:

print("Hello, World!")

That single line prints text to the screen.

One reason Python became so popular is because it allows people to focus on solving problems instead of struggling with difficult syntax rules. Whether someone wants to build websites, automate tasks, develop artificial intelligence systems, or analyze data, Python provides a flexible and accessible starting point.

Python is also open-source, meaning anyone can use it for free. It works across major operating systems including Windows, Linux, and macOS.

Today, Python is used in many areas such as:

Web development
Cybersecurity
Automation
Artificial intelligence
Machine learning
Data analytics

Among these areas, data analytics has become one of Python’s strongest and fastest-growing fields.

Understanding Data Analytics

Before exploring Python’s role in analytics, it is important to understand what data analytics actually means.

Data analytics refers to the process of examining data to discover patterns, trends, relationships, and useful insights. Organizations use analytics to make better decisions and improve performance.

For example, a company may analyze customer purchasing data to answer questions like:

Which products sell the most?
Which customers are likely to stop buying?
Which marketing campaigns perform best?
What sales trends appear during different seasons?

Data analytics usually involves several steps:

Collecting data
Cleaning and organizing data
Exploring data
Analyzing patterns
Visualizing results
Reporting insights

Python can support all these stages efficiently, which is one of the reasons it has become so valuable in the analytics industry.

Why Python Is So Popular in Data Analytics

Python’s popularity in data analytics continues to grow rapidly. Several factors contribute to this success.

1. Python Is Beginner-Friendly

One of Python’s greatest strengths is its simplicity.

Compared to programming languages such as Java or C++, Python uses cleaner and more readable syntax. Beginners can often understand Python code even with limited programming experience.

For example:

numbers = [10, 20, 30, 40]
average = sum(numbers) / len(numbers)

print(average)

Even someone new to coding can understand that this program calculates an average.

Because of its simplicity, many universities and online learning platforms now teach Python as a first programming language.

2. Python Has a Huge Community

Python has millions of users worldwide. This large community makes learning easier because beginners can quickly find tutorials, videos, forums, and documentation online.

If a learner encounters an error or problem, chances are high that someone else has already solved it and shared the solution online.

Popular learning platforms include:

This strong community support makes Python easier to learn compared to many technical tools.

3. Python Has Powerful Data Libraries

One major reason analysts love Python is its collection of specialized libraries.

Libraries are pre-written collections of code that help programmers complete tasks more efficiently. Instead of writing everything from scratch, analysts can use these libraries to process and analyze data quickly.

Some important Python libraries include:

Pandas
NumPy
Matplotlib
Seaborn
Scikit-learn

These libraries have transformed Python into one of the most powerful analytics tools available today.

4. Python Saves Time Through Automation

Many data tasks are repetitive and time-consuming.

For example, analysts may need to:

Clean datasets daily
Generate weekly reports
Update dashboards
Organize spreadsheets
Combine multiple files

Doing these tasks manually can waste hours.

Python helps automate these processes using scripts, reducing human error and improving efficiency.

Automation is one reason companies increasingly prefer analysts with Python skills.

5. Python Is Used Across Industries

Python is not limited to one field.

Industries using Python for analytics include:

Finance
Healthcare
E-commerce
Marketing
Transportation
Telecommunications
Education

Because Python is widely applicable, learning it opens opportunities across many career paths.

Important Python Libraries for Data Analytics

Python’s true strength in analytics comes from its libraries.

Pandas

Pandas is one of the most important Python libraries for data analytics.

It allows analysts to:

Read CSV and Excel files
Clean messy data
Filter rows and columns
Merge datasets
Calculate statistics
Organize information into tables

Pandas uses structures called DataFrames, which resemble Excel spreadsheets.

Example:

python id="p7t9q1"
import pandas as pd

data = {
    "Name": ["Alice", "Brian", "Cynthia"],
    "Age": [24, 29, 31]
}

df = pd.DataFrame(data)

print(df)

This creates a simple table of names and ages.

Pandas is essential for almost every data analytics project.

NumPy

NumPy focuses on numerical operations and mathematical calculations.

It is especially useful when handling large amounts of numerical data efficiently.

NumPy helps analysts:

Perform calculations
Work with arrays
Calculate statistics
Handle mathematical functions

Example:

python id="f4w2r8"
import numpy as np

numbers = np.array([5, 10, 15, 20])

print(numbers.mean())

This calculates the average of the numbers.

Matplotlib

Matplotlib is one of the most widely used visualization libraries in Python.

Data visualization helps convert raw data into understandable charts and graphs.

Matplotlib can create:

Line charts
Bar charts
Pie charts
Histograms
Scatter plots

Example:

python id="l1m9s7"
import matplotlib.pyplot as plt

x = [1, 2, 3, 4]
y = [10, 15, 25, 40]

plt.plot(x, y)
plt.show()

This generates a simple line graph.

Visualizations make it easier for businesses and decision-makers to understand data trends.

Seaborn

Seaborn is built on top of Matplotlib and provides more advanced and visually appealing charts.

Analysts use Seaborn for:

Correlation heatmaps
Statistical charts
Distribution plots
Comparative visualizations

Seaborn is popular because it creates professional-looking visuals with relatively little code.

Scikit-learn

Scikit-learn is widely used for machine learning and predictive analytics.

Although machine learning is more advanced than basic analytics, many analysts eventually use Scikit-learn for predictive tasks such as:

Customer churn prediction
Fraud detection
Sales forecasting
Recommendation systems

Scikit-learn simplifies complex machine learning processes for developers and analysts.

How Python Is Used in Data Analytics

Python supports analysts throughout the entire analytics workflow.

Data Collection

The first step in analytics is gathering data.

Python can collect data from:

CSV files
Excel spreadsheets
Databases
APIs
Websites

Analysts often use Python for web scraping, which involves extracting information from websites automatically.

Libraries such as BeautifulSoup and Requests help collect online data efficiently.

For example, companies may scrape product prices, customer reviews, or market information for analysis.

Data Cleaning

Real-world data is rarely perfect.

Datasets often contain:

Missing values
Duplicate records
Formatting problems
Incorrect entries

Poor-quality data can produce misleading insights.

Python makes cleaning easier using Pandas.

Example:

python id="n6r3b5"
df.dropna()

This removes rows with missing values.

Another example:

python id="u8w2y4"
df.drop_duplicates()

This removes duplicate records.

Python helps automate cleaning tasks that would take much longer in spreadsheet software.

Data Exploration

Once data is cleaned, analysts explore it to understand patterns and trends.

This stage includes:

Finding averages
Measuring relationships
Identifying unusual values
Understanding distributions

Python helps generate quick summaries.

Example:

python id="h5t2e1"
df.describe()

This produces statistics such as:

Mean
Minimum value
Maximum value
Standard deviation

Exploring data helps analysts understand what the dataset contains before performing deeper analysis.

Data Visualization

One of the most important parts of analytics is visualization.

Humans understand visuals faster than raw numbers. Charts and graphs simplify complex information.

Python allows analysts to create dashboards and visuals showing:

Sales performance
Customer behavior
Market trends
Financial reports

For example, a company may use graphs to identify which products perform best during certain months.

Good visualizations improve communication and support better business decisions.

Predictive Analytics and Machine Learning

Modern analytics increasingly focuses on predicting future outcomes instead of only analyzing past data.

Python supports predictive analytics using machine learning.

Organizations use Python to predict:

Customer behavior
Product demand
Stock trends
Equipment failures
Fraudulent transactions

This allows businesses to make proactive decisions instead of reacting after problems occur.

Real-World Examples of Python in Data Analytics

Python is used globally across multiple industries.

Healthcare Analytics

Hospitals and healthcare organizations analyze patient data using Python.

Analytics helps healthcare professionals:

Predict disease risks
Monitor patient trends
Improve hospital efficiency
Analyze treatment effectiveness

Python played an important role in analyzing health data during global disease outbreaks.

Finance and Banking

Banks generate massive amounts of financial data daily.

Python is widely used for:

Fraud detection
Risk assessment
Credit scoring
Investment analysis

Financial analysts use Python because it processes large datasets efficiently.

E-Commerce

Online shopping platforms rely heavily on analytics.

Python helps businesses analyze:

Customer purchasing behavior
Product recommendations
Website performance
Sales trends

Recommendation systems used by streaming and shopping platforms are often powered by Python.

Marketing Analytics

Marketing teams use Python to evaluate campaign performance.

Analytics helps companies understand:

Audience engagement
Advertisement performance
Customer interests
Social media trends

This helps businesses improve their marketing strategies.

Transportation and Logistics

Transportation companies use Python for:

Route optimization
Demand forecasting
Delivery tracking
Traffic analysis

Analytics improves efficiency and reduces operational costs.

Common Mistakes Beginners Make When Learning Python

Many beginners struggle because they approach Python the wrong way.

Some common mistakes include:

Trying to Learn Everything at Once

Python is a large language with many areas. Beginners should focus on fundamentals first instead of rushing into advanced topics.

Watching Tutorials Without Practicing

Watching videos alone is not enough.

The best way to learn Python is by writing code regularly and building small projects.

Ignoring Real Datasets

Real learning happens when working with messy, real-world data.

Beginners should practice using datasets from:

Focusing Too Much on Syntax

Many beginners try to memorize every command.

Instead, they should focus on problem-solving and understanding logic.

Even experienced developers frequently search online for syntax help.

Beginner Python Analytics Project Idea

One of the best ways to improve is through projects.

A simple beginner project could involve analyzing sales data.

Steps might include:

Import a CSV file using Pandas
Clean missing values
Calculate total sales
Identify top-selling products
Create charts showing trends

This type of project helps beginners practice real analytics workflows.

Why Beginners Should Learn Python

Learning Python offers many long-term benefits.

Strong Career Opportunities

Many employers actively search for candidates with Python skills.

Python appears frequently in job descriptions for:

Data Analysts
Data Scientists
Business Analysts
Machine Learning Engineers

As data continues growing globally, demand for Python skills is expected to remain strong.

Easy Entry Into Tech

Python provides a relatively accessible entry point into the technology industry.

Because it is beginner-friendly, learners can start building useful projects quickly.

Gateway to Advanced Technologies

Python is heavily used in:

Artificial intelligence
Machine learning
Automation
Deep learning
Data science

Learning Python opens doors to multiple technical career paths.

The Future of Python in Data Analytics

Python continues to grow rapidly in popularity.

As organizations generate more data, the demand for analytical tools increases. Python remains central to this growth because of its flexibility, large community, and extensive library ecosystem.

Emerging fields such as artificial intelligence, automation, and big data analytics continue relying heavily on Python.

It is likely that Python will remain one of the dominant programming languages in analytics for many years.

Conclusion

Python has become one of the most important tools in modern data analytics. Its simplicity, flexibility, and powerful libraries make it suitable for both beginners and experienced professionals.

Through libraries such as Pandas, NumPy, Matplotlib, Seaborn, and Scikit-learn, Python enables analysts to collect, clean, analyze, and visualize data efficiently. Organizations across healthcare, finance, transportation, marketing, and e-commerce rely on Python to make data-driven decisions.

For beginners interested in analytics, learning Python is one of the best investments they can make. It provides practical problem-solving skills, strong career opportunities, and access to advanced technologies such as machine learning and artificial intelligence.

Although learning Python requires patience and practice, consistent effort leads to significant growth over time. Starting with small projects, experimenting with real datasets, and practicing regularly can help beginners build confidence and develop valuable analytical skills.

In a world increasingly driven by data, Python is more than just a programming language. It is a powerful tool that helps transform raw information into meaningful insights that can shape businesses, industries, and society itself.

SQL Subqueries vs CTEs: Types, Differences, Performance, and When to Use Each

Audrine Marion — Tue, 21 Apr 2026 11:44:41 +0000

When working with SQL in analytics, reporting, or data engineering workflows, two powerful tools help structure complex logic:

Subqueries
Common Table Expressions (CTEs)

Understanding when to use each improves query readability, performance, and maintainability—especially when building dashboards or transforming datasets for analysis.

This article explains:

what subqueries are
types of subqueries
when to use subqueries
what CTEs are
types of CTEs
performance comparison
when to choose subqueries vs CTEs

I started paying closer attention to the difference between subqueries and CTEs while working on analytics queries that became harder to debug as they grew more complex.

What Is a Subquery?

A subquery is a query nested inside another SQL query.

It executes first, and its output becomes input for the outer query.

Example:

SELECT name, salary
FROM employees
WHERE salary > (
    SELECT AVG(salary)
    FROM employees
);

Here:

the inner query calculates the average salary
the outer query filters employees earning above average

Subqueries are useful for embedding logic directly inside SQL statements.

Types of Subqueries

Subqueries can return different result structures depending on how they are used.

1. Scalar Subquery

Returns one value

Example:

SELECT name,
       (SELECT AVG(salary) FROM employees) AS avg_salary
FROM employees;

Use case:

Comparisons or calculated columns

2. Single-Row Subquery

Returns exactly one row

Example:

SELECT *
FROM employees
WHERE salary = (
    SELECT MAX(salary)
    FROM employees
);

Common operators:

=  >  <  >=  <=

3. Multi-Row Subquery

Returns multiple rows

Example:

SELECT *
FROM employees
WHERE department_id IN (
    SELECT department_id
    FROM departments
    WHERE location = 'Nairobi'
);

Common operators:

IN
ANY
ALL

4. Correlated Subquery

Runs once for every row in the outer query

Example:

SELECT e.name, e.salary
FROM employees e
WHERE salary > (
    SELECT AVG(salary)
    FROM employees
    WHERE department_id = e.department_id
);

Use carefully with large datasets because they may affect performance.

When Should You Use Subqueries?

Subqueries are best when:

filtering using aggregated values
comparing values dynamically
embedding quick logic inside WHERE clauses
simplifying small calculations
avoiding temporary tables

They work especially well when logic is used once only.

What Is a CTE (Common Table Expression)?

A Common Table Expression (CTE) is a temporary named result set created using the WITH clause.

It improves readability and breaks complex queries into logical steps.

Example:

WITH avg_salary AS (
    SELECT AVG(salary) AS avg_sal
    FROM employees
)

SELECT *
FROM employees
WHERE salary > (SELECT avg_sal FROM avg_salary);

Think of a CTE as a temporary table that exists during query execution.

Types of CTEs

1. Non-Recursive CTE

Used for simplifying complex logic

Example:

WITH department_totals AS (
    SELECT department_id,
           SUM(salary) AS total_salary
    FROM employees
    GROUP BY department_id
)

SELECT *
FROM department_totals;

Best for:

transformations
aggregations
reusable logic

2. Recursive CTE

Used for hierarchical datasets

Example:

WITH RECURSIVE employee_hierarchy AS (

    SELECT id, manager_id, name
    FROM employees
    WHERE manager_id IS NULL

    UNION ALL

    SELECT e.id, e.manager_id, e.name
    FROM employees e
    JOIN employee_hierarchy eh
    ON e.manager_id = eh.id
)

SELECT *
FROM employee_hierarchy;

Common use cases:

organization structures
category trees
folder hierarchies
graph traversal

Subqueries vs CTEs: Key Differences

Feature	Subqueries	CTEs
Readability	Moderate	High
Reusability	Low	High
Debugging	Harder	Easier
Recursion Support	No	Yes
Best Use Case	Simple filtering	Multi-step logic

Performance Comparison

Performance depends on the database system.

General guidance:

Use subqueries when:

logic is simple
result used once
query is short

Use CTEs when:

logic reused multiple times
queries become complex
building layered transformations
working with hierarchies

Test performance using:

EXPLAIN ANALYZE

This helps evaluate execution plans.

When Should You Choose Each?

Choose a subquery if:

quick filtering needed
used inside WHERE clause
logic simple

Choose a CTE if:

readability matters
transformation steps needed
recursion required
logic reused multiple times

Final Thoughts

Both subqueries and CTEs are essential SQL tools for analysts and BI professionals.

Subqueries help embed quick logic inside statements.

CTEs improve structure, readability, and scalability—especially in analytics pipelines and dashboards.

Knowing when to use each makes your SQL cleaner, faster to debug, and easier to maintain.

How to publish a Power BI report and embed it into a website.

Audrine Marion — Sun, 05 Apr 2026 15:58:02 +0000

Microsoft Power BI is a powerful business intelligence tool that enables users to transform raw data into interactive dashboards and reports. One of its most useful features is the ability to publish reports to the Power BI Service and embed them into websites for sharing insights with a wider audience.

This guide walks through the full process step by step: creating a workspace, publishing a report, generating embed code, and embedding the report into a website.

Step 1: Create a Workspace in Power BI Service

A workspace is a collaborative environment where reports, dashboards, and datasets are stored before publishing or sharing.

Steps:

Sign in to Power BI Service (https://app.powerbi.com).
From the left navigation panel, select Workspaces.
Click New workspace.
Enter a workspace name and description.
Click Save.

Then proceed to add New Workspace

Why this matters:
Workspaces help organize reports and control access permissions for collaboration and publishing.

Step 2: Upload and Publish Your Report

Once your workspace is ready, the next step is publishing your Power BI Desktop report.

Steps:

Open your report in Power BI Desktop.
Click File → Publish or select Publish from the Home ribbon.
Choose your created workspace.
Click Select.
Wait for the confirmation message indicating successful publishing.

Why this matters:
Publishing moves your report from local development to the cloud where it can be shared and embedded.

Step 3: Generate the Embed Code

After publishing the report, you can generate embed code from Power BI Service.

Steps:

Open Power BI Service.
Navigate to your workspace.
Select your report.
Click File → Embed report → Publish to web (public).
Confirm the security warning.
Copy the generated HTML iframe embed code.

Why this matters:
The iframe embed code allows your report to be displayed inside a webpage.

Step 4: Embed the Report on a Website

Once the embed code is generated, you can place it inside your website's HTML.

Example HTML Code:

<iframe
    width="800"
    height="600"
    src="https://app.powerbi.com/view?r=YOUR_EMBED_LINK"
    frameborder="0"
    allowFullScreen="true">
</iframe>

Steps:

Open your website HTML file.
Paste the iframe code where the report should appear.
Save changes.
Refresh your webpage.

Why this matters:
Embedding enables stakeholders and users to interact with dashboards without logging into Power BI.

Key Insights and Best Practices

Publishing and embedding Power BI reports makes insights accessible to a broader audience. However, it is important to understand visibility settings before sharing.

Key takeaways:

Workspaces help manage report access and organization.
Publishing moves reports from Desktop to Power BI Service.
Embed codes allow reports to be integrated into websites.
"Publish to web" makes reports publicly accessible, so sensitive data should not be included.

Conclusion

Publishing and embedding Power BI reports is a straightforward process that significantly enhances data accessibility and collaboration. By following these steps, you can successfully move your report from Power BI Desktop to a live website and share insights interactively with your audience.

Adding screenshots at each step improves clarity and helps users follow the workflow more easily.

Understanding Data Modeling in Power BI: Joins, Relationships, and Schemas Explained

Audrine Marion — Sun, 29 Mar 2026 20:30:09 +0000

Data modeling is the backbone of every effective Power BI report. If dashboards feel slow, filters behave incorrectly, or numbers don’t match expectations, the issue is often the data model not the visuals.

This guide explains how data modeling works in Power BI step‑by‑step. You’ll learn SQL joins, relationships, schemas, fact vs dimension tables, role‑playing dimensions, and how everything is created inside Power BI itself.

This article is beginner‑friendly but structured like a professional BI reference.

What is Data Modeling?

Data modeling is the process of organizing tables and defining how they relate so reports are accurate, scalable, and fast.

In Power BI, good data modeling helps you:

connect multiple datasets
control filter behavior
improve performance
enable time intelligence
prevent duplicate counting
support executive‑level reporting

Power BI primarily uses relationships instead of joins during analysis.

SQL Joins Explained (With Real Examples)

SQL joins combine tables physically in Power Query before loading data into the model.

Location in Power BI:

Transform Data → Merge Queries

INNER JOIN

Returns only matching records in both tables.

Example:

Customers Table

Orders Table

Result: Only customers who placed orders appear.

Real‑life analytics use case:

Analyzing purchasing customers only.

LEFT JOIN (LEFT OUTER JOIN)

Returns all rows from the left table and matching rows from the right.

Use case:

Customer engagement analysis including inactive customers

RIGHT JOIN (RIGHT OUTER JOIN)

Returns all rows from the right table and matching rows from the left.

Use case:

Transaction completeness audits

FULL OUTER JOIN

Returns all rows from both tables.

Use case:

Data reconciliation between systems

LEFT ANTI JOIN

Returns rows from the left table with no matches in the right table.

Use case:

Customer churn targeting

RIGHT ANTI JOIN

Returns rows from the right table with no matches in the left table.

Use case:

Data quality audits

Relationships in Power BI

Relationships connect tables logically instead of physically merging them.

Location:

Model View → Drag column between tables

OR

Home → Manage Relationships → New

Types of Relationships

One‑to‑Many (1:M)

Most common relationship type.

Example: DimCustomer → FactSales

DimCustomer (1)
┌──────────────┐
│ Customer ID │
│ Customer Name│
│ Region │
└──────┬───────┘
│
│
│
▼
FactSales (Many)
┌──────────────┐
│ Order ID │
│ Customer ID │
│ Sales Amount │
│ Quantity │
└──────────────┘
One-to-Many relationship: One customer can appear multiple times in the sales table.

Many‑to‑Many (M:M)

Occurs when both tables contain duplicate keys.

Example: Students ↔ Courses

Students
┌──────────────┐
│ Student ID │
│ Student Name │
└──────┬───────┘
│
│
▼
Enrollment (Bridge Table)
┌──────────────┐
│ Student ID │
│ Course ID │
└──────┬───────┘
│
│
▼
Courses
┌──────────────┐
│ Course ID │
│ Course Name │
└──────────────┘
Many-to-Many relationship resolved using a bridge table between Students and Courses.

One‑to‑One (1:1)

Rare but useful.

Example: Employee table ↔ Employee security table
Employees
┌──────────────┐
│ Employee ID │
│ Name │
│ Department │
└──────┬───────┘
│
│
▼
EmployeeSecurity
┌──────────────┐
│ Employee ID │
│ Access Level │
│ Login Role │
└──────────────┘
One-to-One relationship: Each employee record matches exactly one security profile.

Active vs Inactive Relationships

Power BI allows multiple relationships between tables but only one active at a time.
DimDate
┌──────────────┐
│ Date │
└──────┬───────┘
│
(Active) ──┼──────── OrderDate
│
(Inactive) ─ ─┼──────── ShipDate
│
(Inactive) ─ ─┼──────── DeliveryDate
│
FactSales

Active relationship (solid line) filters visuals automatically, while inactive relationships (dashed lines) require USERELATIONSHIP() in DAX.

Cardinality Explained

Cardinality describes table relationship structure:

One‑to‑Many, Many‑to‑One, Many‑to‑Many, One‑to‑One

Location:

Manage Relationships → Cardinality dropdown

Cross Filter Direction

Controls how filters move between tables.

Single Direction (recommended) vs Bi‑Directional

Difference Between Joins and Relationships

Power Query Joins: Combine tables physically, run before loading, increase table size
Relationships: Connect tables logically, run after loading, improve performance

Fact vs Dimension Tables

Fact Tables contain numeric values (Sales Amount, Revenue, Quantity)

Dimension Tables contain descriptive attributes (Customer Name, Product Category, Region)

Star Schema (Recommended Model)

Fact table at center, dimension tables surrounding.

Fast performance, simple relationships, scalable dashboards.

Snowflake Schema

Dimensions split into multiple related tables.

Reduced redundancy but more complex filtering.

                DimDepartment
              ┌──────────────┐
              │ Dept ID      │
              │ Dept Name    │
              └──────┬───────┘
                     │
                     │
                DimCategory
              ┌──────────────┐
              │ Category ID  │
              │ Category Name│
              └──────┬───────┘
                     │
                     │
                DimProduct
              ┌──────────────┐
              │ Product ID   │
              │ Product Name │
              └──────┬───────┘
                     │
                     │
             ┌───────▼────────┐
             │    FactSales   │
             │ Sales Amount   │
             │ Quantity       │
             │ Revenue        │
             │ Order ID       │
             └───────┬────────┘
                     │
                     │
                DimCustomer
              ┌──────────────┐
              │ Customer ID  │
              │ Customer Name│
              │ Region       │
              └──────────────┘

Flat Table (Denormalized / DLAT Model)

All fields stored inside one table.

Simple setup but poor scalability.

             Flat Table (Sales Dataset)
        ┌───────────────────────────────┐
        │ Order ID                      │
        │ Order Date                    │
        │ Customer ID                   │
        │ Customer Name                 │
        │ Region                        │
        │ Product ID                    │
        │ Product Name                  │
        │ Category                      │
        │ Department                    │
        │ Sales Amount                  │
        │ Quantity                      │
        │ Revenue                       │
        └───────────────────────────────┘

Example Flat Table (DLAT Model): All descriptive and transactional fields stored inside a single table without relationships between dimensions.

Role‑Playing Dimensions

One dimension reused multiple times (e.g., Date table as Order Date, Ship Date, Delivery Date).

Common Data Modeling Mistakes in Power BI

Circular relationships
Duplicate keys
Many‑to‑many misuse
Too many bi‑directional filters
Over‑joining tables in Power Query

Step‑by‑Step Modeling Workflow in Power BI

Home → Get Data
Transform Data → Power Query
Merge tables if required
Create relationships
Validate cardinality
Test filter flow

Generally

Strong data modeling transforms dashboards into decision systems.

Master joins, schemas, and relationships to move from report builder to analytics engineer.

How Excel is Used in Real-World Data Analysis

Audrine Marion — Mon, 23 Mar 2026 16:17:18 +0000

Introduction

Excel is one of the most widely used tools for data analysis across industries. Although newer technologies like Python, SQL, and Power BI are becoming more common, Excel remains a foundational skill for anyone working with data. It allows analysts to organize information, clean datasets, perform calculations, explore patterns, and present insights in a structured and accessible way.

Because of its flexibility and ease of use, Excel is often the first tool professionals rely on when working with raw data before moving into more advanced analytics environments.

What is Excel?

Excel is a spreadsheet application developed by Microsoft that allows users to store, organize, calculate, and analyze data using rows and columns. It provides built-in formulas, visualization tools, Pivot Tables, and automation features that help transform raw numbers into meaningful insights.

In real-world data analysis, Excel is commonly used for:

Cleaning messy datasets
Performing calculations
Summarizing large datasets
Identifying trends and patterns
Creating dashboards and reports
Supporting business decision-making

Real-World Applications of Excel in Data Analysis

1. Data Cleaning

Before analysis begins, datasets often contain missing values, duplicates, or formatting issues. Excel helps analysts prepare datasets using tools such as:

Remove Duplicates
Find and Replace
Text-to-Columns
Sorting and Filtering

These features ensure that data is accurate and ready for analysis.

2. Data Summarization Using Pivot Tables

Pivot Tables are one of Excel’s most powerful tools. They allow analysts to quickly summarize large datasets and answer important questions such as:

Which product generated the highest revenue?
Which region had the most customers?
Which month recorded the highest sales?

Instead of manually calculating totals, Pivot Tables automatically group and summarize information.

example of pivot table showing the sum of bonus by location

Pivot Tables help analysts transform raw datasets into structured summaries within seconds.

3. Using Formulas for Analysis

Excel formulas help analysts perform calculations efficiently and accurately.

SUM Function

Used to calculate totals such as total sales or expenses.

Example:

=SUM(B2:B100)

Used in scenarios like:

calculating monthly expenses
total company revenue
inventory totals

IF Function

Used to apply logical conditions to categorize data.

Example:

=IF(B2>50,"Pass","Fail")

Real-world uses include:

grading systems
customer segmentation
performance classification

VLOOKUP Function

Used to retrieve matching values from another table.

Example:

=VLOOKUP(A2,Sheet2!A:B,2,FALSE)

Analysts commonly use this when:

merging customer records
matching product IDs
combining datasets from multiple sheets

4. Data Visualization

Excel allows analysts to create charts such as:

Bar charts
Line charts
Pie charts
Column charts

These visualizations make it easier to communicate findings clearly to stakeholders who may not have technical backgrounds.

Example Excel line chart showing average salary across month of hire.

5. Conditional Formatting for Insight Discovery

Conditional formatting highlights important values automatically.

Example uses:

highlighting top-performing regions
identifying missing values
detecting unusually high expenses
spotting declining performance trends

Example of conditional formatting that highlights performance score greater than 5

This improves readability and makes patterns visible immediately.

Features I Have Learned and Applied When Working with Data

While learning Excel, I gained experience using several important features that support data analysis workflows.

Pivot Tables helped me summarize datasets quickly and identify trends without writing complex formulas.

The IF function allowed me to categorize data into meaningful groups, making it easier to interpret results.

VLOOKUP helped me combine information from multiple tables, which is especially useful when working with relational datasets.

Conditional formatting helped highlight important values such as high-performing regions or missing data points, making datasets easier to understand visually.

These tools significantly improved my ability to explore and interpret datasets efficiently.

Personal Reflection

Learning Excel has changed the way I approach data. Previously, I mainly viewed data as numbers arranged in tables, but Excel helped me understand how structured analysis can reveal meaningful insights and patterns.

Through practicing formulas, Pivot Tables, and visualization techniques, I became more confident in cleaning datasets, summarizing information, and presenting findings clearly. Excel also strengthened my problem-solving skills by teaching me how to break down datasets into manageable steps during analysis.

As someone building a career in data analytics and data science, learning Excel has been an important step in developing my analytical thinking. It has helped me move from simply observing data to actively exploring it and using it to support informed decisions.

Overall, Excel has become a valuable foundation in my data analysis journey, and it continues to support my growth as I expand my skills into more advanced tools and technologies.