DEV Community: Mark Glemba

Window functions vs Aggregate functions

Mark Glemba — Thu, 24 Sep 2026 09:50:31 +0000

Introduction

SQL is often introduced as a language for retrieving data, but for data scientists it is much more than that. It is a powerful analytical tool for transforming raw event data into metrics, comparisons, rankings, cohorts, trends, and features for downstream modeling. Two of the most important tools in that analytical toolkit are aggregate functions and window functions.

Aggregate functions reduce rows into summaries. Window functions calculate summaries while preserving the original rows.

That distinction sounds small, but it changes how you design queries, interpret results, and avoid analytical mistakes.

This article explains the difference, using practical examples and discussing when each approach is appropriate.

Aggregate functions: turning many rows into fewer rows

Aggregate functions summarize multiple rows into a single value. Common examples include:

SUM()
AVG()
COUNT()
MIN()
MAX()

Suppose you have a table called orders:

order_id	customer_id	order_date	category	revenue
1	101	2026-01-01	Electronics	500
2	101	2026-01-05	Books	30
3	102	2026-01-03	Electronics	200
4	103	2026-01-04	Books	45
5	102	2026-01-06	Books	25

If you want the total revenue across every order, an aggregate query is appropriate:

SELECT
    SUM(revenue) AS total_revenue
FROM orders;

The result is one row:

total_revenue
800

The original five order rows have been collapsed into one summary row.

More often, analysts aggregate by a dimension. For example, to calculate revenue by product category:

SELECT
    category,
    SUM(revenue) AS total_revenue,
    AVG(revenue) AS average_order_value,
    COUNT(*) AS order_count
FROM orders
GROUP BY category;

This returns one row for each category:

category	total_revenue	average_order_value	order_count
Books	100	33.33	3
Electronics	700	350.00	2

This is the core behavior of aggregation: the GROUP BY clause establishes the grain of the output.

Before the query, the data was at the order level. After the query, it is at the category level.

For data analysis, this concept is essential. Every table and query has a level of detail:

One row per event
One row per order
One row per customer
One row per day
One row per product category
One row per customer-month

Aggregate functions intentionally change that level of detail. This is useful when the business question is itself summarized:

What was total revenue last month?
How many active users did each country have?
Which product category has the highest sales?

In all these cases, the desired output is a summary table, not a record-level dataset.

The limitation of aggregation

The challenge appears when you want both the original row-level data and a summary statistic.

Imagine you want to look at every individual order while also seeing the customer’s lifetime spending. An ordinary aggregate query cannot do both in one straightforward result.

For example:

SELECT
    customer_id,
    SUM(revenue) AS customer_lifetime_revenue
FROM orders
GROUP BY customer_id;

This correctly calculates customer-level revenue:

customer_id	customer_lifetime_revenue
101	530
102	225
103	45

But all order-level information has disappeared. You can no longer see the individual purchases, dates, or categories associated with each customer.

A common workaround is to write an aggregate query in a common table expression or subquery and join it back to the original table:

WITH customer_totals AS (
    SELECT
        customer_id,
        SUM(revenue) AS customer_lifetime_revenue
    FROM orders
    GROUP BY customer_id
)
SELECT
    o.order_id,
    o.customer_id,
    o.order_date,
    o.category,
    o.revenue,
    ct.customer_lifetime_revenue
FROM orders AS o
JOIN customer_totals AS ct
    ON o.customer_id = ct.customer_id;

This works, but it is more redundant than necessary. It also becomes cumbersome when you need several related metrics: customer totals, category averages, rank within country, previous purchase values, rolling averages, and so on.

This is where window functions become invaluable.

Window functions: calculations without collapsing rows

A window function performs a calculation across a related set of rows while retaining every row in the result.

The general syntax looks like this:

function_name(...) OVER (
    PARTITION BY ...
    ORDER BY ...
    ROWS BETWEEN ...
)

The OVER() clause turns a normal function into a window function.

Here is the earlier customer lifetime revenue example, now written with a window function:

SELECT
    order_id,
    customer_id,
    order_date,
    category,
    revenue,
    SUM(revenue) OVER (
        PARTITION BY customer_id
    ) AS customer_lifetime_revenue
FROM orders;

The result keeps all order-level records:

order_id	customer_id	order_date	category	revenue	customer_lifetime_revenue
1	101	2026-01-01	Electronics	500	530
2	101	2026-01-05	Books	30	530
3	102	2026-01-03	Electronics	200	225
4	103	2026-01-04	Books	45	45
5	102	2026-01-06	Books	25	225

Each order remains visible, but each one now carries an additional feature: the customer’s total spending.

From a data science standpoint, this is powerful because it allows you to add contextual variables without destroying the original dataset grain. These values can support exploratory analysis, customer segmentation, anomaly detection, feature engineering, and model preparation.

The most important distinction: output grain

The easiest way to remember the difference is this:

Function type	What happens to rows?	Typical use
Aggregate function	Rows are collapsed into grouped summaries	Reporting and high-level metrics
Window function	Rows remain; calculations are added alongside them	Comparison, ranking, time-series features, row-level analysis

Consider a question such as: “What percentage of category revenue did each order represent?”

You need the individual order revenue, but you also need total revenue for the order’s category. If you use an aggregate query alone, you lose the individual order values. A window function is ideal:

SELECT
    order_id,
    category,
    revenue,
    SUM(revenue) OVER (
        PARTITION BY category
    ) AS category_revenue,
    100.0 * revenue /
        SUM(revenue) OVER (
            PARTITION BY category
        ) AS percent_of_category_revenue
FROM orders;

For the Electronics category, the two orders contribute to a total of 700. Each row can now be interpreted relative to its category-level context.

This is a recurring pattern in analytical work: retain the observation, then attach the benchmark.

PARTITION BY: defining the comparison group

The PARTITION BY clause divides rows into logical groups for the window calculation.

For example:

AVG(revenue) OVER (
    PARTITION BY category
)

This calculates the average order revenue within each category.

If PARTITION BY is omitted, the window includes all rows returned by the query:

AVG(revenue) OVER () AS overall_average_revenue

That allows direct comparisons between each row and the global average:

SELECT
    order_id,
    customer_id,
    revenue,
    AVG(revenue) OVER () AS overall_average_revenue,
    revenue - AVG(revenue) OVER () AS difference_from_average
FROM orders;

This can be useful for finding unusually large or small transactions.

In practice, data scientists frequently partition by:

Customer ID for customer behavior analysis
User ID for product analytics
Country, region, or market for geographic comparison
Product category for merchandising analysis
Experiment group for A/B test reporting
Account ID for B2B usage behavior
Month or week for period-based comparisons

The choice of partition is analytical, not merely technical. It defines the peer group against which a row is being evaluated.

ORDER BY: making the window sequential

Aggregate functions generally do not care about row order. The sum of sales is the same whether Monday’s data appears before Friday’s.

Window functions often do care about order, especially for time-based analysis.

Suppose you want each customer’s running total spending over time:

SELECT
    customer_id,
    order_date,
    revenue,
    SUM(revenue) OVER (
        PARTITION BY customer_id
        ORDER BY order_date
    ) AS running_customer_revenue
FROM orders;

For customer 101, the results would look like this:

customer_id	order_date	revenue	running_customer_revenue
101	2026-01-01	500	500
101	2026-01-05	30	530

The window is no longer the entire customer history at once. It expands through ordered rows.

This is especially useful for:

Cumulative revenue
Cumulative sign-ups
Cumulative conversion counts
Running account balances
User engagement over time
Inventory movement
Daily active user trends
Progress toward monthly targets

When multiple events can occur at the same timestamp or date, add a tie-breaker to ensure deterministic ordering:

ORDER BY order_date, order_id

Without a stable ordering rule, running totals and row sequence functions can produce ambiguous results.

Window frames: controlling which rows are included

A window frame refines the set of rows considered around the current row.

For example, a rolling three-day average might be written as:

AVG(daily_revenue) OVER (
    ORDER BY order_date
    ROWS BETWEEN 2 PRECEDING AND CURRENT ROW
) AS rolling_3_day_average

This calculates an average using the current row and the two prior rows.

Frames are especially valuable in time-series analysis. Common examples include:

ROWS BETWEEN 6 PRECEDING AND CURRENT ROW

for a seven-row rolling average, or:

ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW

for a running total from the start of the partition.

However, one should be careful with the difference between row-based and time-based frames. A frame of “six preceding rows” is not always equivalent to “the previous six calendar days.” If dates are missing, there may be fewer than seven days represented. Some SQL systems support range-based or interval-based frames, while others require generating a complete date spine first.

This distinction matters a great deal in production metrics. A “seven-day moving average” should normally mean seven calendar days, not simply seven available observations.

Ranking functions: a major window-function use case

Ranking is one of the clearest examples of a task that window functions solve elegantly.

Suppose you want to rank customers by their total spending:

SELECT
    customer_id,
    SUM(revenue) AS customer_revenue,
    RANK() OVER (
        ORDER BY SUM(revenue) DESC
    ) AS revenue_rank
FROM orders
GROUP BY customer_id;

Here, aggregate and window functions work together. The query first produces one row per customer with SUM(revenue). The window function then ranks those customer-level rows.

You might use:

RANK()
DENSE_RANK()
ROW_NUMBER()
NTILE()

The differences matter:

ROW_NUMBER() gives every row a unique sequential number.
RANK() gives tied rows the same rank and leaves gaps afterward.
DENSE_RANK() gives tied rows the same rank but does not leave gaps.
NTILE(n) divides ordered rows into approximately equal-sized buckets.

For example, NTILE(10) is frequently used to assign customers to deciles based on lifetime value, engagement, risk score, or predicted propensity.

A practical customer segmentation query could look like this:

WITH customer_metrics AS (
    SELECT
        customer_id,
        SUM(revenue) AS lifetime_value,
        COUNT(*) AS order_count
    FROM orders
    GROUP BY customer_id
)
SELECT
    customer_id,
    lifetime_value,
    order_count,
    NTILE(4) OVER (
        ORDER BY lifetime_value DESC
    ) AS value_quartile
FROM customer_metrics;

This produces a straightforward first-pass segmentation of customers into value quartiles.

LAG and LEAD: comparing rows across time

Window functions also enable comparisons between one row and nearby rows. LAG() retrieves a previous row’s value; LEAD() retrieves a later row’s value.

For a daily revenue table:

SELECT
    order_date,
    daily_revenue,
    LAG(daily_revenue) OVER (
        ORDER BY order_date
    ) AS previous_day_revenue,
    daily_revenue -
        LAG(daily_revenue) OVER (
            ORDER BY order_date
        ) AS day_over_day_change
FROM daily_sales;

This lets you calculate changes over time without a self-join.

You can also calculate percentage change:

SELECT
    order_date,
    daily_revenue,
    100.0 * (
        daily_revenue -
        LAG(daily_revenue) OVER (ORDER BY order_date)
    ) /
    NULLIF(
        LAG(daily_revenue) OVER (ORDER BY order_date),
        0
    ) AS day_over_day_growth_pct
FROM daily_sales;

NULLIF prevents a division-by-zero error.

For customer behavior, LAG() can help answer questions such as:

How many days passed between purchases?
Was this customer’s order larger than their previous order?
Has user activity increased or declined?
Did a customer upgrade or downgrade their plan?
Did a product’s weekly demand change materially?

Example:

SELECT
    customer_id,
    order_date,
    revenue,
    LAG(order_date) OVER (
        PARTITION BY customer_id
        ORDER BY order_date
    ) AS previous_order_date
FROM orders;

You can then calculate purchase intervals and use them in recency-frequency-monetary analysis, churn modeling, or customer lifecycle reporting.

Aggregates and windows often work best together

The question should not be “Which one should I use?” in an absolute sense. Mature analytical queries often use both.

For instance, imagine you want to calculate each category’s monthly sales, then determine each category’s share of total monthly sales:

WITH monthly_category_sales AS (
    SELECT
        DATE_TRUNC('month', order_date) AS month,
        category,
        SUM(revenue) AS category_revenue
    FROM orders
    GROUP BY
        DATE_TRUNC('month', order_date),
        category
)
SELECT
    month,
    category,
    category_revenue,
    SUM(category_revenue) OVER (
        PARTITION BY month
    ) AS total_monthly_revenue,
    100.0 * category_revenue /
        SUM(category_revenue) OVER (
            PARTITION BY month
        ) AS monthly_revenue_share
FROM monthly_category_sales;

The aggregate stage establishes the desired analytical grain: one row per category per month. The window stage adds context: each category’s contribution to the month’s total.

This two-stage pattern is extremely common in analytics engineering and data science:

Aggregate raw event data to the correct grain.
Use window functions to compare groups, rank them, calculate rolling statistics, or create relative measures.
Feed the resulting table into a dashboard, statistical analysis, or machine-learning workflow.

Common mistakes to avoid

One common mistake is mixing grouped and ungrouped columns incorrectly. This query is invalid in most SQL engines:

SELECT
    customer_id,
    order_date,
    SUM(revenue)
FROM orders
GROUP BY customer_id;

The problem is that order_date is neither grouped nor aggregated. SQL does not know which order date to choose for each customer.

Another common mistake is using an aggregate when a window is needed. If you need to see every transaction plus a customer average, use:

AVG(revenue) OVER (
    PARTITION BY customer_id
)

rather than grouping by customer and losing the transaction records.

A third mistake is filtering window-function results in WHERE. Window functions are calculated after the WHERE clause in the logical query-processing order. For example, this usually will not work:

SELECT
    customer_id,
    revenue,
    RANK() OVER (ORDER BY revenue DESC) AS revenue_rank
FROM orders
WHERE revenue_rank <= 10;

Instead, place the window calculation in a subquery or common table expression:

WITH ranked_orders AS (
    SELECT
        customer_id,
        revenue,
        RANK() OVER (
            ORDER BY revenue DESC
        ) AS revenue_rank
    FROM orders
)
SELECT *
FROM ranked_orders
WHERE revenue_rank <= 10;

Some databases provide a QUALIFY clause that makes this pattern more concise, but it is not universally available.

Finally, be cautious about unintended partitions. Leaving out PARTITION BY means the calculation runs across all rows. Including the wrong partition can create misleading results that appear plausible. Always ask: “Which rows should count as this row’s peers?”

Conclusion(Choosing the right tool)

Use aggregate functions when you want to reduce a dataset into a summary at a new grain. They are excellent for dashboard metrics, grouped reports, and compact analytical tables.

Use window functions when you need row-level detail plus group-level or time-based context. They are particularly valuable for ranks, percent-of-total calculations, running totals, rolling averages, prior-period comparisons, sessionization, and feature engineering.

Aggregate functions tell you what happened at a summary level. Window functions help explain how each observation fits into the broader pattern. Together, they turn SQL from a reporting language into a flexible system for analytical reasoning.

Scikit-Learn, Pipeline Fundamentals: A Titanic Survival Prediction Guide

Mark Glemba — Sat, 12 Sep 2026 20:36:52 +0000

Machine learning workflows often suffer from code duplication, data leakage, and hyperparameter tuning complexities. Scikit-Learn's Pipeline and ColumnTransformer modules simplify this by combining feature preprocessing and model estimation into an integrated, reproducible object.

This guide demonstrates a complete predictive modeling workflow using the classic Titanic: Machine Learning from Disaster dataset, covering exploratory data analysis, pipeline-based feature engineering, model selection, and cross-validation.

1. Environment Setup & Data Loading

Begin by importing the necessary libraries for data manipulation, visualization, preprocessing, modeling, and evaluation.

import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt

# Model selection & evaluation
from sklearn.model_selection import train_test_split, StratifiedKFold, cross_val_score
from sklearn.metrics import roc_auc_score, classification_report

# Preprocessing & Pipelines
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.compose import ColumnTransformer

# Classifiers
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.svm import SVC
from xgboost import XGBClassifier

# Load dataset
titanic_df = pd.read_csv("data/train.csv")

2. Exploratory Data Analysis & Data Sanitation

A quick inspection of the dataset structure reveals the feature types, missing values, and potential columns to drop.

def basic_checks(df):
    print("Shape:", df.shape)
    print("Missing Values:\n", df.isnull().sum())
    print("Data Types:\n", df.dtypes)

basic_checks(titanic_df)

Missing Value Analysis

Age: Missing 19.87% of data (177 missing values).
Embarked: Missing 0.22% of data (2 missing values).
Cabin: Missing 77.10% of data (687 missing values).

Due to the extreme proportion of missing data, the Cabin column is dropped.

titanic_df = titanic_df.drop(columns=['Cabin'])

3. Train-Test Split & Data Partitioning

Identifiers such as PassengerId, Name, and Ticket carry no predictive power for survival and are excluded from the feature set $X$. We split the data using a 80/20 train-test split stratified by the target column Survived to maintain class proportions.

# Separate features and target
X = titanic_df.drop(columns=['PassengerId', 'Name', 'Ticket', 'Survived'])
y = titanic_df['Survived']

# Categorize feature types
numeric_columns = ['Age', 'SibSp', 'Parch', 'Fare']
categorical_columns = ['Pclass', 'Sex', 'Embarked']

# Stratified split
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

4. Building Scikit-Learn Preprocessing Pipelines

To avoid data leakage (such as calculating the mean or standard deviation on the entire dataset before splitting), transformations must be learned only on the training set. Using Pipeline and ColumnTransformer ensures these transformations are safely applied during cross-validation.

# 1. Numerical Pipeline: Impute missing values with median, then scale
numerical_pipeline = Pipeline(steps=[
    ('imputer', SimpleImputer(strategy='median')),
    ('scaler', StandardScaler())
])

# 2. Categorical Pipeline: Impute missing values with mode, then One-Hot Encode
categorical_pipeline = Pipeline(steps=[
    ('imputer', SimpleImputer(strategy='most_frequent')),
    ('onehot', OneHotEncoder(handle_unknown='ignore'))
])

# 3. Combine Preprocessing Steps
preprocessor = ColumnTransformer(transformers=[
    ('num', numerical_pipeline, numeric_columns),
    ('cat', categorical_pipeline, categorical_columns)
])

5. Model Candidate Comparison & Stratified Cross-Validation

Evaluate multiple classification algorithms using 5-Fold Stratified Cross-Validation evaluated on the $F_1$-score metric.

# Define candidate models
models = {
    'Logistic Regression': LogisticRegression(max_iter=1000),
    'K-Nearest Neighbors': KNeighborsClassifier(n_neighbors=7, metric='euclidean'),
    'Random Forest': RandomForestClassifier(n_estimators=300, max_depth=15, random_state=42),
    'Support Vector Machine': SVC(kernel='rbf', probability=True, random_state=42),
    'Gradient Boosting': GradientBoostingClassifier(n_estimators=300, max_depth=15, random_state=42)
}

# Cross-Validation setup
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
results = {}

for name, model in models.items():
    # Chain preprocessing and model estimation into a single pipeline
    full_pipeline = Pipeline(steps=[
        ('preprocessor', preprocessor),
        ('model', model)
    ])

    cv_scores = cross_val_score(full_pipeline, X_train, y_train, cv=cv, scoring='f1', n_jobs=-1)
    results[name] = {
        'Mean F1 Score': cv_scores.mean(),
        'Std Dev': cv_scores.std()
    }

# Display results in a table
results_df = pd.DataFrame(results).T.sort_values(by='Mean F1 Score', ascending=False)

Model Performance Metrics

Model Classifier	Mean $F_1$ Score	Standard Deviation
Support Vector Machine (SVM)	0.7447	0.0311
Random Forest	0.7374	0.0200
Logistic Regression	0.7301	0.0217
K-Nearest Neighbors	0.7194	0.0229
Gradient Boosting	0.7002	0.0233

6. Key Takeaways & Best Practices

Preventing Data Leakage: Wrapping feature engineering steps inside a Pipeline guarantees that statistics (e.g., mean/median for imputation, standard deviations for scaling) are calculated strictly on training folds during cross-validation.
Handling Mixed Data Types: ColumnTransformer cleanly routes numerical features to continuous scaling modules and categorical variables to encoding blocks.
Model Verdict: The Support Vector Machine (SVM) achieved the highest mean $F_1$-score ($0.7447$), with Random Forest displaying the highest stability across folds with the lowest variance ($\sigma = 0.0200$).

Machine Learning and Its Real-World Impacts

Mark Glemba — Sat, 12 Sep 2026 17:47:11 +0000

Introduction:

Machine learning is one of the most important areas of modern technology. It is behind many tools people use every day, including search engines, social media recommendations, voice assistants, online shopping suggestions, fraud detection systems, and navigation apps. Although the phrase may sound complex at first, the central idea is quite simple: machine learning enables computers to learn patterns from data and use those patterns to make predictions, decisions, or recommendations.

For a new student, machine learning can be understood as teaching a computer through examples rather than giving it every rule manually. Instead of writing thousands of instructions explaining how to identify a cat in a photo, developers can show a computer many labeled images of cats and non-cats. Over time, the system learns useful visual patterns, such as ears, fur, eyes, and body shapes. It can then make a reasonable guess when shown a new image.

This article introduces the main ideas of machine learning, explains common types of learning, discusses the usual development process, and explores how machine learning is applied in the real world.

1. What Exactly is Machine Learning?

At its core, Machine Learning is a branch of Artificial Intelligence (AI) focused on building applications that learn from data and improve their accuracy over time without being explicitly programmed.

Machine learning is one way to achieve AI. It focuses on creating algorithms that improve their performance by learning from data.

Machine learning does not mean that a computer thinks exactly like a person. It means that the computer uses mathematical methods to learn from examples and make useful outputs.

Why Data Matters

Data is the foundation of machine learning. A model can only learn from the information it receives, so the quality of the data strongly affects the quality of the result.

Data can come in many forms:

Numbers, such as house prices or temperatures
Text, such as emails, reviews, and news articles
Images, such as medical scans or photographs
Audio, such as voice recordings and music
Video, such as security-camera footage
Records of behavior, such as purchases, clicks, or travel routes

However, more data is not always better if the data is poor, incomplete, outdated, or unfair. If a model is trained using biased data, it may produce biased results. This is why collecting, cleaning, checking, and protecting data are important parts of machine-learning work.

2. The Main Types of Machine Learning

1. Supervised Learning

The algorithm is provided with a training dataset that contains both the input data (features) and the correct output labels (targets). The goal of the algorithm is to learn a function that maps inputs to outputs so accurately that it can predict the label for brand-new, unseen data.

Supervised learning generally splits into two sub-categories:

1. Classification: The output target is a discrete category or class.

Examples: Is this image a cat or a dog?

2. Regression: The output target is a continuous numeric value.

Examples: Predicting house prices based on size and location, or estimating temperature.

2. Unsupervised Learning

Here, the algorithm is given input data without any target labels and is tasked with finding inherent structures, patterns, or groupings on its own.

Key techniques in unsupervised learning include:

1. Clustering: Grouping similar data points together based on shared characteristics.

2. Dimensionality Reduction: Simplifying complex datasets with hundreds of features into fewer features while preserving essential information.

3. Anomaly Detection: Identifying data points that deviate significantly from the norm.

Example: Spotting unusual system logs or structural flaws in industrial equipment.

3. Reinforcement Learning

Reinforcement learning is inspired by learning through experience. In this approach, an agent takes actions in an environment and receives rewards or penalties based on the results.

The agent’s goal is to learn which actions lead to the best long-term reward.

A simple example is training a computer program to play a game. The program tries different moves. If a move helps it win, it receives a positive reward. If it loses, it receives a negative reward. After many attempts, it learns strategies that improve its chances of winning.

Reinforcement learning is used in areas such as:

Game-playing systems
Robotics
Traffic-signal optimization
Resource management
Some autonomous systems

It can be powerful, but it is often more difficult to train than supervised learning because the system may need many attempts before discovering successful behavior.

3. The Lifecycle of a Machine Learning Project

1. Problem Formulation: Defining what we want to predict or discover.

2. Data Collection: Gathering relevant, high-quality data.

3. Data Preprocessing and Cleaning: Cleaning messy data, handling missing values, normalizing numerical scales, and converting raw inputs into mathematical vectors that algorithms can work with.

4. Model Training: Selecting an appropriate algorithm (e.g., Linear Regression, Decision Trees, Neural Networks) and letting it iteratively compute the mathematical weights that best represent the patterns in the training set.

5. Model Evaluation: Testing the trained model against a separate "test dataset" that the model has never seen before. This step checks whether the model actually learned general concepts or merely memorized the training data ( known as overfitting).

6. Deployment and Continuous Monitoring: Integrating the validated model into a production system and monitoring its performance over time to ensure its predictions remain accurate as real-world conditions evolve.

4. Real-World Applications: Machine Learning in Everyday Life

A. Healthcare and Biomedical Science

Machine learning is driving some of its most meaningful breakthroughs in medicine. By acting as an advanced diagnostic assistant, ML empowers medical professionals to detect diseases earlier and tailor treatments to individual patients.

Medical Imaging Analysis: Convolutional Neural Networks (CNNs)—a class of deep learning models designed for processing visual data—can analyze X-rays, MRIs, and CT scans to detect early-stage tumors, bone fractures, or diabetic retinopathy. In many studies, these models achieve diagnostic accuracy on par with seasoned radiologists.

Drug Discovery: ML algorithms can simulate molecular structures, predict how potential drug compounds will interact with biological targets, and narrow down candidate molecules in days rather than years.

Predictive Diagnostics: By evaluating electronic health records, heart rate metrics from wearables, and blood markers, predictive models can flag patients at high risk of developing conditions like sepsis or heart failure before acute symptoms manifest.

B. Finance and Banking

Fraud Detection: Every time you swipe your credit card, a machine learning model evaluates the transaction in milliseconds. It compares the location, dollar amount, merchant type, and purchase velocity against your historical habits.

Algorithmic Trading: Financial institutions use complex ML models to analyze market trends, sentiment from financial news releases, and economic indicators to execute trades at optimized prices and microsecond speeds.

Credit Scoring and Risk Assessment: Rather than relying solely on rigid rule-based credit scores, modern fintech companies use ML models to evaluate alternative data sources, allowing them to extend credit safely to underbanked individuals who lack traditional financial histories.

C. E-Commerce and Personal Entertainment

Streaming Services (Netflix, Spotify): Recommendation systems utilize techniques like Collaborative Filtering and Content-Based Filtering. By comparing your watching or listening history with millions of other users who share similar tastes, the system can predict with surprising accuracy which movie or song you are likely to enjoy next.

Dynamic Pricing: Companies like Uber, Airbnb, and airlines use ML algorithms to adjust prices dynamically based on real-time factors including current demand, local weather conditions, traffic flow, time of day, and historical booking patterns.

D. Autonomous Systems and Transportation

Autonomous Vehicles: Companies building self-driving cars rely on computer vision models to identify lane markings, interpret traffic signals, track pedestrians, and predict the movements of surrounding vehicles in real time.

Smart Logistics: Global shipping companies use machine learning to optimize delivery routes, forecast inventory demands at regional warehouses, and minimize fuel consumption across global supply chains.

E. Natural Language Processing (NLP) and Generative AI

Large Language Models (LLMs) and Assistants: Virtual assistants, automated translation services, and modern generative AI tools utilize massive neural network architectures (specifically Transformers) to parse human context, summarize complex documents, write functional code, and engage in natural conversations.

Sentiment Analysis: Brands use NLP algorithms to analyze millions of social media posts, product reviews, and customer support tickets to gauge public sentiment toward products or services in real time.

5. Key Challenges and Ethical Considerations

┌─────────────────────────────────────────────────────────────┐
│                Ethical & Practical Challenges               │
├──────────────────────────────┬──────────────────────────────┤
│ ⚖️ Algorithmic Bias          │ Models inherit and amplify   │
│                              │ human biases in data.        │
├──────────────────────────────┼──────────────────────────────┤
│ 🔍 Lack of Transparency      │ "Black Box" models make      │
│                              │ decisions hard to explain.   │
├──────────────────────────────┼──────────────────────────────┤
│ 🔒 Privacy Concerns          │ Training requires vast amounts│
│                              │ of sensitive personal data.  │
├──────────────────────────────┼──────────────────────────────┤
│ ⚡ Overfitting & Reliability  │ Models can fail unexpectedly  │
│                              │ when real-world data shifts. │
└──────────────────────────────┴──────────────────────────────┘

1. Data Bias and Fairness

If training data does not represent all groups fairly, the model may perform better for some people than others. For example, a facial-recognition system trained on limited types of faces may be less accurate for underrepresented groups.

2. The "Black Box" Problem (Explainability)

While simple algorithms like decision trees are easy to audit, complex deep neural networks consist of millions—or even billions—of mathematical parameters. This creates a "black box" scenario: the model may yield highly accurate outputs, but engineers cannot easily trace why or how it reached a specific decision. In high-stakes environments like criminal justice sentencing or medical diagnoses, this lack of transparency presents major safety and legal hurdles.

3. Data Privacy

Many machine-learning systems depend on personal data, including location, browsing history, medical records, and purchasing behavior. Organizations must collect and use this data responsibly.

6. Conclusion

Machine learning is a way of building computer systems that learn from data. Instead of writing a separate rule for every possible situation, developers train models using examples. These models can then make predictions, classify information, recognize patterns, and support decisions.

It is a combination of data, mathematics, algorithms, testing, and human judgment. A good machine-learning system depends not only on technical skill, but also on responsible design, high-quality data, fairness, privacy, and careful evaluation.

It is already used in healthcare, banking, transportation, education, agriculture, entertainment, cybersecurity, and many other fields. Its influence will likely continue to grow as more data, computing power, and research become available.

Learning the basics of machine learning opens the door to understanding many technologies that shape daily life. It also gives students the opportunity to think critically about how these technologies should be designed and used in the future.

Understanding Unsupervised Machine Learning

Mark Glemba — Wed, 09 Sep 2026 09:57:48 +0000

Introduction

Machine learning is a branch of artificial intelligence that allows computers to learn patterns from data. Instead of writing every rule manually, we provide a computer with examples and let it discover useful relationships. Machine learning is used in many parts of daily life, including recommendation systems, fraud detection, voice assistants, online shopping, medical research, social media, and banking.

There are several major types of machine learning. One of the most important is called unsupervised machine learning. It is especially useful when we have a large amount of data but do not already know the correct answers or categories for that data.

What is Unsupervised Machine Learning?

In traditional Supervised Machine Learning, models learn with a teacher or a supervisor. We feed the computer input data paired with correct answers (called labels). For example, we show thousands of pictures labeled "Cat" or "Dog", and the model learns the relationship between the pixels and the labels.In Unsupervised Machine Learning, there is no teacher, no supervisor, and—most importantly—no ground truth labels.We feed the algorithm raw data without target outputs. The algorithm's sole goal is to inspect the data, uncover hidden mathematical structures, detect repeating patterns, and group similar data points together.

Supervised vs. Unsupervised Learning

Why Do We Need Unsupervised Learning?

Data Labeling is Expensive and Slow: In the real world, human annotation is time-consuming and costly. Unsupervised learning allows us to make sense of vast datasets before or without labeling them.

Discovering Unknown Patterns: Humans are inherently limited by their own biases and domain knowledge. Unsupervised learning can discover connections in data that humans never thought to look for.

Data Compression and Feature Extraction: Unsupervised techniques help simplify complex datasets, making them easier to visualize, store, and feed into downstream predictive models.

Core Branch 1: Clustering

Clustering is the task of partitioning a dataset into distinct groups (or clusters) such that data points in the same group are more similar to each other than to those in other groups.

Three most important clustering algorithms.

A. K-Means Clustering

K-Means is the workhorse of clustering algorithms due to its speed and simplicity.
How K-Means Works Step-by-Step:

Choose $K$ : Decide how many clusters ($K$) you want to discover.
Initialize Centroids: Randomly place $K$ points in the feature space. These points act as the initial center points (centroids) of your clusters.
Assign Points: Calculate the distance (usually Euclidean distance) between every data point and all $K$ centroids. Assign each data point to its nearest centroid.
Update Centroids: Recompute the position of each centroid by taking the average (mean) of all data points assigned to that cluster.
Repeat: Repeat steps 3 and 4 until the centroids stop moving (convergence) or a maximum number of iterations is reached.

The Math Behind Distance Calculation

To measure how close two data points $A = (a_1, a_2, \dots, a_n)$ and $B = (b_1, b_2, \dots, b_n)$ are, K-Means uses the standard Euclidean distance formula:

$d(A, B) = \sqrt{\sum_{i=1}^{n} (a_i - b_i)^2}$$

Finding the Right $K$:

The Elbow Method

We plot the Within-Cluster Sum of Squares (WCSS) against various values of $K$ . As $K$ increases, WCSS decreases because clusters become smaller and tighter. The optimal $K$ is located at the "elbow" point—where the rate of decrease dramatically slows down.

$$\text{WCSS} = \sum_{k=1}^{K} \sum_{x \in C_k} \vert{}\vert{}x - \mu_k\vert{}\vert{}^2$$

(Where $C_k$ is the set of points in cluster $k$ , and $\mu_k$
is the mean/centroid of cluster $k$ .)

B. Hierarchical Clustering

Unlike K-Means, which requires you to pre-define $K$ , Hierarchical Clustering creates a nested hierarchy of clusters presented as a tree-like diagram called a Dendrogram.
There are two primary approaches:

Agglomerative (Bottom-Up): Starts with every single data point as its own individual cluster. In each step, the two closest clusters are merged until only one grand cluster remains.

Divisive (Top-Down): Starts with all data points inside a single master cluster and recursively splits them into smaller sub-clusters.

What is a Dendrogram?
A dendrogram illustrates how clusters are progressively combined or split. By drawing a horizontal line across the dendrogram at a specific height threshold, you can "cut" the tree and choose the number of clusters that best fits your problem.

C. DBSCAN (Density-Based Spatial Clustering of Applications with Noise)

Both K-Means and Hierarchical Clustering struggle when clusters have arbitrary, non-spherical shapes (like concentric circles or crescent shapes) or when data contains significant background noise.DBSCAN clusters points based on local data density rather than centroid distances.
Key Concepts of DBSCAN:

$\epsilon$ (Epsilon): The maximum radius around a point to search for neighbors.

MinPts: The minimum number of points required within the $\epsilon$-neighborhood to consider that area a "dense region".

Point Classification in DBSCAN:

1. Core Point: Has at least MinPts within its $\epsilon$ -radius.

2. Border Point: Lies within the $\epsilon$ -radius of a Core Point but has fewer than MinPts in its own radius.

3. Noise Point (Outlier): Any point that is neither a Core Point nor a Border Point.

Advantage of DBSCAN: It automatically identifies outliers as noise and can discover complex cluster topologies without needing $K$ specified upfront

Core Branch 2: Dimensionality Reduction

Modern datasets often suffer from the Curiosity of High Dimensionality.
Dimensionality Reduction reduces the number of random variables under consideration by obtaining a set of principal features, compressing the data while retaining as much critical information as possible.

A. Principal Component Analysis (PCA)

PCA is a linear dimensionality reduction technique. It reorients the dataset into a new coordinate system such that maximum variance is captured in the fewest possible axes.

How PCA Works (Intuition):

Center the Data: Subtract the mean from each feature vector so the data centers around the origin $(0,0)$ .
Compute Covariance Matrix: Calculate how each variable correlates with every other variable.
Find Eigenvectors and Eigenvalues:
- Eigenvectors represent the directions of the new coordinate axes (called Principal Components).
- Eigenvalues represent the amount of variance captured along each Principal Component.
Select Top Components: Sort the components by their eigenvalues and select the top $k$ components that explain most of the total variance (e.g., $95\%$ of original variance).

B. t-SNE (t-Distributed Stochastic Neighbor Embedding)

While PCA searches for global linear relationships, t-SNE is a non-linear technique designed specifically for visualizing high-dimensional data in 2D or 3D space.

Intuition Behind t-SNE:

Computes similarity probabilities between pairs of data points in high-dimensional space using a Gaussian distribution.
Constructs a low-dimensional map (usually 2D) with points positioned such that neighboring points in the high-dimensional space remain close neighbors in the low-dimensional map.
Uses a Student-t distribution in the low-dimensional space to solve the "crowding problem," allowing clusters to spread out nicely for clear visualization.

Core Branch 3: Anomaly Detection

Anomaly Detection (or Outlier Detection) is the process of identifying rare events, items, or observations that raise suspicion by differing significantly from the vast majority of the data.
Because anomalies are rare by definition, labeled anomaly datasets are extremely scarce. Unsupervised algorithms excel here by learning what "normal" data looks like and flagging anything that deviates from the norm.

Key Algorithms for Anomaly Detection:

Isolation Forest: An tree-based algorithm that isolates anomalies instead of profiling normal points. Because anomalies are rare and different, they require fewer splits in a decision tree to be isolated compared to normal points.

One-Class SVM: A variation of Support Vector Machines that fits a tight boundary around normal data points. Anything falling outside this decision boundary is flagged as an anomaly.

 Isolation Forest Split Tree Depth:

       Normal Points:  Root -> Split 1 -> Split 2 -> Split 3 -> Split 4 (Deep)
       Anomaly Point:  Root -> Split 1 (Isolated early!)

Core Branch 4: Autoencoders (Deep Unsupervised Learning)

When we combine unsupervised concepts with Deep Neural Networks, we get Autoencoders.
An Autoencoder is a neural network designed to copy its input to its output through a constrained bottleneck layer.

 [ Input Data X ] ---> ( Encoder ) ---> [ Bottleneck / Latent Space ] ---> ( Decoder ) ---> [ Reconstruction X' ]

The Architecture:

1. Encoder: A series of layers that compresses the input data $X$ into a lower-dimensional representation (the Latent Space or Code).

2. Bottleneck: The narrowest layer of the network that restricts the flow of information, forcing the network to learn only the most essential features.

3. Decoder: A series of layers that attempts to reconstruct the original input from the latent code.

Loss Function:

The loss function measures how closely the reconstructed output $\hat{X}$ matches the original input $X$ (often measured via Mean Squared Error):

$$\text{Reconstruction Loss} = \vert{}\vert{}X - \hat{X}\vert{}\vert{}^2$$

Applications of Autoencoders:

Image Denoising: Train the network using noisy images as input and clean images as output; the network learns to strip out background noise.

Dimensionality Reduction: Non-linear compression that often outperforms linear PCA.

Anomaly Reconstruction: If an autoencoder is trained only on normal data, it will fail to reconstruct abnormal inputs, yielding a high reconstruction error that alerts operators.

Real-World Applications

Unsupervised Machine Learning powers dozens of critical services across modern industries:

Customer Segmentation in Marketing: E-commerce platforms group customers by browsing behavior, purchase history, and spending habits to build personalized marketing campaigns.

Fraud Detection in Banking: Credit card processors monitor transactional patterns and flag unusual purchases occurring in unexpected geographical locations.

Gene Expression Analysis in Genomics: Scientists cluster human genetic patterns to uncover previously unknown biological sub-types of complex diseases.

Recommendation Engine Pre-processing: Streaming platforms use dimensionality reduction to handle massive user-item interaction matrices before generating recommendations.

Document Topic Modeling: Natural Language Processing (NLP) models organize millions of unstructured news articles into distinct topic groups automatically.

Summary & Key Takeaways

Unsupervised machine learning provides the toolkit to transform raw, unlabeled chaos into structured knowledge.

Clustering (K-Means, Hierarchical, DBSCAN) groups similar data points together based on distance or density metrics.
Dimensionality Reduction (PCA, t-SNE) compresses high-dimensional feature spaces down to manageable components while preserving critical structural information.
Anomaly Detection isolates rare events by measuring deviations from learned normal distributions.
Autoencoders leverage deep learning bottlenecks to extract rich latent representations from complex, unstructured data like images and video.

Conclusion

Unsupervised machine learning is a powerful way of learning from data that does not have predefined answers or labels. Instead of being told exactly what to look for, the model searches for hidden structures, similarities, relationships, and unusual observations.

Its main uses include clustering similar items, finding products that occur together, simplifying complex data, and detecting unusual behavior. It is useful in business, health, cybersecurity, research, education, online platforms, and many other fields.

The most important idea to remember is this:

Unsupervised machine learning helps computers discover patterns in data when no one has already provided the correct categories or answers.

Understanding Neural Networks and The Core Idea Behind Them

Mark Glemba — Mon, 07 Sep 2026 22:18:22 +0000

Introduction

Neural networks work by building a system inspired by the human brain that can learn how to solve the problem by looking at examples.If you are new to machine learning, neural networks might seem like an impenetrable black box full of dense mathematics. However, the core idea behind them is surprisingly simple, intuitive, and beautiful.

In this guide, we will unpack neural networks from first principles, stripping away the hype to see how they truly work.

1. The Core Idea: Function Approximation and Pattern Recognition

Mathematically, a neural network is just a giant, adjustable mathematical function.

A neural network is a tool for Function Approximation. It begins as a blank slate—a mathematical machine full of adjustable "knobs" and "dials." Initially, these dials are set randomly, meaning the network gives terrible guesses. But by feeding it thousands of examples (inputs paired with correct outputs), we gradually tweak those knobs until the network’s output matches the desired result.In essence: A neural network learns to approximate the invisible mathematical rule that connects inputs to outputs.

2. The Biological Inspiration: From Neurons to Code

The name "Neural Network" comes from biological neuroscience. Our brains contain approximately 86 billion interconnected biological cells called neurons.Biological Neuron Flow:
[ Dendrites ] ---> ( Cell Body ) ---> [ Axon ] ---> Synapses (Processing) (Output) (Connections)

Dendrites receive electrical signals from other neurons.The Cell Body aggregates these incoming signals. If the combined electrical charge crosses a certain threshold, the neuron "fires". The Axon carries the fired signal down toward the end of the cell.Synapses transmit the signal across junctions to neighboring neurons.In artificial intelligence, we do not build actual biological structures. Instead, we create a mathematical simplified abstraction called an Artificial Neuron (or a Perceptron).

3. The Anatomy of an Artificial Neuron

To understand how a complex deep network functions, we must first understand its smallest building block: a single artificial neuron.

An artificial neuron performs four fundamental operations:

Receives Inputs, Multiplies each input by a Weight, Adds a constant called Bias, Passes the result through an Activation Function, to produce the final Output.

Let’s break down each of these components in plain language:

A. Inputs ( $x$ ) Inputs represent the features of the data you want to analyze.

B. Weights ( $w$ ) Weights are the "knobs and dials" of the network. They determine how much influence a given input has on the neuron’s final decision: A large positive weight means the input strongly drives the decision upward. A weight close to zero means the input is mostly ignored. A negative weight means as the input increases, the prediction goes down.

Mathematically, we multiply each input by its weight

C. Bias ( $b$ ) The Bias is an extra adjustable parameter added to the weighted sum. It acts as a base threshold, shifting the calculation up or down regardless of the inputs.

It gives the neuron the freedom to trigger even when inputs are low, or remain quiet even when inputs are high.Adding the bias gives us the linear equation:

$z = \left( \sum_{i=1}^{n} w_i x_i \right) + b$

D. Activation Function ( $f(z)$ )

If a neural network consisted only of weights and biases, it would just be doing linear algebra—drawing straight lines through data. But real-world data is non-linear, full of curves, twists, and subtle thresholds.

The Activation Function introduces non-linearity into the network. It decides whether the neuron should activate and by how much.

Here are three common activation functions:

1. The Step FunctionHistorically, early neurons used a binary switch: if $z > 0$ , output $1$ ; otherwise output $0$ . This mirrors a simple on/off switch, but it lacks nuance because small changes in inputs can cause sudden, wild jumps in output.

2. The Sigmoid Function The Sigmoid function squashes any real number input into a smooth range between 0 and 1:

$\sigma(z) = \frac{1}{1 + e^{-z}}$

. It is often used when predicting probabilities

3. ReLU (Rectified Linear Unit) Despite its fancy name, ReLU is remarkably simple:

$f(z) = \max(0, z)$

If $z$ is negative, output 0. If $z$ is positive, return $z$ as it is.

Because it is computationally efficient and works incredibly well, ReLU is the most popular activation function used inside modern neural networks today.

4. Tanh(Hyperbolic Tangent)

$\tanh(x) = \frac{e^z - e^{-z}}{e^z + e^{-z}}$

Output range for the hyperbolic tangent is between (-1, 1)
Still suffers from vanishing gradients at the extremes.

5. Leaky ReLU

$f(x) = \max(\alpha x, x)$

Output range for leaky relu is -∞ to ∞
A small modificationof ReLU that allows a small, non-zero gradient when the input is negative
Helps mitigate the dying ReLU problem

6. SoftMax

$\sigma (\mathbf{z})_{i}=\frac{e^{z_{i}}}{\sum _{j=1}^{K}e^{z_{j}}}$

Output range is between 0 and 1 and they sum up to 1.

Converts a vector of raw scores(logits) into a probability distribution across multiple classes.
Used almost exclusively in the output layer for multiclass classification problems.

4. Organizing Neurons into Layers

A single neuron can make basic linear decisions (like drawing a straight line to divide two groups of points). However, real intelligence requires combining hundreds, thousands, or millions of these neurons into a network.A typical neural network is structured in vertical slices called Layers:

1. The Input Layer: This is where raw data enters the network. It doesn't perform calculations; it simply passes feature values forward.

2. The Hidden Layer: Located between input and output, these layers are called "hidden" simply because their intermediate processing values are not directly observed in the dataset.
Early hidden layers learn simple, fundamental patterns (e.g., detecting horizontal, vertical, or diagonal edges in an image).
Middle hidden layers combine those basic edges to detect shapes (e.g., circles, corners, texture patterns).
Deeper hidden layers combine shapes into high-level concepts (e.g., eyes, noses, wheels, or ears).
When a network has multiple hidden layers stacked on top of each other, we call it a Deep Neural Network — hence the term Deep Learning.

3. The Output Layer: The final layer produces the network's answer.For a regression task (predicting a house price), it might contain 1 single neuron outputting a raw number like $350,000.
For a classification task (identifying an animal as a Cat, Dog, or Bird), it might contain 3 neurons, each outputting a probability score for its category.

5. How Neural Networks Learn:The Training Loop

When you initialize a neural network, its weights ( $w$ ) and biases ( $b$ ) are filled with random numbers. Learning happens through a continuous loop consisting of four major steps:

Step 1: Forward Propagation

During forward propagation, data flows in one direction—from input to output: Inputs are fed into the input layer. Neurons calculate weighted sums, add biases, apply activation functions, and pass their results forward to the next layer.
This process continues layer by layer until the final output layer generates a prediction.

Step 2: Measuring Error (The Loss Function)

Once the network makes a prediction, we compare it against the real target answer ( $y$ ) using a Loss Function (also called a Cost Function).
The Loss Function measures how wrong the network is.
A simple example is Mean Squared Error (MSE), commonly used for predicting numerical values:

$\text{Loss} = \frac{1}{2} (\hat{y} - y)^2$

The goal of training is simple: Adjust the network's weights and biases to make the Loss as close to zero as possible.

Step 3: Gradient Descent (Finding the Way Down)

Mathematically, the gradient is calculated using calculus (derivatives). It tells us two crucial pieces of information: Direction: Which way does the error increase or decrease?
Steepness: How fast is the error changing relative to changes in weight?
If we move our parameters in the opposite direction of the gradient, we walk downhill toward lower loss. The Learning Rate ( $\eta$ ) The size of the step we take in each iteration is controlled by a hyperparameter called the Learning Rate ( $\eta$ ): If the learning rate is too small: The network takes tiny steps and takes days or weeks to train.If the learning rate is too large: The network takes giant leaps and might overshoot the valley completely, bouncing wildly without ever learning.

Step 4: Backpropagation (Assigning Blame)

Backpropagation uses the mathematical Chain Rule from calculus to work backward through the network: Calculate the final error at the output layer, Determine how much the output layer weights contributed to that error, Pass the error metric back to the previous hidden layer, Determine how much that hidden layer's weights contributed and Repeat all the way back to the input layer.
By working backward, every single weight in the network receives a precise update instruction based on how much it contributed to the overall error.

6. Key Challenges in Neural Networks

While neural networks are extremely powerful, they are not without flaws. Working with them involves handling several distinct challenges:

1. Overfitting:

Overfitting happens when a network learns its training data too well. Instead of discovering general rules, it essentially memorizes the training examples, including their noise and random quirks.

2. Analogy:

A student who memorizes every practice question and answer key word-for-word, but fails completely on the actual test when questions are rephrased.

3. Solution:

Techniques like Regularization, Dropout (randomly disabling neurons during training), and collecting more diverse training data help prevent overfitting.

The Black Box Problem
When a traditional software system makes a decision, a developer can read the lines of code to understand why. But when a deep neural network with 175 billion weights makes a decision, it is nearly impossible for a human to look at those floating-point numbers and understand its exact reasoning. This lack of interpretability is a challenge in fields like healthcare, finance, and legal decision-making.

Data and Compute Requirements
Unlike simple statistical models, neural networks thrive on massive amounts of data. They also require significant computational power, often requiring specialized hardware like GPUs (Graphics Processing Units) or TPUs (Tensor Processing Units) to perform billions of matrix multiplications efficiently.

7. Summary Checklist

Conclusion

A neural network is a system of connected mathematical units that learns patterns by adjusting numerical weights after making predictions and receiving feedback.

It receives information, processes that information through layers, makes a prediction, measures its error, and adjusts itself. Repeated enough times with good data, this process can produce systems capable of solving complex problems.

Neural networks are not copies of the human brain. They are tools built from mathematics, data, and computation. Their power comes from their ability to learn complicated patterns that would be difficult to describe using hand-written rules.

Python Distributions and Their Impact on Data Science

Mark Glemba — Mon, 07 Sep 2026 19:42:26 +0000

Understanding Statistical Distributions and Their Impact on Data Science

In the world of data science, data is the foundation of every decision, model, and prediction. However, raw data on its own often appears chaotic and difficult to interpret. This is where statistical distributions become important. Statistical distributions help data scientists understand how data is spread, identify patterns, and make informed decisions based on probabilities. Without understanding distributions, analyzing data effectively would be nearly impossible.

Statistical distributions are one of the most fundamental concepts in statistics and data science. They describe how values in a dataset are arranged and how frequently they occur. From predicting customer behavior to detecting fraud, statistical distributions influence nearly every data-driven process.

What is a Statistical Distribution?

A statistical distribution is a mathematical representation of how data points are spread across possible values. It shows the frequency or probability of different outcomes.

For example, consider the heights of students in a class. If you plot these heights on a graph, you may notice that most students have average heights, while only a few are extremely short or tall. This pattern forms a distribution.

A distribution answers important questions such as:

Where is the center of the data?
How spread out is the data?
Are there unusual values (outliers)?
Is the data symmetric or skewed?

These insights are essential for data analysis and machine learning.

Key Components of a Distribution

To understand distributions, it is important to know the following concepts:

1. Mean

The mean is the average value of the data. It represents the central point of a distribution.

Formula:

[
Mean = \frac{\sum x}{n}
]

Example:
If the values are 2, 4, 6, 8:

Mean = (2 + 4 + 6 + 8) / 4 = 5

2. Median

The median is the middle value when data is arranged in order. It is less affected by outliers.

Example:
For 2, 4, 6, 100:

Median = 5
Mean = 28

This shows how outliers affect the mean.

3. Variance

Variance measures how far data points are from the mean.

Low variance means data points are close together.
High variance means data points are spread out.

4. Standard Deviation

This is the square root of variance and gives a measure of spread in the same units as the data.

5. Skewness

Skewness tells us whether the distribution leans left or right.

Positive skew: Tail on the right
Negative skew: Tail on the left

6. Kurtosis

Kurtosis measures how heavy the tails of a distribution are compared to a normal distribution.

Types of Statistical Distributions

There are many statistical distributions, but some are more common in data science.

1. Normal Distribution

The Normal Distribution is the most important distribution in statistics. It is also called the Gaussian distribution.

It has these properties:

Bell-shaped curve
Symmetrical around the mean
Mean = Median = Mode

Examples:

Human heights
Exam scores
Measurement errors

Why it matters in data science:
Many machine learning algorithms assume data is normally distributed. Examples include linear regression and logistic regression.

The probability density function is:

[
f(x) = \frac{1}{σ\sqrt{2π}} e^{-\frac{(x-μ)^2}{2σ^2}}
]

2. Uniform Distribution

The Uniform Distribution occurs when all values have an equal probability of occurring.

Example:
Rolling a fair dice.

Each number (1–6) has an equal chance.

Impact in data science:
Used in random sampling and simulations.

3. Binomial Distribution

The Binomial Distribution represents the probability of success in a fixed number of independent trials.

Example:
Flipping a coin 10 times.

Possible outcomes:
How many heads occur?

Applications:

A/B testing
Customer conversion rates

4. Poisson Distribution

The Poisson Distribution models the number of events occurring in a fixed interval.

Example:

Number of website visits per minute
Number of accidents in a day

Applications:
Useful in event prediction.

5. Exponential Distribution

The Exponential Distribution models the time between events.

Example:
Time between customer arrivals.

Applications:

Queue systems
Reliability analysis

6. Power Law Distribution

The Power Law describes data where a small number of occurrences are very common while most are rare.

Examples:

Social media followers
Wealth distribution

Impact:
Helps in network analysis and anomaly detection.

Why Statistical Distributions Matter in Data Science

Understanding distributions has a major impact on data science workflows.

1. Data Cleaning

Distributions help detect anomalies and outliers.

Example:
If salaries in a dataset are mostly between $500 and $2000 but one entry is $500,000, this may be an error.

Outlier detection improves model quality.

2. Feature Engineering

Data transformations are often applied based on distributions.

Examples:

Log transformation for skewed data
Normalization for scaling

This makes data easier for models to learn.

3. Choosing the Right Model

Different models assume different distributions.

Examples:

Linear regression assumes normality
Naive Bayes often assumes Gaussian distribution
Poisson regression assumes Poisson distribution

Using the wrong assumptions can reduce accuracy.

4. Hypothesis Testing

Many statistical tests depend on distributions.

Examples:

t-test uses normal distribution
Chi-square test uses chi-square distribution

These tests help validate findings.

5. Probability Predictions

Distributions allow us to calculate probabilities.

Example:
What is the probability that sales exceed 100 units?

This supports risk assessment and forecasting.

Real-Life Applications in Data Science

Finance

Stock returns are often analyzed using distributions to measure volatility and risk.

Relevant entity: Quantitative Finance

Healthcare

Patient recovery times may follow an exponential distribution.

Relevant entity: Biostatistics

E-commerce

Customer purchase behavior often follows a power law distribution.

Companies like Amazon use distribution analysis for recommendations.

Fraud Detection

Unusual patterns that deviate from expected distributions may indicate fraud.

Banks use this to detect suspicious transactions.

Machine Learning

Algorithms like Gradient Descent perform better when data is standardized and distributed properly.

Visualizing Distributions

Data scientists use visualization tools to understand distributions:

Histograms
Box plots
Density plots
Scatter plots

In Python, libraries such as Matplotlib and Pandas help visualize distributions.

Example:

import matplotlib.pyplot as plt
import pandas as pd

data = [12, 15, 18, 20, 22, 22, 23, 24, 30]

plt.hist(data)
plt.show()

This creates a histogram to show how values are distributed.

Challenges in Understanding Distributions

Although distributions are useful, they can be challenging because:

Real-world data is often messy
Some datasets do not follow standard distributions
Mixed distributions can complicate analysis
Small datasets may not show clear patterns

Data scientists must often transform data before analysis.

Conclusion

Statistical distributions are the backbone of data science. They provide a framework for understanding how data behaves, identifying trends, and making predictions. Whether it is the normal distribution for machine learning, the binomial distribution for testing, or the Poisson distribution for event modeling, distributions guide decision-making in almost every area of data science.

A strong understanding of statistical distributions allows data scientists to clean data more effectively, choose better models, improve accuracy, and generate meaningful insights. As data continues to grow in importance across industries, mastering statistical distributions remains an essential skill for every aspiring data scientist.

Statistics, Parametric and Non-Parametric Tests in Python

Mark Glemba — Tue, 07 Jul 2026 20:01:44 +0000

Introduction
Statistics is one of the fundamental pillars of data science, machine learning, artificial intelligence, business intelligence, scientific research, healthcare analytics, finance, engineering, and many other disciplines. Every day, organizations collect enormous amounts of data from customers, sensors, websites, social media platforms, financial transactions, and business operations. However, raw data alone provides little value unless it can be analyzed and interpreted correctly. Statistics provides the mathematical framework that enables analysts and researchers to extract meaningful insights from data, identify patterns, test hypotheses, and make evidence-based decisions.
Python has become one of the world’s most popular programming languages for statistical analysis due to its simplicity, extensive libraries, and powerful data manipulation capabilities. Libraries such as NumPy, Pandas, SciPy, Statsmodels, and Scikit-learn provide researchers and analysts with comprehensive tools for performing descriptive statistics, inferential statistics, hypothesis testing, regression analysis, and machine learning.
Among the most important concepts in statistical analysis are parametric and non-parametric statistical tests. Choosing the correct statistical test is essential because using an inappropriate test may produce misleading conclusions. Parametric tests assume that the underlying data follow specific statistical distributions, usually the normal distribution, while non-parametric tests make fewer assumptions and are particularly useful when those assumptions are violated.
This article explores statistics, parametric tests, non-parametric tests, their assumptions, advantages, disadvantages, practical applications, and demonstrates how these tests can be implemented using Python.

Understanding Statistics
Statistics is the science of collecting, organizing, analyzing, interpreting, and presenting data. It helps transform raw numbers into meaningful information that supports decision-making.
Statistics is generally divided into two major branches:

1. Descriptive Statistics
Descriptive statistics summarize and describe the characteristics of a dataset without making conclusions beyond the data collected.
Common descriptive statistics include:
• Mean
• Median
• Mode
• Range
• Variance
• Standard deviation
• Quartiles
• Percentiles
• Skewness
• Kurtosis
For example, a company may calculate the average monthly salary of employees or the average customer spending per transaction.
Python Example:

2. Inferential Statistics
Inferential statistics goes beyond describing data by drawing conclusions about an entire population using a sample.
Inferential statistics helps answer questions such as:
• Is a new medicine more effective than the old one?
• Does education level influence salary?
• Is customer satisfaction different between two stores?
• Does gender affect purchasing behavior?
Inferential statistics relies heavily on probability theory and hypothesis testing.

Hypothesis Testing
Hypothesis testing is a statistical method used to determine whether there is sufficient evidence to support a claim.
Every hypothesis test begins with two hypotheses.

Null Hypothesis (H₀)
The null hypothesis states that there is no significant difference or relationship.
Example:
“There is no difference in average salaries between male and female employees.”
Alternative Hypothesis (H₁)
The alternative hypothesis states that there is a significant difference or relationship.
Example:
“There is a significant difference in average salaries between male and female employees.”

Understanding the p-value
The p-value measures how likely the observed data would occur if the null hypothesis were true.
General interpretation:
• p ≤ 0.05 → Reject H₀
• p > 0.05 → Fail to reject H₀
For example:

Parametric Statistical Tests
Parametric tests are statistical methods that make assumptions about the population distribution.
The most common assumption is that the data follow a normal distribution.
Because they use more information from the data, parametric tests are generally more powerful than non-parametric tests when their assumptions are satisfied.

Assumptions of Parametric Tests
Before applying a parametric test, several assumptions should be checked.

1. Normality
Data should approximately follow a normal distribution.
Python example:

If
p > 0.05
the data are considered approximately normal.

2. Homogeneity of Variance
Groups should have similar variances.
Python example:

3. Independence
Observations should be independent of one another.

4. Continuous Data
Most parametric tests require interval or ratio-level data.

Common Parametric Tests
1. Independent Samples t-Test
Purpose:
Compare the means of two independent groups.
Example:
Compare salaries of male and female employees
Python:

2. Paired t-Test
Purpose:
Compare measurements taken from the same individuals before and after an intervention.
Example:
Employee productivity before and after training.
Python:

3. One-Sample t-Test
Purpose:
Determine whether a sample mean differs from a known population mean.
Python:

4. ANOVA (Analysis of Variance)
Purpose:
Compare means of three or more groups.
Example:
Compare salaries across departments.
Python:

5. Pearson Correlation
Purpose:
Measure the linear relationship between two continuous variables.
Python:

Correlation values range from -1 to +1.

6. Linear Regression
Purpose:
Model the relationship between independent and dependent variables.
Python:

Regression is widely used for prediction and identifying significant predictors.

Advantages of Parametric Tests
Some benefits include:
• Higher statistical power
• More precise estimates
• Efficient with normally distributed data
• Suitable for predictive modeling
• Widely accepted in scientific research

Disadvantages of Parametric Tests
Limitations include:
• Sensitive to outliers
• Require normality assumptions
• Require continuous data
• Can produce misleading results when assumptions are violated

Non-Parametric Statistical Tests
Non-parametric tests are statistical procedures that do not assume a specific population distribution.
They are often called distribution-free tests.
These tests are especially useful when:
• Data are skewed
• Sample size is small
• Data contain outliers
• Data are ordinal
• Normality assumptions fail

Advantages of Non-Parametric Tests
Advantages include:
• Few assumptions
• Handle skewed data
• Robust against outliers
• Suitable for ordinal data
• Useful with small sample sizes

Disadvantages of Non-Parametric Tests
Disadvantages include:
• Lower statistical power than parametric tests
• May ignore some information contained in the data
• Less effective for normally distributed datasets

Common Non-Parametric Tests
1. Mann-Whitney U Test
Equivalent to the independent t-test.
Used for comparing two independent groups.
Python:

2. Wilcoxon Signed-Rank Test
Equivalent to the paired t-test.
Python:

3. Kruskal-Wallis Test
Equivalent to one-way ANOVA.
Python:

4. Friedman Test
Equivalent to repeated-measures ANOVA.
Python:

5. Spearman Rank Correlation
Equivalent to Pearson correlation when data are not normally distributed.
Python:

6. Chi-Square Test of Independence
Used for categorical variables.
Example:
Determine whether education level is associated with employment status.
Python:

Choosing Between Parametric and Non-Parametric Tests
Selecting the appropriate statistical test depends on several factors, including the type of data, the number of groups, whether observations are independent or paired, and whether the assumptions of parametric tests are satisfied.

Before choosing a test, analysts should perform exploratory data analysis (EDA), inspect visualizations such as histograms and box plots, and conduct assumption checks including tests for normality and homogeneity of variance. If assumptions hold, parametric tests are generally preferred because they provide greater statistical power. When assumptions are violated or the data are ordinal or heavily skewed, non-parametric tests offer a more reliable alternative.

Python Libraries for Statistical Analysis
Python offers a rich ecosystem of libraries that simplify statistical analysis and hypothesis testing.

NumPy
Used for numerical computations and array operations.

Pandas
Provides powerful data structures for data manipulation and descriptive statistics.

SciPy
Contains numerous statistical functions and hypothesis tests.

Statsmodels
Used for regression analysis, ANOVA, generalized linear models, and other advanced statistical techniques.

Matplotlib
Creates visualizations that aid in understanding data distributions and statistical results.

These libraries work seamlessly together, making Python one of the most comprehensive environments for statistical computing.

Practical Applications of Statistical Tests
Parametric and non-parametric tests are widely used across industries to solve real-world problems.
In healthcare, researchers use t-tests and ANOVA to compare treatment effectiveness between patient groups, while non-parametric tests are applied when medical data are skewed or involve ordinal pain scores.
In finance, analysts use correlation and regression to examine relationships between stock prices, interest rates, and economic indicators. Statistical tests help determine whether observed trends are significant or due to random variation.
Businesses rely on hypothesis testing to evaluate marketing campaigns, compare customer satisfaction across branches, and assess employee performance after training programs. A/B testing, a common practice in digital marketing, is fundamentally based on statistical hypothesis testing.
Educational institutions analyze examination results to compare teaching methods or determine whether interventions improve student performance. Government agencies apply statistical methods to census data, unemployment rates, and public health studies to guide policy decisions.

Best Practices for Statistical Testing in Python
To ensure valid and reliable results, analysts should follow several best practices:

Understand the research question before selecting a statistical test.
Perform exploratory data analysis to identify missing values, outliers, and unusual patterns.
Check assumptions such as normality, independence, and equal variances before applying parametric tests.
Choose non-parametric alternatives when assumptions are violated or when working with ordinal data.
Report both the test statistic and the p-value, and where appropriate include confidence intervals and effect sizes to provide a more complete interpretation of the findings.
Interpret statistical significance alongside practical significance, recognizing that a statistically significant result may not always have meaningful real-world implications.
Document the methodology and Python code to ensure that analyses are transparent, reproducible, and easy to verify.

Conclusion
Statistics plays an indispensable role in transforming raw data into meaningful knowledge. Through descriptive statistics, analysts summarize and understand datasets, while inferential statistics enables them to make predictions and draw conclusions about larger populations from sample data. Hypothesis testing forms the foundation of inferential analysis by providing an objective framework for evaluating claims and determining whether observed differences or relationships are statistically significant.
Parametric tests, including t-tests, ANOVA, Pearson correlation, and linear regression, are powerful methods when their underlying assumptions—such as normality, homogeneity of variance, and independence—are satisfied. Their higher statistical power makes them the preferred choice for many research applications involving continuous, normally distributed data. In contrast, non-parametric tests such as the Mann-Whitney U test, Wilcoxon signed-rank test, Kruskal-Wallis test, Friedman test, Spearman correlation, and Chi-square test provide flexible and robust alternatives when data violate these assumptions or are measured on an ordinal scale.
Python has revolutionized statistical analysis by offering an accessible and efficient ecosystem of libraries, including NumPy, Pandas, SciPy, Statsmodels, and Matplotlib. These tools enable users to clean data, perform sophisticated statistical tests, visualize results, and build predictive models with relatively little code. As organizations increasingly rely on data-driven decision-making, proficiency in statistical methods and Python programming has become an essential skill for data scientists, researchers, analysts, and professionals across numerous fields. Understanding when to apply parametric or non-parametric methods ensures that analyses are both statistically valid and practically meaningful, ultimately leading to more accurate conclusions and better-informed decisions.

Understanding DDL, DML, and Key SQL Concepts

Mark Glemba — Tue, 14 Apr 2026 07:11:57 +0000

Introduction
Structured Query Language (SQL) is essential for managing and manipulating data in relational databases. In this article, we explore two important categories of SQL commands—Data Definition Language (DDL) and Data Manipulation Language (DML)—along with practical operations such as CREATE, INSERT, UPDATE, DELETE, filtering with WHERE, and the use of CASE WHEN for data transformation.

What DDL and DML Are (and Their Differences)

Data Definition Language (DDL) refers to SQL commands used to define and manage the structure of a database. These commands deal with creating, modifying, and deleting database objects such as tables.

Common DDL commands include

CREATE – used to create new tables or databases
ALTER – used to modify existing structures
DROP– used to delete tables or databases

For example, using "CREATE TABLE" allows you to define columns, data types, and constraints for storing data.

Data Manipulation Language (DML), on the other hand, is used to manage the data within those structures. It focuses on inserting, updating, retrieving, and deleting records.

Common DML commands include:

INSERT – adds new records
UPDATE– modifies existing data
DELETE – removes records
SELECT– retrieves data

Key Difference:
DDL deals with the structure of the database, while DML deals with the data inside the database.

Using CREATE, INSERT, UPDATE, and DELETE

In a typical database assignment, these commands are used as follows:

CREATE: I used this command to define tables such as students, subjects, or exam results. It involved specifying column names, data types (e.g., INTEGER, VARCHAR), and primary keys.
INSERT: This command was used to add records into the tables. For example, inserting student names, subject details, and exam scores into the respective tables.
UPDATE: I used UPDATE to modify existing records. For instance, correcting a student’s score or updating a subject name.
DELETE: This command helped remove unwanted or incorrect records from the database, such as deleting a student entry or clearing outdated data.

These commands are fundamental for maintaining accurate and up-to-date data in any database system.

Filtering Data with WHERE

The WHERE clause is used in SQL to filter records based on specific conditions. It helps retrieve only the data that meets certain criteria.

Some commonly used operators include:

= (Equal to): Selects records that match a specific value
Example: "WHERE score = 80"
> (Greater than): Selects values greater than a given number
Example: "WHERE score > 70"
BETWEEN: Filters values within a range
Example: "WHERE score BETWEEN 50 AND 90"
IN: Matches values within a list
Example: "WHERE subject IN ('Math', 'Science')"
LIKE: Used for pattern matching
Example: "WHERE name LIKE 'J%'" (names starting with J)

The WHERE clause is powerful because it allows precise data retrieval, making queries more meaningful and efficient.

How CASE WHEN Helps in Transforming Data The CASE WHEN statement in SQL is used to perform conditional logic within queries. It allows you to transform data by assigning values based on conditions.

For example, you can categorize student performance:

If score ≥ 70 → “Pass”
If score < 70 → “Fail”

This helps in:

Creating calculated columns
Categorizing or grouping data
Improving readability of query results

CASE WHEN is especially useful in reports, where raw data needs to be interpreted into meaningful insights.

Conclusion

Understanding DDL and DML is crucial for working with databases effectively. While DDL defines the structure of the database, DML allows you to interact with the data itself. Commands like CREATE, INSERT, UPDATE, and DELETE form the backbone of database operations. Additionally, tools like the WHERE clause and CASE WHEN statement enhance your ability to filter and transform data, making SQL a powerful language for data management and analysis.

HOW TO PUBLISH A POWER BI REPORT AND EMBED IT INTO A WEBSITE

Mark Glemba — Mon, 06 Apr 2026 14:10:43 +0000

INTRODUCTION
Power BI is an analytics tool developed by Microsoft that lets a user visualize and share insights on certain data. Power BI is also used transform and clean data using Power Query Editor. Data transformation occurs through; shaping data by filtering, sorting and grouping, adding columns and changing data types through conversion to; numeric, text etc. Data cleaning is done through; handling blanks, standardizing and removing errors.

PUBLISHING PROCESS IN POWER BI

Create a workspace A workspace is like a folder where one stores reports, dashboards and datasets.

. Go to [https://app.powerbi.com/]
. Sign into your account
. Once your logged in, on the left far side click on workspaces

. Click on; + New workspace

2 .Uploading and publishing
. In the power bi desktop in the home page click on publish on the far right.

. Select your workspace where the report will be uploaded
. Once uploaded the dataset will appear on your workspace.

Generating embedding code . Open the report you want embedded . Click on File ~~~~ Embed report . Choose embedding option; either Public (Publish to Web), OR Secure Embed; which requires log in and is more secure. . After clicking on Publish to web, Power BI will generate an embed link + iframe code.

4.Embedding code into website

. Copy entire code
. Open your website HTML file
. Paste the iframe code where you want your report to appear

. Save and open your website where your report will display.

CONCLUSION
Using Power BI with embedding capabilities promotes better collaboration, transparency and efficiency. This empowers users to embrace data-driven solutions that support smarter strategies and improved outcomes.

HOW EXCEL IS USED IN REAL-WORLD DATA ANALYSIS

Mark Glemba — Tue, 31 Mar 2026 19:39:06 +0000

INTRODUCTION
Excel is a tool one can use to organize data for work or any other specific software functions. It allows one to develop graphs for comparison of data and create charts to organize and visualize data.

HOW EXCEL IS USED IN REAL WORLD SCENARIOS

DATA CLEANING
Excel is in various ways for data cleaning by performing functions such as; removing duplicates, trimming spaces, fixing formatting, finding errors, standardizing data, splitting data and filling blanks.
DATA SUMMARIZATION
Excel summarizes large data sets to offer quick insights by use of; pivot tables, formulas and functions, instead of users manually going through large data sets.
VISUALIZING TRENDS
Done by the use of charts; pie, line and bar charts.

FORMULAS USED IN EXCEL
Basic formulas in excel include:

Add: Used to add values of two or more cells. The (+) is used.
Subtract: Used to subtract values of two or more cells. The (-) is used.
Multiply: Used to multiply values of two or more cells. The (*) is used.
Divide: Used to divide values of two or more cells. The (/) is used.

BASIC FUNCTIONS USED IN EXCEL
Functions are used to automate tasks normally performed by formulas in excel. Some popular and basic functions include:

AVERAGE: It sums up and calculate the mean of values within a range given. For example,=AVERAGE(A1:A15).
SUM: It adds up a range of cells. For example,=SUM(A1:A15).
IF: Used to arrange values based on a logical test. The syntax for IF ; =IF(logical_test, value_if_true, [value_if_false]). For example, =IF(N2>30, "young", "old").
COUNTIF: Used to return the number of cells that meeet a certain criteria. The syntax is; =COUNTIF(range, criteria). For example, =COUNTIF(A1:A27, "Nairobi"). NOTE: COUNTIFS function counts the number of cells that meet multiple criteria.
VLOOKUP: Allows one to search for a value on a spreadsheet. Its syntax ; =VLOOKUP(lookup_value, table_array, col_index_num, [range_lookup]). For example, =VLOOKUP(300,A1:A20, 3, "False").

CONCLUSION
In conclusion Excel's versatility makes it an efficient tool for quick analysis and reporting. Its availability allows users to efficiently handle large data sets through processes such as performing calculations and creating meaningful visuals.

Understanding Data Modelling in Power BI: Joins, Relationships and Schemes Explained

Mark Glemba — Tue, 31 Mar 2026 19:13:12 +0000

DATA MODELLING
This is a detailed process that involves creating a visual representation of data and its relationships.

TYPES OF TABLES
FACT TABLE
It contains quantitative data for analysis.
DIMENSIONS TABLE
Provides context to fact table data.
STAR SCHEMA
It contains a single fact table in the center that connects to multiple other dimensiontables.

SNOW FLAKE
Is an extension of a star schema where dimension tables are broken down into subdimensions.

FLAT TABLE
A table that displays data in a simple, two-dimensional format without any relationships to other tables.

SQL JOINS IN POWER BI

INNER JOIN: Returns rows present in both tables if there is a match.
LEFT JOIN: Returns all rows present in the left table and matching rows from the right table.
RIGHT JOIN: Returns matching rows from the left table and all rows present in the SQL right table.
FULL OUTER: Returns all rows present in both right and left tables.
LEFT ANTI: Returns rows from the left table that don't have matches in the right table.
RIGHT ANTI:Returns rows from the right table that don't have matches in the left table.

POWER BI RELATIONSHIPS

One-to-Many (1:M)
One row in table A matches many in table B. For example,

Many-to Many (M:M)
Here one is recommended to use a bridge table.

0ne-to-One (1:1)
One row in table A matches one in table B. For example,

Many-to-One (M:M)
Many rows in table A match one in table B. For example,

Active and Inactive relationships
The active relationship is used by default in reports and calculations, while the inactive relationship is not used unless specified. It is also useful for alternate paths such as multiple dates.

DIFFERENCE BETWEEN JOINS AND RELATIONSHIPS
Joins; combine tables based on a condition, result in new tables with combined columns and are used in power query steps. Relationships; define connections between tables in a data model and are used for; filtering, calculations and visuals across tables.