DEV Community: Jonathan kip

How machines learn to see, interpret, and understand images

Jonathan kip — Mon, 21 Sep 2026 10:13:40 +0000

Understanding Convolutional Neural Networks and Computer Vision

Computer Vision is a field of Artificial Intelligence (AI) that enables computers to process and interpret visual information from images and videos.

It powers applications such as facial recognition, medical imaging, autonomous vehicles, security systems, OCR, and image search.

One of the most important technologies behind modern Computer Vision is the Convolutional Neural Network (CNN).

What Is Computer Vision?

Computers do not see images the way humans do. An image is represented as numerical values called pixels.

A grayscale image typically has one channel, while an RGB image has three:

Red
Green
Blue

An RGB image can therefore be represented as:

H × W × C

Where:

H = height
W = width
C = number of channels

For example:

224 × 224 × 3

represents an image that is 224 pixels high, 224 pixels wide, and has three color channels.

Computer Vision algorithms process these values to identify patterns and produce meaningful results.

Common Computer Vision Tasks

Some common tasks include:

Image classification — assigning a label to an image.
Object detection — identifying objects and their locations.
Image segmentation — classifying individual pixels.
Face recognition — identifying or verifying faces.
Optical Character Recognition (OCR) — extracting text from images.
Pose estimation — detecting body positions.
Video analysis — understanding movement across frames.

For example, classification might tell us that an image contains a dog, while object detection can identify several dogs and locate each one.

What Is a CNN?

A Convolutional Neural Network is a deep-learning architecture designed to process grid-like data, especially images.

A traditional fully connected neural network can become computationally expensive when processing images because every pixel may be connected to many neurons.

CNNs solve this problem by processing local regions of an image using learnable filters, also called kernels.

A typical CNN contains:

Input Image
     ↓
Convolution
     ↓
Activation Function
     ↓
Pooling / Downsampling
     ↓
More Convolution Layers
     ↓
Prediction Layer
     ↓
Output

The key advantage is that CNNs learn useful visual features automatically rather than requiring us to manually define them.

How Convolution Works

A convolutional filter is a small matrix that moves across an image.

At each position, the filter performs element-wise multiplication with the image region and sums the results. This produces a feature map.

A simplified convolution can be represented as:

Y(i, j) = Σₘ Σₙ X(i + m, j + n)K(m, n)

Where:

X = input image or feature map
K = convolutional kernel
Y(i, j) = output value at position (i, j)

During training, the CNN learns the values of these filters.

Some filters may learn to detect:

Edges
Corners
Curves
Textures
Shapes
Object parts

This leads to one of the most important concepts in CNNs: hierarchical feature learning.

The CNN Feature Hierarchy

CNNs generally learn increasingly complex features as information moves through the network.

Layer	Typical Features
Early	Edges, colors, textures
Middle	Curves, shapes, patterns
Deep	Eyes, wheels, windows, object parts
Final	Complete objects or categories

For example, in a cat-classification model:

Pixels
  ↓
Edges
  ↓
Textures and shapes
  ↓
Eyes, ears, fur
  ↓
Cat

The network learns this hierarchy automatically from training data.

Activation Functions

After convolution, CNNs typically apply an activation function to introduce nonlinearity.

One of the most common is the Rectified Linear Unit (ReLU):

ReLU(x) = max(0, x)

For example:

ReLU(-5) = 0
ReLU( 7) = 7

ReLU allows neural networks to learn complex relationships that cannot be represented using only linear operations.

Padding and Stride

Two important convolution parameters are padding and stride.

Padding adds pixels around the border of an image. It can help preserve spatial dimensions and ensure that border information is processed.

Stride determines how far the filter moves during convolution.

The output size can be calculated using:

Output size = floor((N − F + 2P) / S) + 1

Where:

N = input size
F = filter size
P = padding
S = stride

For example, with a 32 × 32 input, a 3 × 3 filter, padding of 1, and stride of 1:

Output = floor((32 − 3 + 2) / 1) + 1
       = 32

The spatial dimensions therefore remain 32 × 32.

Pooling

Pooling reduces the spatial size of feature maps.

The two common types are max pooling and average pooling.

Max Pooling

Max pooling selects the largest value:

[1  3]
[2  4] → 4

Average Pooling

Average pooling calculates the average:

[1  3]
[2  4] → 2.5

Pooling reduces computational requirements and can make models less sensitive to small changes in object position.

However, excessive downsampling can remove useful spatial information, which is why some modern architectures use other downsampling techniques.

How CNNs Learn

CNN training involves repeatedly making predictions and correcting errors.

The process is roughly:

Pass an image through the network.
Generate a prediction.
Calculate the loss.
Use backpropagation to calculate gradients.
Update the model's parameters.
Repeat over many batches and epochs.

For multiclass classification, a common loss function is cross-entropy:

L = −Σᵢ yᵢ log(ŷᵢ)

A simplified gradient-descent update is:

θₜ₊₁ = θₜ − η∇θL

Where:

θ = model parameters
η = learning rate
∇θL = gradient of the loss

Over many training iterations, the network learns filters that produce better predictions.

Softmax and Classification

For a classification problem, the final layer can produce a score for each class.

A softmax function converts these scores into probabilities:

P(y = i | x) = eᶻⁱ / Σⱼ eᶻʲ

For example:

Cat:    0.82
Dog:    0.12
Rabbit: 0.06

The model predicts Cat because it has the highest probability.

Training, Overfitting, and Generalization

CNNs require representative training data.

For a vehicle-recognition model, the dataset should ideally contain vehicles under different:

Lighting conditions
Viewing angles
Backgrounds
Weather conditions
Resolutions
Levels of obstruction

If a model performs extremely well on training data but poorly on unseen data, it may be overfitting.

A model that performs well on unseen data is said to generalize effectively.

Data Augmentation

Data augmentation can improve generalization by creating variations of existing images.

Common techniques include:

Flipping
Cropping
Rotation
Scaling
Translation
Brightness adjustment
Contrast adjustment

For example, a dog image can be flipped, slightly rotated, or cropped to create additional training examples.

The transformations should remain realistic. An inappropriate transformation can change the meaning of an image and confuse the model.

Transfer Learning

Training a CNN from scratch can require substantial data and computational resources.

Transfer learning provides a practical alternative.

A pretrained model has already learned general visual features such as:

Edges → Textures → Shapes → Object parts

These learned features can be reused for a new task.

Transfer learning is particularly useful when:

The dataset is small.
Computing resources are limited.
Training time needs to be reduced.
The new task is related to the original training task.

CNN Applications

CNNs are used across many areas of Computer Vision.

Image Classification

Examples include:

Animal classification
Traffic-sign recognition
Handwritten-digit recognition
Medical image classification

Object Detection

Object detection identifies objects and their locations.

A system might produce:

Person: 0.96
Bounding box: (x1, y1, x2, y2)

Bicycle: 0.91
Bounding box: (x1, y1, x2, y2)

This is useful in robotics, autonomous systems, surveillance, and retail analytics.

Image Segmentation

Segmentation assigns labels to individual pixels.

Applications include:

Medical imaging
Autonomous driving
Satellite imagery
Industrial inspection

Facial Analysis

CNNs can be used for face detection, verification, and recognition.

However, these systems require careful consideration of privacy, consent, security, and fairness.

Evaluating CNN Models

Different Computer Vision tasks require different evaluation metrics.

For classification, common metrics include:

Accuracy
Precision
Recall
F1 score
Top-k accuracy

For object detection, Intersection over Union (IoU) measures the overlap between a predicted bounding box and the ground-truth box:

IoU = Area of Intersection / Area of Union

IoU ranges from:

0 → No overlap
1 → Perfect overlap

For segmentation, commonly used metrics include:

Pixel accuracy
IoU
Dice coefficient
Mean IoU

Accuracy should not always be used alone. For example, in a medical-diagnosis application, false negatives may be particularly important.

Limitations of CNNs

CNNs are powerful, but they have limitations.

Data Dependence

A model can struggle when it encounters conditions that are significantly different from its training data.

Dataset Bias

Unbalanced or incomplete datasets can cause differences in model performance across groups or environments.

Adversarial Inputs

Small changes to an image can sometimes cause a model to produce an incorrect prediction.

Interpretability

CNNs can be difficult to interpret because their learned representations are distributed across many layers and parameters.

Techniques such as Grad-CAM can help visualize which image regions influenced a prediction.

Computational Requirements

Large CNNs can require significant memory and processing power.

Techniques such as quantization, pruning, model compression, and efficient architectures can help make models more suitable for edge devices.

CNNs and Modern Vision Models

CNNs remain important, but modern Computer Vision is expanding beyond convolutional architectures.

Other approaches include:

Vision Transformers (ViTs)
Attention mechanisms
Generative models
Multimodal models
CNN-transformer hybrid architectures

CNNs are particularly good at learning local visual patterns, while transformer-based models can capture relationships between distant regions of an image.

Many modern systems combine ideas from both approaches.

Conclusion

Computer Vision enables machines to process and interpret visual information, while Convolutional Neural Networks provide a powerful way to learn visual features directly from image data.

CNNs progressively transform raw pixels into increasingly meaningful representations:

Pixels
  ↓
Edges
  ↓
Textures
  ↓
Shapes
  ↓
Object Parts
  ↓
Objects
  ↓
Prediction

Understanding convolution, activation functions, pooling, backpropagation, training data, and evaluation provides a strong foundation for working with Computer Vision.

CNNs are not the end of Computer Vision, but they remain an important foundation for understanding how modern visual AI systems work.

If you found this article useful, consider following for more articles about machine learning, deep learning, and computer vision.

Neural Networks: Inspiration and Main Components

Jonathan kip — Mon, 21 Sep 2026 09:02:19 +0000

Neural Networks: Inspiration and Main Components

A neural network is a computer model that learns from examples. It is inspired by the human brain, where interconnected biological neurons receive information, process it, and send signals to one another. In artificial intelligence, neural networks use mathematical calculations to recognize patterns, make predictions, and support decision-making.

Although neural networks are inspired by the brain, they are not exact copies of biological neurons. They are simplified computational systems made up of connected processing units called artificial neurons.

Biological Inspiration

The human brain contains many biological neurons connected through junctions called synapses. A neuron receives signals through its dendrites, processes them in the cell body, and sends signals through its axon.

Artificial neural networks simplify this process:

Biological brain	Artificial neural network
Biological neuron	Artificial neuron
Dendrites receive signals	Inputs receive data
Synapse strength	Weight
Cell-body processing	Mathematical calculation
Neuron firing	Activation output
Learning through experience	Adjusting weights

The main idea is that many simple units can work together to solve complicated problems. Artificial neural networks use this idea to learn patterns from data.

Structure of a Neural Network

A neural network is normally organized into layers:

Input layer
Hidden layer or layers
Output layer

Information moves through these layers until the network produces a prediction.

Input Layer

The input layer receives the information used by the network. Each input represents a feature.

For example, a network predicting whether a student will pass may receive:

Hours studied.
Attendance percentage.
Previous examination marks.

In an image-recognition system, the inputs may represent pixel values.

Hidden Layers

Hidden layers process the information received from the input layer. They identify relationships and patterns within the data.

For example, when identifying an image:

One layer may detect lines.
Another layer may detect shapes.
A later layer may detect parts of an object.
The final hidden layers may recognize the complete object.

A neural network with many hidden layers is called a deep neural network, while the use of these networks is generally referred to as deep learning.

Output Layer

The output layer gives the final answer.

For example, a network classifying an image may produce:

Class	Probability
Cat	0.85
Dog	0.15

The network would classify the image as a cat because it has the higher probability.

The output layer can produce different types of results:

A yes-or-no answer.
A category among several choices.
A numerical value, such as a price or temperature.
Several values for a multi-output prediction.

The Artificial Neuron

An artificial neuron receives input values and combines them to produce an output. It does this using weights, a bias, and an activation function.

The basic calculation is:

z = w₁x₁ + w₂x₂ + ... + wₙxₙ + b

The neuron then applies an activation function:


a = f(z)

In simple terms, the neuron:

Receives information.
Determines the importance of each input.
Combines the inputs.
Adds a bias.
Produces an output.

Inputs

Inputs are the data given to the neural network. They can be numbers, words, sounds, or images.

Weights

Weights determine the importance of different inputs. A large positive weight means that an input has a strong positive influence. A negative weight can reduce or reverse the influence of an input.

For example, in a spam-detection model, words such as “free” or “prize” may receive important weights if they frequently occur in spam emails.

Bias

A bias is an additional value that makes the neuron more flexible. It helps the neuron adjust its output even when the input values are small or zero.

Activation Functions

An activation function controls the output of a neuron. It also allows the neural network to learn complex, non-linear patterns rather than only simple relationships.

Common activation functions include:

ReLU: Produces zero for negative values and keeps positive values.
Sigmoid: Produces values between 0 and 1 and is often used for probabilities.
Tanh: Produces values between -1 and 1.
Softmax: Converts several outputs into probabilities that add up to 1.

An activation function can be viewed as a decision rule that determines whether a neuron should produce a weak, strong, or zero signal.

How Neural Networks Learn

Neural networks learn by studying examples. This process is similar to teaching a student through practice.

Suppose a neural network is being trained to identify cats and dogs. The training process works as follows:

The network receives an image.
It makes a prediction.
The prediction is compared with the correct answer.
The error is measured.
The network adjusts its weights.
It repeats the process with many other examples.

At the beginning, the network may make many mistakes. Gradually, it changes its weights and learns which patterns are important.

Loss and Error

The difference between the network’s prediction and the correct answer is called the error or loss.

For example, if the correct label is “cat” but the network predicts “dog,” the loss function measures how incorrect the prediction was.

The purpose of training is to reduce the loss. A smaller loss usually means that the network’s predictions are becoming more accurate.

Common loss functions include:

Mean squared error: Used mainly for numerical predictions.
Binary cross-entropy: Used for two-class problems.
Categorical cross-entropy: Used for problems involving several classes.

Backpropagation

Once the network calculates its error, it determines which weights contributed to that error. It then sends the error information backward through the network.

This process is called backpropagation. It calculates how the weights should change so that the network can make better predictions in the future.

An optimization algorithm, such as gradient descent or Adam, applies these changes.

The overall learning cycle can be represented as:

Input → Prediction → Error → Weight adjustment
This cycle is repeated many times during training.

Simple Example

Consider a neural network designed to predict whether a student will pass an examination.

Inputs

Hours studied: 6.
Attendance: 85%.
Previous score: 70%.

The input layer receives these values. The hidden layers examine relationships between the features. The weights determine how important each feature is.

The output layer may produce:

Probability of passing: 0.82

This means the network estimates an 82% chance that the student will pass. If the student eventually fails, the network uses that incorrect prediction to adjust its weights during training.

Types of Neural Networks

Feedforward Neural Networks

In a feedforward network, information moves in one direction from the input layer to the output layer. These networks are commonly used for basic classification and regression tasks.

Convolutional Neural Networks

Convolutional neural networks, or CNNs, are designed mainly for image and spatial data. They identify local patterns such as edges, textures, and shapes.

Applications include:

Image classification.
Object detection.
Medical-image analysis.
Satellite-image interpretation.

Recurrent Neural Networks

Recurrent neural networks, or RNNs, are designed for sequential information. They can use information from earlier parts of a sequence.

They have been used for:

Speech recognition.
Text processing.
Time-series forecasting.
Language translation.

Transformers

Transformers use attention mechanisms to determine which parts of an input are most important in relation to other parts. They are widely used in language processing and other modern artificial-intelligence applications.

Autoencoders

Autoencoders learn to compress information and then reconstruct it. They are useful for dimensionality reduction, noise removal, anomaly detection, and feature learning.

Advantages and Limitations

Neural networks have several advantages:

They can learn complicated relationships.
They can process images, text, sound, and numerical data.
They can automatically discover useful features.
Their performance may improve with suitable training data.

They also have limitations:

They may require a large amount of data.
Training can require significant computing resources.
They can learn bias or errors found in the training data.
Their decisions may be difficult to explain.
They may perform poorly on unfamiliar examples.
They can overfit the training data.

Conclusion

Neural networks are computer models inspired by the way biological neurons connect and communicate. They are made up of artificial neurons organized into input, hidden, and output layers.

The main components include inputs, weights, biases, activation functions, neurons, loss functions, and optimization methods. A neural network learns by making predictions, measuring errors, and adjusting its weights through backpropagation.

In simple terms, a neural network is like a learner that improves through practice. It studies examples, discovers important patterns, learns from its mistakes, and uses what it has learned to make predictions about new data.

Using Scikit-learn Pipelines: A Practical Technical Guide

Jonathan kip — Mon, 31 Aug 2026 06:26:41 +0000

Think of a scikit-learn Pipeline as a “recipe” for your machine learning workflow: it lists the steps (cleaning, transforming, modeling) in order and runs them automatically every time you train or predict. This keeps your code tidy, prevents mistakes, and makes your models easier to reuse.

What Problem Do Pipelines Solve?

Without pipelines, you might:

Impute missing values, then scale, then train a model—using separate commands.
Accidentally fit preprocessing on the whole dataset (including test data), which leaks information.
Repeat the same steps for training, validation, and deployment.

Pipelines fix this by:

Bundling all steps into one object.
Ensuring preprocessing is fit only on training data during cross-validation.
Letting you call fit and predict once on the whole workflow.

The Basic Idea: Steps in Order

A pipeline is just an ordered list of named steps:

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

pipe = Pipeline([
    ("scaler", StandardScaler()),
    ("model", LogisticRegression())
])

pipe.fit(X_train, y_train)
y_pred = pipe.predict(X_test)

What happens inside:

On fit: StandardScaler learns mean and std from X_train, transforms X_train, then LogisticRegression fits on the scaled data.
On predict: StandardScaler transforms X_test using the training statistics, then LogisticRegression predicts.

You never manually scale test data; the pipeline does it consistently.

A Simpler Constructor: make_pipeline

If you don’t care about step names, use make_pipeline:

from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

pipe = make_pipeline(StandardScaler(), LogisticRegression())

scikit-learn names the steps automatically (standardscaler, logisticregression).

Handling Different Column Types: ColumnTransformer

Real data has numeric and categorical columns that need different treatment. ColumnTransformer lets you apply different preprocessing to different columns, then combines them.

from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder, StandardScaler
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline

numeric_features = ["age", "income"]
categorical_features = ["city", "subscription"]

numeric_pipe = Pipeline([
    ("imputer", SimpleImputer(strategy="median")),
    ("scaler", StandardScaler())
])

categorical_pipe = Pipeline([
    ("imputer", SimpleImputer(strategy="most_frequent")),
    ("onehot", OneHotEncoder(handle_unknown="ignore"))
])

preprocess = ColumnTransformer([
    ("num", numeric_pipe, numeric_features),
    ("cat", categorical_pipe, categorical_features)
])

model = Pipeline([
    ("preprocess", preprocess),
    ("clf", LogisticRegression())
])

model.fit(X_train, y_train)
y_pred = model.predict(X_test)

Key idea: define “what to do” for each group of columns once, then reuse.

Why This Prevents Data Leakage

Data leakage happens when information from the test set influences training. Common mistake:

# Wrong: scaling before split or on full X
scaler = StandardScaler().fit(X)
X_scaled = scaler.transform(X)

Correct pattern with pipelines:

model = Pipeline([
    ("scaler", StandardScaler()),
    ("clf", LogisticRegression())
])

model.fit(X_train, y_train)  # scaler learns only from X_train
y_pred = model.predict(X_test)  # scaler uses training stats on X_test

During cross-validation, scikit-learn refits the entire pipeline on each training fold, so preprocessing never sees the validation fold.

Tuning Hyperparameters Across Steps

Pipelines let you tune parameters from any step using stepname__param syntax.

from sklearn.model_selection import GridSearchCV

param_grid = {
    "scaler__with_mean": [True, False],
    "clf__C": [0.1, 1, 10]
}

grid = GridSearchCV(model, param_grid, cv=5)
grid.fit(X_train, y_train)

This searches over both scaling options and model regularization jointly.

Custom Steps When You Need Them

Sometimes you need your own transformation. You can:

Wrap a simple function with FunctionTransformer.
Or write a small class that follows scikit-learn’s transformer interface.

Simple function example:

from sklearn.preprocessing import FunctionTransformer
import numpy as np

pipe = Pipeline([
    ("log", FunctionTransformer(np.log1p, validate=False)),
    ("clf", LogisticRegression())
])

For more complex logic, define a class with fit and transform methods and use it like any other step.

Saving and Reusing Your Model

Once your pipeline is trained, save the whole thing:

import joblib

joblib.dump(model, "model_pipeline.joblib")

# Later, in another script or service:
loaded = joblib.load("model_pipeline.joblib")
y_pred = loaded.predict(X_new)

You don’t need to re-implement preprocessing; it’s all inside the saved pipeline.

Minimal Complete Example

from sklearn.datasets import fetch_openml
from sklearn.model_selection import train_test_split
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder, StandardScaler
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline
from sklearn.ensemble import RandomForestClassifier
import pandas as pd

# Load a sample dataset
data = fetch_openml("adult", version=2, as_frame=True)
X = data.data
y = (data.target == ">50K").astype(int)

num_cols = X.select_dtypes(include=["int64", "float64"]).columns
cat_cols = X.select_dtypes(include=["object", "category"]).columns

num_pipe = Pipeline([
    ("imputer", SimpleImputer(strategy="median")),
    ("scaler", StandardScaler())
])

cat_pipe = Pipeline([
    ("imputer", SimpleImputer(strategy="most_frequent")),
    ("onehot", OneHotEncoder(handle_unknown="ignore"))
])

preprocess = ColumnTransformer([
    ("num", num_pipe, num_cols),
    ("cat", cat_pipe, cat_cols)
])

model = Pipeline([
    ("preprocess", preprocess),
    ("clf", RandomForestClassifier(n_estimators=200, random_state=42))
])

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

model.fit(X_train, y_train)
print("Test accuracy:", model.score(X_test, y_test))

This single model object handles imputation, encoding, scaling, and prediction.

Quick Checklist for Using Pipelines

Put all preprocessing inside the pipeline, not outside.
Fit only on X_train, y_train; use the pipeline to predict on new data.
Use ColumnTransformer when you have mixed column types.
Use make_pipeline for quick experiments; use Pipeline when you want clear step names.
Save the entire pipeline for deployment.

Why Hypothesis Testing is the Backbone of Data Science

Jonathan kip — Fri, 07 Aug 2026 11:35:44 +0000

Why Hypothesis Testing is the Backbone of Data Science

Hypothesis testing is the backbone of data science because it provides a rigorous, structured way to distinguish real patterns from random noise and to make decisions backed by statistical evidence rather than intuition. It allows data scientists to validate assumptions, quantify uncertainty, and determine whether observed results are genuine or just due to chance—making it essential for trustworthy models, experiments, and business decisions. openstax

What Is Hypothesis Testing?

Hypothesis testing is a statistical method used to evaluate claims about a population using sample data. It works by setting up two mutually exclusive hypotheses: openstax

Null hypothesis (H₀): The default assumption (e.g., “There is no difference between two groups”). dasca
Alternative hypothesis (H₁): The claim you want to test (e.g., “There is a difference between two groups”). dasca

By analyzing sample data, you assess the likelihood of observing your results under the assumption that H₀ is true, and decide whether there’s enough evidence to reject it. openstax

Why Hypothesis Testing Matters in Data Science

Hypothesis testing is not just a math exercise—it’s the foundation of evidence-based decision-making in data science. Here’s why it’s so critical: dasca

1. Separates Signal from Noise

In the era of big data, patterns can appear by pure luck. Hypothesis testing answers a key question: “Is this pattern real, or did it happen by chance?” This helps you avoid acting on false positives or misleading trends.

2. Validates Assumptions and Models

Data science workflows rely on assumptions—about data distributions, feature importance, or model performance. Hypothesis testing provides a structured way to: openstax

Validate whether a new model or feature actually improves performance. dasca
Confirm that observed differences (e.g., in conversion rates) are statistically significant. dasca
Test assumptions required by statistical methods (e.g., normality, equal variances). geeksforgeeks

3. Enables Data-Driven Decision-Making

Instead of relying on gut feelings or anecdotes, hypothesis testing transforms assumptions into evidence. It allows you to: dasca

Make quantifiable, data-backed decisions. linkedin
Support business choices (e.g., launch a feature, change pricing) with statistical rigor. dasca
Build confidence among stakeholders by showing that results are validated, not guessed. geeksforgeeks

4. Forms the Foundation of Experimentation

Hypothesis testing is the engine behind A/B testing, clinical trials, and scientific experiments. It helps you: dasca

Design experiments in a methodical way. dasca
Determine whether changes in one variable directly impact another (causality, not just correlation). dasca
Calculate sample sizes and interpret p-values to draw statistically valid inferences. dasca

5. Quantifies Uncertainty and Risk

Every data-driven decision carries risk. Hypothesis testing quantifies that uncertainty by: openstax

Providing p-values that measure the strength of evidence against the null hypothesis. openstax
Controlling error rates (Type I and Type II) to balance false positives and false negatives. dasca
Generating confidence intervals that show the range of plausible values for an effect. geeksforgeeks

Core Concepts in Hypothesis Testing

To use hypothesis testing effectively, you need to understand these key concepts:

Null and Alternative Hypotheses

Null hypothesis (H₀): Represents the status quo (e.g., “The new feature has no effect on conversion rate”). dasca
Alternative hypothesis (H₁): Represents the effect you’re testing for (e.g., “The new feature increases conversion rate”). dasca

P-Value and Significance Level

P-value: The probability of observing your data (or more extreme) if H₀ is true. openstax
Significance level (α): A threshold (commonly 0.05) below which you reject H₀. dasca
If p < α, the result is considered statistically significant, and you reject H₀. openstax

Type I and Type II Errors

Type I error (false positive): Rejecting H₀ when it’s actually true. dasca
Type II error (false negative): Failing to reject H₀ when it’s actually false. dasca Balancing these errors is crucial—especially in high-stakes domains like healthcare or finance. dasca

Common Hypothesis Tests

Data scientists frequently use:

t-tests: Compare means between two groups (e.g., treatment vs control).
Z-tests: Similar to t-tests but for large samples or known variance.
Chi-square tests: Test relationships between categorical variables.
ANOVA: Compare means across three or more groups.
Mann-Whitney U / Wilcoxon: Non-parametric alternatives when assumptions are violated.

Real-World Applications in Data Science

Hypothesis testing powers decisions across industries and use cases:

A/B Testing: Validate whether a new UI, pricing strategy, or marketing campaign actually improves metrics. dasca
Feature Selection: Test whether a new feature significantly improves model performance. dasca
Medical Studies: Determine if a new treatment is more effective than a placebo. dasca
Manufacturing & Quality Control: Verify whether a process change reduces defects.
Customer Analytics: Test if churn rates differ between user segments.

Common Pitfalls and How to Avoid Them

Even with its power, hypothesis testing can be misused. Watch out for:

P-hacking: Running many tests until you find a “significant” result by chance.
Ignoring effect size: A tiny effect can be statistically significant with a large sample but meaningless in practice. geeksforgeeks
Misinterpreting p-values: A p-value is not the probability that H₀ is true—it’s the probability of your data under H₀. geeksforgeeks
Overlooking assumptions: Many tests assume normality, equal variance, or independence—violating these can invalidate results. openstax

Bottom Line

Hypothesis testing is the powerful backbone of modern data science because it turns observations into evidence and assumptions into validated insights. It enables data scientists to: dasca

Distinguish real patterns from random noise.
Make decisions with quantified uncertainty. openstax
Build trustworthy models and experiments. dasca

Without hypothesis testing, data science would rely on guesswork and anecdote—undermining the credibility, reliability, and impact of every analysis.

Understanding Statistical Distributions and Their Impact on Data Science

Jonathan kip — Fri, 07 Aug 2026 11:31:13 +0000

Understanding Statistical Distributions and Their Impact on Data Science

Statistical distributions are the backbone of data science—they describe how data values are spread, how likely certain outcomes are, and how uncertain our measurements really are. Misunderstanding or misapplying a distribution can lead to invalid tests, biased models, and poor business decisions, while using the right one improves accuracy, interpretability, and model robustness. sciencedirect

What Is a Statistical Distribution?

A statistical distribution is a mathematical function that assigns probabilities to possible outcomes of a random variable. It describes how values are spread across a dataset—where most observations cluster, how much variation exists, and whether extreme values (outliers) are common or rare. sciencedirect

Distributions answer key questions like:

How likely is a particular event?
What values are typical or extreme?
How uncertain is our data? sciencedirect

Types of Distributions

Distributions are broadly categorized by the type of data they model. sciencedirect

Discrete vs Continuous

Discrete distributions: Model countable outcomes (e.g., number of purchases, clicks, defects). Examples: Binomial, Poisson, Bernoulli. sciencedirect
Continuous distributions: Model measurable quantities (e.g., height, temperature, time). Examples: Normal, Uniform, Exponential, Log-Normal. sciencedirect

Key Distributions in Data Science

Here are some of the most important distributions and their typical use cases:

Distribution	Type	Key Parameters	Typical Use Cases
Normal (Gaussian)	Continuous	Mean (μ), Standard Deviation (σ)	Residuals in regression, many natural phenomena, Z-tests, confidence intervals sciencedirect
Uniform	Continuous or Discrete	Min, Max	Random sampling, baseline models, Monte Carlo simulations towardsdatascience
Binomial	Discrete	Trials (n), Success probability (p)	Binary outcomes (e.g., click/no-click, defect/no defect) www3.cs.stonybrook
Poisson	Discrete	Rate (λ)	Count of events in fixed time (e.g., calls per hour, arrivals) www3.cs.stonybrook
Exponential	Continuous	Rate (λ)	Time between events (e.g., time between customer arrivals) towardsdatascience
Log-Normal	Continuous	Mean, Std Dev (in log scale)	Right-skewed data (e.g., income, time-to-failure) arounddatascience
Student’s t	Continuous	Degrees of freedom	Small-sample inference, hypothesis testing when variance is unknown towardsdatascience

Why Distributions Matter in Data Science

Understanding the underlying distribution of your data is critical because it affects nearly every stage of analysis and modeling. github

1. Guides Exploratory Data Analysis (EDA)

Distributions help you:

Spot patterns that simple summary statistics (mean, median) might hide. dev
Detect outliers, data entry errors, and unexpected observations. github
Choose the right summary measures (mean vs median, standard deviation vs IQR). linkedin

2. Determines Statistical Assumptions

Many statistical tests and models assume specific distributions:

Normal distribution: Assumed by linear regression residuals, t-tests, ANOVA, and many parametric methods. towardsdatascience
Poisson distribution: Used for count-based models and event-rate analysis. towardsdatascience
Binomial distribution: Underpins binary classification and A/B testing. towardsdatascience

If these assumptions are violated, results can be misleading or invalid. arounddatascience

3. Influences Model Performance

Choosing the wrong distribution can lead to biased estimators, poor model fit, and inaccurate predictions. arounddatascience
Correct distribution choice improves model accuracy, interpretability, and robustness. arounddatascience

For example, applying linear regression to heavily right-skewed data without transformation often produces biased estimates and unreliable confidence intervals. arounddatascience

4. Shapes Feature Engineering and Preprocessing

Understanding distribution helps you:

Decide whether to apply log, square root, or Box-Cox transformations to normalize skewed data. towardsdatascience
Choose between standardization (mean=0, std=1) and normalization (min–max scaling). linkedin
Handle outliers appropriately (cap, remove, or model with heavy-tailed distributions). linkedin

How to Analyze Distributions in Practice

1. Visualize the Data

Start with simple plots to understand the shape of your data:

Histograms: Show the frequency of values across bins. youtube
Box plots: Highlight median, quartiles, and outliers. youtube
Q-Q plots: Compare your data to a theoretical distribution (e.g., Normal). youtube

2. Compute Descriptive Statistics

Key metrics that describe distribution shape:

Central tendency: Mean, median, mode. linkedin
Dispersion: Variance, standard deviation, interquartile range (IQR). linkedin
Shape:
- Skewness: Measures asymmetry (positive = right tail, negative = left tail). linkedin
- Kurtosis: Measures tail heaviness (high = more outliers, low = flatter distribution). linkedin

3. Test for Distribution Fit

Use statistical tests to check if your data matches a theoretical distribution:

Shapiro-Wilk: Tests for normality. youtube
Kolmogorov-Smirnov: Compares data to a reference distribution. youtube
Anderson-Darling: Another normality test, sensitive to tails. youtube

These tests help you decide whether parametric methods (e.g., t-tests) are appropriate or if non-parametric alternatives are needed. youtube

Practical Implications for Data Science Workflows

A/B Testing

Conversion rates typically follow a Binomial distribution.
Sample size calculations and p-values depend on correct distribution assumptions. towardsdatascience

Regression Models

Linear regression assumes normally distributed residuals with mean zero.
Violations lead to biased coefficients and unreliable confidence intervals. towardsdatascience

Anomaly Detection

Heavy-tailed or skewed distributions suggest more frequent outliers.
Understanding kurtosis helps you choose robust detection methods (e.g., IQR, isolation forests). linkedin

Time-to-Event Modeling

Exponential and Poisson distributions model event rates and inter-arrival times.
Useful in churn prediction, customer arrivals, or failure analysis. towardsdatascience

Monte Carlo Simulations

Simulations often start by generating random variables from a Uniform or Normal distribution.
Accurate simulations depend on realistic distribution choices. towardsdatascience

Common Mistakes and How to Avoid Them

Assuming normality blindly: Always check distribution shape with plots and tests before applying parametric methods. www3.cs.stonybrook
Ignoring skewness and outliers: Skewed data can bias mean and standard deviation; use median and IQR for robust summaries. linkedin
Using the wrong distribution for counts or rates: Apply Poisson or Binomial models instead of continuous ones for discrete events. towardsdatascience
Overlooking heavy tails: High kurtosis means more extreme values—use robust models or transformations. linkedin

Bottom Line

Statistical distributions are not just theoretical—they’re the foundation of every model, test, and insight in data science. They guide how you explore data, which methods you use, how you preprocess features, and how you interpret results. arounddatascience

Before building any model or running any test, always ask: “What is my data really telling me about its distribution?” linkedin

Python for Data Analytics: A Beginner’s Complete Guide

Jonathan kip — Fri, 07 Aug 2026 11:28:07 +0000

Python for Data Analytics: A Beginner’s Complete Guide

Python has become the go-to language for data analytics because it’s easy to learn, has powerful libraries, and works well with data of all sizes. This guide walks you through everything you need to know to get started—from setting up your environment to building your first data analysis project. realpython

Why Python for Data Analytics?

Python is popular for data analytics because:

Simple syntax: It’s beginner-friendly and reads like plain English.
Rich ecosystem: Libraries like pandas, NumPy, and Matplotlib handle everything from data cleaning to visualization.
Community and resources: Millions of tutorials, courses, and free projects make it easy to learn and troubleshoot.
Integration: Works well with databases, Excel, APIs, and big data tools.

Step 1: Set Up Your Python Environment

Option A: Anaconda (Recommended for Beginners)

Anaconda bundles Python, Jupyter, and data science libraries in one installer. ibm

Download Anaconda from anaconda.com/download. tableau
Run the installer and follow the default steps. tableau
Open Anaconda Navigator and launch Jupyter Notebook. tableau

Option B: Python + VS Code + Jupyter

Install Python from python.org/downloads. youtube
Install VS Code from code.visualstudio.com. youtube
Add the Python and Jupyter extensions in VS Code. ibm
Create a virtual environment to keep your packages organized. ibm

Install Essential Libraries

If you’re not using Anaconda, install key packages with pip: realpython

pip install pandas numpy matplotlib seaborn jupyter

Step 2: Learn Python Basics

Before diving into data analysis, get comfortable with Python fundamentals. w3resource

Variables and data types: Numbers, strings, booleans. w3resource
Data structures: Lists, dictionaries, tuples, sets. w3resource
Control flow: if statements, for and while loops. youtube
Functions: Reusable blocks of code with def. w3resource
List comprehensions: Compact way to create lists. youtube

Practice by writing small scripts—like a grade calculator or a text analyzer. tableau

Step 3: Core Libraries for Data Analytics

NumPy: Numerical Computing

NumPy is the foundation for numerical operations in Python. realpython

Creates and manipulates arrays (vectors and matrices).
Performs fast mathematical operations.
Used by pandas and other libraries under the hood.

import numpy as np

arr = np.array([1, 2, 3, 4])
print(arr.mean())  # 2.5

pandas: Data Manipulation

pandas is the most important library for data analysis. realpython

Loads data from CSV, Excel, SQL, and more.
Cleans, filters, and transforms data using DataFrames.
Summarizes and aggregates data with groupby operations.

import pandas as pd

df = pd.read_csv('sales.csv')
print(df.head())          # First 5 rows
print(df.describe())      # Summary statistics
print(df.isnull().sum())  # Missing values per column

Key pandas skills:

Data loading: read_csv(), read_excel(), read_sql(). pmc.ncbi.nlm.nih
Inspection: head(), info(), describe(). tableau
Cleaning: dropna(), fillna(), drop_duplicates(). pmc.ncbi.nlm.nih
Filtering: Boolean masks, loc, iloc. pmc.ncbi.nlm.nih
Aggregation: groupby(), pivot_table(). ibm

Matplotlib & Seaborn: Data Visualization

Matplotlib and Seaborn help you create charts and graphs. realpython

Matplotlib: Basic plotting (line, bar, scatter, histograms).
Seaborn: Beautiful statistical plots with less code.

import matplotlib.pyplot as plt
import seaborn as sns

sns.histplot(df['salary'], bins=20)
plt.show()

Common chart types:

Histograms: Distribution of a single variable.
Bar charts: Compare categories.
Scatter plots: Relationship between two variables.
Box plots: Show median, quartiles, and outliers. pmc.ncbi.nlm.nih

Step 4: The Data Analysis Workflow

A typical data analysis project in Python follows these steps: realpython

1. Define Your Objective

Ask a clear question: “What are the top-selling products?” or “Which factors affect customer churn?” realpython

2. Load and Inspect the Data

df = pd.read_csv('data.csv')
df.head()
df.info()
df.describe()

Check for missing values, data types, and overall structure. tableau

3. Clean the Data

Data cleaning often takes the most time. ibm

Handle missing values: Remove or fill with mean/median.
Remove duplicates: df.drop_duplicates().
Fix data types: Convert strings to dates, numbers to categories.
Treat outliers: Filter or cap extreme values. ibm

4. Explore and Analyze (EDA)

Use summary statistics and visualizations to understand patterns. ibm

Univariate analysis: Distribution of one variable.
Bivariate analysis: Relationships between two variables.
Correlation analysis: Which variables move together?
GroupBy operations: Compare metrics by category. ibm

5. Visualize Insights

Create charts that tell a clear story. ibm

Use Seaborn for clean, publication-ready plots.
Label axes, add titles, and keep charts simple.
Highlight key findings (e.g., top categories, trends over time).

6. Draw Conclusions and Communicate

Summarize your findings in a Jupyter notebook or report. ibm

What did you discover?
What business decisions could be made?
What are the limitations or next steps?

Step 5: Build a Beginner Project

Here’s a simple end-to-end project to add to your portfolio: youtube

Find a dataset on Kaggle (e.g., sales, salaries, or customer reviews). tableau
Load and explore the data with head(), describe(), and isnull(). tableau
Clean the data: Remove or fill missing values, fix data types, drop duplicates. tableau
Analyze: Calculate averages, medians, and distributions. Identify outliers. tableau
Visualize: Create at least 3 charts that tell a clear story. tableau
Conclude: Write a short summary of your findings and potential business actions. tableau

Step 6: Learning Roadmap (2026)

If you’re planning your learning path, here’s a suggested timeline: ibm

Month 1–2: Python basics + NumPy + pandas fundamentals
Month 3: Data cleaning, EDA, and visualization (Matplotlib/Seaborn/Plotly)
Month 4: Advanced pandas (merge, pivot, groupby) + performance tools like Polars/DuckDB
Month 5: Statistics, hypothesis testing, and basic Scikit-learn
Month 6+: Build end-to-end projects, integrate SQL, and create a portfolio ibm

With 1–2 hours of daily practice, you can reach a functional level in 3–4 months and be job-ready in 6–12 months. tableau

Next Steps

Follow free courses on YouTube (e.g., freeCodeCamp, Simplilearn, Luke Barousse). w3resource
Practice on real datasets from Kaggle or Google Dataset Search. tableau
Share your projects on GitHub and LinkedIn to build a portfolio. youtube

SQL Subqueries vs CTEs: Types, Differences, Performance, and When to Use Each

Jonathan kip — Fri, 07 Aug 2026 11:18:21 +0000

SQL Subqueries vs CTEs: Types, Differences, Performance, and When to Use Each

Subqueries and Common Table Expressions (CTEs) are two of the most common ways to break complex SQL logic into smaller, reusable pieces. While they often produce the same results, they differ in readability, reusability, and—in some cases—performance. learnsql

What Are Subqueries?

A subquery is a query nested inside another query. It can appear in the SELECT, FROM, WHERE, or HAVING clauses. learnsql

Types of Subqueries

Non-correlated subqueries: Run independently once, then return results to the outer query. dev
Correlated subqueries: Reference columns from the outer query and run once per row—often slower on large datasets. dev

Example:

-- Non-correlated: find employees earning more than average salary
SELECT name, salary
FROM employees
WHERE salary > (SELECT AVG(salary) FROM employees);

-- Correlated: find employees earning more than their department average
SELECT e.name, e.salary, e.department
FROM employees e
WHERE e.salary > (
    SELECT AVG(salary) FROM employees WHERE department = e.department
);

What Are CTEs?

A Common Table Expression (CTE) is a named temporary result set defined using the WITH clause. It exists only for the duration of the query and can be referenced multiple times. learnsql

Example:

WITH avg_salary AS (
    SELECT AVG(salary) AS avg_sal FROM employees
)
SELECT name, salary
FROM employees, avg_salary
WHERE salary > avg_sal;

Key Differences Between Subqueries and CTEs

Feature	Subqueries	CTEs
Definition location	Inline, inside the main query	Defined at the top with `WITH`
Naming	Not required (except in PostgreSQL)	Must be named
Reusability	Can be used only once per occurrence	Can be referenced multiple times in the same query
Readability	Can become hard to read when deeply nested	Breaks logic into named, readable steps
Use in `WHERE` with `IN`/`EXISTS`	Yes	No (CTEs must be joined or referenced differently)
Recursive queries	Not supported	Supported via recursive CTEs
Performance (modern DBs)	Often identical to CTEs	Often identical to subqueries; older PostgreSQL materialized CTEs

Performance: Subqueries vs CTEs

The Short Answer

In modern databases (PostgreSQL 12+, MySQL 8+, SQL Server 2019+, Snowflake, BigQuery), there is typically no performance difference between equivalent subqueries and CTEs. The query optimizer treats them similarly and often generates identical execution plans. dev

When Performance Can Differ

Older PostgreSQL versions (<12): CTEs were always materialized (computed once and stored), which could make them slower than inlined subqueries. dev
Repeated logic: If the same subquery appears multiple times, the database may recompute it each time, whereas a CTE is computed once and reused. dev
Correlated subqueries: These can be slow on large tables because they run once per row, but modern optimizers sometimes rewrite them as joins. dev

What Actually Affects Performance

Performance depends more on:

Indexing and join conditions
Data size and distribution
Execution plan chosen by the optimizer
Partitioning and statistics

—not just whether you use a subquery or CTE. linkedin

When to Use Subqueries

Use a subquery when:

You need a single value or quick filter (e.g., WHERE salary > (SELECT AVG(salary) ...)). learnsql
The logic is short, simple, and used only once. datawithsarah
You’re using IN or EXISTS in a WHERE clause. learnsql
You’re writing a quick ad-hoc query where readability isn’t critical. dev

When to Use CTEs

Use a CTE when:

Your query is long or has multiple logical steps. learnsql
You need to reference the same intermediate result multiple times. learnsql
You want to improve readability by naming each step. datawithsarah
You’re writing recursive queries (e.g., hierarchical data like org charts). learnsql
You’re building production-level queries or ETL pipelines where maintainability matters. datawithsarah

Practical Example: Subquery vs CTE

Subquery version:

SELECT name, salary
FROM employees
WHERE salary > (
    SELECT AVG(salary) FROM employees
)
AND department IN (
    SELECT department FROM departments WHERE location = 'New York'
);

CTE version (more readable):

WITH avg_salary AS (
    SELECT AVG(salary) AS avg_sal FROM employees
),
ny_departments AS (
    SELECT department FROM departments WHERE location = 'New York'
)
SELECT e.name, e.salary
FROM employees e
JOIN ny_departments nd ON e.department = nd.department
CROSS JOIN avg_salary
WHERE e.salary > avg_sal;

Bottom Line

Subqueries are great for simple, one-off logic and inline filters. datawithsarah
CTEs shine when queries get complex, need multiple steps, or require reuse. learnsql
Performance is usually similar in modern databases—focus on readability and maintainability. dev

Choose the tool that makes your SQL clearer and easier to maintain, and always check the execution plan if performance is a concern.

How to Publish a Power BI Report and Embed it into a Website

Jonathan kip — Fri, 07 Aug 2026 11:10:32 +0000

How to Publish a Power BI Report and Embed It into a Website

Publishing a Power BI report and embedding it on a website is a straightforward process that lets you share interactive dashboards with anyone online. Below is a step-by-step guide covering both public and secure embedding options. learn.microsoft

Step 1: Prepare and Publish Your Report to Power BI Service

Before you can embed a report, it must be published to the Power BI cloud service.

Open your report in Power BI Desktop

Make sure your .pbix file is complete and saved. dev
Click Publish

In the Home tab, click the Publish button. dev
Sign in and select a workspace

If prompted, sign in to your Power BI account, then choose the workspace where you want to publish the report. dev
Confirm the upload

Once published, you’ll see a success message with a link to open the report in the Power BI service. dev

Step 2: Generate the Embed Code

You have two main options depending on whether you want the report to be publicly accessible or restricted to authenticated users.

Option A: Publish to Web (Public)

Use this if you’re okay with anyone on the internet viewing your report—ideal for portfolios, blogs, or public dashboards. learn.microsoft

Open your report in Power BI Service

Go to app.powerbi.com, navigate to your workspace, and open the report. learn.microsoft
Click File > Embed report > Publish to web (public)

This opens the embed dialog. learn.microsoft
Review the warning

Power BI will warn you that the report will be public. Confirm only if your data is safe to share. learn.microsoft
Create embed code

Click Create embed code, then copy the <iframe> HTML snippet provided. dev

Option B: Secure Embed (Website or Portal)

Use this for internal dashboards where viewers must log in to Power BI. learn.microsoft

Open your report in Power BI Service

Navigate to your workspace and open the report. learn.microsoft
Click File > Embed report > Website or portal

This opens the secure embed dialog. learn.microsoft
Copy the link or iframe code

You can either copy the embed link or the <iframe> HTML snippet to paste into your site. learn.microsoft

Step 3: Embed the Report on Your Website

Now that you have the embed code, you can add it to your website.

For Custom HTML Sites

Open your HTML file

Use a code editor like VS Code or your CMS editor. dev
Paste the iframe code

Insert the <iframe> snippet inside the <body> section where you want the report to appear. dev
Save and publish

Upload the updated file to your web server or publish the changes in your CMS. dev

For WordPress or Other CMS Platforms

Open the page editor

Go to the page where you want the report. dev
Add a Custom HTML block

In WordPress, insert a “Custom HTML” block. dev
Paste the iframe code

Drop the code into the block and update or publish the page. dev

Important Considerations

Data privacy: Only use “Publish to web” if your report contains no sensitive or confidential data. learn.microsoft
Admin settings: Your organization’s Power BI admin may need to enable “Publish to web” in the admin portal. youtube
Responsiveness: You can adjust the width and height attributes in the iframe to fit your site’s layout. learn.microsoft

With these steps, your Power BI report will be live and interactive on your website, letting visitors explore your data in real time. dev

Machine Learning: The Invisible Helper

Jonathan kip — Fri, 07 Aug 2026 10:55:37 +0000

Machine Learning: The Invisible Helper

You probably use machine learning dozens of times a day without even realizing it. From unlocking your phone with your face to getting uncannily accurate music recommendations, machine learning is the quiet intelligence powering the apps and devices you rely on every day.

What Is Machine Learning

Machine learning is a type of artificial intelligence that lets computers learn from experience instead of being programmed with rigid rules. Instead of giving the computer step-by-step instructions, we feed it lots of examples—like photos, emails, or songs—and let it figure out the patterns on its own. youtube

How Machine Learning Works: A Simple Breakdown

You can think of machine learning as a four-step process that happens behind the scenes of your favorite apps. youtube

1. Collect and Prepare Data

Everything starts with data—numbers, photos, text, or sensor readings. For example, Netflix gathers data on what you watch, when you pause, and what you rate. This data is cleaned and organized so the computer can learn from it. youtube

2. Train the Model

A “model” is the brain the computer builds from the data. Programmers choose an algorithm (a learning method) and feed it the training data. The algorithm sifts through the examples, looking for patterns—like which words often appear in spam emails or which movies people who like action films also enjoy. youtube

3. Test and Improve

Before using the model in real life, it’s tested on new data it hasn’t seen before. If it makes mistakes, engineers tweak it—adjusting settings or feeding it more data—until it gets better. This is why your recommendations get smarter the more you use an app. medium

4. Make Predictions

Once trained, the model is deployed to make predictions or decisions on new inputs. When a new email arrives, the spam filter uses its learned patterns to guess if it’s junk. When you open Spotify, the recommendation engine predicts which songs you’ll like next. youtube

Everyday Things You Use That Rely on Machine Learning

Your Smartphone

Face unlock: The phone learns your facial features from many photos to securely unlock when it sees you.
Photo organization: Apps like Google Photos group pictures by person or pet by learning visual patterns.
Predictive text and autocorrect: Your keyboard learns your typing style to suggest words and fix mistakes.
Voice assistants: Siri and Google Assistant use ML to understand your spoken commands and respond.

Streaming and Entertainment

Netflix and YouTube recommendations: These platforms analyze your watch history and compare it to millions of other users to suggest content. medium
Spotify and Apple Music playlists: “Discover Weekly” and “For You” mixes are built using ML that learns your listening habits.
Social media feeds: Facebook, Instagram, and TikTok use ML to decide which posts and ads appear in your feed.

Email and Messaging

Spam filters: Gmail learns from emails labeled as “spam” or “not spam” to spot suspicious messages. medium
Smart replies: Gmail suggests quick responses like “Thanks!” or “Sounds good” based on the email’s content.
Auto-transcription: Apps like Slack and YouTube generate captions and transcriptions of spoken words.

Shopping and Online Services

Product recommendations: Amazon suggests items based on your browsing and purchase history.
Delivery time estimates: Apps predict when your food or package will arrive using traffic and order data.
Chatbots: Customer service bots on websites handle routine questions using ML-powered natural language processing.

Maps and Navigation

Google Maps and Waze: These apps predict traffic jams, estimate travel times, and suggest the fastest routes.
Ride-hailing apps: Uber and Lyft use ML to match drivers with riders and set dynamic pricing.

Banking and Payments

Fraud alerts: Your bank flags unusual transactions by learning your normal spending patterns.
Credit card approvals: ML models assess risk when you apply for loans or credit cards.

Health and Fitness

Step and workout tracking: Fitness apps learn your activity patterns to give personalized goals.
Health insights: Wearables detect irregular heartbeats or sleep issues using ML on sensor data.

Why It Matters to You

Machine learning makes your devices smarter, your apps more personalized, and your daily tasks easier. It’s the reason your phone recognizes your face, your music app knows your mood, and your bank stops fraud before it hits your account. In short, machine learning is the invisible helper that quietly powers the technology you use every single day.

Understanding Data Modeling in Power BI: Joins, Relationships, and Schemas

Jonathan kip — Sun, 29 Mar 2026 11:59:55 +0000

Power BI is a platform that enables users to connect to various data sources, transform data, create interactive visualizations, and share insights through dashboards and reports.

Data modeling refers to the process of defining how tables connect, interact, and filter each other to enable accurate calculations, fast performance, and intuitive reporting.

SQL JOINS

INNER JOIN-Returns matching rows from both tables.
LEFT JOIN-All rows from left table, matching from right, NULLs for no match.
RIGHT JOIN-All rows from right table, matching from left, NULLs for no match.
FULL OUTER-All rows from both, NULLs where no match.
LEFT ANTI-Rows in left table with no match in right.
RIGHT ANTI-Rows in right table with no match in left.

POWERBI RELATIONSHIP
Power BI relationships connect tables in the data model to enable dynamic filtering and accurate aggregations across visuals without physically merging data.

Relationship Cardinality
Cardinality defines how many rows in one table match rows in another.

1:1 (One-to-One): Each row in Table A matches exactly one row in Table B (e.g., Employee to EmployeeDetails via unique EmployeeID); rare, used for normalized data or security filtering.

1:M (One-to-Many): One row in Table A matches multiple in Table B (e.g., Product to Sales via ProductID); standard for star schemas.

M:M (Many-to-Many): Multiple rows in both tables match (e.g., Students to Classes); avoid direct M:M—instead use a bridge table for performance and clarity.

Active vs Inactive Relationships
Active: Default, single relationship per table pair; filter context propagates automatically in visuals and DAX (solid line).

Inactive: Additional relationships (dashed line); ignored by default—activate via USERELATIONSHIP() DAX function for specific measures (e.g., role-playing dates).

Cross-Filter Direction
Controls filter propagation:

Single: Filters flow one way (default for 1:M, e.g., Dimension → Fact).

Both: Bi-directional (e.g., Fact ↔ Dimension); use cautiously to avoid ambiguity or performance hits.

The difference between joins and relationships

Fact vs Dimension Tables
Fact tables store quantitative metrics (e.g., SalesAmount, Quantity) with foreign keys to dimensions; they grow vertically with transactions.

Dimension tables hold descriptive attributes (e.g., ProductName, CustomerCity) with hierarchies; they grow horizontally and less frequently.

Star Schema: Central fact table surrounded by denormalized dimension tables; simplest and fastest for Power BI queries.
Use Case: Sales dashboards needing quick slicing by product/customer.

Snowflake Schema: Dimensions normalized into sub-tables (e.g., Product → Category); saves storage but slower joins.
Use Case: Data warehouses with complex hierarchies; avoid in pure Power BI models.

Flat Table (DLAT): Single denormalized table blending facts/dimensions; easy but inefficient for filtering.
Use Case: Small, simple datasets before unpivoting into star schema.

Role-Playing Dimensions
A single dimension (e.g., Date table) connects to a fact table multiple times for different roles (OrderDate, ShipDate). Use inactive relationships + USERELATIONSHIP() DAX to activate contextually.

Common Modeling Issues
Circular dependencies: Resolved by star schema or inactive relationships.
Bi-directional filter loops: Set single direction.
Wrong cardinality: Causes incorrect aggregations; verify in properties.
Ambiguous paths: Hide unused columns or use bridge tables.