Fundamentals of Computer Vision

Computer vision is a discipline within artificial intelligence aimed at enabling computers to process visual information and take actions according to their analysis. An image is simply a collection of numbers for a computer. Computer vision is the body of techniques that help convert all these numbers into an interpretation of the objects, people, movements, depth, and meaning contained in the scene.

This article explores some of the essential components of the domain, from image formation itself to the way in which modern deep learning-based models understand the content of the image.

What Is Computer Vision?

Ultimately, computer vision is concerned with the problem of interpreting visual information. Humans are naturally able to do this by virtue of years of experience with the world, but computers must learn to interpret visual input.

Computer Vision vs. Image Processing

Here, it is necessary to demarcate the boundaries clearly. The technique of image processing converts one image into another, making it sharper, noise-free, and contrast-enhanced. Computer vision moves beyond image processing; it seeks to analyze the content of the image, for instance, "there is a car in this picture" or "this X-ray shows a fracture." Image processing may simply be a preprocessing phase of computer vision.

Where It Fits in AI

There are numerous areas of knowledge, like artificial intelligence, machine learning, mathematics specifically linear algebra and geometry, signal processing, and, lately, deep learning based on neuroscience, that fall under this field. However, due to its multidisciplinary nature, computer vision might appear quite complex initially, yet the basic concepts are quite simple.

Real-World Applications

Computer vision has moved from research labs into everyday products. A few of the most common domains:

Healthcare — tumor detection in MRI/CT scans, automated pathology, AI-guided surgery.

Automotive — lane detection, pedestrian recognition, full autonomous driving stacks.

Manufacturing — visual defect inspection, quality control on production lines.

Retail — automated checkout, shelf-stock monitoring, visual search.

Smartphones — face unlock, portrait mode, QR code scanning, panorama stitching.

Security — surveillance analytics, facial recognition, anomaly detection.

Seeing this breadth helps explain why computer vision curricula cover such a wide range of topics — the underlying techniques are shared across wildly different use cases.

Math & Programming Prerequisites

You don't need a PhD to start learning computer vision, but a working comfort with a few areas of math makes the concepts click faster.

The Math You'll Actually Use

Linear algebra — vectors, matrices, and transformations underpin image representation and camera geometry

Probability and statistics — needed for noise models, evaluation metrics, and probabilistic reasoning

Calculus — gradients and optimization show up constantly, especially in deep learning

Tools of the Trade

The programming language that has become the standard in the field, due to tools such as OpenCV, NumPy, and PyTorch, is Python. Basic knowledge of any programming language should be sufficient to get started; knowledge of CV concepts is what is important. As with most computer vision development work, fluency with these tools matters less than a solid grasp of the underlying concepts they implement.

How Images Are Formed and Represented

For the computer to "see" anything, there must first be light gathered and then translated into a digital signal. This is where the camera comes into play; light enters the camera lens, lands on an image sensor, and then is transformed into numbers, which are called pixels.

Pixels and Color

Each pixel contains intensity data. The value of the intensity for a grayscale image is a single number, indicating the brightness of the pixel. In the case of color images, each pixel normally holds three numbers — red, green, and blue — that are used together to cover the entire visible spectrum. Some specific applications use multispectral or hyperspectral images, covering out-of-spectrum wavelengths.

Resolution and File Formats

Resolution — the number of pixels in an image — determines how much detail is captured. File formats like JPEG, PNG, and TIFF determine how that pixel data is compressed and stored:

JPEG — lossy compression, well suited for photographs

PNG — lossless, supports transparency

TIFF — high quality, common in professional and scientific imaging

Camera Characteristics That Matter Later

Focal length, aperture, exposure, and sensor size all affect how a scene translates into pixel data. These properties become critical later, when dealing with camera calibration and 3D reconstruction.

Color Spaces

Not all color representations are created equal for computer vision tasks.

RGB, familiar from displays, encodes color as a mix of red, green, and blue. It's intuitive, but not always ideal for analysis, because brightness and color information are tangled together.

HSV (hue, saturation, value) separates color identity from intensity, which makes it much easier to do things like color-based segmentation — isolating everything "red" in an image regardless of lighting conditions.

LAB goes further, separating lightness from color channels in a way that closely mirrors human perception. It's often preferred for color-thresholding tasks where consistent segmentation across lighting conditions matters.

Choosing the right color space is a small decision that can dramatically simplify downstream processing — it's one of the first practical skills learners pick up when working hands-on with OpenCV.

Image Processing Fundamentals

Once an image is captured, it usually needs to be cleaned up or enhanced before further analysis. This is the domain of classical image processing.

Filtering and Convolution

A filter (or kernel) is a small matrix slid across the image to compute a new value for each pixel based on its neighbors. Depending on the kernel, this can blur an image (smoothing, useful for noise reduction), sharpen it, or highlight edges. Convolution is the math operation that makes this possible — and it's also the foundation of convolutional neural networks used later in deep learning.

Morphological Operations

Operations like erosion and dilation work on binary images to remove small noise artifacts, separate touching objects, or fill small holes in detected shapes. These are especially useful in industrial inspection and document analysis.

Thresholding

Thresholding converts a grayscale image into a binary one by classifying each pixel as foreground or background based on intensity. It's a simple but powerful first step toward segmentation, especially in controlled environments with consistent lighting.

Feature Detection & Extraction

A "feature" in computer vision is a distinctive, identifiable part of an image — an edge, a corner, a blob, or a specific texture pattern. Feature detection matters because it's far more efficient to match and compare a handful of distinctive points between two images than to compare every pixel.

Edges and Corners

Edge detection algorithms identify sharp changes in intensity, typically marking the boundaries of objects. Corner detection finds points where two edges meet — points that tend to be stable and easy to re-identify across different images of the same scene.

Feature Descriptors

More sophisticated descriptors — like SIFT (Scale-Invariant Feature Transform), ORB (Oriented FAST and Rotated BRIEF), and HOG (Histogram of Oriented Gradients) — encode a feature's local appearance in a way that's robust to changes in scale, rotation, and lighting. These are what allow two photos of the same object, taken from different angles, to be matched reliably — a capability underlying panorama stitching, object tracking, and 3D reconstruction.

Connected Component Analysis

This groups adjacent pixels that share properties (like being part of the same detected object) into labeled regions — a key step before counting, measuring, or classifying objects in an image.

Image Segmentation

Segmentation divides an image into meaningful regions — separating a foreground object from its background, or splitting a medical scan into distinct anatomical structures.

Classical Segmentation

Traditional methods rely on thresholding, clustering (like k-means on pixel colors), or edge-based region growing. These work well in controlled, predictable environments but tend to struggle with complex, cluttered real-world scenes.

Learning-Based Segmentation

Modern segmentation increasingly relies on deep learning models trained to output a pixel-by-pixel classification map, identifying exactly which class each pixel belongs to. This is the technology behind features like background removal in video calls and organ segmentation in medical imaging.

Camera Geometry & 3D Vision

A single 2D image discards a huge amount of information: depth. Recovering the 3D structure of a scene from 2D images is one of the more mathematically rich areas of computer vision.

Calibration and Stereo Vision

The camera calibration process involves figuring out the intrinsic properties of the camera (focal length and distortion in the lens), ensuring that any measurement done from the image can be trusted. The stereo vision approach involves using two cameras in order to achieve a human-like binocular effect.

Structure from Motion

This technique reconstructs 3D scene geometry from a sequence of 2D images captured from different positions — the same principle that powers 3D scanning apps and some autonomous vehicle perception systems. Underlying all of this is epipolar geometry, the mathematical relationship between two camera views of the same scene, which constrains where a point in one image can appear in the other.

Motion Analysis & Tracking

Many computer vision applications deal with video, not static images, which introduces the dimension of time.

Optical Flow

Optical flow estimates the apparent motion of pixels between consecutive frames, revealing how objects (or the camera itself) are moving through the scene.

Object Tracking

From this, tracking extends to tracking a particular object across multiple frames despite variations in the size or position of the object, or its temporary occlusion. This is key in applications such as sports analysis, surveillance, and autonomous navigation systems where the direction an object is traveling is just as important as its identity.

Object Recognition & Detection

This is often what people picture when they think of "computer vision" — a system that can look at an image and say what's in it.

Image classification is the task of assigning a label to a complete image (for example, "this is a cat"). On the other hand, object detection takes it a step ahead and identifies and labels several objects in a single image by placing a box around it. They constitute the building blocks of several applications, including photo labeling, and also mark the introduction of deep learning for most students.

Deep Learning for Computer Vision

It is the key reason behind the sudden improvement in the performance of computer vision during the last decade and is still being used in current state-of-the-art computer vision systems.

How a CNN Builds Understanding

A CNN applies learned convolutional filters across an image, building up increasingly abstract representations layer by layer:

Early layers tend to detect edges

Middle layers tend to detect shapes or textures

Deeper layers tend to detect entire objects or faces

This shift, powered by large labeled datasets and GPU-accelerated training, drove the dramatic jump in computer vision performance over the past decade and remains the dominant paradigm for state-of-the-art systems today.

Evaluation Metrics

Building a model is only half the job — you need to know how well it actually performs.

Precision and recall — how many detections were correct, and how many actual objects were found

mAP (mean Average Precision) — a standard benchmark for object detection quality

AUC (Area Under the Curve) — summarizes classifier performance across thresholds

FAR/FRR (False Accept/Reject Rate) — critical in biometric systems like facial recognition

Choosing the right metric depends heavily on the application. A medical diagnosis system and a photo-tagging app tolerate very different kinds of errors.

Ethical Considerations

Computer vision systems don't operate in a vacuum, and their limitations carry real consequences.

Facial recognition and surveillance systems have documented accuracy disparities across demographic groups, raising fairness concerns. Widespread image and video capture also raises legitimate questions about privacy, consent, and data ownership — especially in healthcare, security, and social media contexts. Anyone building or deploying computer vision systems should treat bias auditing and privacy safeguards as core requirements, not afterthoughts.

Tools, Libraries & Learning Resources

Libraries

OpenCV — the most widely used open-source library for classical computer vision tasks

PyTorch — dominant framework for deep learning-based vision work

Reference Material

For deeper theoretical grounding, Szeliski's Computer Vision: Algorithms and Applications and Snyder & Qi's Fundamentals of Computer Vision are frequently recommended academic references, alongside university courses from institutions like Columbia, CMU, and Berkeley that offer structured, rigorous introductions to the field.

Getting Started: A Practical Learning Path

A sensible progression looks like this:

Build comfort with the math prerequisites

Learn image representation and basic processing operations hands-on with OpenCV

Work through feature detection and classical techniques

Move into CNNs and modern deep learning architectures once the fundamentals feel solid

There's an obvious temptation to jump straight into deep learning; however, the classical theories will make it much easier to know why the current models work that way and, more importantly, how to debug them if they don't work that way.

Computer vision may seem to be a big field, but it is based on a relatively small number of key theories: the formation and representation of images, their processing and feature extraction, 3D recovery and understanding of motions, and training of computer vision systems to recognize and label objects.