In 2001, Paul Viola and Michael Jones described a face detector that ran at 15 frames per second on 384×288 images using a 700 MHz Pentium III (Viola & Jones, CVPR 2001). Roughly a quarter-century later, Meta's 2025 SAM 3 takes the phrase "yellow school bus" and returns masks for the object instances it matches to that concept across an image or video (Carion et al., SAM 3, 2025).
If you build things, the interesting gap between those two systems is not faster chips. It is where the visual knowledge is stored, and who or what put it there. That single question — where does the model keep what it knows? — is the cleanest way I know to hold the whole history in your head.
For a long time, the intelligence lived in the head of the engineer who designed the features. Then it moved into convolutional filters learned from labeled examples. Then into deeper architectures and scaling rules, into transformer attention, and finally into large pretrained models that connect pixels to regions, language, prompts, and video. Each move changed the hard part of the job. None of them retired the old problems of data quality, geometry, evaluation, and knowing when a system is about to be wrong.
Era 1: engineers wrote the features down by hand
Many classical pipelines turned pixels into human-designed signals — edges, corners, gradients, contrast patterns, local descriptors — before any matching or classification happened. Most of the design work lived in feature extraction, not in the final decision rule.
The roots go back further than the deep-learning timeline usually admits. Marr and Hildreth modeled edges as zero-crossings after Laplacian-of-Gaussian filtering in 1980 (Marr & Hildreth, 1980); Canny formalized edge detection in 1986 (Canny, 1986); Harris and Stephens gave us a corner detector in 1988 (Harris & Stephens, 1988). And convolution-flavored hierarchies predate 2012 by decades: Fukushima's 1980 neocognitron used alternating shift-tolerant S- and C-cell stages (Fukushima, 1980). It anticipated ideas central to CNNs, though it was not a modern backprop-trained network.
There is a debugging lesson buried in this era that still applies. When a hand-designed feature extractor discarded information the task needed, a downstream classifier working only from that representation could not get it back. So engineers had to reason about invariance up front — scale, rotation, lighting, occlusion — because nothing downstream would rediscover what the front end threw away.
Between 2001 and 2006, four methods turned that philosophy into practical systems:
- Viola–Jones (2001) paired an integral image, AdaBoost feature selection, and a rejection cascade so most of the image failed cheap tests early (paper). The 15 fps figure is a historical result on historical hardware, not a modern benchmark.
- SIFT (2004) built keypoints designed for scale and rotation invariance and robustness to viewpoint, noise, lighting, clutter, and partial occlusion (Lowe, IJCV 2004). The paper claims robust and partial invariance, not perfect invariance. SIFT first appeared in a 1999 ICCV paper; the 2004 journal article is the mature four-stage version.
- HOG (2005) described local shape with dense, locally normalized gradient-orientation histograms, evaluated on pedestrian detection (Dalal & Triggs, CVPR 2005).
- SURF (2006) chased SIFT's invariance goals with speed as an explicit target (Bay et al., ECCV 2006). It is not universally better than SIFT; the two trade off by workload.
Calling these "primitive AI" gets it backwards. They were the moment the field learned to state its assumptions out loud.
Era 2: CNNs changed who designed the representation
The convenient legend is that CNNs arrived in 2012 and swept the old methods away. Both halves need qualifying.
CNNs already worked long before 2012 — LeCun and colleagues described gradient-based learning with convolutional features for document recognition in 1998 (LeCun et al., 1998). And it helps to keep three ImageNet milestones separate, because they get merged constantly: the dataset paper (5,247 synsets, 3.2M images) appeared in 2009 (Deng et al., CVPR 2009); the annual ILSVRC challenge began in 2010 (Russakovsky et al.); and the result that turned heads landed in 2012.
That result was AlexNet: 15.3% top-5 test error versus 26.2% for the runner-up (Krizhevsky, Sutskever & Hinton, NeurIPS 2012). Two cautions ride along with that number. It is a competition-system result on one benchmark, not a value to line up against later scores from other tasks and protocols. And the win was a convergence — a large-capacity CNN, a GPU implementation, ReLUs, dropout, and 1.2 million labeled images — not the invention of convolution.
The lasting consequence was a change in job description. Instead of hand-designing a feature hierarchy, practitioners increasingly specify data, objectives, architecture, and optimization, and let the hierarchy emerge through training. Representation design did not vanish. It moved.
Once a network produced rich features, people asked what else those features could do. "What's in this picture?" is one question; here is a compact taxonomy of the tasks that followed, each with a primary source that produced that output type:
- Classification assigns an image-level category (AlexNet).
- Detection predicts object categories with locations (YOLO).
- Semantic segmentation assigns classes at pixel level (U-Net).
- Instance segmentation separates masks for individual object instances (Mask R-CNN).
The detection lineage is a clean case study in removing duplicated work. R-CNN transferred CNN features into localization but ran the network many times over overlapping regions (Girshick et al., 2014). Fast R-CNN computed one image-level feature map and pooled regions from it (Fast R-CNN, 2015) — though its headline inference time excluded external proposal generation. Faster R-CNN learned the proposals on the shared features (Ren et al., 2015). Three papers, one direction: stop recomputing the same thing. YOLO took the other route, regressing straight from the full image to boxes and class probabilities in a single pass, trading some localization precision on small or clustered objects for speed (Redmon et al., 2015/2016). That was the original YOLO; later versions changed substantially, so don't read the first paper's numbers as describing anything that shipped afterward.
Era 3: depth broke, and then it didn't
If a few conv layers were good, more should be better. Reality disagreed.
VGG argued for depth from discipline — stacks of small 3×3 convolutions whose representations transferred well (Simonyan & Zisserman, 2015). Inception widened under a compute budget with parallel multi-scale branches and 1×1 projections (Szegedy et al., 2015); its famous 6.67% top-5 came from an ensemble of seven models with 144 crops per image, not one model on one crop.
Then ResNet named the real problem. In its experiments, deeper plain networks had higher training error than shallower ones — the degradation problem, which the authors explicitly distinguish from overfitting. Residual learning fixed it by having each block learn a residual relative to its input, with an identity shortcut carrying the input forward (He et al., 2016). Worth getting right, because the shorthand does damage: the paper does not claim "skip connections solved vanishing gradients." It notes normalization and initialization had already helped convergence, and frames the contribution as making very deep networks (up to 152 layers) optimizable at all. EfficientNet later treated accuracy and efficiency as a joint problem, balancing depth, width, and resolution with one compound coefficient (Tan & Le, ICML 2019) — the balance is the idea, and its reported numbers are tied to the authors' specific setup, not a current leaderboard.
Era 4: transformers, language, and promptable segmentation
DETR reframed detection as direct set prediction: a transformer encoder-decoder with learned object queries and a set-matching loss, which removed several hand-designed components like anchors and non-maximum suppression (Carion et al., 2020). It kept a convolutional backbone and had slow-convergence and small-object limits; the contribution was the formulation. ViT then fed raw image patches as tokens to a near-standard transformer (Dosovitskiy et al., 2021). Its strongest results depended heavily on large-scale pretraining — the paper says so plainly. ViT did not prove attention is inherently smarter than convolution, and it did not retire CNNs; it offered different priors and a different scaling path that paid off where that path fit.
Then the label space stopped being a fixed list. CLIP aligned images and text from 400 million web pairs in a shared embedding space, enabling classification against categories described in words (Radford et al., 2021). "Zero-shot" here means no task-specific gradient update for new categories — not learning from nothing. The same paper documents prompt sensitivity, uneven fine-grained performance, and inherited web bias. CLIP is an alignment and embedding model, not a guarantee of correct visual reasoning. DINOv2 is a useful counterweight: general-purpose visual features from self-supervised ViT training on curated data, no captions at all (Oquab et al., 2023). And Flamingo (Alayrac et al., 2022) and LLaVA (Liu et al., 2023) connected vision encoders to language models so you could prompt and instruct them in plain language.
Here is the part to tattoo on your deployment checklist: fluent multimodal answers are not proof of grounded perception. A language interface can hide uncertainty rather than remove it, so an answer still has to be checked against the underlying visual task and evidence. Articulate output is not verification.
Segmentation changed its contract in parallel. The original Segment Anything introduced promptable masks from points, boxes, or a rough mask — not unrestricted text — trained on SA-1B, reported as 1B+ masks over 11M licensed, privacy-respecting images (Kirillov et al., ICCV 2023). SAM 2 added streaming memory to track prompted objects through video (Ravi et al., 2024). SAM 3 added concept prompts — noun phrases and visual exemplars — unifying detection, segmentation, and tracking, with a data engine built around 4 million unique concept labels (Carion et al., 2025). As of this article's research cutoff, the most recent official step is SAM 3.1: Meta's repository records a March 27, 2026 release adding shared-memory joint multi-object tracking, intended to improve speed without sacrificing accuracy (release note; Meta blog). Two things are true at once: it is an official project-team release, and it is a vendor claim rather than an independent evaluation.
The caution generalizes across the whole family. A mask is a claim about which pixels belong together. It is not, by itself, a verified statement about what the object is or what it means in the scene, and video tracking stays workload-dependent as scenes and visibility change. Segmentation solved a real problem. It did not quietly solve perception.
The takeaway: the bottleneck keeps moving
Read across the timeline and one pattern holds. The hardest part of building a vision system keeps relocating:
- Classical: inventing robust features by hand.
- Deep learning: getting enough labeled data and GPU compute to train large networks.
- Deeper networks: optimizing depth and efficiency so models trained at all and ran in budget.
- Foundation models: pretraining reusable representations and promptable interfaces that many tasks share.
And one methodological warning has to travel with all of it: the accuracy and speed numbers across these papers are not points on a single curve. ImageNet top-5 error, VOC mAP, COCO mask AP, segmentation IoU, and fps measure different tasks on different datasets with different protocols and hardware, sometimes single models and sometimes big ensembles. This is a synthesis across the cited primary papers, not a result from any one of them. Lining them up as one rising line tells a satisfying story and a dishonest one.
The deepest continuity is the one that's easy to miss: modern systems are full of hand-designed choices. They just live in the data curation, the training objectives, the augmentation, the prompt design, the architecture, and the evaluation protocol rather than in a gradient operator. The engineer who once wrote down what an edge was now writes down what counts as a good training example, which concepts the data engine should cover, and which benchmark actually reflects the deployment. The judgment did not leave. It changed address — which, if you maintain these systems, is either reassuring or ominous depending on the day.
Sources link inline to primary papers, official repositories, and project pages. Benchmark figures are historical and specific to their original tasks, datasets, and hardware, and are not directly comparable across eras. A longer technical-history edition with the full milestone timeline is linked at the canonical URL above.
Top comments (0)