π How AI Got Smarter (Not Just Bigger): The Quiet Revolution in Efficient Deep Learning
Five papers that changed how we build AI β and why the future isn't about 100-billion-parameter models
By Rayyan Β· 10 min read Β· Published 19 oct 2026
For years, the story of AI was a story of scale.
Bigger models. More data. More GPUs. GPT-3 had 175 billion parameters. ViT-Huge had 632 million. The assumption was simple: if you want human-level AI, you need to build something enormous.
But here's what most people missed: while everyone was watching the giants, a quieter revolution was happening. A handful of researchers were asking a different question:
What if we didn't need to make models bigger β just smarter?
The answer, as it turns out, is yes. And it's changing everything.
π The Problem with Big
Let's be honest about what "big" actually means.
Training GPT-3 cost an estimated $4.6 million. Running it requires specialized hardware that most companies don't have. And deploying it on a phone? Forget about it.
But the real issue is deeper. Big models are:
Β· Expensive to train and run
Β· Slow at inference
Β· Impossible to deploy on edge devices
Β· Environmentally costly β training one large model emits as much COβ as five cars over their lifetimes
And here's the kicker: bigger doesn't always mean better. A model with 175 billion parameters can still fail at basic reasoning. Scale alone doesn't solve intelligence.
So researchers started asking: what if we could get the same performance with 10Γ less compute? 100Γ less?
That's the story of the five papers below.
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β THE EFFICIENCY FRONTIER β
β β
β Accuracy β
β β² β
β β β
EfficientNet β
β β β
β
β β β
β
β β β
β
β β β
β
β β β
β
β β β
β
β ββββββββββββββββββββββββββββββββββββββββββββΊ FLOPs β
β β
β EfficientNet pushes the frontier: more accuracy, β
β less compute. The gap widens at every budget level. β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
𧩠Paper 1: The Transformer Goes to Vision (ViT, 2021)
For a decade, computer vision belonged to CNNs. Convolutional Neural Networks β with their sliding filters and hierarchical features β were the undisputed kings of image recognition.
Then a team at Google asked a heretical question: what if we just⦠didn't use convolutions at all?
Their idea, called the Vision Transformer (ViT), was almost absurdly simple. Take an image. Cut it into 16Γ16 patches. Treat each patch like a word in a sentence. Feed the sequence into a standard Transformer β the same architecture that powers GPT and BERT.
That's it. No convolutions. No hand-crafted features. Just patches and attention.
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β VISION TRANSFORMER (ViT) β
β β
β Input Image (224Γ224) β
β ββββββ¬βββββ¬βββββ¬βββββ β
β β P1 β P2 β P3 β P4 β β 16Γ16 patches β
β ββββββΌβββββΌβββββΌβββββ€ β
β β P5 β P6 β P7 β P8 β β
β ββββββΌβββββΌβββββΌβββββ€ β
β β P9 βP10 βP11 βP12 β β
β ββββββΌβββββΌβββββΌβββββ€ β
β βP13 βP14 βP15 βP16 β β
β ββββββ΄βββββ΄βββββ΄βββββ β
β β β
β βΌ β
β βββββββββββββββββββββββββββββββββββββββ β
β β Linear Projection (each patch β β β
β β a vector of 768 numbers) β β
β βββββββββββββββββββββββββββββββββββββββ β
β β β
β βΌ β
β βββββββββββββββββββββββββββββββββββββββ β
β β Add [CLS] token + Positional β β
β β Embeddings β β
β βββββββββββββββββββββββββββββββββββββββ β
β β β
β βΌ β
β βββββββββββββββββββββββββββββββββββββββ β
β β Transformer Encoder (12 layers) β β
β β βββββββββββββββββββββββββββββββββ β β
β β β Multi-Head Self-Attention β β β
β β β (every patch attends to β β β
β β β every other patch) β β β
β β βββββββββββββββββββββββββββββββββ β β
β β βββββββββββββββββββββββββββββββββ β β
β β β Multi-Layer Perceptron β β β
β β βββββββββββββββββββββββββββββββββ β β
β βββββββββββββββββββββββββββββββββββββββ β
β β β
β βΌ β
β βββββββββββββββββββββββββββββββββββββββ β
β β Classification Head β β
β β (uses [CLS] token output) β β
β βββββββββββββββββββββββββββββββββββββββ β
β β β
β βΌ β
β "Dog" (88.5% confidence) β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
The results were stunning. ViT-Huge achieved 88.5% top-1 accuracy on ImageNet β state of the art at the time β and it required 4Γ less compute to train than comparable CNNs.
Why it matters: ViT proved that Transformers are general-purpose. They're not just for language. They can see, too.
But there was a catch: ViT is data-hungry. It needs 300 million images to work well. On small datasets, it underperforms CNNs. The architecture had won, but it was expensive.
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β THE DATA SCALE HYPOTHESIS β
β β
β Accuracy β
β β² β
β β β ViT β
β β / β
β β / β
β β / β
β β / β
β β / β
β β / β ResNet β
β β / / β
β β / / β
β β / / β
β β / / β
β β / / β
β β / / β
β β / / β
β βββββββββ/βββ/βββββββββββββββββββββββββββββΊ Data Size β
β 1.3M 14M 300M β
β β
β On small data (1.3M), ResNet wins. β
β On large data (300M), ViT overtakes dramatically. β
β Inductive biases are a shortcut for small data. β
β Flexible architectures win with big data. β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
𧩠Paper 2: Making Transformers Efficient (Swin, 2021)
ViT had a problem: its self-attention is quadratic in image size. Double the image size, and you quadruple the compute. That makes it impractical for high-resolution images β the kind you need for object detection and segmentation.
Microsoft's Swin Transformer solved this with a clever trick: shifted window attention.
Instead of computing attention across the entire image, Swin computes it within local windows. Then, in the next layer, it shifts the windows by half a window β so information can flow between them. The result is linear complexity instead of quadratic.
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β SWIN: SHIFTED WINDOWS β
β β
β Layer 1: Standard Windows (7Γ7 patches each) β
β βββββββββββ¬ββββββββββ¬ββββββββββ β
β β β β β β
β β Win 1 β Win 2 β Win 3 β β
β β β β β β
β βββββββββββΌββββββββββΌββββββββββ€ β
β β β β β β
β β Win 4 β Win 5 β Win 6 β β
β β β β β β
β βββββββββββΌββββββββββΌββββββββββ€ β
β β β β β β
β β Win 7 β Win 8 β Win 9 β β
β β β β β β
β βββββββββββ΄ββββββββββ΄ββββββββββ β
β β
β Layer 2: Shifted Windows (by half a window) β
β ββββββ¬ββββββββββ¬ββββββββββ¬βββββ β
β β β β β β β
β β β Win A β Win B β β β
β ββββββΌββββββββββΌββββββββββΌβββββ€ β
β β β β β β β
β βWin Cβ Win D β Win E βWin Fβ β
β ββββββΌββββββββββΌββββββββββΌβββββ€ β
β β β β β β β
β β β Win G β Win H β β β
β ββββββ΄ββββββββββ΄ββββββββββ΄βββββ β
β β
β Patches that were in different windows now interact. β
β Information flows across the entire image over layers. β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
Swin became a general-purpose backbone. It hit 87.3% top-1 on ImageNet, beat state-of-the-art on object detection (+2.7 mAP), and crushed segmentation (+3.2 mIoU). It won ICCV 2021 Best Paper.
Why it matters: Swin made Transformers practical for real-world vision β not just classification, but detection and segmentation too.
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β SWIN: HIERARCHICAL FEATURES β
β β
β Stage 1 Stage 2 Stage 3 Stage 4 β
β H/4 Γ W/4 H/8 Γ W/8 H/16 Γ W/16 H/32ΓW/32β
β βββββββββββ βββββββββββ βββββββββββ βββββββ β
β β β β β β β β ββ ββ β β β ββββ ββ β βββββββ β
β β β β β β β β ββ ββ β β β ββββ ββ β βββββββ β
β β β β β β β β ββ ββ β β β ββββ ββ β βββββββ β
β βββββββββββ βββββββββββ βββββββββββ βββββββ β
β 96 channels 192 channels 384 channels 768 ch. β
β β
β Like CNNs, Swin builds a feature pyramid: β
β β’ High resolution at stage 1 β good for small objects β
β β’ Low resolution at stage 4 β good for large objects β
β This is exactly what object detection and segmentation β
β need. It makes Swin a drop-in replacement for ResNet. β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
𧩠Paper 3: Scaling Intelligently (EfficientNet, 2020)
Before EfficientNet, scaling CNNs was an art, not a science. Want a better model? Make it deeper (ResNet). Or wider (WideResNet). Or higher-resolution. But nobody knew the optimal way to scale.
Google's EfficientNet introduced compound scaling: a single coefficient that uniformly scales depth, width, and resolution using fixed ratios.
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β COMPOUND SCALING β
β β
β Traditional scaling (independent): β
β βββββββββββ βββββββββββ βββββββββββ β
β β Deeper β β Wider β β Higher β β
β β only β β only β β Res onlyβ β
β βββββββββββ βββββββββββ βββββββββββ β
β Each gives diminishing returns. β
β β
β Compound scaling (together): β
β βββββββββββββββββββββββββββββββββββββββ β
β β Depth Γ Width Γ Resolution β β
β β d = Ξ±^Ο β β
β β w = Ξ²^Ο β β
β β r = Ξ³^Ο β β
β β subject to: Ξ± Β· Ξ²Β² Β· Ξ³Β² β 2 β β
β βββββββββββββββββββββββββββββββββββββββ β
β Multiplicative benefits. β
β β
β Intuition: Like a camera. To take the best photo, β
β you need a bigger sensor (width), a better zoom β
β (resolution), AND more lenses (depth) β together. β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
The result was a family of models that achieved state-of-the-art accuracy with up to 10Γ better efficiency. EfficientNet-B0 matched ResNet-50 with 5Γ fewer parameters and 10Γ fewer FLOPs. EfficientNet-B7 hit 84.3% top-1 while being 8.4Γ smaller than the previous best.
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β EFFICIENTNET: THE FAMILY β
β β
β Model Resolution Params FLOPs Top-1 β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β B0 224 5.3M 0.39B 77.1% β
β B1 240 7.8M 0.70B 79.1% β
β B2 260 9.2M 1.0B 80.1% β
β B3 300 12M 1.8B 81.6% β
β B4 380 19M 4.2B 82.9% β
β B5 456 30M 9.9B 83.6% β
β B6 528 43M 19B 84.0% β
β B7 600 66M 37B 84.3% β
β β
β B0 matches ResNet-50 with 5Γ fewer params, 10Γ fewer FLOPsβ
β B7 matches GPipe (556M params) with 8.4Γ fewer params. β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
Why it matters: EfficientNet showed how to scale optimally, not just more. It's the difference between throwing more GPUs at a problem and actually designing a better model.
𧩠Paper 4: Learning Without Labels (BEiT, 2022)
ViT needed 300 million labeled images. But labels are expensive β someone has to manually annotate every image.
Microsoft's BEiT asked: can we learn from unlabeled images, like BERT learns from unlabeled text?
The answer was yes. BEiT adapts BERT's masked language modeling to vision:
- Mask ~40% of image patches.
- Train the model to predict the visual tokens of masked patches (using a discrete VAE).
- Learn rich, semantic representations without any labels.
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β BEiT: MASKED IMAGE MODELING β
β β
β Step 1: Tokenize image with discrete VAE β
β ββββββ¬βββββ¬βββββ¬βββββ β
β β T1 β T2 β T3 β T4 β β Each patch β one visual token β
β ββββββΌβββββΌβββββΌβββββ€ β
β β T5 β T6 β T7 β T8 β β
β ββββββΌβββββΌβββββΌβββββ€ β
β β T9 βT10 βT11 βT12 β β
β ββββββΌβββββΌβββββΌβββββ€ β
β βT13 βT14 βT15 βT16 β β
β ββββββ΄βββββ΄βββββ΄βββββ β
β β
β Step 2: Mask ~40% of patches β
β ββββββ¬βββββ¬βββββ¬βββββ β
β β T1 β????β T3 β????β β ???? = [MASK] token β
β ββββββΌβββββΌβββββΌβββββ€ β
β β????β T6 β????β T8 β β
β ββββββΌβββββΌβββββΌβββββ€ β
β β T9 β????βT11 β????β β
β ββββββΌβββββΌβββββΌβββββ€ β
β βT13 βT14 β????βT16 β β
β ββββββ΄βββββ΄βββββ΄βββββ β
β β
β Step 3: Predict visual tokens of masked patches β
β ββββββ¬βββββ¬βββββ¬βββββ β
β β T1 β T2'β T3 β T4'β β T2', T4' = predictions β
β ββββββΌβββββΌβββββΌβββββ€ β
β β T5'β T6 β T7'β T8 β β
β ββββββΌβββββΌβββββΌβββββ€ β
β β T9 β T10βT11 β T12β β
β ββββββΌβββββΌβββββΌβββββ€ β
β βT13 βT14 βT15'βT16 β β
β ββββββ΄βββββ΄βββββ΄βββββ β
β β
β Loss: cross-entropy between predicted and true tokens β
β No labels needed. The true tokens come from the VAE. β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
BEiT-L achieved 85.2% top-1 on ImageNet β outperforming supervised ViT (82.3%) β and matched models trained on 10Γ more data.
Why it matters: BEiT enabled learning from unlabeled data. That's how humans learn β by observing the world, not by being told the answer to everything.
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β PREDICT TOKENS, NOT PIXELS β
β β
β Predicting pixels: β
β β’ Forces model to learn edges, colors, textures β
β β’ Low-level details, not semantics β
β β’ Like predicting individual letters of a word β
β β
β Predicting visual tokens: β
β β’ Forces model to learn what is in the patch β
β β’ High-level semantics β
β β’ Like predicting whole words β
β β
β This is why BEiT outperforms pixel-prediction methods. β
β It's exactly the same reason BERT predicts words, β
β not characters. β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
𧩠Paper 5: Making It Deployable (TinyBERT, 2020)
BERT is powerful. BERT is also huge β 110 million parameters. Too slow for real-time apps. Too big for a phone.
TinyBERT solved this with knowledge distillation: train a small model (the student) to mimic a large model (the teacher). Two stages: general distillation on unlabeled text, then task-specific distillation.
Distillation happens at four levels: embeddings, attention matrices, hidden states, and predictions. The result: 7.5Γ smaller, 9.4Γ faster, while retaining 96.8% of BERT's performance.
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β TINYBERT: FOUR-LEVEL DISTILLATION β
β β
β Teacher (BERT-Base, 110M params) β
β βββββββββββββββββββββββββββββββββββββββ β
β β Layer 1 β Layer 2 β ... β Layer 12β β
β βββββββββββββββββββββββββββββββββββββββ β
β β β β β
β βΌ βΌ βΌ β
β βββββββββββββββββββββββββββββββββββββββ β
β β Student (TinyBERT, 14.5M params) β β
β β Layer 1 β Layer 2 β Layer 3βL4 β β
β βββββββββββββββββββββββββββββββββββββββ β
β β
β Distillation happens at 4 levels: β
β 1. Embedding layer βββββΊ match embeddings β
β 2. Attention matrices βββββΊ match where teacher looks β
β 3. Hidden states βββββΊ match internal representationsβ
β 4. Prediction logits βββββΊ match final outputs β
β β
β Two stages: β
β Stage 1: General distillation (unlabeled text) β
β Stage 2: Task-specific distillation (labeled task data) β
β β
β Result: 96.8% of BERT's accuracy, 7.5Γ smaller, 9.4Γ fasterβ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
Why it matters: TinyBERT made powerful models actually deployable. It's the bridge between research and reality.
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β TINYBERT VS DISTILBERT β
β β
β Model GLUE Avg Params Speedup β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β BERT-Base 79.2 110M 1Γ β
β DistilBERT 77.0 66M 1.6Γ β
β TinyBERT-4 76.8 14.5M 9.4Γ β
β TinyBERT-6 79.0 67M 2.3Γ β
β β
β TinyBERT-4 is 4.5Γ smaller than DistilBERT β
β but nearly as accurate (76.8 vs 77.0). β
β Why? Four-level distillation transfers more knowledge β
β than logit-only distillation. β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
π The Through-Line
Together, these five papers tell a story:
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β THE EFFICIENT AI STORY ARC β
β β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β ViT (2021) β β
β β "Transformers can see" β β
β β Proved the architecture is general-purpose β β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β β
β βΌ β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β Swin (2021) β β
β β "Transformers can be efficient" β β
β β Linear complexity + hierarchical features β β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β β
β βΌ β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β EfficientNet (2020) β β
β β "Scaling can be principled" β β
β β Compound scaling: depth + width + resolution β β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β β
β βΌ β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β BEiT (2022) β β
β β "Models can learn without labels" β β
β β Masked image modeling β like BERT for vision β β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β β
β βΌ β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β TinyBERT (2020) β β
β β "Models can be compressed dramatically" β β
β β Four-level distillation β deployable on any device β β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β
β Together: Smarter, not just bigger. β
β This is the real research direction pushing AI toward β
β near-human capability. β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
This is the real research direction pushing AI toward near-human capability. Not by making models bigger β but by making them smarter, more efficient, and more general.
β οΈ The Gaps That Remain
Despite this progress, big problems remain:
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β SEVEN RESEARCH GAPS β
β β
β 1. NO UNIFIED EFFICIENCY BENCHMARK β
β Every paper uses different metrics. β
β We can't compare fairly. β
β β Solution: Standardized benchmark (like MLPerf) β
β β
β 2. MULTI-MODAL DISTILLATION MISSING β
β TinyBERT = text; ViT/Swin = vision. β
β No one distills CLIP or GPT-4V. β
β β Solution: Multi-modal distillation frameworks β
β β
β 3. HARDWARE-AWARE EFFICIENCY β
β FLOPs β real latency. β
β Paper-efficient β phone-efficient. β
β β Solution: Hardware-aware NAS and benchmarking β
β β
β 4. SELF-SUPERVISION IS EXPENSIVE β
β BEiT is powerful but costly to pre-train. β
β β Solution: Efficient self-supervised methods β
β β
β 5. NO EFFICIENT VISION-LANGUAGE MODELS β
β CLIP and GPT-4V are huge. β
β We need small, fast equivalents. β
β β Solution: Efficient multi-modal architectures β
β β
β 6. INTERPRETABILITY IS IGNORED β
β None of these papers address explainability. β
β β Solution: XAI for efficient models β
β β
β 7. CONTINUAL LEARNING β
β All assume static datasets. β
β The real world changes. β
β β Solution: Efficient continual learning β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
π What's Next
The next frontier is efficient, general-purpose, multi-modal AI β models that:
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β THE NEXT FRONTIER β
β β
β β
Match human performance on any task β
β β
Run on any device (phone, sensor, laptop) β
β β
Learn from any data (labeled or not) β
β β
Adapt continuously without retraining β
β β
Explain their decisions β
β β
β This is the direction AI is heading. β
β And it's not about making models bigger. β
β It's about making them smarter. β
β β
β The giants will always have their place. β
β But the future belongs to the efficient. β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
π‘ The Bottom Line
For years, we thought bigger meant better. But the real breakthroughs β the ones pushing AI toward human-level capability β came from a different direction.
Β· ViT showed Transformers could see.
Β· Swin made them efficient.
Β· EfficientNet showed how to scale.
Β· BEiT learned without labels.
Β· TinyBERT made it deployable.
Together, they define the efficient AI movement. And they prove something important:
You don't need a 175-billion-parameter model to be smart. You need a better architecture, a smarter training method, and a principled way to scale.
That's the quiet revolution. And it's just getting started.
If you found this useful, share it with someone who thinks AI is just about bigger models. And if you're building AI systems β I'd love to hear what efficiency trade-offs you're making.
π References
- Dosovitskiy, Alexey; Beyer, Lucas; Kolesnikov, Alexander; Weissenborn, Dirk; Zhai, Xiaohua; Unterthiner, Thomas; Dehghani, Mostafa; Minderer, Matthias; Heigold, Georg; Gelly, Sylvain; Uszkoreit, Jakob; and Houlsby, Neil (2021). An Image is Worth 16Γ16 Words: Transformers for Image Recognition at Scale. International Conference on Learning Representations (ICLR) 2021. arXiv identifier: 2010.11929.
- Liu, Ze; Lin, Yutong; Cao, Yue; Hu, Han; Wei, Yixuan; Zhang, Zheng; Lin, Stephen; and Guo, Baining (2021). Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. International Conference on Computer Vision (ICCV) 2021. arXiv identifier: 2103.14030.
- Tan, Mingxing and Le, Quoc V. (2020). EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. International Conference on Machine Learning (ICML) 2019 / Springer. arXiv identifier: 1905.11946.
- Bao, Hangbo; Dong, Li; Piao, Songhao; and Wei, Furu (2022). BEiT: BERT Pre-Training of Image Transformers. International Conference on Learning Representations (ICLR) 2022. arXiv identifier: 2106.08254.
- Jiao, Xiaoqi; Yin, Yichun; Shang, Lifeng; Jiang, Xin; Chen, Xiao; Li, Linlin; Wang, Fang; and Liu, Qun (2020). TinyBERT: Distilling BERT for Natural Language Understanding. International Conference on Learning Representations (ICLR) 2020 / Springer. arXiv identifier: 1909.10351.
Tags: #AI #DeepLearning #MachineLearning #VisionTransformer #EfficientAI #Transformers #TechBlog #ArtificialIntelligence #SwinTransformer #EfficientNet #BEiT #TinyBERT
-
Top comments (0)