The latest articles on DEV Community by Rayyan (@rayyan231503101). https://dev.to/rayyan231503101 https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4193401%2F5bc1d1a9-9f77-4641-a82a-2f9371a4e009.png https://dev.to/rayyan231503101 en Rayyan Mon, 19 Oct 2026 06:32:29 +0000 https://dev.to/rayyan231503101/an-analysis-of-5-key-papers-1me8 https://dev.to/rayyan231503101/an-analysis-of-5-key-papers-1me8 <p>🚀 How AI Got Smarter (Not Just Bigger): The Quiet Revolution in Efficient Deep Learning</p> <p>Five papers that changed how we build AI — and why the future isn't about 100-billion-parameter models</p> <p>By Rayyan · 10 min read · Published 19 oct 2026</p> <p>For years, the story of AI was a story of scale.</p> <p>Bigger models. More data. More GPUs. GPT-3 had 175 billion parameters. ViT-Huge had 632 million. The assumption was simple: if you want human-level AI, you need to build something enormous.</p> <p>But here's what most people missed: while everyone was watching the giants, a quieter revolution was happening. A handful of researchers were asking a different question:</p> <p>What if we didn't need to make models bigger — just smarter?</p> <p>The answer, as it turns out, is yes. And it's changing everything.</p> <p>📊 The Problem with Big</p> <p>Let's be honest about what "big" actually means.</p> <p>Training GPT-3 cost an estimated $4.6 million. Running it requires specialized hardware that most companies don't have. And deploying it on a phone? Forget about it.</p> <p>But the real issue is deeper. Big models are:</p> <p>· Expensive to train and run<br> · Slow at inference<br> · Impossible to deploy on edge devices<br> · Environmentally costly — training one large model emits as much CO₂ as five cars over their lifetimes</p> <p>And here's the kicker: bigger doesn't always mean better. A model with 175 billion parameters can still fail at basic reasoning. Scale alone doesn't solve intelligence.</p> <p>So researchers started asking: what if we could get the same performance with 10× less compute? 100× less?</p> <p>That's the story of the five papers below.<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>┌─────────────────────────────────────────────────────────────┐ │ THE EFFICIENCY FRONTIER │ │ │ │ Accuracy │ │ ▲ │ │ │ ★ EfficientNet │ │ │ ★ │ │ │ ★ │ │ │ ★ │ │ │ ★ │ │ │ ★ │ │ │ ★ │ │ └──────────────────────────────────────────► FLOPs │ │ │ │ EfficientNet pushes the frontier: more accuracy, │ │ less compute. The gap widens at every budget level. │ └─────────────────────────────────────────────────────────────┘ </code></pre> </div> <p>🧩 Paper 1: The Transformer Goes to Vision (ViT, 2021)</p> <p>For a decade, computer vision belonged to CNNs. Convolutional Neural Networks — with their sliding filters and hierarchical features — were the undisputed kings of image recognition.</p> <p>Then a team at Google asked a heretical question: what if we just… didn't use convolutions at all?</p> <p>Their idea, called the Vision Transformer (ViT), was almost absurdly simple. Take an image. Cut it into 16×16 patches. Treat each patch like a word in a sentence. Feed the sequence into a standard Transformer — the same architecture that powers GPT and BERT.</p> <p>That's it. No convolutions. No hand-crafted features. Just patches and attention.<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>┌─────────────────────────────────────────────────────────────┐ │ VISION TRANSFORMER (ViT) │ │ │ │ Input Image (224×224) │ │ ┌────┬────┬────┬────┐ │ │ │ P1 │ P2 │ P3 │ P4 │ ← 16×16 patches │ │ ├────┼────┼────┼────┤ │ │ │ P5 │ P6 │ P7 │ P8 │ │ │ ├────┼────┼────┼────┤ │ │ │ P9 │P10 │P11 │P12 │ │ │ ├────┼────┼────┼────┤ │ │ │P13 │P14 │P15 │P16 │ │ │ └────┴────┴────┴────┘ │ │ │ │ │ ▼ │ │ ┌─────────────────────────────────────┐ │ │ │ Linear Projection (each patch → │ │ │ │ a vector of 768 numbers) │ │ │ └─────────────────────────────────────┘ │ │ │ │ │ ▼ │ │ ┌─────────────────────────────────────┐ │ │ │ Add [CLS] token + Positional │ │ │ │ Embeddings │ │ │ └─────────────────────────────────────┘ │ │ │ │ │ ▼ │ │ ┌─────────────────────────────────────┐ │ │ │ Transformer Encoder (12 layers) │ │ │ │ ┌───────────────────────────────┐ │ │ │ │ │ Multi-Head Self-Attention │ │ │ │ │ │ (every patch attends to │ │ │ │ │ │ every other patch) │ │ │ │ │ └───────────────────────────────┘ │ │ │ │ ┌───────────────────────────────┐ │ │ │ │ │ Multi-Layer Perceptron │ │ │ │ │ └───────────────────────────────┘ │ │ │ └─────────────────────────────────────┘ │ │ │ │ │ ▼ │ │ ┌─────────────────────────────────────┐ │ │ │ Classification Head │ │ │ │ (uses [CLS] token output) │ │ │ └─────────────────────────────────────┘ │ │ │ │ │ ▼ │ │ "Dog" (88.5% confidence) │ └─────────────────────────────────────────────────────────────┘ </code></pre> </div> <p>The results were stunning. ViT-Huge achieved 88.5% top-1 accuracy on ImageNet — state of the art at the time — and it required 4× less compute to train than comparable CNNs.</p> <p>Why it matters: ViT proved that Transformers are general-purpose. They're not just for language. They can see, too.</p> <p>But there was a catch: ViT is data-hungry. It needs 300 million images to work well. On small datasets, it underperforms CNNs. The architecture had won, but it was expensive.<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>┌─────────────────────────────────────────────────────────────┐ │ THE DATA SCALE HYPOTHESIS │ │ │ │ Accuracy │ │ ▲ │ │ │ ● ViT │ │ │ / │ │ │ / │ │ │ / │ │ │ / │ │ │ / │ │ │ / ○ ResNet │ │ │ / / │ │ │ / / │ │ │ / / │ │ │ / / │ │ │ / / │ │ │ / / │ │ │ / / │ │ └────────/───/────────────────────────────► Data Size │ │ 1.3M 14M 300M │ │ │ │ On small data (1.3M), ResNet wins. │ │ On large data (300M), ViT overtakes dramatically. │ │ Inductive biases are a shortcut for small data. │ │ Flexible architectures win with big data. │ └─────────────────────────────────────────────────────────────┘ </code></pre> </div> <p>🧩 Paper 2: Making Transformers Efficient (Swin, 2021)</p> <p>ViT had a problem: its self-attention is quadratic in image size. Double the image size, and you quadruple the compute. That makes it impractical for high-resolution images — the kind you need for object detection and segmentation.</p> <p>Microsoft's Swin Transformer solved this with a clever trick: shifted window attention.</p> <p>Instead of computing attention across the entire image, Swin computes it within local windows. Then, in the next layer, it shifts the windows by half a window — so information can flow between them. The result is linear complexity instead of quadratic.<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>┌─────────────────────────────────────────────────────────────┐ │ SWIN: SHIFTED WINDOWS │ │ │ │ Layer 1: Standard Windows (7×7 patches each) │ │ ┌─────────┬─────────┬─────────┐ │ │ │ │ │ │ │ │ │ Win 1 │ Win 2 │ Win 3 │ │ │ │ │ │ │ │ │ ├─────────┼─────────┼─────────┤ │ │ │ │ │ │ │ │ │ Win 4 │ Win 5 │ Win 6 │ │ │ │ │ │ │ │ │ ├─────────┼─────────┼─────────┤ │ │ │ │ │ │ │ │ │ Win 7 │ Win 8 │ Win 9 │ │ │ │ │ │ │ │ │ └─────────┴─────────┴─────────┘ │ │ │ │ Layer 2: Shifted Windows (by half a window) │ │ ┌────┬─────────┬─────────┬────┐ │ │ │ │ │ │ │ │ │ │ │ Win A │ Win B │ │ │ │ ├────┼─────────┼─────────┼────┤ │ │ │ │ │ │ │ │ │ │Win C│ Win D │ Win E │Win F│ │ │ ├────┼─────────┼─────────┼────┤ │ │ │ │ │ │ │ │ │ │ │ Win G │ Win H │ │ │ │ └────┴─────────┴─────────┴────┘ │ │ │ │ Patches that were in different windows now interact. │ │ Information flows across the entire image over layers. │ └─────────────────────────────────────────────────────────────┘ </code></pre> </div> <p>Swin became a general-purpose backbone. It hit 87.3% top-1 on ImageNet, beat state-of-the-art on object detection (+2.7 mAP), and crushed segmentation (+3.2 mIoU). It won ICCV 2021 Best Paper.</p> <p>Why it matters: Swin made Transformers practical for real-world vision — not just classification, but detection and segmentation too.<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>┌─────────────────────────────────────────────────────────────┐ │ SWIN: HIERARCHICAL FEATURES │ │ │ │ Stage 1 Stage 2 Stage 3 Stage 4 │ │ H/4 × W/4 H/8 × W/8 H/16 × W/16 H/32×W/32│ │ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────┐ │ │ │ █ █ █ █ │ │ ██ ██ █ │ │ ████ ██ │ │█████│ │ │ │ █ █ █ █ │ │ ██ ██ █ │ │ ████ ██ │ │█████│ │ │ │ █ █ █ █ │ │ ██ ██ █ │ │ ████ ██ │ │█████│ │ │ └─────────┘ └─────────┘ └─────────┘ └─────┘ │ │ 96 channels 192 channels 384 channels 768 ch. │ │ │ │ Like CNNs, Swin builds a feature pyramid: │ │ • High resolution at stage 1 → good for small objects │ │ • Low resolution at stage 4 → good for large objects │ │ This is exactly what object detection and segmentation │ │ need. It makes Swin a drop-in replacement for ResNet. │ └─────────────────────────────────────────────────────────────┘ </code></pre> </div> <p>🧩 Paper 3: Scaling Intelligently (EfficientNet, 2020)</p> <p>Before EfficientNet, scaling CNNs was an art, not a science. Want a better model? Make it deeper (ResNet). Or wider (WideResNet). Or higher-resolution. But nobody knew the optimal way to scale.</p> <p>Google's EfficientNet introduced compound scaling: a single coefficient that uniformly scales depth, width, and resolution using fixed ratios.<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>┌─────────────────────────────────────────────────────────────┐ │ COMPOUND SCALING │ │ │ │ Traditional scaling (independent): │ │ ┌─────────┐ ┌─────────┐ ┌─────────┐ │ │ │ Deeper │ │ Wider │ │ Higher │ │ │ │ only │ │ only │ │ Res only│ │ │ └─────────┘ └─────────┘ └─────────┘ │ │ Each gives diminishing returns. │ │ │ │ Compound scaling (together): │ │ ┌─────────────────────────────────────┐ │ │ │ Depth × Width × Resolution │ │ │ │ d = α^φ │ │ │ │ w = β^φ │ │ │ │ r = γ^φ │ │ │ │ subject to: α · β² · γ² ≈ 2 │ │ │ └─────────────────────────────────────┘ │ │ Multiplicative benefits. │ │ │ │ Intuition: Like a camera. To take the best photo, │ │ you need a bigger sensor (width), a better zoom │ │ (resolution), AND more lenses (depth) — together. │ └─────────────────────────────────────────────────────────────┘ </code></pre> </div> <p>The result was a family of models that achieved state-of-the-art accuracy with up to 10× better efficiency. EfficientNet-B0 matched ResNet-50 with 5× fewer parameters and 10× fewer FLOPs. EfficientNet-B7 hit 84.3% top-1 while being 8.4× smaller than the previous best.<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>┌─────────────────────────────────────────────────────────────┐ │ EFFICIENTNET: THE FAMILY │ │ │ │ Model Resolution Params FLOPs Top-1 │ │ ───────────────────────────────────────────────────── │ │ B0 224 5.3M 0.39B 77.1% │ │ B1 240 7.8M 0.70B 79.1% │ │ B2 260 9.2M 1.0B 80.1% │ │ B3 300 12M 1.8B 81.6% │ │ B4 380 19M 4.2B 82.9% │ │ B5 456 30M 9.9B 83.6% │ │ B6 528 43M 19B 84.0% │ │ B7 600 66M 37B 84.3% │ │ │ │ B0 matches ResNet-50 with 5× fewer params, 10× fewer FLOPs│ │ B7 matches GPipe (556M params) with 8.4× fewer params. │ └─────────────────────────────────────────────────────────────┘ </code></pre> </div> <p>Why it matters: EfficientNet showed how to scale optimally, not just more. It's the difference between throwing more GPUs at a problem and actually designing a better model.</p> <p>🧩 Paper 4: Learning Without Labels (BEiT, 2022)</p> <p>ViT needed 300 million labeled images. But labels are expensive — someone has to manually annotate every image.</p> <p>Microsoft's BEiT asked: can we learn from unlabeled images, like BERT learns from unlabeled text?</p> <p>The answer was yes. BEiT adapts BERT's masked language modeling to vision:</p> <ol> <li>Mask ~40% of image patches.</li> <li>Train the model to predict the visual tokens of masked patches (using a discrete VAE).</li> <li>Learn rich, semantic representations without any labels. </li> </ol> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>┌─────────────────────────────────────────────────────────────┐ │ BEiT: MASKED IMAGE MODELING │ │ │ │ Step 1: Tokenize image with discrete VAE │ │ ┌────┬────┬────┬────┐ │ │ │ T1 │ T2 │ T3 │ T4 │ ← Each patch → one visual token │ │ ├────┼────┼────┼────┤ │ │ │ T5 │ T6 │ T7 │ T8 │ │ │ ├────┼────┼────┼────┤ │ │ │ T9 │T10 │T11 │T12 │ │ │ ├────┼────┼────┼────┤ │ │ │T13 │T14 │T15 │T16 │ │ │ └────┴────┴────┴────┘ │ │ │ │ Step 2: Mask ~40% of patches │ │ ┌────┬────┬────┬────┐ │ │ │ T1 │????│ T3 │????│ ← ???? = [MASK] token │ │ ├────┼────┼────┼────┤ │ │ │????│ T6 │????│ T8 │ │ │ ├────┼────┼────┼────┤ │ │ │ T9 │????│T11 │????│ │ │ ├────┼────┼────┼────┤ │ │ │T13 │T14 │????│T16 │ │ │ └────┴────┴────┴────┘ │ │ │ │ Step 3: Predict visual tokens of masked patches │ │ ┌────┬────┬────┬────┐ │ │ │ T1 │ T2'│ T3 │ T4'│ ← T2', T4' = predictions │ │ ├────┼────┼────┼────┤ │ │ │ T5'│ T6 │ T7'│ T8 │ │ │ ├────┼────┼────┼────┤ │ │ │ T9 │ T10│T11 │ T12│ │ │ ├────┼────┼────┼────┤ │ │ │T13 │T14 │T15'│T16 │ │ │ └────┴────┴────┴────┘ │ │ │ │ Loss: cross-entropy between predicted and true tokens │ │ No labels needed. The true tokens come from the VAE. │ └─────────────────────────────────────────────────────────────┘ </code></pre> </div> <p>BEiT-L achieved 85.2% top-1 on ImageNet — outperforming supervised ViT (82.3%) — and matched models trained on 10× more data.</p> <p>Why it matters: BEiT enabled learning from unlabeled data. That's how humans learn — by observing the world, not by being told the answer to everything.<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>┌─────────────────────────────────────────────────────────────┐ │ PREDICT TOKENS, NOT PIXELS │ │ │ │ Predicting pixels: │ │ • Forces model to learn edges, colors, textures │ │ • Low-level details, not semantics │ │ • Like predicting individual letters of a word │ │ │ │ Predicting visual tokens: │ │ • Forces model to learn what is in the patch │ │ • High-level semantics │ │ • Like predicting whole words │ │ │ │ This is why BEiT outperforms pixel-prediction methods. │ │ It's exactly the same reason BERT predicts words, │ │ not characters. │ └─────────────────────────────────────────────────────────────┘ </code></pre> </div> <p>🧩 Paper 5: Making It Deployable (TinyBERT, 2020)</p> <p>BERT is powerful. BERT is also huge — 110 million parameters. Too slow for real-time apps. Too big for a phone.</p> <p>TinyBERT solved this with knowledge distillation: train a small model (the student) to mimic a large model (the teacher). Two stages: general distillation on unlabeled text, then task-specific distillation.</p> <p>Distillation happens at four levels: embeddings, attention matrices, hidden states, and predictions. The result: 7.5× smaller, 9.4× faster, while retaining 96.8% of BERT's performance.<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>┌─────────────────────────────────────────────────────────────┐ │ TINYBERT: FOUR-LEVEL DISTILLATION │ │ │ │ Teacher (BERT-Base, 110M params) │ │ ┌─────────────────────────────────────┐ │ │ │ Layer 1 │ Layer 2 │ ... │ Layer 12│ │ │ └─────────────────────────────────────┘ │ │ │ │ │ │ │ ▼ ▼ ▼ │ │ ┌─────────────────────────────────────┐ │ │ │ Student (TinyBERT, 14.5M params) │ │ │ │ Layer 1 │ Layer 2 │ Layer 3│L4 │ │ │ └─────────────────────────────────────┘ │ │ │ │ Distillation happens at 4 levels: │ │ 1. Embedding layer ────► match embeddings │ │ 2. Attention matrices ────► match where teacher looks │ │ 3. Hidden states ────► match internal representations│ │ 4. Prediction logits ────► match final outputs │ │ │ │ Two stages: │ │ Stage 1: General distillation (unlabeled text) │ │ Stage 2: Task-specific distillation (labeled task data) │ │ │ │ Result: 96.8% of BERT's accuracy, 7.5× smaller, 9.4× faster│ └─────────────────────────────────────────────────────────────┘ </code></pre> </div> <p>Why it matters: TinyBERT made powerful models actually deployable. It's the bridge between research and reality.<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>┌─────────────────────────────────────────────────────────────┐ │ TINYBERT VS DISTILBERT │ │ │ │ Model GLUE Avg Params Speedup │ │ ───────────────────────────────────────────────────── │ │ BERT-Base 79.2 110M 1× │ │ DistilBERT 77.0 66M 1.6× │ │ TinyBERT-4 76.8 14.5M 9.4× │ │ TinyBERT-6 79.0 67M 2.3× │ │ │ │ TinyBERT-4 is 4.5× smaller than DistilBERT │ │ but nearly as accurate (76.8 vs 77.0). │ │ Why? Four-level distillation transfers more knowledge │ │ than logit-only distillation. │ └─────────────────────────────────────────────────────────────┘ </code></pre> </div> <p>🔗 The Through-Line</p> <p>Together, these five papers tell a story:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>┌─────────────────────────────────────────────────────────────┐ │ THE EFFICIENT AI STORY ARC │ │ │ │ ┌─────────────────────────────────────────────────────┐ │ │ │ ViT (2021) │ │ │ │ "Transformers can see" │ │ │ │ Proved the architecture is general-purpose │ │ │ └─────────────────────────────────────────────────────┘ │ │ │ │ │ ▼ │ │ ┌─────────────────────────────────────────────────────┐ │ │ │ Swin (2021) │ │ │ │ "Transformers can be efficient" │ │ │ │ Linear complexity + hierarchical features │ │ │ └─────────────────────────────────────────────────────┘ │ │ │ │ │ ▼ │ │ ┌─────────────────────────────────────────────────────┐ │ │ │ EfficientNet (2020) │ │ │ │ "Scaling can be principled" │ │ │ │ Compound scaling: depth + width + resolution │ │ │ └─────────────────────────────────────────────────────┘ │ │ │ │ │ ▼ │ │ ┌─────────────────────────────────────────────────────┐ │ │ │ BEiT (2022) │ │ │ │ "Models can learn without labels" │ │ │ │ Masked image modeling — like BERT for vision │ │ │ └─────────────────────────────────────────────────────┘ │ │ │ │ │ ▼ │ │ ┌─────────────────────────────────────────────────────┐ │ │ │ TinyBERT (2020) │ │ │ │ "Models can be compressed dramatically" │ │ │ │ Four-level distillation — deployable on any device │ │ │ └─────────────────────────────────────────────────────┘ │ │ │ │ Together: Smarter, not just bigger. │ │ This is the real research direction pushing AI toward │ │ near-human capability. │ └─────────────────────────────────────────────────────────────┘ </code></pre> </div> <p>This is the real research direction pushing AI toward near-human capability. Not by making models bigger — but by making them smarter, more efficient, and more general.</p> <p>⚠️ The Gaps That Remain</p> <p>Despite this progress, big problems remain:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>┌─────────────────────────────────────────────────────────────┐ │ SEVEN RESEARCH GAPS │ │ │ │ 1. NO UNIFIED EFFICIENCY BENCHMARK │ │ Every paper uses different metrics. │ │ We can't compare fairly. │ │ → Solution: Standardized benchmark (like MLPerf) │ │ │ │ 2. MULTI-MODAL DISTILLATION MISSING │ │ TinyBERT = text; ViT/Swin = vision. │ │ No one distills CLIP or GPT-4V. │ │ → Solution: Multi-modal distillation frameworks │ │ │ │ 3. HARDWARE-AWARE EFFICIENCY │ │ FLOPs ≠ real latency. │ │ Paper-efficient ≠ phone-efficient. │ │ → Solution: Hardware-aware NAS and benchmarking │ │ │ │ 4. SELF-SUPERVISION IS EXPENSIVE │ │ BEiT is powerful but costly to pre-train. │ │ → Solution: Efficient self-supervised methods │ │ │ │ 5. NO EFFICIENT VISION-LANGUAGE MODELS │ │ CLIP and GPT-4V are huge. │ │ We need small, fast equivalents. │ │ → Solution: Efficient multi-modal architectures │ │ │ │ 6. INTERPRETABILITY IS IGNORED │ │ None of these papers address explainability. │ │ → Solution: XAI for efficient models │ │ │ │ 7. CONTINUAL LEARNING │ │ All assume static datasets. │ │ The real world changes. │ │ → Solution: Efficient continual learning │ └─────────────────────────────────────────────────────────────┘ </code></pre> </div> <p>🚀 What's Next</p> <p>The next frontier is efficient, general-purpose, multi-modal AI — models that:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>┌─────────────────────────────────────────────────────────────┐ │ THE NEXT FRONTIER │ │ │ │ ✅ Match human performance on any task │ │ ✅ Run on any device (phone, sensor, laptop) │ │ ✅ Learn from any data (labeled or not) │ │ ✅ Adapt continuously without retraining │ │ ✅ Explain their decisions │ │ │ │ This is the direction AI is heading. │ │ And it's not about making models bigger. │ │ It's about making them smarter. │ │ │ │ The giants will always have their place. │ │ But the future belongs to the efficient. │ └─────────────────────────────────────────────────────────────┘ </code></pre> </div> <p>💡 The Bottom Line</p> <p>For years, we thought bigger meant better. But the real breakthroughs — the ones pushing AI toward human-level capability — came from a different direction.</p> <p>· ViT showed Transformers could see.<br> · Swin made them efficient.<br> · EfficientNet showed how to scale.<br> · BEiT learned without labels.<br> · TinyBERT made it deployable.</p> <p>Together, they define the efficient AI movement. And they prove something important:</p> <p>You don't need a 175-billion-parameter model to be smart. You need a better architecture, a smarter training method, and a principled way to scale.</p> <p>That's the quiet revolution. And it's just getting started.</p> <p>If you found this useful, share it with someone who thinks AI is just about bigger models. And if you're building AI systems — I'd love to hear what efficiency trade-offs you're making.</p> <p>📚 References</p> <ol> <li>Dosovitskiy, Alexey; Beyer, Lucas; Kolesnikov, Alexander; Weissenborn, Dirk; Zhai, Xiaohua; Unterthiner, Thomas; Dehghani, Mostafa; Minderer, Matthias; Heigold, Georg; Gelly, Sylvain; Uszkoreit, Jakob; and Houlsby, Neil (2021). An Image is Worth 16×16 Words: Transformers for Image Recognition at Scale. International Conference on Learning Representations (ICLR) 2021. arXiv identifier: 2010.11929.</li> <li>Liu, Ze; Lin, Yutong; Cao, Yue; Hu, Han; Wei, Yixuan; Zhang, Zheng; Lin, Stephen; and Guo, Baining (2021). Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. International Conference on Computer Vision (ICCV) 2021. arXiv identifier: 2103.14030.</li> <li>Tan, Mingxing and Le, Quoc V. (2020). EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. International Conference on Machine Learning (ICML) 2019 / Springer. arXiv identifier: 1905.11946.</li> <li>Bao, Hangbo; Dong, Li; Piao, Songhao; and Wei, Furu (2022). BEiT: BERT Pre-Training of Image Transformers. International Conference on Learning Representations (ICLR) 2022. arXiv identifier: 2106.08254.</li> <li>Jiao, Xiaoqi; Yin, Yichun; Shang, Lifeng; Jiang, Xin; Chen, Xiao; Li, Linlin; Wang, Fang; and Liu, Qun (2020). TinyBERT: Distilling BERT for Natural Language Understanding. International Conference on Learning Representations (ICLR) 2020 / Springer. arXiv identifier: 1909.10351.</li> </ol> <p>Tags: #AI #DeepLearning #MachineLearning #VisionTransformer #EfficientAI #Transformers #TechBlog #ArtificialIntelligence #SwinTransformer #EfficientNet #BEiT #TinyBERT</p> <p>-</p> ai deeplearning machinelearning