DEV Community: Vishal Chandak

What if corporate bureaucracy and data center networking suffered from the exact same problem? AWS just slashed 69% of their networking hardware by flattening their data centers and embracing quasi-random graph math. Read about how it really works here.

Vishal Chandak — Sat, 22 Aug 2026 15:18:55 +0000

Vishal Chandak

Jul 27

Why AWS Is Tearing Down the "Tree"?

#aws #networktopology #networking #cloudcomputing

5 min read

From Massive to Miniature: How Small Language Models Are Engineered

Vishal Chandak — Sat, 22 Aug 2026 14:36:47 +0000

In Part 1, we started with a simple question:

Does every AI task need the biggest model we can afford?

Often, the answer is no.

A model that is considerably smaller than a frontier model can be the better choice for a well-defined workload—especially when latency, cost, privacy, or on-device execution matters.

But that immediately creates a harder question:

How do we build a smaller model that is still good enough?

This is where the conversation moves from strategy to engineering.

You cannot take a 400-billion-parameter model, delete 99% of its parameters, and expect the remaining billion parameters to magically retain everything the original model knew.

Model compression is not a file-size problem.

It is a capability-preservation problem.

And there are several different tools for attacking it.

The most important ones are:

Knowledge distillation
Quantization
Pruning
Fine-tuning
Parameter-efficient fine-tuning, especially LoRA and QLoRA

They sound similar because they all help make AI systems more efficient.

Under the hood, however, they do very different things.

First, let's separate two problems

Before getting into the techniques, there is an important distinction.

Suppose you have a 7B model and your application needs a 3B model.

There are actually two questions:

How do I make the model cheaper to run?

and:

How do I make the model better at my particular task?

Quantization primarily attacks the first problem.

Fine-tuning primarily attacks the second.

Distillation can address both, because it can transfer useful behavior from a larger teacher into a smaller student.

This distinction is useful because you can combine these techniques.

For example:

            Large Teacher
                  |
                  | Distillation
                  v
           Small Base Model
                  |
                  | Fine-tuning / LoRA
                  v
          Domain Specialist
                  |
                  | Quantization
                  v
         Efficient Deployment

That is not one optimization.

It's a pipeline.

1. Knowledge Distillation: Teach the Small Model

Let's start with the technique most closely associated with the idea of transferring capability from a larger model.

Knowledge distillation uses a larger model—the teacher —to help train a smaller student model.

The concept predates today's LLMs. Hinton, Vinyals, and Dean described a method for transferring knowledge from an ensemble of models into a smaller model that is easier to deploy.

The basic idea is beautifully simple.

Instead of asking the student to learn only from the original training labels, we also let it learn from the behavior of the teacher.

Conceptually:

        Input
          |
    +-----+------+
    | |
    v v
 Teacher Student
    | |
    v v
Teacher output Student output
    | |
    +-----+------+
          |
          v
       Loss
          |
          v
   Update Student

The teacher already contains useful information about the task.

The student learns to approximate that behavior.

Hard targets vs. soft targets

This is one of the most important ideas in traditional knowledge distillation.

Imagine a classification problem with three classes:

Cat 0.92

Dog 0.07

Rabbit 0.01

The hard label might simply be:

Cat

The hard label tells the student which answer is correct.

The soft distribution tells it something more subtle:

"This is overwhelmingly a cat, but it has some resemblance to a dog and very little resemblance to a rabbit."

That additional structure can contain useful information.

The original distillation work showed how these soft targets can transfer information from a larger model into a smaller one.

For modern language models, the picture becomes more complicated because the output is a sequence of tokens rather than a single class.

But the principle remains:

Don't just teach the student the answer. Teach it something about how the teacher behaves.

Distillation for LLMs is more complicated

With a language model, the teacher might generate:

"The crash is most likely caused by an invalid pointer dereference in the native library."

A straightforward approach is to train the student to reproduce that response.

But there are many ways to distill an LLM.

You can transfer:

generated answers,
token-level probabilities,
reasoning-oriented examples,
task-specific demonstrations,
intermediate representations,
preferences,
or specialized behavior.

Modern LLM distillation research has become a large field in its own right, with different approaches targeting algorithms, skills, and domain specialization.

This gives us an important correction to a common oversimplification:

Distillation isn't simply "copy the big model into the small model."

It is a family of training techniques for transferring useful behavior.

Why distillation is so attractive for SLMs

Imagine a large model is excellent at a particular task but far too expensive to deploy at scale.

You can use that model offline as a teacher.

Generate high-quality examples.

Train a smaller model on those examples.

Then deploy the smaller model for the high-volume workload.

The expensive teacher doesn't necessarily have to serve every production request.

That gives us an architecture like this:

          EXPENSIVE / OFFLINE

         +---------------+
         | Large Teacher |
         +-------+-------+
                 |
         Generate examples
                 |
                 v
         +---------------+
         | Training Data |
         +-------+-------+
                 |
                 v
         +---------------+
         | Small Student |
         +-------+-------+
                 |
                 v

          CHEAP / ONLINE

         Millions of requests
                 |
                 v
            Small Model

This separation between expensive capability acquisition and cheap production inference is one of the reasons distillation is so interesting for SLMs.

But distillation has a catch

A student cannot learn what the teacher never demonstrates.

If your distillation dataset contains only easy questions, the student may become excellent at easy questions and terrible at edge cases.

If the teacher itself makes systematic mistakes, those mistakes can be transferred too.

And if the student is substantially smaller than the teacher, there is a limit to how much information it can absorb.

So a good distillation pipeline isn't:

Teacher → dump outputs → train student → ship.

It's closer to:

Teacher
   |
   v
Generate candidate data
   |
   v
Filter / validate
   |
   v
Balance easy + hard examples
   |
   v
Train student
   |
   v
Evaluate against real workloads
   |
   +----> Fail? ----> Improve dataset
   |
   v
Deploy

The dataset becomes part of the engineering

2. Quantization: Fewer Bits, Less Baggage

Distillation changes the model itself.

Quantization attacks representation.

Neural-network parameters are stored as numerical values.

A model might commonly use formats such as:

FP32 — 32-bit floating point
FP16 — 16-bit floating point
BF16 — 16-bit brain floating point
INT8 — 8-bit integer
INT4 — 4-bit integer

The basic intuition is straightforward:

If we can represent the model's numbers using fewer bits, we can reduce its memory footprint.

For example, ignoring overhead and implementation details, storing a billion parameters at 16 bits requires roughly 2 GB just for the weights.

At 8 bits, it is roughly 1 GB.

At 4 bits, roughly 0.5 GB.

Those are simplified calculations, because real systems contain additional metadata, scaling factors, buffers, KV cache, runtime overhead, and other components.

But the intuition is important.

The numerical representation matters

Quantization isn't just "round the numbers"

A naive approach would be:

FP16 value
    |
    v
Round it
    |
    v
INT4 value

But neural networks contain distributions that aren't always friendly to naive quantization.

Some values matter disproportionately.

That means good quantization methods use scaling, calibration, mixed precision, or other techniques to preserve important information.

LLM.int8(), for example, demonstrated an approach that handled outlier features separately while performing the majority of multiplication in 8-bit precision. The authors reported that this allowed large models to be run with substantially lower memory requirements without the performance degradation they observed from simpler approaches.

The broader lesson is:

Quantization is an optimization problem, not simply a bit-counting exercise.

What quantization buys you

Depending on the model, hardware, and runtime, quantization can provide:

lower model memory,
lower bandwidth requirements,
the ability to run models on smaller GPUs,
improved feasibility for CPU or edge inference,
and potentially higher throughput.

But there are trade-offs.

Quantization can affect:

accuracy,
perplexity,
reasoning performance,
generation quality,
and sometimes latency.

And the trade-off isn't identical for every model.

A 4-bit model isn't automatically "better" than a 8-bit model.

You need to measure.

3. Pruning: Remove What You Don't Need

Quantization changes how parameters are represented.

Pruning tries to remove parameters or structures altogether.

The intuition is similar to trimming a tree.

Some branches contribute more than others.

If certain weights contribute very little to the final behavior, perhaps they can be removed.

There are several forms of pruning, including:

unstructured pruning,
structured pruning,
neuron pruning,
head pruning,
layer pruning,
and other architecture-specific approaches.

The trade-off is that sparsity is only useful if the hardware and runtime can exploit it.

Imagine removing 50% of the weights but still performing essentially the same dense matrix multiplication.

You may have a smaller file.

You may not have a faster model.

This is an important engineering distinction:

Compression does not automatically translate into acceleration.

A technique can reduce storage while providing little real-world latency benefit.

The three techniques so far

At this point, we can summarize the difference:

Technique	What changes?	Primary goal
Distillation	Training behavior	Transfer capability
Quantization	Numerical representation	Reduce memory/compute cost
Pruning	Model structure	Remove unnecessary computation
Fine-tuning	Model behavior	Specialize for a task
LoRA / QLoRA	Trainable parameters	Make adaptation cheaper

And these techniques can be combined.

That's where things get powerful.

4. Fine-tuning: Make the Model Care About Your Problem

Pre-trained language models are generalists.

They have learned from enormous and diverse datasets.

But your application probably doesn't need a generalist.

It needs something specific.

Imagine a device diagnostics application.

The model doesn't need to be an expert in Shakespeare.

It needs to understand:

crash signatures,
error messages,
device metadata,
component names,
severity levels,
and your organization's troubleshooting vocabulary.

Fine-tuning allows us to adapt a pre-trained model to a particular task or domain.

Instead of starting from zero:

Random weights
      |
      v
Train enormous model
      |
      v
General-purpose model

we start from an existing model:

Pre-trained model
       |
       v
Task-specific data
       |
       v
Fine-tuned specialist

This is dramatically more practical.

But traditional fine-tuning still has a problem.

You have to update the model's parameters.

For a large model, that can be expensive.

That's where parameter-efficient fine-tuning enters.

5. LoRA: Don't Rewrite the Whole Model

LoRA—Low-Rank Adaptation of Large Language Models—takes a clever approach.

Instead of updating all of the original model weights, LoRA freezes the pre-trained model and introduces small trainable matrices into the model's layers.

Conceptually:

          Original Model
         +--------------+
         | Frozen |

The base model stays frozen.

The adapter learns the task-specific modification.

The original LoRA paper demonstrated that this can dramatically reduce the number of trainable parameters and memory requirements compared with full fine-tuning while achieving comparable or better quality on the tasks they evaluated.

This changes the economics of specialization.

Instead of storing a complete copy of a model for every task, you can conceptually maintain:

             Base Model
                 |
    +------------+------------+
    | | |
    v v v
 Adapter A Adapter B Adapter C
 Medical Support Coding

The same base model can therefore support multiple specialized behaviors.

6. QLoRA: Quantization Meets LoRA

Now combine two ideas.

LoRA says:

Don't update the entire model.

Quantization says:

Don't store the model using unnecessarily high numerical precision.

QLoRA combines these ideas.

The base model is loaded in a quantized representation, while LoRA adapters are trained on top of it.

The QLoRA paper demonstrated that this approach could reduce memory requirements enough to fine-tune a 65B-parameter model on a single 48 GB GPU while maintaining the authors' reported 16-bit fine-tuning performance on their evaluated setup.

QLoRA introduced several components, including:

4-bit NormalFloat (NF4),
double quantization,
paged optimizers,
and LoRA adapters.

The important architectural idea is simpler than the terminology:

Keep the expensive base model compressed and frozen; learn a small amount of task-specific information on top.

This is where SLM engineering gets interesting

Now we can combine the techniques.

Suppose you want a specialized model for a production application.

A possible pipeline is:

            Large Teacher
                 |
                 | Distillation
                 v
          Small Base Model
                 |
                 | QLoRA / Fine-tuning
                 v
         Domain Specialist
                 |
                 | Quantization
                 v
         Deployment Model
                 |
         +-------+-------+
         | |
         v v
      Cloud Edge

Notice what happened.

We didn't simply "make an LLM smaller."

We built a specialized model optimized for a workload.

That is a fundamentally different mindset.

The hidden trade-off: every optimization changes something

There is a tendency in AI discussions to present optimization techniques as if they are free.

They're not.

Every technique introduces a trade-off.

Distillation

Can reduce model size while transferring useful behavior.

But the student can lose capabilities the teacher had, particularly outside the distilled distribution.

Quantization

Can dramatically reduce memory requirements.

But aggressive quantization can affect model quality, and the impact varies by model and task.

Pruning

Can reduce the number of parameters or operations.

But irregular sparsity may not translate into real speedups on the target hardware.

Fine-tuning

Can dramatically improve performance on a domain.

But a poorly designed dataset can cause overfitting, unwanted behavior, or loss of general capabilities.

LoRA

Can make specialization much cheaper.

But the adapter still depends on the underlying base model, and the chosen rank, target modules, training data, and task determine how effective it is.

QLoRA

Can make fine-tuning much more memory-efficient.

But quantized training introduces its own numerical and implementation considerations.

There is no magic compression button.

The benchmark trap

This is where experienced engineers should be particularly skeptical.

Suppose someone tells you:

"Our 3B model is almost as good as a 70B model."

The next question should be:

At what?

A model can perform extremely well on one benchmark while failing badly on another.

Even worse, a benchmark may not resemble your production workload.

Consider a model used for structured extraction.

The benchmark might measure semantic accuracy.

Your application might require:

{
  "severity": "critical",
  "component": "camera",
  "confidence": 0.93
}

If the model instead produces:

The severity appears to be critical and the affected componentis probably the camera. I would estimate confidence at around 93%.

a human might consider that a good answer.

Your parser might consider it a complete failure.

This is why application-level evaluation matters.

For an SLM, you should measure things like:

task accuracy,
structured-output validity,
latency,
memory usage,
throughput,
energy consumption where relevant,
failure rate,
escalation rate,
and recovery behavior.

The best model is the one that performs well across the metrics your application actually cares about.

A production SLM is more than a model file

This is perhaps the most important engineering lesson from Part 2.

When you deploy an SLM, you aren't deploying:

"model.bin"

You're deploying a system.

That system may include:

                User Request
                     |
                     v
             +---------------+
             | Preprocessor |
             +-------+-------+
                     |
                     v
             +---------------+
             | SLM |
             +-------+-------+
                     |
                     v
             +---------------+
             | Schema |
             | Validation |
             +-------+-------+
                     |
          +----------+----------+
          | |
        Valid Invalid
          | |
          v v
       Accept Retry /
                            Escalate
                                |
                                v
                              LLM

This is especially important for on-device applications.

A model might produce a semantically correct answer but violate the application's output contract.

Your runtime needs to handle that.

A model might run beautifully for a 2K-token context and then exhaust memory at 16K.

Your application needs to handle that.

A quantized model might be fast on one device and slower on another because the runtime doesn't have optimized kernels.

Your deployment system needs to handle that.

The model is only one component.

So, how do you actually choose?

There isn't one universally optimal compression strategy.

Instead, start with the deployment constraint.

If memory is the primary problem

Start by investigating quantization.

If task performance is the primary problem

Investigate fine-tuning or distillation.

If training cost is the primary problem

Investigate LoRA/QLoRA and parameter-efficient methods.

If the model contains unnecessary capacity

Investigate distillation or pruning.

If the device is extremely constrained

Combine techniques.

For example:

Distill → specialize → quantize → benchmark on the actual device.

And don't forget the final step.

Benchmark on the actual device.

A model that looks fantastic on an A100 benchmark may behave very differently on a phone.

The real optimization target

At the beginning of this article, we asked:

How do we make a model smaller without losing everything useful?

The answer isn't a single algorithm.

It's a sequence of trade-offs.

Distillation asks:

What knowledge can we transfer?

Quantization asks:

How precisely do we need to represent it?

Pruning asks:

What computation can we remove?

Fine-tuning asks:

What behavior does this application actually need?

LoRA asks:

How little of the model do we need to change?

QLoRA asks:

Can we do that while keeping the base model heavily compressed?

Together, these techniques let us move from a general-purpose model toward a model that is smaller, more specialized, and easier to deploy.

But that still leaves one major problem.

We've optimized the model.

We haven't yet optimized the system.

From model optimization to system optimization

Imagine we have three models:

Small model → Fast, cheap, limited reasoning
Medium model → Balanced
Large model → Expensive, powerful, broad reasoning

Which one should receive the next request?

If we always choose the large model, we've thrown away much of the benefit of SLMs.

If we always choose the small model, we'll eventually encounter tasks it can't handle.

The interesting solution is to make the system decide.

                   Request
                      |
                      v
                +-----------+
                | Router |
                +-----+-----+
                      |
         +------------+------------+
         | | |
         v v v
       Small Medium Large
       Model Model Model

Now the optimization problem becomes much more interesting.

We're no longer asking:

"How do I make one model do everything?"

We're asking:

"How do I use the right amount of intelligence for every request?"

That is the bridge between Part 2 and Part 3.

And it leads us to the final—and perhaps most important—idea in this series:

The future of efficient AI may not be a smaller model. It may be a system that knows when to use a small model.

What's next?

In Part 3, we move from model engineering to system orchestration.

We'll look at:

intelligent model routing;
SLM + LLM hybrid architectures;
confidence-based escalation;
agents and tool use;
local versus cloud execution;
measuring energy and water consumption;
the difference between model efficiency and system efficiency;
and whether an SLM-first architecture actually makes AI more sustainable.

Because "small model = green AI" is an appealing story.

But the real story is much more complicated.

And much more interesting.

Part 3: The Orchestrated Future

When the smartest AI system isn't the one with the smartest model—but the one that knows which model to use.

The Small Language Model Revolution: Why Fit Beats Force

Vishal Chandak — Tue, 11 Aug 2026 19:53:22 +0000

For the last few years, AI engineering has operated under a surprisingly simple assumption:

Bigger models are better models.

More parameters. More training data. More GPUs. More compute.

And to be fair, that strategy has worked remarkably well. Scaling has produced huge improvements in language understanding, coding, reasoning, multimodal capabilities, and general-purpose AI.

But engineering is rarely about maximizing one metric.

Eventually, the question changes from:

"Can the model solve this?"

to:

"What does it cost us to make the model solve this?"

That's where Small Language Models become interesting.

The problem with using a giant model for everything

Imagine a production system processing millions of AI requests every day.

A request might ask the model to:

classify a support ticket;
extract a few fields from a document;
identify the language of a message;
summarize a paragraph;
detect a known failure pattern in a device log.

These are useful AI workloads.

But they aren't all difficult reasoning problems.

If every request is sent to the largest model available, the architecture is effectively saying:

"Every problem deserves our most expensive reasoning engine."

That can work.

It can also be a terrible optimization strategy.

A better question is:

What is the smallest model that can solve this task reliably?

That question is at the heart of the SLM approach.

The goal isn't to prove that a 1B model is "as intelligent" as a 100B model. It isn't.

The goal is to recognize that model capability exists on a spectrum, while application requirements are usually much narrower.

A support-ticket classifier doesn't need to write a novel.

A document extractor doesn't need to solve an Olympiad problem.

A device assistant doesn't necessarily need the world's broadest knowledge.

If a smaller model can reliably do the job, using a larger one may simply be unnecessary.

So, what exactly is an SLM?

Here's where things get slightly messy.

There isn't one universally accepted parameter-count boundary that separates an SLM from an LLM. Different researchers and vendors use different definitions. Some emphasize parameter count. Others focus on memory, latency, deployment environment, or computational constraints.

For engineers, I find a capability-and-deployment definition more useful:

An SLM is a language model designed to provide useful language capabilities within a substantially smaller computational and memory footprint than large general-purpose models.

The important part is not the exact number of parameters. Because parameter count alone doesn't tell you how practical a model is.

Runtime memory depends on much more:

parameter precision;
architecture;
context length;
KV-cache size;
batch size;
inference runtime;
hardware.

That becomes especially important when the target isn't a GPU cluster.

It's a phone.

Small doesn't necessarily mean weak

Reducing model size doesn't necessarily mean randomly throwing away capability.

There are several techniques for improving the efficiency of language models.

Knowledge distillation transfers useful behavior from a stronger teacher model into a smaller student.

Quantization represents model parameters using fewer bits, reducing memory requirements and potentially improving inference efficiency.

Pruning removes parameters or structures that contribute less to computation.

Fine-tuning and parameter-efficient methods specialize an existing model for a particular domain or workflow.

None of these magically turns a small model into a frontier model.

What they can do is improve the amount of useful capability we get for a given resource budget.

And that is a much more interesting metric.

The device changes the equation

The difference becomes especially interesting when inference moves from the data center to the device.

A phone has a finite amount of:

RAM;
compute;
battery;
storage;
memory bandwidth;
thermal capacity.

And there is another memory consumer that developers sometimes underestimate:

the KV cache.

As context grows, the KV cache grows too. A model that comfortably fits into memory with a short prompt can behave very differently when an application starts processing long conversations or documents.

So the engineering question isn't simply:

"Can I fit the model on the phone?"

It's:

"Can I fit the entire inference workload on the phone while keeping the application responsive?"

A 2026 practitioner case study on integrating Qwen3 0.6B and Gemma 4 E2B into a production Android word-guessing game illustrates this very well. The authors encountered output-format violations, constraint violations, context degradation, latency problems, and model-selection instability. The final architecture deliberately reduced the amount of work delegated to the model and added deterministic fallbacks.

That gives us an important engineering lesson:

Putting a model on a device is an application-engineering problem, not just a model-download problem.

Current Scenario:

Use case: "We need to classify ERROR vs WARNING."

Engineering Team: Let's deploy the frontier model.

Three reasons engineers should care

The appeal of SLMs isn't simply that they're smaller.

Their smaller footprint can change three important characteristics of an AI application: cost, latency, and data locality.

A fourth benefit—deployment flexibility—often follows from those three.

1. Cost

Inference requires compute, and compute costs money.

At low request volumes, the difference between models may not matter much.

At very high volumes, however, even modest differences in per-request compute can become significant.

Suppose an application receives 100 million requests.

If a large fraction of those requests can be handled accurately by a smaller model, there is little reason to automatically send all 100 million to the most expensive model available.

Instead, expensive inference can be reserved for requests that actually need it.

This leads to an architectural principle we'll keep returning to:

Use expensive intelligence where expensive intelligence is actually necessary.

That doesn't mean "always use the cheapest model."

A cheap model that fails frequently can become expensive once retries, fallbacks, downstream failures, and human review are included.

So a more useful metric is: Cost per successful task.

2. Latency

Latency is another reason smaller or local models can be attractive.

A cloud request can involve network transfer, queuing, inference scheduling, generation, and response transmission.

A local model changes that equation.

It moves computation closer to the user.

For mobile assistants, interactive applications, robotics, device diagnostics, and other latency-sensitive workloads, that architectural difference can matter.

But we should avoid simplistic statements such as:

"SLMs are always X times faster."

They aren't.

Actual latency depends on:

model architecture;
quantization;
hardware;
inference runtime;
context length;
concurrency;
workload.

The defensible statement is:

Smaller models expand the range of workloads for which local and edge inference becomes practical

3. Privacy and data locality

Now consider an application processing sensitive information.

Perhaps it is analyzing:

proprietary source code;
internal documents;
customer information;
device telemetry;
industrial data;
confidential material.

A cloud architecture requires that information to cross a network boundary.

That doesn't automatically make the architecture insecure. Cloud AI systems can have strong security controls.

But it introduces another boundary that needs to be governed.

A local model offers another option:

Bring the model to the data instead of bringing the data to the model.

Again, this isn't a magic security solution.

A local model doesn't protect an insecure application.

But it can change the threat model and reduce the amount of information that needs to leave the device.

The scalpel and the Swiss Army knife

This is where the distinction between SLMs and large general-purpose models becomes useful.

Think of a large general-purpose model as a Swiss Army knife.

It has a huge range of capabilities and is useful when the shape of the problem is unknown.

An SLM is more like a specialized tool.

It may have considerably less general capability, but if the task falls within its intended operating range, it can be a much more efficient choice.

A larger model makes sense when:

the problem is genuinely open-ended;
complex multi-step reasoning is required;
the request combines many capabilities;
the cost of an incorrect answer justifies additional model capability.

A smaller model becomes attractive when:

the task is well defined;
the workload is repetitive;
request volume is high;
latency matters;
privacy or data locality matters;
the task can be evaluated automatically;
the model can be specialized.

The question therefore isn't:

"Which model is smarter?"

It's:

"Which model is sufficient?"

But there is a catch: the capability cliff

We also need to resist the hype.

SLMs aren't miniature versions of frontier models with exactly the same capabilities.

Smaller models can be remarkably capable on constrained or specialized workloads. But reducing model capacity can affect performance on unfamiliar problems, complex reasoning, and tasks requiring broad knowledge.

Compare:

"Classify this support ticket."

with:

"Read these conflicting reports, identify the hidden assumption, construct a counterexample, and explain why the conclusion doesn't follow."

Both requests involve language.

The second requires substantially more reasoning.

That's why benchmark results need context.

A small model can perform extremely well on a particular benchmark and still be a poor choice for your workload.

The right question isn't:

"What's this model's benchmark score?"

It's:

"How does this model perform on the tasks my application actually needs?"

Don't choose one model. Build a system.

This is where things get really interesting.

The future doesn't have to look like this:

                Every Request
                     |
                     v
                 +-----------+
                |    LLM    |
                +-----------+

Instead, introduce a routing layer:

                User Request
                     |
                     v
               +-----------+
               |   Router  |
               +-----+-----+
                     |
            +--------+--------+
            |                 |
            v                 v
       +---------+       +---------+
       |   SLM   |       |   LLM   |
       +---------+       +---------+
       Routine work     Complex work

A routine classification task might remain local.

A complicated reasoning problem can be escalated.

A privacy-sensitive request might stay on-device.

A request that fails validation can be retried or sent to a stronger model.

This is more powerful than simply choosing "the best model."

You're building a model hierarchy.

And that idea becomes central in Part 3.

What about sustainability?

AI systems ultimately consume physical resources.

Behind an inference request are processors, memory, networking, cooling, electricity, and physical infrastructure.

It's tempting to jump from that observation to:

"Small models are green."

That's too simplistic.

A smaller model may require less computation for a comparable workload, but environmental impact depends on the complete system.

We need to consider:

model architecture;
precision;
hardware;
utilization;
context length;
generated tokens;
retries;
routing;
data-center efficiency;
electricity source.

A small model that fails repeatedly and requires escalation may not be more efficient than a larger model that succeeds on the first attempt.

The environmental question therefore isn't simply:

"How much energy does this model use?"

It's:

"How much useful work does the complete system deliver per unit of resource consumption?"

We'll return to that question in Part 3 of this series.

The principle of least intelligence

There is a broader architectural principle hiding underneath all of this:

Give each task the minimum model capability required to solve it reliably.

Notice the important word:

reliably.

Sometimes a deterministic program is better than an SLM.

Sometimes an SLM is better than a large model.

Sometimes the large model is exactly what the task requires.

The engineering challenge is knowing which is which.

That means evaluating models against real workloads, not just parameter counts or headline benchmarks.

It also means building systems that can recover when the smaller model isn't good enough.

That might involve:

schema validation;
confidence estimation;
retries;
deterministic post-processing;
fallback models;
human review;
escalation.

The future of SLMs isn't just about making models smaller.

It's about making systems smarter about when and where intelligence is used.

What's next?

We've answered the why.

But there's an obvious question:

How do you actually make a model smaller without throwing away everything that makes it useful?

That's where things get interesting.

In Part 2 of this series, we'll open the machine shop.

We'll look at knowledge distillation, quantization, pruning, fine-tuning, LoRA, and QLoRA—and, more importantly, understand what each technique actually changes.

We'll also look at a question that's easy to gloss over:

When you make a model smaller, what capability did you actually lose?

If this helped you understand why SLMs matter, follow me for Part 2.

In a couple of days, we'll move from the architecture diagram to the actual engineering behind the model.

Part 2: From Massive to Miniature — How We Build Small Language Models.- Coming Soon

Why AWS Is Tearing Down the "Tree"?

Vishal Chandak — Mon, 27 Jul 2026 18:40:24 +0000

Amazon CEO Andy Jassy has been on a highly publicized crusade lately to crush corporate bureaucracy. His goal? Flatten the org chart, remove the layers of middle management that slow things down, and let individual teams move faster.

As it turns out, AWS’s network engineers took his mandate incredibly literally—they just applied it to their data centers.

For thirty years, the networking industry has been stuck with a rigid hierarchy known as the "fat-tree." In this model, data center architecture looks exactly like an aging, bloated corporate org chart. To move a packet of data from one server rack to its neighbor, the data can’t just walk across the hallway. It has to climb the corporate ladder—from Top-of-Rack (ToR) switches, up to aggregation layers, and finally to the "executive" spine switches—before trickling back down to its destination.

Every layer in this network acts exactly like a middle manager: it adds latency, drains power, and serves as a potential bottleneck. While the fat-tree has been the workhorse of the cloud, AWS is officially tearing it down. They are replacing it with a radically flat alternative called the Resilient Network Graph (RNG). By taking quasi-random graph theory out of academia and into hyperscale production, AWS has figured out how to commoditize randomness.

Here is how AWS is slashing their hardware footprint while actually making the network faster.

Takeaway 1: Why "Random" Beats "Ordered"

In traditional networking, order equals efficiency. But RNG proves the exact opposite: randomly connecting routers is mathematically superior to stacking them in a neat hierarchy.

In a traditional fat-tree, traffic is limited to strict pathways. Think of it like forcing all highway traffic through a few toll booths; it creates "small cuts," or bottlenecks where upper-layer links are jammed while other routes sit completely idle. Because of this, a fat-tree can strand up to 60% of its capacity.

In an RNG fabric, those toll booths don't exist. Every node has high bandwidth to all other nodes. This is powered by the Expansion Property of random graphs, creating "capacity fungibility"—a fancy way of saying no bandwidth ever goes to waste.

So why didn't we do this decades ago? Because it looked terrible on paper. As Giacomo Bernardi, AWS Principal Applied Scientist, noted:

*“It was typical for academia. Everybody's excited, but then the real world hits.”*

The real world meant an unmanageable hairball of physical cables and routing memory requirements that no standard switch could handle. Until now.

Takeaway 2: The "Magic Number" 69%

By firing the "middle managers" (removing the aggregation and spine layers entirely), AWS is pushing more data using drastically less metal. This shift to RNG is one of the most massive infrastructure optimizations in cloud history.

The impact breaks down into four staggering metrics:

69% fewer networking devices (routers and switches).
Up to 33% higher throughput compared to traditional fat-trees.
40% less power consumption for network equipment.
9% to 45% in cost savings , depending on the setup.

Beyond the balance sheet, eliminating thousands of power-hungry routers directly shrinks the carbon footprint of AWS's global grids. After all, the greenest energy is the power you never have to use in the first place.

Takeaway 3: The ShuffleBox (Taming the Spaghetti Monster)

The biggest barrier to flat networks has always been the physical wiring. Connecting routers hundreds of meters apart in a truly random pattern usually results in a logistical nightmare affectionately known as the "spaghetti monster."

AWS solved this with a secret weapon called the ShuffleBox.

Think of the ShuffleBox as a master illusionist. It’s a passive optical device that internally scrambles fiber wiring into a deterministic, random pattern. To the network, the topology looks wonderfully chaotic and random; to the technician on the floor, the cabling looks perfectly structured, neat, and maintainable. Because it’s completely passive, it requires no power and has no active parts that can fail.

Crucially, the ShuffleBox fixes the expansion problem. In older random graphs, adding a new server rack meant unplugging live cables, risking a total outage. With ShuffleBoxes, technicians can plug in new racks just like they always have, growing the network without disrupting live traffic.

🎥 Watch: How AWS Tames the Data Center "Spaghetti Monster" 📦⚡

Takeaway 4: Spraypoint Routing (Taking the Side Streets)

To navigate a structure-less network, AWS had to throw out the traditional networking rulebook. Standard routing logic uses too much memory—scaling it up would require 20 to 80 times more memory than commodity switches actually have.

Instead, AWS built a lightweight protocol called Spraypoint. It works on a simple "Spray and Point" strategy:

Spray: The source router essentially blasts the data out to all its immediate neighbors.
Point: The packets bounce through "rings" of waypoints surrounding the destination, guiding the data in from every possible angle.

It sounds counterintuitive. Why take a longer, zig-zagging path? Because in a flat network, taking a dozen different side streets is actually faster than getting stuck in a traffic jam on the main corporate highway.

🎥 Watch: SprayPoint Explained

Takeaway 5: Surviving the Blast Radius

The best argument for RNG brings us right back to Andy Jassy’s war on corporate structure: removing single points of failure.

In a hierarchical fat-tree, if a critical "VP" spine router fails, it can wipe out half the capacity for massive regions of a data center. The blast radius is catastrophic.

In an RNG topology, because there is no "top" of the network, a failure is barely a blip. If 1% of the routers fail, you only lose 1% of your capacity.

The network degrades proportionally, making outages virtually invisible to the end user. As Matt Rehder, VP of Network Engineering, put it:

“You turn it on, it functions. It's not something we want our customers to think about at all.”

The Future of the Flat Fabric

As of April 2026, this isn't just an ambitious science experiment. RNG is the global default for all new general-compute AWS data center builds. The network "tree" has been officially chopped down for standard workloads.

The final boss is specialized AI training. Because massive AI clusters require highly synchronized, centralized traffic patterns, they still rely on AWS's structured UltraServer architecture. But as RNG research evolves to handle these coordinated workloads, even those AI islands might eventually get flattened.

AWS has proven that chaos—when properly managed—scales better than order. The question now is whether competitors will follow them into the mesh, or remain stubbornly stuck in the trees while AWS enjoys a 69% hardware advantage.

Serverless vs Managed Servers

Vishal Chandak — Mon, 14 Dec 2020 05:56:43 +0000

Serverless seems really nice to begin with but does it make your bills a lot more heavier as you grow?

Uing AWS Lambda when the traffic received is around 100 transactions per seconds works fine and costs nominal but if the Transactions per seconds were to grow ten fold (or more) is Lambda still a viable option?

Along with Lamba, the Cloud watch costs extra to house the logs this is not an issue with managed servers.

Demystifying Machine Learning

Vishal Chandak — Thu, 29 Oct 2020 07:26:51 +0000

Even though a quick google search can give you plenty of “answers” for this question, it appears that they have actually not been satisfactory enough. The fact that you are reading this post should be enough to back my claim but if you are still not convinced, take a look at the google trends report for the term “Machine Learning”.

So what is Machine Learning? Let’s find out!

Before you go any further, I should warn you that this post is intended for people with little or no idea about the topic and thus has a super-simplified view of what Machine learning is.

To put it in Layman’s terms— Machine Learning is the ability to let machine form an equation to correctly solve any problem with little or no human intervention.

With Machine Learning, instead of programming the machine to do something, we let the machine learn to find a solution to the problem by itself.

“It sounds cool, but how is that it even possible.” — Your Brain

Hold your horses, we will get to that question in a minute. Before that, let’s understand how is it different than the “usual” way of solving a problem.

To understand this analogy better, let’s take a simple real-world example of — “Classifying shapes”.

Problem:

Given an image of a particular shape, let’s find out if the image is for a Square or Triangle.

Non Machine Learning Approach

The pseudo-code (fancy term programmers use to define “steps”) for that would look like:

Take the input Image
Find the number of corners in the image. This operation would basically include: (Don’t worry if you don’t understand all the terms below, neither do I)
- Convert the image from RGB (i.e. Colored Image) to Grayscale (i.e. B/w)
- Smooth-en out the image
- Use thresholding to convert the image to Bi-Level Binary Image
- Use an edge-detector algorithm to get the list of edges using pixel gradients
- Use contours to get the list of corners in the image
Write an if-else block to return the shape of the object based on the number of corners returned

Here’s a small visualization to help you better visualize what the above steps might look like:

Huh! You must be thinking — That seems a lot of work for such a trivial task. You are not entirely wrong but this has been the general approach to solve such problems until a few years back. Even though this approach “worked”, it had a lot of issues. For starters, it only worked well with perfectly captured images i.e. pictures taken under proper light and clear background which means that it will have a high percentage of failure in real-world scenarios.

Machine Learning Approach to the rescue

Let’s see what the same would look like with Machine Learning:

Gather some images for different shapes — squares and triangles.
Feed it to a ML Algorithm (more technical term would be a Convolutional Neural Network)
Run the image to be tested on the above trained model
Sit back and relax!

Now you must be thinking “Ok. That seems to be awfully easy. But how does that even work?” Let me answer that question with a question of my own — How would you teach a child a difference between a square and a triangle?

You won’t start by telling him about the number of corners or that the angles between them should be 90; Rather you would show him some images and help him relate that one of them is called a square and other is called a triangle.

Exactly! This is the way Machine Learning works too — based on feedback from the label provided with the data.

The below image might help understand it better:

Let’s take off the training wheels and see how far our newly trained child can go:

“All that’s cool but I don’t need a machine to categorize shapes.” — Almost everyone

To give a little more perspective on its potential - Machine Learning is the magic behind how Google lets you find your favorite cat photos (Who doesn’t love cats!) on the internet. If you look closely, almost everything you interact with on a day-to-day basis has machine learning weaved into it — Amazon product recommendations, Netflix movie recommendations, predicting stock prices, Alexa Voice, Sales forecasting, Automatic subtitle generation, Language translation and a lot more. The details pertaining to those is outside the scope of this post.
The major advantages with the Machine Learning approach is that:

You don’t need any domain knowledge to get started
The same classifier which can recognize shapes can also recognize numerical digits, cats vs dogs etc. since the Machine trains itself based on the input data
Can be continuously improved based on its own performance

The few disadvantages with Machine Learning are:

Needs a lot of training data for better predictions
Needs a lot of processing power for training the model and deriving inferences

However, the advent of the information age, providing us with humongous amount of data, and the drastic reduction in CPU and GPU costs over the years have helped solve both the above problems. Thus, it makes sense now more than ever, to have a generalized solution for such cases rather than a problem-specific solution.

That’s where Machine Learning comes into play. Sounds really cool, doesn’t it? If it’s so fascinating, why isn’t everyone trying to get their hands dirty with Machine Learning?

I will be addressing this question in my next post. Stay Tuned!

Until then, That’s all folks!

DEV Community: Vishal Chandak

What if corporate bureaucracy and data center networking suffered from the exact same problem? AWS just slashed 69% of their networking hardware by flattening their data centers and embracing quasi-random graph math. Read about how it really works here.

Why AWS Is Tearing Down the "Tree"?

From Massive to Miniature: How Small Language Models Are Engineered

First, let's separate two problems

1. Knowledge Distillation: Teach the Small Model

Hard targets vs. soft targets

Distillation for LLMs is more complicated

Why distillation is so attractive for SLMs

But distillation has a catch

2. Quantization: Fewer Bits, Less Baggage

Quantization isn't just "round the numbers"

What quantization buys you

3. Pruning: Remove What You Don't Need

The three techniques so far

4. Fine-tuning: Make the Model Care About Your Problem

5. LoRA: Don't Rewrite the Whole Model

6. QLoRA: Quantization Meets LoRA

This is where SLM engineering gets interesting

The hidden trade-off: every optimization changes something

Distillation

Quantization

Pruning

Fine-tuning

LoRA

QLoRA

The benchmark trap

A production SLM is more than a model file

So, how do you actually choose?

If memory is the primary problem

If task performance is the primary problem

If training cost is the primary problem

If the model contains unnecessary capacity

If the device is extremely constrained

The real optimization target

From model optimization to system optimization

What's next?

Part 3: The Orchestrated Future

The Small Language Model Revolution: Why Fit Beats Force

The problem with using a giant model for everything

So, what exactly is an SLM?

Small doesn't necessarily mean weak

The device changes the equation

Three reasons engineers should care

1. Cost

2. Latency

3. Privacy and data locality

The scalpel and the Swiss Army knife

But there is a catch: the capability cliff

Don't choose one model. Build a system.

What about sustainability?

The principle of least intelligence

What's next?

Why AWS Is Tearing Down the "Tree"?

​Takeaway 1: Why "Random" Beats "Ordered"

​Takeaway 2: The "Magic Number" 69%

​Takeaway 3: The ShuffleBox (Taming the Spaghetti Monster)

​Takeaway 4: Spraypoint Routing (Taking the Side Streets)

​Takeaway 5: Surviving the Blast Radius

​The Future of the Flat Fabric

Serverless vs Managed Servers

Demystifying Machine Learning

“It sounds cool, but how is that it even possible.” — Your Brain

Problem:

Non Machine Learning Approach

Machine Learning Approach to the rescue

“All that’s cool but I don’t need a machine to categorize shapes.” — Almost everyone

Takeaway 1: Why "Random" Beats "Ordered"

Takeaway 2: The "Magic Number" 69%

Takeaway 3: The ShuffleBox (Taming the Spaghetti Monster)

Takeaway 4: Spraypoint Routing (Taking the Side Streets)

Takeaway 5: Surviving the Blast Radius

The Future of the Flat Fabric