DEV Community

Julien Simon
Julien Simon

Posted on • Originally published at julsimon.Medium on

Deep Dive: Model Distillation with DistillKit

In this deep dive video, we zoom in on model distillation, an advanced technique to build high-performance small language models at a reasonable cost.

We first explain what a model distillation is. Then, we introduce two popular techniques for distillation, logits distillation, and hidden states distillation. We study in detail how they work, and how they’re implemented in the Arcee DistillKit open-source library. Finally, we look at two Arcee models built with distillation, Arcee SuperNova 70B and Arcee SuperNova Medius 14B.

Top comments (0)

A Workflow Copilot. Tailored to You.

Pieces.app image

Our desktop app, with its intelligent copilot, streamlines coding by generating snippets, extracting code from screenshots, and accelerating problem-solving.

Read the docs

👋 Kindness is contagious

Engage with a sea of insights in this enlightening article, highly esteemed within the encouraging DEV Community. Programmers of every skill level are invited to participate and enrich our shared knowledge.

A simple "thank you" can uplift someone's spirits. Express your appreciation in the comments section!

On DEV, sharing knowledge smooths our journey and strengthens our community bonds. Found this useful? A brief thank you to the author can mean a lot.

Okay