Everything About ___
Live briefing

Everything about Knowledge distillation

Machine learning method to transfer knowledge from a large model to a smaller one

In machine learning, knowledge distillation or model distillation is the process of transferring knowledge from a large model to a smaller one. While large models have more knowledge capacity than small models, this capacity might not be fully utilized. It can be just as computationally expensive to evaluate a model even if it utilizes little of its knowledge capacity. Knowledge distillation transfers knowledge from a large model to a smaller one with the aim of retaining predictive performance. As smaller models are less expensive to evaluate, they can be deployed on less powerful hardware. Wikipedia →

🧭 Start here: a 6-step reading path

  1. Read the overview: Knowledge distillationA 5-minute orientation
  2. Skim a recent paper: Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New OutlooksInfluential work since 2022
  3. Pick up a classic book: The English house-wifeThe most-reprinted book on the subject
  4. Read the landmark paper: Knowledge Distillation: A SurveyCited 3,856 times
  5. Go sideways: Tensors in curvilinear coordinatesA closely related topic
  6. Go sideways: Infinitesimal strain theoryA closely related topic

📖 Definitions

No dictionary entry for this exact phrase.

🗂️ The subject at a glance

  1. Methods
  2. History
  3. Large language models

🏆 Landmark papers (most cited)

🆕 Influential since 2022

📚 Books worth reading

Ranked by number of editions on Open Library, a rough measure of how lasting a book is.

🔗 Related topics