Key facts
| Definition | Training a smaller student model to imitate a larger teacher model's behaviour |
| Why it matters | Lower inference cost and latency at equal quality on a narrow task |
| Training signal | Teacher outputs, logits or preference pairs; quality of data dominates results |
| Best fit | High-volume, well-defined tasks such as classification, extraction or routing |
| Limits | A student cannot exceed the teacher, and narrow training narrows generality |
| Plugsky today | Generate teacher data with live chat completions; distillation is a coming-soon endpoint |
| Models | Use a strong teacher and a small target from the 30+ model catalogue |
| Cost structure | Upfront data generation and training cost against ongoing inference savings |
TL;DR
- Distillation trades training effort for cheaper, faster inference.
- It pays off on narrow, high-volume tasks — not general assistants.
- Data quality and coverage matter more than training tricks.
- The student never exceeds the teacher's ceiling.
- Plugsky distillation is coming soon; build teacher datasets now with live chat.
How it works, step by step
- Pick a narrow task with high request volume and a measurable success criterion.
- Generate teacher outputs for a diverse, representative prompt set through the chat API.
- Filter aggressively: keep examples where the teacher was correct and consistent.
- Train or fine-tune a smaller model on the filtered dataset with held-out validation.
- Evaluate the student against the teacher on your metric, not on loss alone.
- Deploy the student for the narrow task and fall back to the teacher on low confidence.
Try it yourself
Open the LLM cost calculator →
How distillation works
A teacher model — large, expensive, accurate — processes a large set of prompts. Its outputs, probability distributions or preference rankings become training data for a smaller student model. The student learns to reproduce the teacher's behaviour on that distribution. Because the task is narrower than open-ended chat, a small model can match the teacher on it while running faster and cheaper. The method is data-centric: coverage of real inputs matters more than exotic training objectives.
When distillation is worth it
- High volume, low variety: classification, tagging, extraction and routing.
- Latency sensitivity: interactive features where a smaller model responds faster.
- Cost pressure: unit economics that a frontier model cannot meet at scale.
- Stable task definition: a schema or policy that does not change weekly.
- On-device or edge: deployment targets where only small models fit.
Distillation is a poor fit for open-ended assistants, rapidly changing requirements, or tasks where the teacher itself is unreliable.
Common mistakes
- Training on unfiltered teacher output, including the teacher's mistakes.
- Covering only easy examples, so the student fails on real edge cases.
- Evaluating with training loss instead of task accuracy against the teacher.
- Forgetting that a distilled model is frozen to its training distribution.
- Underestimating data generation cost — teachers bill per token like any other call.
Distillation and Plugsky
Distillation and fine-tuning are documented as coming-soon endpoints, so managed training is not part of the live stack today. What is live is the data-generation path: use chat completions with a strong model to produce and label teacher outputs, embeddings to deduplicate and select representative examples, and function calling to enforce output schemas during generation. Export the dataset and run training with your own tooling. When managed distillation becomes available, the same dataset will be reusable.
Honest comparison
| Approach | Distilled student | Teacher model directly | Prompt engineering |
|---|---|---|---|
| Inference cost | Lowest | Highest | Unchanged |
| Latency | Fastest | Slowest | Unchanged |
| Quality ceiling | Approaches teacher on narrow tasks | Highest | Depends on prompt |
| Upfront work | Dataset plus training | None | Prompt iteration |
| Flexibility | Frozen to training task | General | General |
Frequently asked questions
What is distillation in AI?
Training a smaller student model to reproduce the behaviour of a larger teacher model on a defined task, so inference becomes cheaper and faster while quality stays close on that task.
Does distillation reduce model quality?
On the narrow task it is trained for, quality can approach the teacher. Outside that distribution it degrades quickly, so do not use a distilled model as a general assistant.
How much data does distillation need?
Enough diverse examples to cover real inputs, typically thousands rather than dozens. Coverage and correct labels matter more than raw volume.
Can I distill using Plugsky today?
Plugsky's distillation endpoint is coming soon. You can build the dataset now with live chat completions and embeddings, then train with external tooling.
How is distillation different from fine-tuning?
Fine-tuning adapts a model to your data directly. Distillation specifically transfers behaviour from a larger teacher into a smaller student, often using the teacher's outputs as labels.
When should I not distill?
When the task is open-ended, the requirements change frequently, volumes are low, or the teacher is not accurate enough to be a reliable label source.