LLMs are now available in snack size but digest with care

While distillation enables cost savings, faster inference, and better operational efficiency, distilled models inherit many security risks from their teacher models, along with a few others of their own.

Students take on the teacher’s burden

Distilled models inherit a huge part of their teacher model’s behavior, including any security risks embedded in their training data. These risks include intellectual property theft, privacy leaks, and model inversion attacks.

“Typical model distillation uses the training data originally consumed by the larger teacher model alongside the teacher model’s predictions of valid possible outputs (i.e. the probability distribution of outputs),” Brauchler said. “Consequently, the student model has the opportunity to memorize many of the same behaviors as the teacher model, including sensitive data in the training sets.”