Compressing Streaming Neural Audio Encoders via Latent-Space Distillation

Summary

Apple’s on-device dictation tokenizer maps waveform to latent representations for a sparsely activated language model, and compressing it reduces memory and latency demands. The authors compress it via latent-space distillation, using the teacher’s pre-quantizer latent as the supervision target rather than tokens or output distributions. A student encoder is trained to regress the teacher’s per-frame latent with a squared-error objective, and an affine layer absorbs width mismatch. This recipe works for both token interfaces and for tokenizers pretrained alone or jointly with a language model, requiring no fine-tuning. At 2.8x compression, the student stays within 1.9% relative WER of its teacher on five of six pairs and outperforms an independently trained tokenizer of identical capacity by 3.9% relative.

Takeaways

1. 1

   The distillation supervision target is the teacher’s pre-quantizer latent representation per frame, not discrete tokens or output distributions, enabling a single recipe for both token interfaces.

2. 2

   At 2.8x compression, the distilled student stays within 1.9% relative WER of its teacher on five of six teacher–student pairs without any fine-tuning.

3. 3

   The distilled student outperforms an independently trained tokenizer of the same capacity by 3.9% relative WER.

FAQ