---
title: "Compressing Streaming Neural Audio Encoders via Latent-Space Distillation"
description: "Apple’s on-device dictation tokenizer maps waveform to latent representations for a sparsely activated language model, and compressing it reduces memory and latency demands. The authors compress it via latent-space distillation, using the teacher’s pre-quantizer latent as the supervision target rather than tokens or output distributions. A student encoder is trained to regress the teacher’s per-frame latent with a squared-error objective, and an affine layer absorbs width mismatch. This recipe works for both token interfaces and for tokenizers pretrained alone or jointly with a language model, requiring no fine-tuning. At 2.8x compression, the student stays within 1.9% relative WER of its teacher on five of six pairs and outperforms an independently trained tokenizer of identical capacity by 3.9% relative."
---

# Compressing Streaming Neural Audio Encoders via Latent-Space Distillation

[Apple](https://yomu.fyi/company/apple) · Sep 24, 2026

**Type:** Problem & solution

## Summary

Apple’s on-device dictation tokenizer maps waveform to latent representations for a sparsely activated language model, and compressing it reduces memory and latency demands. The authors compress it via latent-space distillation, using the teacher’s pre-quantizer latent as the supervision target rather than tokens or output distributions. A student encoder is trained to regress the teacher’s per-frame latent with a squared-error objective, and an affine layer absorbs width mismatch. This recipe works for both token interfaces and for tokenizers pretrained alone or jointly with a language model, requiring no fine-tuning. At 2.8x compression, the student stays within 1.9% relative WER of its teacher on five of six pairs and outperforms an independently trained tokenizer of identical capacity by 3.9% relative.

## Context

On-device dictation on Apple devices uses a neural tokenizer to map speech waveforms to latent representations for a sparsely activated language model. The always-on tokenizer competes for DRAM with the language model’s sparse experts, so its parameter count directly impacts power and latency.

## Approach / What changed

Train a student encoder to regress the teacher’s per-frame latent representation using a squared-error objective, with a single affine layer to absorb width mismatch. This targets the pre-quantizer latent shared by both token interfaces, allowing one recipe for both token types and for pretrained or jointly trained tokenizers.

## Takeaways

- The distillation supervision target is the teacher’s pre-quantizer latent representation per frame, not discrete tokens or output distributions, enabling a single recipe for both token interfaces.
- At 2.8x compression, the distilled student stays within 1.9% relative WER of its teacher on five of six teacher–student pairs without any fine-tuning.
- The distilled student outperforms an independently trained tokenizer of the same capacity by 3.9% relative WER.

## FAQ

### What supervision target is used for distillation?

The teacher's pre-quantizer latent representation per frame, rather than discrete tokens or output distributions, because it is the last representation shared by both token interfaces before the quantizer and LM bridge.

### How is the teacher–student width mismatch handled?

A single affine layer is inserted into the student to absorb the difference in width, allowing regression with squared-error loss without further architectural changes.

### Does the method work for tokenizers jointly trained with a language model?

Yes, the recipe applies both to a tokenizer pretrained alone and to one jointly trained with a language model, requiring no fine-tuning afterward.

**Tags:** [Distillation](https://yomu.fyi/topic/distillation), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Neural Audio Encoders](https://yomu.fyi/topic/neural-audio-encoders)

- Source: [Apple](https://machinelearning.apple.com/research/latent-space-distillation)
- Source URL: https://machinelearning.apple.com/research/latent-space-distillation
- Ingested by Yomu: 2026-09-24T18:00:08.917Z

[Read original post](https://machinelearning.apple.com/research/latent-space-distillation)