Topic

Distillation

2 posts about Distillation. Every summary links to the original.

Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.

Newest first

Posts

Apple
Normalizing Trajectory Models
Diffusion models decompose sampling into many small Gaussian denoising steps, an assumption that breaks down when generation is compressed to a few coarse transitions. Existing few-step methods address this through distillation, consistency training, or adversarial objectives, but sacrifice the likelihood framework. Normalizing Trajectory Models (NTM) models each reverse step as an expressive conditional normalizing flow with exact likelihood training. Architecturally, NTM combines shallow invertible blocks within each step with a deep parallel predictor across the trajectory, forming an end-to-end network trainable from scratch or initializable from pretrained flow-matching models. On text-to-image benchmarks, NTM matches or outperforms strong image generation baselines in just four sampling steps while uniquely retaining exact likelihood over the generative trajectory.

Apple
Compressing Streaming Neural Audio Encoders via Latent-Space Distillation
Apple’s on-device dictation tokenizer maps waveform to latent representations for a sparsely activated language model, and compressing it reduces memory and latency demands. The authors compress it via latent-space distillation, using the teacher’s pre-quantizer latent as the supervision target rather than tokens or output distributions. A student encoder is trained to regress the teacher’s per-frame latent with a squared-error objective, and an affine layer absorbs width mismatch. This recipe works for both token interfaces and for tokenizers pretrained alone or jointly with a language model, requiring no fine-tuning. At 2.8x compression, the student stays within 1.9% relative WER of its teacher on five of six pairs and outperforms an independently trained tokenizer of identical capacity by 3.9% relative.

The best engineering reads, once a week.

Topic

2 posts about Distillation. Every summary links to the original.

Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.

Newest first

Apple
Normalizing Trajectory Models
Diffusion models decompose sampling into many small Gaussian denoising steps, an assumption that breaks down when generation is compressed to a few coarse transitions. Existing few-step methods address this through distillation, consistency training, or adversarial objectives, but sacrifice the likelihood framework. Normalizing Trajectory Models (NTM) models each reverse step as an expressive conditional normalizing flow with exact likelihood training. Architecturally, NTM combines shallow invertible blocks within each step with a deep parallel predictor across the trajectory, forming an end-to-end network trainable from scratch or initializable from pretrained flow-matching models. On text-to-image benchmarks, NTM matches or outperforms strong image generation baselines in just four sampling steps while uniquely retaining exact likelihood over the generative trajectory.

Apple
Compressing Streaming Neural Audio Encoders via Latent-Space Distillation
Apple’s on-device dictation tokenizer maps waveform to latent representations for a sparsely activated language model, and compressing it reduces memory and latency demands. The authors compress it via latent-space distillation, using the teacher’s pre-quantizer latent as the supervision target rather than tokens or output distributions. A student encoder is trained to regress the teacher’s per-frame latent with a squared-error objective, and an affine layer absorbs width mismatch. This recipe works for both token interfaces and for tokenizers pretrained alone or jointly with a language model, requiring no fine-tuning. At 2.8x compression, the student stays within 1.9% relative WER of its teacher on five of six pairs and outperforms an independently trained tokenizer of identical capacity by 3.9% relative.

The best engineering reads, once a week.