AppleProblem & solution
Split-and-Scale Latent 3D Representations
Summary
Latent 3D representations enable compact encoding of scenes and support generative models, but they struggle to capture fine details due to limited latent capacity. Scaling capacity at training is computationally expensive and often infeasible under memory limits, while naively increasing latents at inference fails to use added capacity effectively. Split-and-scale addresses both challenges by representing 3D scenes as continuous functions of 3D coordinates, allowing a single pretrained tokenizer to be applied at inference to arbitrarily small sub-regions. This allocates the same latent capacity to a smaller spatial extent, capturing far higher detail. Extensive ablations show inference-time scaling achieves quality competitive with much larger models that exceed single-GPU memory limits, and the method supports high-quality generative modeling with an autoregressive 3D generator competitive with state-of-the-art baselines.
Context
Latent 3D representations compactly encode scenes but are limited in capacity, making it difficult to capture fine details. Scaling capacity at training is memory-prohibitive, and naive inference scaling is ineffective.
Approach / What changed
Split-and-scale: at inference, apply a pretrained tokenizer to arbitrarily small sub-regions of a 3D scene, allocating the same latent capacity to a smaller spatial extent to capture higher detail.
Takeaways
Split-and-scale effectively expands latent capacity at inference while keeping training cost tractable.
The method achieves reconstruction quality competitive with much larger models that exceed single-GPU memory limits.
The approach supports high-quality generative modeling, with an autoregressive 3D generator performing competitively with state-of-the-art baselines.
FAQ
How does split-and-scale increase latent capacity without retraining?
By representing 3D scenes as continuous functions of 3D coordinates, a pretrained tokenizer can be applied to arbitrarily small sub-regions, allocating the same capacity to a smaller spatial extent.
What trade-offs does the method study?
The paper conducts extensive ablations on trade-offs between compute, memory, latent capacity, and reconstruction quality.
Can split-and-scale be used for generative modeling?
Yes, the method supports high-quality generative modeling; an autoregressive 3D generator trained with it performs competitively with state-of-the-art baselines.
Source and freshness
- Source
- machinelearning.apple.com
- Published
- Added to Yomu
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.