Luce: Relightable Gaussians for 3D Asset Generation

Summary

High-fidelity image-to-3D asset generation requires 3D representations that capture geometry alongside physically based rendering modalities for seamless pipeline integration and relighting. Luce addresses this challenge by unifying geometry and material properties into a voxelized multimodal Gaussian cloud with dedicated primitives for albedo, metallic-roughness, and surface normals. A variational autoencoder compresses these multimodal primitives into a unified material-aware latent space. A rectified-flow transformer subsequently generates this latent from a single input image using multi-layer features extracted from a pretrained image encoder. On the Toys4K dataset, Luce improves FID by 28% over the leading baseline while maintaining fine details such as text and logos.

Context

High-fidelity image-to-3D generation requires 3D representations that capture geometry and appearance while supporting relighting and standard rendering pipeline integration via physically based rendering (PBR) modalities such as albedo, metallic-roughness, and surface normals.

Approach / What changed

Luce unifies geometry and PBR materials within a voxelized multimodal Gaussian cloud using dedicated Gaussian primitives for each modality. A variational autoencoder compresses the representation into a unified material-aware latent space, which a rectified-flow transformer generates from a single image conditioned on multi-layer features from a pretrained image encoder. The generated latent decodes into relightable PBR Gaussians and an optional textured mesh with a tangent-space normal map.

Takeaways

• Luce represents geometry and physically based rendering modalities, including albedo, metallic-roughness, and surface normals, using dedicated primitives inside a voxelized multimodal Gaussian cloud.
• The architecture uses a variational autoencoder to compress multimodal 3D Gaussians into a latent space and a rectified-flow transformer conditioned on multi-layer image encoder features for generation.
• On the Toys4K benchmark, Luce improves FID by 28% over the strongest baseline and achieves a CLIP image-alignment score of 0.8519 compared to 0.8299 on AI-generated images.

Related reading

• Apple · Aug 25, 2026

  STARFlow2: Bridging Language Models and Normalizing Flows for Unified Multimodal Generation

  Existing multimodal models struggle with structural fragmentation, often sacrificing visual fidelity via discrete tokenization or breaking structural symmetry by pairing causal text generation with diffusion-based denoising. STARFlow2 addresses these limitations by leveraging autoregressive normalizing flows, which share the causal mask, key-value cache mechanism, and left-to-right structure of large language models. Built upon the Pretzel architecture, the system vertically interleaves a frozen pretrained vision-language model stream with a TARFlow stream through residual skip connections under a single causal mask. A deep-shallow flow design combined with a unified FAE latent space allows both text and continuous visual outputs to enter the key-value cache directly without re-encoding. Benchmark experiments validate strong performance across multimodal understanding and continuous image generation tasks.