Luce: Relightable Gaussians for 3D Asset Generation

Summary

High-fidelity image-to-3D asset generation requires 3D representations that capture geometry alongside physically based rendering modalities for seamless pipeline integration and relighting. Luce addresses this challenge by unifying geometry and material properties into a voxelized multimodal Gaussian cloud with dedicated primitives for albedo, metallic-roughness, and surface normals. A variational autoencoder compresses these multimodal primitives into a unified material-aware latent space. A rectified-flow transformer subsequently generates this latent from a single input image using multi-layer features extracted from a pretrained image encoder. On the Toys4K dataset, Luce improves FID by 28% over the leading baseline while maintaining fine details such as text and logos.

Takeaways

1. 1

   Luce represents geometry and physically based rendering modalities, including albedo, metallic-roughness, and surface normals, using dedicated primitives inside a voxelized multimodal Gaussian cloud.

2. 2

   The architecture uses a variational autoencoder to compress multimodal 3D Gaussians into a latent space and a rectified-flow transformer conditioned on multi-layer image encoder features for generation.

3. 3

   On the Toys4K benchmark, Luce improves FID by 28% over the strongest baseline and achieves a CLIP image-alignment score of 0.8519 compared to 0.8299 on AI-generated images.