Beyond human imagination: the art of creating prompt-driven 3D scenes with Generative AI

Federico G., Carrara F., Amato G., Di Benedetto M.

Generative AI, Computer Graphics, Denoising Diffusion Probabilistic Model, Gaussian Splatting, NeRF, Signed Distance Field, Video Reconstruction, Deep Learning, Machine Learning, Artificial Intelligence, Text-to-3D, Image-to-3D, Urban Environment, Score Distillation Sampling 

Reconstructing large-scale outdoor environments is essential for advancing XR applications but is hindered by the high cost and limitations of traditional methods like LiDAR, depth sensors, and photogrammetry. We propose generative neural architectures to address these issues. Our initial Spatio-Temporal Diffusion model combines temporal image sequences and coarse spatial data with a novel SDF_MIP representation for efficient training. Building on this, we introduce Neural-Clipmap, a scalable framework using an enhanced octree structure and Triplane representations to refine 3D reconstructions iteratively. Additionally, we leverage monocular RGB image sequences with 2D diffusion priors via Score Distillation Sampling (SDS) to reconstruct missing data, addressing challenges like initialization coherence and color accuracy through a multi-phase inpainting process. These approaches reduce resource requirements while enabling efficient, high-quality reconstructions.

Source: VTT TECHNOLOGY, vol. 432, pp. 201-206. Athens, Greece, 27-29 Novembre 2024