Spatio-temporal 3D reconstruction from frame sequences and feature points

Federico G., Carrara F., Amato G., Di Benedetto M.

Denoising diffusion probabilistic model, Signed distance field, 3D reconstruction, Video reconstruction, Deep Learning, Machine Learning, Artificial Intelligence 

Reconstructing a large real environment is a fundamental task to promote eXtended Reality adoption in industrial and entertainment fields. However, the short range of depth cameras, the sparsity of LiDAR sensors, and the huge computational cost of Structure-from-Motion pipelines prevent scene replication in near real time. To overcome these limitations, we introduce a spatio-temporal diffusion neural architecture, a generative AI technique that fuses temporal information (i.e., a short temporally-ordered list of color photographs, like sparse frames of a video stream) with an approximate spatial resemblance of the explored environment. Our aim is to modify an existing 3D diffusion neural model to produce a Signed Distance Field volume from which a 3D mesh representation can be extracted. Our results show that the hallucination approach of diffusion models is an effective methodology where a fast reconstruction is a crucial target.

Publisher: Association for Computing Machinery

Metrics