Deep Hybrid Self-Prior for Full 3D Mesh Generation


Fudan University     Zhejiang University     Google


We propose to utilize the deep hybrid 2D-3D self-prior in neural networks to generate the high-quality textured 3D mesh model from the sparse colored point cloud.



Abstract

We present a deep learning pipeline that leverages network self-prior to recover a full 3D model consisting of both a triangular mesh and a texture map from the colored 3D point cloud. Different from previous methods either exploiting 2D self-prior for image editing or 3D self-prior for pure surface reconstruction, we propose to exploit a novel hybrid 2D-3D self-prior in deep neural networks to significantly improve the geometry quality and produce a high-resolution texture map, which is typically missing from the output of commodity-level 3D scanners. In particular, we first generate an initial mesh using a 3D convolutional neural network with 3D self-prior, and then encode both 3D information and color information in the 2D UV atlas, which is further refined by 2D convolutional neural networks with the self-prior. In this way, both 2D and 3D self-priors are utilized for the mesh and texture recovery. Experiments show that, without the need of any additional training data, our method recovers the 3D textured mesh model of high quality from sparse input, and outperforms the state-of-the-art methods in terms of both the geometry and texture quality.




Network Architecture

Overview of our method. Our full model contains two building blocks, namely, 3D deep self-prior network, and 2D deep self-prior network, which run iteratively to improve the geometry and texture outputs.



Paper, Code and Data

X. Wen, Z. Chen, Y. Fu., Z. Cui, Y. Zhang

Deep Hybrid Self-Prior for Full 3D Mesh Generation.

ICCV, 2021.

[arXiv]     [Bibtex]     Codes coming soon    



Results


Comparison between our method and other surface reconstruction methods with real scans.



Acknowledgements

This work was supported in part by NSFC Project under Grant 62076067. The websiteis modified from this template.