Abstract

The accurate reconstruction of per-pixel depth for an image is vital for many tasks in computer graphics, computer vision, and robotics. In this paper, we present a novel approach to generate view consistent and detailed depth maps from a number of posed images. We leverage advances in monocular depth estimation, which generate topologically complete, but metrically inaccurate depth maps and refine them in a two-stage optimization process based on a differentiable renderer. Taking the monocular depth map as input, we first transform the map to a triangle surface mesh and scale this map to absolute distances based on structure-from-motion. We then refine this depth mesh in a local optimization, enforcing photometric and geometric consistency.

Our evaluation shows that our method is able to generate dense, detailed, high-quality depth maps, also in challenging indoor scenarios, and outperforms state-of-the-art depth reconstruction approaches.

Pipeline

Overview of our method: We employ monocular depth estimation for a relative, but topologically complete depth map. Results from Structure-from-Motion are used to scale the depth map to absolute space. Following, we convert the depth map to a surface mesh which is refined via differentiable rendering. The refinement is done in two consecutive steps: first, we learn a mapping function that smoothly aligns the depth map to the sparse point cloud and, second, we refine per-vertex positions resulting in accurate, absolute depth maps.

Coarse and Local Refinement

After the initial scale approximated from the sparse point cloud, we refine the depth map in two conscutive steps. First, we train a shallow MLP that coarsely align the initialization to the the sparse point cloud. Second, the meshed depth map is optimized on a vertex-basis to achieve multi-view consistency.

Citation

Acknowledgements

We would like to thank all members of the Visual Computing Lab Erlangen for the fruitful discussions. Specifically, we appreciate Darius Rückert's support during late night debugging sessions.
The authors gratefully acknowledge the scientific support and HPC resources provided by the National High Performance Computing Center of the Friedrich-Alexander-Universität Erlangen-Nürnberg (NHR@FAU) under the project b212dc. NHR funding is provided by federal and Bavarian state authorities. NHR@FAU hardware is partially funded by the German Research Foundation (DFG) – 440719683. Linus Franke was supported by the 5G innovation program of the German Federal Ministry for Digital and Transport under the funding code 165GU103B. Joachim Keinert was supported by the Free State of Bavaria in the DSAI project.

The website template was adapted from VET, who borrowed from Zip-NeRF, who borrowed from Michaël Gharbi and Ref-NeRF. Image sliders are from BakedSDF.

References