LIDAR and Monocular Camera Fusion: On-road Depth Completion for Autonomous Driving

October 2019

LIDAR and Monocular Camera Fusion: On-road Depth Completion for Autonomous Driving

Authors:

Chen Fu, Christoph Mertz, and John M. Dolan

Abstract:

LIDAR and RGB cameras are commonly used sensors in autonomous vehicles. However, both of them have limitations: LIDAR provides accurate depth but is sparse in vertical and horizontal resolution; RGB images provide dense texture but lack depth information. In this paper, we fuse LIDAR and RGB images by a deep neural network, which completes a denser pixel-wise depth map. The proposed architecture reconstructs the pixel-wise depth map, taking advantage of both the dense color features and sparse 3D spatial features. We applied the early fusion technique and fine-tuned the ResNet model as the encoder. The designed Residual Up-Projection block recovers the spatial resolution of the feature map and captures context within the depth map. We introduced a depth feature tensor which propagates context information from encoder blocks to decoder blocks. Our proposed method is evaluated on the large-scale indoor NYUdepthV2 and KITTI odometry datasets, on which it outperforms the state-of-the-art single RGB image and depth fusion method. The proposed method is also evaluated on a reduced-resolution KITTI dataset which synthesizes the planar LIDAR and RGB image fusion.
@conference{Fu-2019-118749,
author = {Chen Fu And Christoph Mertz And John M. Dolan},
title = {LIDAR and Monocular Camera Fusion: On-road Depth Completion for Autonomous Driving},
booktitle = {Proceedings of IEEE Intelligent Transportation Systems Conference (ITSC '19)},
year = {2019},
month = {October},
pages = {273 - 278},
keywords = {autonomous driving, perception, LIDAR, computer vision, depth completion, sensor fusion, deep learning},
}
Copyright notice: This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. These works may not be reposted without the explicit permission of the copyright holder.