Simple Yet Effective Way for Improving the Performance of Depth Map Super-Resolution

Depth information is essential for many computer vision applications, such as autonomous navigation, virtual reality, and 3D reconstruction. Currently, many low-cost but low-resolution (LR) depth sensors are widely used, and they are generally embodied with high-resolution (HR) color sensors [21]–[22][23]. Therefore, many studies have examined the super-resolution (SR) of depth images using HR color images as guidance [1]–[2][3][4][5][6][7][8][9], [16]–[17][18][19].

As in the case with other computer vision tasks, convolutional neural networks (CNNs) are currently dominating depth SR [1]–[2][3][4][5][6][7][8]. Motivated by classical color-guided depth SR methods [16]–[17][18][19], CNN-based methods attempt to use color features such that the resultant HR depth image can inherit the HR details of the color image. The main difference in the existing CNN-based methods involves the use of color features for depth SR.

Hui et al. [1] achieved promising SR results by using a multi-scale guided network (MSG-Net) that used the HR intensity image as guidance to complement depth features. Considering that the edges play the most important role in depth SR, Wen et al. [4] developed a handcrafted preprocessing method that refined the initially interpolated depth map using the corresponding color pixels to preserve details. On the other hand, some approaches inferred the HR depth edge map using the HR color image and the LR depth map, and used the HR edge map as the supplementary information to refine the boundaries of the depth map [5], [6]. To further improve the performance, Guo et al. [7] employed rich hierarchical features from the HR intensity image through the residual U-Net [24] architecture. Moreover, with the HR intensity image, Zuo et al. [8] progressively upscaled the LR depth map with global and local residual learning. In addition, Zhao et al. [9] simultaneously increased the resolution of the color image and the depth map using the generative adversarial network [20]. Most of the aforementioned methods [1]–[2][3], [6]–[7][8][9] used the guide features simply by concatenating them with the depth features. Therefore, the depth SR network needs to learn how to use two types of features properly, which remains a challenge. Indeed, the sharpness of the depth boundaries can be lost without sufficient usage of the guide features. Otherwise, if the guide features are used excessively in flat areas, unnecessary high frequency color details can be transferred from the guide image to the depth map.