Spatial-Angular Versatile Convolution for Light Field Reconstruction

Spatial-angular separable convolution (SAS-conv) has been widely used for efficient and effective 4D light field (LF) feature embedding in different tasks, which mimics a 4D convolution by alternatively operating on 2D spatial slices and 2D angular slices. In this paper, we argue that, despite its global intensity modeling capabilities, SAS-conv can only embed local geometry information into the features, resulting in inferior performances in the regions with textures and occlusions. Because the epipolar lines are highly related to the scene depth, we introduce the concept of spatial-angular correlated convolution (SAC-conv). By alternating 2D convolutions on the vertical and horizontal epipolar slices, SAC-conv can embed global and robust geometry information into the features. We verify that SAS-conv and SAC-conv are skilled at different aspects of 4D LF feature embedding through a detailed feature and error analysis. Based on their complementarity, we further combine SAS-conv and SAC-conv by a parallel residual connection, forming a new spatial-angular versatile convolution (SAV-conv) module. We conduct comprehensive experiments on two representative LF reconstruction tasks, i.e., LF angular super-resolution and LF spatial super-resolution. Both the quantitative and qualitative results demonstrate that, without any extra parameters, networks upgraded with our proposed SAV-conv notably outperform those upgraded with SAS-conv and achieve a new state-of-the-art performance.

The 4D light field (LF), first introduced in [1], records the intensity of the light rays emitted from different positions and along different angles. With such redundant 4D spatio-angular information, LF imaging makes many new applications possible and popular, such as post-capture refocusing [2], seeing through occlusions [3] and stereoscopic displays [4]. Moreover, many classic computer vision tasks, such as super-resolution, depth estimation, and material recognition, benefit from 4D LFs compared with 2D images [5], [6], [7], [8], [9] due to the additional geometric information encoded in multiple adjacent views. With the prosperity of deep learning techniques, convolutional neural networks (CNNs) now dominate most LF processing tasks. For these tasks, efficient and effective modeling of the redundant 4D information for LF feature embedding is essential and has attracted increased research attention recently.