Fusion based hyperspectral image (HSI) super-resolution method, which obtains a spatially high-resolution (HR) HSI by fusing a low-resolution (LR) HSI and an HR conventional image, has been a prevalent method for HSI super-resolution. One effective fusion based method is to cast HSI super-resolution into a unified optimization problem, where handcrafted priors such as sparse prior or low rank prior are always adopted to regularize the latent HR HSI to be optimized. However, these priors show limitations in generalizing to challenging cases due to the heuristic assumption on image statistics as well as the restricted expressiveness capacity of the shallow structure. Taking advantages of the powerful expression ability of deep learning based method, a new HSI super-resolution network is proposed which implicitly incorporates a deep structure as the regularizer/prior. Specifically, we reformulate the original unified optimization problem into three sub-optimization problems, one is related with the regularizer and the others are without. Thanks to the fact that the one related with the regularizer naturally equals to a denoising problem, a recursive residual network is proposed for this sub-optimization problem. In addition, we unfold the other sub-optimization problems into network representations, with which the original unified optimization problem can be represented into a fully end-to-end network. Experimental results shows the superiority of the proposed method for HSI super-resolution on three benchmark datasets.
