Panoramic videos are becoming more and more easily obtained for common users. Although these videos have 360∘ field of view, they are usually displayed with perspective views, which needs the saliency informations for viewing angle selection. In this paper, we propose a saliency prediction network for 360∘ videos. Our network takes video frames and optical flows in cube map format as input, thus it does not suffer from image distorations of panoramic frames.