Dense Video Captioning Using Graph-Based Sentence Summarization

You are here

Top Reasons to Join SPS Today!

1. IEEE Signal Processing Magazine
2. Signal Processing Digital Library*
3. Inside Signal Processing Newsletter
4. SPS Resource Center
5. Career advancement & recognition
6. Discounts on conferences and publications
7. Professional networking
8. Communities for students, young professionals, and women
9. Volunteer opportunities
10. Coming soon! PDH/CEU credits
Click here to learn more.

Dense Video Captioning Using Graph-Based Sentence Summarization

By: 
Zhiwang Zhang; Dong Xu; Wanli Ouyang; Luping Zhou

Recently, dense video captioning has made attractive progress in detecting and captioning all events in a long untrimmed video. Despite promising results were achieved, most existing methods do not sufficiently explore the scene evolution within an event temporal proposal for captioning, and therefore perform less satisfactorily when the scenes and objects change over a relatively long proposal. To address this problem, we propose a graph-based partition-and-summarization (GPaS) framework for dense video captioning within two stages. For the “partition” stage, a whole event proposal is split into short video segments for captioning at a finer level. For the “summarization” stage, the generated sentences carrying rich description information for each segment are summarized into one sentence to describe the whole event. We particularly focus on the “summarization” stage, and propose a framework that effectively exploits the relationship between semantic words for summarization. We achieve this goal by treating semantic words as the nodes in a graph and learning their interactions by coupling Graph Convolutional Network (GCN) and Long Short Term Memory (LSTM), with the aid of visual cues. Two schemes of GCN-LSTM Interaction (GLI) modules are proposed for seamless integration of GCN and LSTM. The effectiveness of our approach is demonstrated via an extensive comparison with the state-of-the-arts methods on the two benchmarks ActivityNet Captions dataset and YouCook II dataset.

SPS on Twitter

  • SPS is proud to participate in IEEE's new Multiple Society Discount Program! Join two or more participating societi… https://t.co/BnwcM7O7iu
  • IEEE Day is October 4th. Celebrate IEEE Day by attending a local event. Visit the IEEE Day site for a complete list… https://t.co/mESJHTn7ek
  • The Biomedical Imaging and Signal Processing Webinar Series continues on Tuesday, 4 October when Selin Aviyente pre… https://t.co/Gl4bHlWbqh
  • On Wednesday, 26 October, join Dr. DeLiang Wang for a new SPS webinar, "Neural Spectrospatial Filter" - register no… https://t.co/vUkiWC4Am8
  • Join Dr. Peilan Wang and Dr Jun Fang for "Channel State Information Acquisition for Intelligent Reflecting Surface-… https://t.co/jOhyA10xuG

SPS Videos


Signal Processing in Home Assistants

 


Multimedia Forensics


Careers in Signal Processing             

 


Under the Radar