LD-MAN: Layout-Driven Multimodal Attention Network for Online News Sentiment Recognition

You are here

Top Reasons to Join SPS Today!

1. IEEE Signal Processing Magazine
2. Signal Processing Digital Library*
3. Inside Signal Processing Newsletter
4. SPS Resource Center
5. Career advancement & recognition
6. Discounts on conferences and publications
7. Professional networking
8. Communities for students, young professionals, and women
9. Volunteer opportunities
10. Coming soon! PDH/CEU credits
Click here to learn more.

LD-MAN: Layout-Driven Multimodal Attention Network for Online News Sentiment Recognition

Wenya Guo; Ying Zhang; Xiangrui Cai; Lei Meng; Jufeng Yang; Xiaojie Yuan

The prevailing use of both images and text to express opinions on the web leads to the need for multimodal sentiment recognition. Some commonly used social media data containing short text and few images, such as tweets and product reviews, have been well studied. However, it is still challenging to predict the readers’ sentiment after reading online news articles, since news articles often have more complicated structures, e.g., longer text and more images. To address this problem, we propose a layout-driven multimodal attention network (LD-MAN) to recognize news sentiment in an end-to-end manner. Rather than modeling text and images individually, LD-MAN uses the layout of online news to align images with the corresponding text. Specifically, it exploits a set of distance-based coefficients to model the image locations and measure the contextual relationship between images and text. LD-MAN then learns the affective representations of the articles from the aligned text and images using a multimodal attention mechanism. Considering the lack of relevant datasets in this field, we collect two multimodal online news datasets, containing a total of 14,566 articles with 56,260 images and 251,202 words. Experimental results demonstrate that the proposed method performs favorably compared with state-of-the-art approaches. We will release all the codes, models and datasets to the community.

SPS on Twitter

  • The Brain Space Initiative Talk Series continues on Friday, 29 October when Dr. Selin Aviyente presents "Cross-Freq… https://t.co/Jxgu2soJCc
  • Join the Brain Space Initiative for another virtual mixing event on Wednesday, 27 October! Grab a coffee and meet w… https://t.co/KA3kuPUGw0
  • We're proud to sponsor a new journal, IEEE Transactions on Quantum Engineering, publishing regular, review, and tut… https://t.co/cZskrh9cvX
  • We are now seeking mentors and students for the launch of a new initiative, Mentoring Experiences for Underrepresen… https://t.co/i9SarNyKm9
  • This Wednesday, 13 October, join the Women in Signal Processing Committee for an IEEE Day webinar, "Promoting Diver… https://t.co/HrtVGqpwFx

SPS Videos

Signal Processing in Home Assistants


Multimedia Forensics

Careers in Signal Processing             


Under the Radar