Learning Semantic Text Features for Web Text-Aided Image Classification

You are here

Top Reasons to Join SPS Today!

1. IEEE Signal Processing Magazine
2. Signal Processing Digital Library*
3. Inside Signal Processing Newsletter
4. SPS Resource Center
5. Career advancement & recognition
6. Discounts on conferences and publications
7. Professional networking
8. Communities for students, young professionals, and women
9. Volunteer opportunities
10. Coming soon! PDH/CEU credits
Click here to learn more.

Learning Semantic Text Features for Web Text-Aided Image Classification

Dongzhe Wang; Kezhi Mao;

The good generalization performance of conventional pattern classifiers often relies on the size of training data labeled by costly human labor. These days, publicly available web resources grow explosively, and this allows us to easily obtain abundant and cheap web data. Yet, web data are usually not as cooperative as human labeled data. In this paper, we explore the use of web text data to aid image classification. Without requiring the previous collection of auxiliary data from the web, we directly retrieve the web text information with the aid of the powerful reverse image search engine. We develop a novel textual modeling method named semantic matching neural network (SMNN) that is capable of learning semantic features from the associated text of web images. The SMNN text features have improved reliability and applicability, compared to the text features obtained from other methods. The SMNN text features and convolutional neural network (CNN) visual features are merged into a shared representation, which learns to capture the correlations between the two modalities. Experimental results on benchmark UIUC-Sports, Scene-15, Caltech-256, and Pascal VOC-2012 data sets show that the visual and text modalities of data from different sources are remarkably complementary and the fusion of them achieves substantial performance improvement.

SPS on Twitter

  • The virtual conference is powered by leading companies who have continued to show their support for our t… https://t.co/Jz7mMCqqhJ
  • The virtual conference begins today! Join us all week long for an exciting week of inspiring, innovative… https://t.co/xQY7lS2nmw
  • Learn what's new with Society activities during the SPS Town Hall! Join SPS President Ahmed Tewfik on Wednesday, 28… https://t.co/l5mKvqaskO
  • DEADLINE EXTENDED: The application deadline for PROGRESS at has been extended to Tuesday, 20 October. Det… https://t.co/pO5s3ief9t
  • On Wednesday, 28 October, join Stefania Sardellitti for a new SPS Webinar, “Joint Optimization of Radio and Computa… https://t.co/yGKACShQzu

SPS Videos

Signal Processing in Home Assistants


Multimedia Forensics

Careers in Signal Processing             


Under the Radar