A Brain-Media Deep Framework Towards Seeing Imaginations Inside Brains

You are here

Top Reasons to Join SPS Today!

1. IEEE Signal Processing Magazine
2. Signal Processing Digital Library*
3. Inside Signal Processing Newsletter
4. SPS Resource Center
5. Career advancement & recognition
6. Discounts on conferences and publications
7. Professional networking
8. Communities for students, young professionals, and women
9. Volunteer opportunities
10. Coming soon! PDH/CEU credits
Click here to learn more.

A Brain-Media Deep Framework Towards Seeing Imaginations Inside Brains

By: 
Jianmin Jiang; Ahmed Fares; Sheng-Hua Zhong

While current research on multimedia is essentially dealing with the information derived from our observations of the world, internal activities inside human brains, such as imaginations and memories of past events etc., could become a brand new concept of multimedia, for which we coin as “brain-media”. In this paper, we pioneer this idea by directly applying natural images to stimulate human brains and then collect the corresponding electroencephalogram (EEG) sequences to drive a deep framework to learn and visualize the corresponding brain activities. By examining the relevance between the visualized image and the stimulation image, we are able to assess the performance of our proposed deep framework in terms of not only the quality of such visualization but also the feasibility of introducing the new concept of “brain-media”. To ensure that our explorative research is meaningful, we introduce a dually conditioned learning mechanism in the proposed deep framework. One condition is analyzing EEG sequences through deep learning to extract a more compact and class-dependent brain features via exploiting those unique characteristics of human brains such as hemispheric lateralization and biological neurons myelination (neurons importance), and the other is to analyze the content of images via computing approaches and extract representative visual features to exploit artificial intelligence in assisting our automated analysis of brain activities and their visualizations. By combining the brain feature space with the associated visual feature space of those images that are candidates of the stimuli, we are able to generate a combined-conditional space to support the proposed dual-conditioned and lateralization-supported GAN framework. Extensive experiments carried out illustrate that our proposed deep framework significantly outperforms the existing relevant work, indicating that our proposed does provide a good potential for further research upon the introduced concept of “brain-media”, a new member for the big family of multimedia. To encourage more research along this direction, we make our source codes publicly available for downloading at GitHub.

SPS on Twitter

  • CALL FOR PROPOSALS: The IEEE Workshop on Automatic Speech Recognition and Understanding is now soliciting proposals… https://t.co/gzYreLyroa
  • authors have started uploading their conference slides and posters to IEEE SPS SigPort! Get a sneak pea… https://t.co/XGvnfdrHIb
  • DEADLINE EXTENDED: The IEEE Journal of Selected Topics in Signal Processing is accepting papers for a Special Issue… https://t.co/E89M7bEFlu
  • Voting for the IEEE SPS 5-Minute Video Clip Contest is now live! Check out the three finalists and cast your vote f… https://t.co/fbqgHY1tw7
  • CALL FOR PROPOSALS: Now seeking proposals for the 2024 IEEE International Workshop on Machine Learning for Signal P… https://t.co/l7V1bF2qhT

SPS Videos


Signal Processing in Home Assistants

 


Multimedia Forensics


Careers in Signal Processing             

 


Under the Radar