A Brain-Media Deep Framework Towards Seeing Imaginations Inside Brains

You are here

Top Reasons to Join SPS Today!

1. IEEE Signal Processing Magazine
2. Signal Processing Digital Library*
3. Inside Signal Processing Newsletter
4. SPS Resource Center
5. Career advancement & recognition
6. Discounts on conferences and publications
7. Professional networking
8. Communities for students, young professionals, and women
9. Volunteer opportunities
10. Coming soon! PDH/CEU credits
Click here to learn more.

A Brain-Media Deep Framework Towards Seeing Imaginations Inside Brains

By: 
Jianmin Jiang; Ahmed Fares; Sheng-Hua Zhong

While current research on multimedia is essentially dealing with the information derived from our observations of the world, internal activities inside human brains, such as imaginations and memories of past events etc., could become a brand new concept of multimedia, for which we coin as “brain-media”. In this paper, we pioneer this idea by directly applying natural images to stimulate human brains and then collect the corresponding electroencephalogram (EEG) sequences to drive a deep framework to learn and visualize the corresponding brain activities. By examining the relevance between the visualized image and the stimulation image, we are able to assess the performance of our proposed deep framework in terms of not only the quality of such visualization but also the feasibility of introducing the new concept of “brain-media”. To ensure that our explorative research is meaningful, we introduce a dually conditioned learning mechanism in the proposed deep framework. One condition is analyzing EEG sequences through deep learning to extract a more compact and class-dependent brain features via exploiting those unique characteristics of human brains such as hemispheric lateralization and biological neurons myelination (neurons importance), and the other is to analyze the content of images via computing approaches and extract representative visual features to exploit artificial intelligence in assisting our automated analysis of brain activities and their visualizations. By combining the brain feature space with the associated visual feature space of those images that are candidates of the stimuli, we are able to generate a combined-conditional space to support the proposed dual-conditioned and lateralization-supported GAN framework. Extensive experiments carried out illustrate that our proposed deep framework significantly outperforms the existing relevant work, indicating that our proposed does provide a good potential for further research upon the introduced concept of “brain-media”, a new member for the big family of multimedia. To encourage more research along this direction, we make our source codes publicly available for downloading at GitHub.

SPS on Twitter

  • The SPACE Webinar series continues Tuesday, 18 May at 10:00 AM EST when Dr. Rebecca Willet presents "Machine Learni… https://t.co/jdUjHQpoaf
  • Join us on Friday, 21 May at 1:00 PM EST when Dr. Amir Asif (York University) shares his journey and the importance… https://t.co/SLJGLI3K8u
  • There's still time to apply for PROGRESS! Visit https://t.co/0h4GgRY1Jr to connect with signal processing leaders a… https://t.co/dQNnkxpv8f
  • This Saturday, 8 May, join the SPS JSS Academy of Technical Education Noida Student Branch Chapter in collaboration… https://t.co/lFVmmVucvG
  • The SPACE Webinar Series continues this Tuesday, 4 May at 10:00 AM Eastern when Dr. Lei Tian presents "Modeling and… https://t.co/9emEVjOInK

SPS Videos


Signal Processing in Home Assistants

 


Multimedia Forensics


Careers in Signal Processing             

 


Under the Radar