Reconstructing and semantically classifying observed natural images from novel (unknown) fMRI brain recordings is a milestone for developing brain-machine interfaces and for the study of consciousness. Unfortunately, acquiring sufficient “paired” training examples (images with their corresponding fMRI recordings) to span the huge space of natural images and their semantic classes is prohibitive, even in the largest image-fMRI datasets. We present a self-supervised deep learning approach that overcomes this barrier, giving rise to better generalizing fMRI-to-image decoders. This is obtained by enriching the scarce paired training data with additional easily accessible “unpaired” data from both domains (i.e., images without fMRI, and fMRI without images). Our approach achieves state-of-the-art results in image reconstruction from fMRI responses, as well as unprecedented large-scale (1000-way) semantic classification of never-before-seen classes. References (ranked by relevance):- Image Reconstruction - project webpage- Self-Supervised Natural Image Reconstruction and Rich Semantic Classification from Brain Activity. bioRxiv'20- From voxels to pixels and back: Self-supervision in natural-image reconstruction from fMRI. 2019, NeurIPS- More than meets the eye: Self-supervised depth reconstruction from brain activity. arXiv'21- Convergent evolution of face spaces across human face-selective neuronal groups and deep convolutional networks. 2019, Nature Comm.
Guy Gaziv is a postdoctoral fellow in the Michal Irani Computer Vision group at The Weizmann Institute of Science. This spring he will be starting his postdoc at the DiCarlo Lab at MIT.His PhD research focused on the intersection between machine and human vision, and specifically on decoding visual experience from brain activity.
Guy earned his BSc in Electrical and Computer Engineering from The Hebrew University of Jerusalem, and his MSc in Physics and PhD in Computer Science from The Weizmann Institute of Science.