5,710
edits
Changes
→Conclusions and Future Directions
This piece has a few obvious limitations, but also many strengths, pointing to future directions of work as the technical infrastructure develops. In aestheticizing “the look” of the Nishimoto video reconstructions or “the sound” of the Pasley audio reconstructions, we are dwelling on artifacts of the data collection process/training dataset/reconstruction algorithm more than anything specific to the mind of the subject. Ideally we could image some higher-level responses in the viewing subject (beyond primary visual areas). I’m sure many neuroscientists would also like to do this. Decoding higher-level processing would give us more meat for the project, more of a cognitive response as opposed to an evoked perceptual one.
Another shortcoming of the project is its essential theatricality. The videos are produced from averaged, highly correlated 10 second clips of source materials. Similar aesthetic effects to the ghostlike compositing could be achieved with simple algorithmic processescomputer programs and none of the technical infrastructure. So in essence, the project relies on the presence of the MRI magnet , functional imaging, and the gesture of “peering inside the brain” of another as central vehicles for meaning.
There is much positive potential in this project. Each resulting audio-visual piece produced reconstruction is uniquely tied to the person of the test subject, and the particular particulars of the session, stimulus material, and context training dataset that produced it. There is a great opportunity for dissonance between source material and filtered signal. Rather than settling for samples extracted from Apple movie trailers and Youtube(Nishimoto et al), you could for instance evoke reactions to video reconstruct footage of gory monkey-brain surgery and train/reconstruct the imagery car crash disasters from a training database of national geographic factory farm production. In the Nishimoto videosit is possible to see traces of the original material poking through.
Future directions to explore include the modeling and reconstruction of perceptions by non-human subjects (simulating what your dog sees), implementing audio reconstruction with non-invasive methods (i.e. not ECoG), attempting to evoke and reconstruct responses in other sensory modalities. Additionally with the improvements in the portability of MRI technology or the development of over other sensing modalities (EEG) there could be new opportunities for reconstructing internal, sensory experience.