File:Figure S7.png

This file contains additional information, probably added from the digital camera or scanner used to create or digitize it. If the file has been modified from its original state, some details may not fully reflect the modified file.

Image title	<p>Audio playback of reconstructed speech. The audio file contains a sequence of six isolated words that were reconstructed from single trial neural activity. Single trial reconstructions are generally not intelligible. However, coarse features such as syllable structure may be discerned. In addition, up and down frequency sweeps (corresponding to faster temporal rates) are more evident in the modulation reconstructions compared to the spectrogram reconstructions. Perceptual similarities between original and reconstructed words can be more easily recognized after first listening to the original sound. In the audio file, each word is presented as a sequence of the original sound heard by the participant, followed by the spectrogram (linear) reconstruction, followed by the modulation (nonlinear) reconstruction. The figure shows the spectrograms of the original and reconstructed words. For audio playback, the spectrogram or modulation representations must be converted to an acoustic waveform, a transformation that requires both magnitude and phase information. Because the reconstructed representations are magnitude-only, the phase must be estimated. In general, this is known as the phase retrieval problem<xref ref-type="bibr" rid="pbio.1001251-Geschwind1">[8]</xref>. To recover the acoustic waveform from the spectrogram, we used an iterative projection algorithm to estimate the phase<xref ref-type="bibr" rid="pbio.1001251-Geschwind1">[8]</xref>. This step introduces additional acoustic artifacts that can distort the auditory features reconstructed directly from neural responses. Consequently, the audio file is an accurate but not perfect reflection of the reconstructed speech representation. A similar algorithm can be used to recover the spectrogram from the modulation representation<xref ref-type="bibr" rid="pbio.1001251-Geschwind1">[8]</xref>. For the purposes of this demo, we instead projected the spectrogram reconstruction into the (complex) modulation domain, extracted the phase, and then combined the extracted phase with the reconstructed magnitude of the modulation representation. With both phase and magnitude information, an invertible transformation can then be used to convert the (complex) modulation representation back to the spectrogram<xref ref-type="bibr" rid="pbio.1001251-Geschwind1">[8]</xref>. Finally, to aid perceptual inspection of the reconstructions, the sample rate of the audio file is slightly slower (14 kHz) than that presented to participants (16 kHz).</p><p>(TIF)</p>
Copyright holder	Creative Commons Attribution License
Pixel composition	RGB
Short title	Figure S7
Source	info:doi/10.1371/journal.pbio.1001251
Horizontal resolution	236.22 dpi
Vertical resolution	236.22 dpi