Christoph Guger, Gerwin Schalk, Christoph Kapeller
In 1985, Dr. Oliver Sacks published “The Man Who Mistook His Wife for a Hat,” which included the case of a man with visual agnosia who had trouble identifying faces. This case made many people think about what a remarkable challenge our brain routinely solves every day. We can recognize faces so easily and automatically that it’s hard to appreciate the complex brain processes that are involved. Although over 30 years have passed since this famous book was published, many questions about face processing are unanswered. For example: do regions involved in face processing (in particular the fusiform face area (FFA)) play an important role in processing other objects, or are these regions exclusively responsible for face processing (1-3)?
We had an opportunity to study this question in a human patient with epilepsy who had electrodes implanted to map the areas of his brain that were responsible for his seizures and important functions such as face and color processing (4-8). Thus, by studying – and even temporarily modifying – these areas of this patients’ brain, we were able to learn more about how to help clinicians optimize the patient’s surgery while simultaneously studying fundamental questions about how we all see the world.
First, we studied the activity at the different electrode sites while the patient was shown both grayscale and color versions of different visual images, including faces. Specifically, we showed the patient a box, ball, face, or word. Figure 1 shows that areas corresponding to the FFA were highly selective for faces, and three adjacent areas were preferentially responsive to color. Other areas of the brain’s surface did not show a strong preference for either faces or colors. The location and pattern of responses was consistent with the established view that brain regions that are responsible for different aspects of visual processing are relatively specific.
Figure 1: This figure shows the location of the electrodes on ventral areas of the patient’s cortex. Some electrodes are colored red to indicate that they showed a preferential response to faces over other objects. Other electrodes, in blue, were more active for color stimuli than grayscale stimuli. The remaining (white) electrodes did not show a strong preference for color or faces. The yellow lightning symbols with black lines between electrodes reflect electrode pairs that were stimulated, producing illusory face or color percepts.
After studying how different areas responded to different stimuli, we next explored how stimulating these areas would change the patient’s perception. We asked the patient to describe his experience when we stimulated pairs of electrodes while he observed a box, ball, face, or word. Previous studies in monkeys already established that stimulation of face-specific areas changed face perception (11-13), but they could not determine whether that stimulation replaced existing percepts with a face percept or merely added a face-like percept – a “facephene” – to the perceived world. What is this like? Asking this question of a human participant should yield more detailed replies than asking a monkey.
Figure 2 shows that stimulation of different brain areas led to specific changes in perception. Without stimulation, the patient correctly identified all four types of stimuli. When the face area of the brain was stimulated, he reported face-like changes in perception, regardless of what he actually saw! Specifically, when he saw a face, his ability to recognize faces was distorted, and he said that “I don’t know what’s going on.” When he saw any of the other three stimuli, he reported seeing faces or parts of faces, which is consistent with prior work that also reported distorted facial perception during face stimulation (4,5,9,10), but added the new information that the face percepts were superimposed to the existing stimuli. We also stimulated the areas responsible for color processing and asked the patient to report his experience. Across all four types of stimuli, he reported seeing colors in addition to the real world.
Thus, stimulating brain regions that responded to faces or colors produced highly specific percepts involving faces or colors, respectively, and those percepts were superimposed to existing percepts. These results further support our view that these brain regions are involved in highly specific processing of distinct types of visual stimuli, and suggest that they play a specific causal role in perception. We also stimulated areas that were not responsive to faces or colors, which did not elicit consistent perception of faces or colors.
Figure 2: The four rows each present photographs of the subject observing four types of stimuli: a box, ball, face, or monitor that displayed a word. The left and right columns of text present the subject’s verbal reports while face-specific (left column) or color-specific (right column) brain areas are stimulated. For example, the top left column of text shows that the subject reported seeing an eye and a mouth when the face region was stimulated, even though he was observing a box.
These results were enabled by a fairly rare opportunity to use invasive methods with a human participant. Non-invasive methods like fMRI and EEG cannot be used to directly stimulate the brain and/or have sufficiently high resolution. However, work with invasive methods also entail major challenges since patients undergoing neurosurgery are only available for a limited time (often just one recording session). Since each patient’s electrodes are placed over different areas (depending on the patient’s specific clinical needs), replicating results in other people can be challenging.
Further details can be found in:
Gerwin Schalk, Christoph Kapeller, Christoph Guger, Satoru Hiroshima, Rosa Lafer-Sousa, Zeynep M. Saygin, Kysousuke Kamada, Nanxy Kanwisher, Facephenes and rainbows: Causal evidence for functional and anatomical specificity of face and color processing in the human brain, PNAS 2017 114 (46) 12285-12290; published ahead of print October 30, 2017, doi:10.1073/pnas.1713447114
Videos of the experiment are also available online.
- Haxby, JV., et al. Distributed and Overlapping Representations of Faces and Objects in Ventral Temporal Cortex. Science. 2001;293(5539):2425-30.
- Reddy, L., Kanwisher N. Category Selectivity in the Ventral Visual Pathway Confers Robustness to Clutter and Diverted Attention. Curr Biol. 2007;17(23):2067-72.
- Kanwisher, N., McDermott J, Chun MM. The Fusiform Face Area: A Module in Human Extrastriate Cortex Specialized for the Perception of Faces. J Neurosci. 1997;17(11):4302-11.
- Allison, T. et al. J Neurophysiol. 1994;71(2):821-5.
- Parvizi, J. et al. Electrical Stimulation of Human Fusiform Face-Selective Regions Distorts Face Perception. J Neurosci. 2012;32(43):14915-20.
- McKeefry DJ, Zeki S. The position and topography of the human colour centre as revealed by functional magnetic resonance imaging. Brain. 1997;120 (Pt 12):2229-42.
- Lafer-Sousa, R., Conway BR, Kanwisher N. Color-Biased Regions of the Ventral Visual Pathway Lie between Face- and Place-Selective Regions in Humans, as in Macaques. J Neurosci. 2016;36(5):1682-97.
- Murphey, DK, Maunsell JH, Beauchamp MS, Yoshor D. Perceiving electrical stimulation of identified human visual areas. PNAS 2009;106(13):5389-93.
- Mundel, T. et al. Face memory in MRI-positive and MRI-negative temporal lobe epilepsy. Transient inability to distinguish between faces: electro- physiologic studies. J Clin Neurophysiol. 2003;20(2):102-10.
- Rangarajan, V. et al. Electrical Stimulation of the Left and Right Human Fusiform Gyrus Causes Different Effects in Conscious Face Perception. J Neurosci. 2014;34(38):12828-36.
- Sadagopan S, Zarco W, Freiwald WA. A causal relationship between face-patch activity and face-detection behavior. Elife. 2017;6.
- Afraz SR, Kiani R, Esteky H. Microstimulation of inferotemporal cortex influences face categorization. Nature. 2006;442(7103):692-5.
- Moeller S, Crapse T, Chang L, Tsao DY. The effect of face patch microstimulation on perception of faces and objects. Nat Neurosci. 2017;20:743-52.
About the Authors
Dr. Christoph Guger studied biomedical engineering at the Technical University of Graz (TUG) and Johns Hopkins University in Baltimore, USA. He then carried out research work at the Department of Medical Informatics (Prof. Pfurtscheller) at TUG and received his PhD in 1999. The topic of his PhD work was the design of the first EEG-based BCI with continuous feedback. In recent years, he has conducted research in about 30 international research projects (non-invasive and invasive BCIs) and is running g.tec medical engineering in Austria, Spain and the USA. Member, IEEE.
Dr. Christoph Kapeller studied Biomedical Engineering at TUG and recently received his Ph.D. at Johannes Kepler University in Linz. He is a software engineer, research scientist, and part of the scientific development of BCI and tools for planning neurosurgeries. He has been especially active using high gamma mapping to study activity related to movement, perception, and attention. He gained extensive experience during recent projects in the USA and Japan in collaboration with renowned neurosurgeons.
Dr. Gerwin Schalk obtained his M.S. in Electrical Engineering and Computer Science from Graz University of Technology in Austria, his M.S. in Information Technology from Rensselaer Polytechnic Institute (RPI) in Troy, New York, and his Ph.D. in Computer and Systems Engineering from RPI. He is interested in integrating and advancing scientific, engineering, and clinical concepts to innovate, develop and test new neurotechnologies and to apply them to basic and applied research. He authored or co-authored >120 peer-reviewed publications, one book and 14 chapters, has >13000 total citations and an H factor of 46, and has given more than 210 invited lectures world-wide. His work is funded by the NIH and the US Army and has been extensively featured by the media. He is listed in Who’s Who in the World and Who’s Who in America, and received several awards for his work. Member, IEEE.