Lin Yao, Ning Jiang
Brain-computer Interface (BCI) permits a direct channel between the brain and the external environment, bypassing the physiological channel for such interaction, i.e. the neuromuscular system. This technology can be useful in medical applications, including locked-in syndrome , stroke , spinal cord injury , and cerebral palsy , as well as applications of a more general purpose such as education, ergonomics, and manufacturing. Event-related desynchronization (ERD) and synchronization (ERS) of brain signals ,  and movement-related cortical potentials (MRCP) , , both of which are generated during motor imagery tasks (MI), have been shown to allow real-time, direct BCI control. These two approaches are fundamentally different from other BCI paradigms, such as P300  or SSVEP , which rely on external stimuli. Rather, users operate these MI-based BCIs only with covert mental activities, not dependent on any external stimulus.
The highly integrated sensory-motor system gives rise to the questions: can we also decode covert sensory attention non-invasively? It has been showed that similar EEG dynamics exists during somatosensory stimulation , . About six years ago, we started to explore this question, with a series of research studies. The high-level outlook of this initiative is shown in Fig. 1.
Figure 1. Illustration of the relationship between the BCIs based motor tasks and somatosensory tasks. In either case, both overt and covert mental tasks can be used for BCI. The shaded area is for BCIS based on motor tasks and the unshaded area is for BCIs based somatosensory tasks, which we have been focused for the last few years.
First, we proposed a tactile BCI based on oscillatory dynamics from the somatosensory area of the cortex, which we termed selective sensation (SS) tactile BCI –. This approach was based on the fact that ERD/ERS in sensory-motor rhythm (SMR) band (8~30 Hz) not only has a strong correlation with real or imagined movement but also with the processing of afferent inflow in the human somatosensory system , . The somatosensory attention could also modulate the amplitudes of ERD/ERS . We demonstrated that the integration of stimulus-induced oscillatory dynamics in tactile SS substantially outperformed previous tactile BCI systems in decoding accuracy , –, making tactile BCI potentially applicable to a large number of users. Motivated by the similarity of SMR between motor execution (ME) and MI in BCI systems based on motor tasks, we further hypothesized that there are also similar characteristics in the brain dynamics elicited by tactile SS and those elicited by sensory processing tasks where no explicit tactile stimulus exists. For example, a BCI user would shift and maintain somatosensory attention by imagining the sensation of tactile stimulation applied to different body parts. We termed such covert sensory processing tasks as somatosensory attentional orientation (SAO) , . Through a large number of experimental studies, we found that EEG signals generated by SAO can be classified with an accuracy >75% for two classes –. The major advantage of SAO over SS is the fact that SAO does not rely on physical sensory stimulation. Therefore, it can be used to develop fully independent BCIs, similar to that of MI-based BCIs.
Research in the past two decades clearly showed that a significant portion of users cannot use specific BCI systems, even after extensive training both from the user side and from the algorithm side –. Three major MI studies on 193 subjects , 80 subjects  and 52 subjects , have specifically investigated this problem. These studies reported that between ~40% and ~60% of the subjects investigated could achieve an average accuracy in BCI control of only ~50% in the classic two-class scenario . This problem and the relatively low decoding accuracy of independent BCIs are the two major ongoing challenges the BCI community is facing. We are addressing these challenges through a series of experimental studies in the following three aspects: 1) we proposed new BCI modalities to broaden BCI application; 2) we proposed a new hybrid approach to improve the classification accuracy of the two-class BCI system; 3) we proposed a new sensory stimulation training and calibration framework to further improve BCI performance.
Tactile somatosensory BCI modality: We compared the performance and the portion of users with low performance (<70%) of tactile selective sensation (SS) and motor imagery (MI) BCI, for a large pool of subjects . We analyzed 80 experimental sessions from 57 subjects with two-class SS protocols. For SS, the group average performance was 79.8±10.6% (with several experiments with or without feedback). When compared to previous results, this tactile BCI outperformed all other tactile BCIs currently available , –. We also analyzed 63 experiment sessions from 43 subjects with two-class MI BCI protocols, where the group average performance was 77.2±13.3% (also with several experiments with or without feedback). We found that the BCI performance was superior with SS than MI especially in the beta frequency band (p<0.05). These results demonstrated that the tactile SS-based BCI provides a good candidate to diversify BCI modalities, particularly those based on users’ own volition. We also showed that SS-based BCI can provide more classes for control , and can be translated to decoding covert somatosensory attention, i.e. covert volition in the absence of external stimuli , . This new BCI paradigm is based on somatosensory processing (with or without actual stimuli), which induces ERD/ERS in EEG recorded over the somatosensory cortex. It differs from tactile BCI based on steady-state somatosensory evoked potential (SSSEP) , , , where the exact stimulation frequency is to be detected in EEG. Both requirements are limiting factors for possibly BCI application scenario. This new BCI paradigm also differs from tactile P300 BCI , which is based on the oddball paradigm, similar to the visual P300 BCI . Both SSSEP and tactile P300 BCI require physical stimulation, while this novel BCI works even when physical stimuli are absent, making it truly independent.
Figure 2. Overview of our proposed BCI approaches to combat some of the key issues affecting BCI performance and wider range of applications.
A hybrid approach to improve BCI performance: In our experimental study, we instructed subjects to perform the mental tasks according to the randomly presented cues (i.e. left-hand motor imagery, right-hand motor imagery, left stimulation sensation or right stimulation sensation). As anticipated, the hybrid BCI can improve the BCI performance. In this new hybrid modality , e.g. subjects perform left hand stimulation sensation and right hand motor imagery tasks, the performance of the hybrid BCI was 11.13% higher on average than the MI group, and 10.45% higher than the SS group, potentially reducing the portion of subjects with performance <70% from 63.6% in MI to 9.1% in hybrid modality (study from 11 subjects). Moreover, our study on a stimulus-independent hybrid BCI further validated the added value of a hybrid approach for improved BCI performance , showing a 7.8% improvement as compared with MI and 7.2% improvement as compared with SAO.
Sensory stimulation training and calibration framework: Because SAO is a purely covert mental process, similar to MI, one challenge of applying these types of BCIs is the user compliance during training, i.e. it is difficult to ensure that users perform consistently the desired tasks during training. To address this challenge, we hypothesized that the brain oscillatory patterns induced by SS (with physical stimulus) can be utilized to calibrate SAO-based BCI systems , , with the previous knowledge that there exist similarities between the two. The system was configured with an SS calibration phase and a post-calibration SAO phase for testing. Our experimental studies validated this idea and showed that the real-time BCI performance across all subjects reached an average accuracy of 78.8±13.1%, with 12 subjects exceeding 70%, and 4 subjects exceeding 90%. Moreover, we propose a novel sensory stimulation training (SST) approach to improve the somatosensory BCI performance . In a controlled study, twenty BCI naïve subjects were recruited and randomly divided into a Control-Group and an SST-Group. In the Control-Group, subjects performed left hand and right-hand SAO tasks in six consecutive runs (with 40 trials in each run), divided into three blocks having two runs each. For the SST-Group, two runs included real tactile stimulation to the left or right hand (SST training block), between the first two (Pre-SST block) and the last two SAO runs (Post-SST block). Results showed that the SST-Group had a significantly improved performance by 9.4% between the last block and the first block after SST training, with enhanced the R2 information especially in the somatosensory cortex; in contrast, no significant difference in BCI performance and R2 distribution, was found in the Control-Group between the first, second and the last block.
Complementary to the ERD/ERS induced by overt or covert motor tasks, in the series of studies described above we have systematically investigated the EEG oscillatory dynamics induced by real or imagined tactile stimulation, showing that the contralateral band power decrease (ERD) of the sensorimotor rhythms reflects the afferent signal processing of the brain, which is in accordance with the literature , , . By utilizing the brain signal mainly from the sensory cortex, we have validated that sensory attention can be decoded in single trial and real-time, with or without physical sensory stimuli. We termed the proposed BCI modality as somatosensory BCI. The somatosensory BCI provides a novel BCI modality. It is different from previous tactile-based BCIs, such as SSSEP (steady state somatosensory evoked potential) ,  and tactile P300 BCI , as it works without external stimuli. It is also different from BCIs based EEG dynamics from real or imaginary movement processing. We demonstrated that with this new BCI modality, we can improve BCI performance, through sensory stimulation training and calibration methodology. The new modality can also be combined with MI-based BCI to provide powerful multi-class BCIs, independent of external stimuli. We believe that the proposed somatosensory BCI will have the potential to make BCI more feasible for a wide range of applications.
- J. R. Wolpaw, N. Birbaumer, D. J. McFarland, G. Pfurtscheller, T. M. Vaughan, and others, “Brain-computer interfaces for communication and control,” Clin. Neurophysiol., vol. 113, no. 6, pp. 767–791, Jun. 2002, doi:10.1016/S1388-2457(02)00057-3.
- A. Biasiucci, R. Leeb, I. Iturrate, S. Perdikis, A. Al-Khodairy, T. Corbet, A. Schnider, T. Schmidlin, H. Zhang, M. Bassolino, D. Viceic, P. Vuadens, A. G. Guggisberg, and J. D. R. Millán, “Brain-actuated functional electrical stimulation elicits lasting arm motor recovery after stroke,” Nat. Commun., 2018, doi:10.1038/s41467-018-04673-z.
- G. Pfurtscheller, P. Linortner, R. Winkler, G. Korisek, and G. Müller-Putz, “Discrimination of motor imagery-induced EEG patterns in patients with complete spinal cord injury,” Comput. Intell. Neurosci., 2009, doi:10.1155/2009/104180.
- I. Daly, M. Billinger, J. Laparra-Hernández, F. Aloise, M. L. García, J. Faller, R. Scherer, and G. Müller-Putz, “On the control of brain-computer interfaces by users with cerebral palsy,” Clin. Neurophysiol., 2013, doi:10.1016/j.clinph.2013.02.118.
- G. Pfurtscheller, “Spatiotemporal ERD/ERS patterns during voluntary movement and motor imagery,” Suppl. Clin. Neurophysiol., vol. 53, pp. 196–198, 2000.
- G. Pfurtscheller, C. Brunner, A. Schlögl, and F. H. Lopes da Silva, “Mu rhythm (de)synchronization and EEG single-trial classification of different motor imagery tasks,” Neuroimage, vol. 31, no. 1, pp. 153–159, 2006, doi:10.1016/j.neuroimage.2005.12.003.
- R. Xu, N. Jiang, C. Lin, N. Mrachacz-Kersting, K. Dremstrup, and D. Farina, “Enhanced low-latency detection of motor intention from EEG for closed-loop brain-computer interface applications.,” IEEE Trans. Biomed. Eng., vol. 61, no. 2, pp. 288–96, Feb. 2014, doi:10.1109/TBME.2013.2294203.
- I. K. Niazi, N. Jiang, O. Tiberghien, J. F. Nielsen, K. Dremstrup, and D. Farina, “Detection of movement intention from single-trial movement-related cortical potentials.,” J. Neural Eng., vol. 8, no. 6, p. 066009, Dec. 2011, doi:10.1088/1741-2560/8/6/066009.
- E. Donchin, K. M. Spencer, and R. Wijesinghe, “The mental prosthesis: Assessing the speed of a P300-based brain- computer interface,” IEEE Trans. Rehabil. Eng., 2000, doi:10.1109/86.847808.
- M. Cheng, X. Gao, S. Gao, and D. Xu, “Design and implementation of a brain-computer interface with high transfer rates,” Biomed. Eng. IEEE Trans., vol. 49, no. 10, pp. 1181–1186, 2002.
- E. Houdayer, E. Labyt, F. Cassim, J. L. Bourriez, and P. Derambure, “Relationship between event-related beta synchronization and afferent inputs: Analysis of finger movement and peripheral nerve stimulations,” Clin. Neurophysiol., vol. 117, no. 3, pp. 628–636, Mar. 2006, doi:10.1016/j.clinph.2005.12.001.
- C. Nangini, B. Ross, F. Tam, and S. J. Graham, “Magnetoencephalographic study of vibrotactile evoked transient and steady-state responses in human somatosensory cortex.,” Neuroimage, vol. 33, no. 1, pp. 252–262, Oct. 2006, doi:10.1016/j.neuroimage.2006.05.045.
- L. Yao, J. Meng, D. Zhang, X. Sheng, and X. Zhu, “Selective Sensation Based Brain-Computer Interface via Mechanical Vibrotactile Stimulation,” PLoS One, vol. 8, no. 6, 2013, doi:10.1371/journal.pone.0064784.
- L. Yao, J. Meng, D. Zhang, X. Sheng, and X. Zhu, “Combining motor imagery with selective sensation toward a hybrid-modality BCI,” IEEE Trans. Biomed. Eng., vol. 61, no. 8, pp. 2304–2312, 2014, doi:10.1109/TBME.2013.2287245.
- L. Yao, M. L. Chen, X. Sheng, N. Mrachacz-Kersting, X. Zhu, D. Farina, and N. Jiang, “A Multi-Class Tactile Brain-Computer Interface Based on Stimulus-Induced Oscillatory Dynamics,” IEEE Trans. Neural Syst. Rehabil. Eng., vol. 26, no. 1, pp. 3–10, Jan. 2018, doi:10.1109/TNSRE.2017.2731261.
- N. Sharma, V. M. Pomeroy, and J.-C. Baron, “Motor imagery: a backdoor to the motor system after stroke?,” Stroke., vol. 37, no. 7, pp. 1941–52, Jul. 2006, doi:10.1161/01.STR.0000226902.43357.fc.
- E. Houdayer, a Degardin, J. Salleron, J. L. Bourriez, L. Defebvre, F. Cassim, and P. Derambure, “Movement preparation and cortical processing of afferent inputs in cortical tremor: an event-related (de)synchronization (ERD/ERS) study.,” Clin. Neurophysiol., vol. 123, no. 6, pp. 1207–15, Jun. 2012, doi:10.1016/j.clinph.2011.10.043.
- C. Dockstader, D. Cheyne, and R. Tannock, “Cortical dynamics of selective attention to somatosensory events.,” Neuroimage, vol. 49, no. 2, pp. 1777–85, Jan. 2010, doi:10.1016/j.neuroimage.2009.09.035.
- L. Yao, X. Sheng, N. Mrachacz-Kersting, X. Zhu, D. Farina, and N. Jiang, “Performance of Brain-Computer Interfacing Based on Tactile Selective Sensation and Motor Imagery,” IEEE Trans. Neural Syst. Rehabil. Eng., vol. 4320, no. c, pp. 1–1, 2017, doi:10.1109/TNSRE.2017.2769686.
- S. Ahn, M. Ahn, H. Cho, S. Chan Jun, and S. C. Jun, “Achieving a hybrid brain-computer interface with tactile selective attention and motor imagery,” J. Neural Eng., vol. 11, no. 6, p. 66004, Dec. 2014, doi:10.1088/1741-2560/11/6/066004.
- A.-M. Brouwer and J. B. F. van Erp, “A tactile P300 brain-computer interface.,” Front. Neurosci., vol. 4, p. 19, Jan. 2010, doi:10.3389/fnins.2010.00019.
- G. R. Muller-Putz, R. Scherer, C. Neuper, and G. Pfurtscheller, “Steady-State Somatosensory Evoked Potentials: Suitable Brain Signals for Brain-computer Interfaces?,” IEEE Trans. Neural Syst. Rehabil. Eng., vol. 14, no. 1, pp. 30–37, Mar. 2006, doi:10.1109/TNSRE.2005.863842.
- L. Yao, X. Sheng, D. Zhang, N. Jiang, D. Farina, and X. Zhu, “A BCI System Based on Somatosensory Attentional Orientation,” IEEE Trans. Neural Syst. Rehabil. Eng., vol. 25, no. 1, pp. 81–90, Jan. 2017, doi:10.1109/TNSRE.2016.2572226.
- L. Yao, N. Mrachacz-Kersting, X. Sheng, X. Zhu, D. Farina, and N. Jiang, “A Multi-class BCI based on Somatosensory Imagery,” IEEE Trans. Neural Syst. Rehabil. Eng., 2018.
- L. Yao, X. Sheng, D. Zhang, N. Jiang, N. Mrachacz-Kersting, X. Zhu, and D. Farina, “A Stimulus-Independent Hybrid BCI Based on Motor Imagery and Somatosensory Attentional Orientation,” IEEE Trans. Neural Syst. Rehabil. Eng., vol. 25, no. 9, pp. 1674–1682, Sep. 2017, doi:10.1109/TNSRE.2017.2684084.
- C. Vidaurre, C. Sannelli, K.-R. Müller, and B. Blankertz, “Co-adaptive calibration to improve BCI efficiency.,” J. Neural Eng., vol. 8, no. 2, p. 025009, Apr. 2011, doi:10.1088/1741-2560/8/2/025009.
- C. Vidaurre and B. Blankertz, “Towards a cure for BCI illiteracy,” Brain Topogr., vol. 23, no. 2, pp. 194–198, 2010.
- B. Z. Allison, C. Brunner, V. Kaiser, G. R. Müller-Putz, C. Neuper, and G. Pfurtscheller, “Toward a hybrid brain-computer interface based on imagined movement and visual attention,” J. Neural Eng., vol. 7, no. 2, p. 26007, 2010.
- L. Yao, J. Meng, X. Sheng, D. Zhang, and X. Zhu, “A novel calibration and task guidance framework for motor imagery BCI via a tendon vibration induced sensation with kinesthesia illusion.,” J. Neural Eng., vol. 12, no. 1, p. 016005, Dec. 2015, doi:10.1088/1741-2560/12/1/016005.
- B. Z. Allison and C. Neuper, “Could anyone use a BCI?,” Brain-computer interfaces, pp. 35–54, 2010.
- G. Edlinger, B. Z. Allison, and C. Guger, “How many people can use a BCI system?,” in Clinical Systems Neuroscience, 2015, pp. 33–66.
- C. Guger, G. Edlinger, W. Harkam, I. Niedermayer, and G. Pfurtscheller, “How many people are able to operate an EEG-based brain-computer interface (BCI)?,” IEEE Trans. Neural Rehabil. Syst. Eng., vol. 11, no. 2, pp. 145–7, Jun. 2003, doi:10.1109/TNSRE.2003.814481.
- C. Guger, S. Daban, E. Sellers, C. Holzner, G. Krausz, R. Carabalona, F. Gramatica, and G. Edlinger, “How many people are able to control a P300-based brain-computer interface (BCI)?,” Neurosci. Lett., vol. 462, no. 1, pp. 94–98, 2009, doi:10.1016/j.neulet.2009.06.045.
- B. Allison, T. Luth, D. Valbuena, A. Teymourian, I. Volosyak, and A. Graser, “BCI Demographics: How many (and what kinds of) people can use an SSVEP BCI?,” Neural Syst. Rehabil. Eng. IEEE Trans., vol. 18, no. 2, pp. 107–116, 2010.
- C. Guger, B. Z. Allison, B. Großwindhager, R. Prückl, C. Hintermüller, C. Kapeller, M. Bruckner, G. Krausz, and G. Edlinger, “How many people could use an SSVEP BCI?,” Front. Neurosci., vol. 6, no. NOV, pp. 2–7, 2012, doi:10.3389/fnins.2012.00169.
- B. Blankertz, C. Sannelli, S. Halder, E. M. Hammer, A. Kübler, K. R. Müller, G. Curio, and T. Dickhaus, “Neurophysiological predictor of SMR-based BCI performance,” Neuroimage, vol. 51, no. 4, pp. 1303–1309, 2010, doi:10.1016/j.neuroimage.2010.03.022.
- M. Ahn, H. Cho, S. Ahn, and S. C. Jun, “High theta and low alpha powers may be indicative of BCI-illiteracy in motor imagery,” PLoS One, vol. 8, no. 11, p. e80886, 2013, doi:10.1371/journal.pone.0080886.
- S. Tobimatsu, Y. M. Zhang, and M. Kato, “Steady-state vibration somatosensory evoked potentials: physiological characteristics and tuning function,” Clin. Neurophysiol., vol. 110, no. 11, pp. 1953–1958, Nov. 1999, doi:10.1016/S1388-2457(99)00146-7.
- M. Severens, J. Farquhar, P. Desain, J. Duysens, and C. Gielen, “Transient and steady-state responses to mechanical stimulation of different fingers reveal interactions based on lateral inhibition,” Clin. Neurophysiol., vol. 121, no. 12, pp. 2090–2096, Dec. 2010, doi:10.1016/j.clinph.2010.05.016.
- L. Yao, X. Sheng, N. Mrachacz-Kersting, X. Zhu, D. Farina, and N. Jiang, “Decoding covert somatosensory attention by a BCI system calibrated with tactile sensation,” IEEE Trans. Biomed. Eng., vol. 9294, no. c, pp. 1–1, 2017, doi:10.1109/TBME.2017.2762461.
- L. Yao, X. Sheng, N. Mrachacz-Kersting, X. Zhu, D. Farina, and N. Jiang, “Sensory Stimulation Training for BCI System based on Somatosensory Attentional Orientation,” IEEE Trans. Biomed. Eng., 2018.
- S. Haegens, B. F. Händel, and O. Jensen, “Top-down controlled alpha-band activity in somatosensory areas determines behavioral performance in a discrimination task,” J. Neurosci., vol. 31, no. 14, pp. 5197–5204, 2011.
- S. Haegens, L. Luther, and O. Jensen, “Somatosensory anticipatory alpha activity increases to suppress distracting input.,” J. Cogn. Neurosci., vol. 24, no. 3, pp. 677–85, Mar. 2012, doi:10.1162/jocn_a_00164.
- G. R. Muller-Putz, R. Scherer, C. Neuper, G. Pfurtscheller, G. R. Müller-Putz, R. Scherer, C. Neuper, and G. Pfurtscheller, “Steady-state somatosensory evoked potentials: Suitable brain signals for brain-computer interfaces?,” Neural Syst. Rehabil. Eng. IEEE Trans., vol. 14, no. 1, pp. 30–37, 2006, doi:10.1109/TNSRE.2005.863842.
- C. Breitwieser, C. Pokorny, and G. R. Müller-Putz, “A hybrid three-class brain-computer interface system utilizing SSSEPs and transient ERPs,” J. Neural Eng., vol. 13, no. 6, p. 066015, 2016, doi:10.1088/1741-2560/13/6/066015.
About the Authors
Lin Yao received the Bachelor’s degree in mechanical design, manufacturing and automation from Sichuan University, Sichuan, China, in 2009, the Master’s and Ph.D. degrees in mechanical engineering from Shanghai Jiao Tong University, Shanghai, China, in 2015. Dr. Yao was a Research Scientist with the Institute of Neurorehabilitation Systems, University Medical Center Göttingen, Georg-August University, Göttingen, Germany, within the Bernstein Center for Computational Neuroscience from Apr. 2015 to Aug. 2016. There, he was the Head of the Brain-computer Interface Laboratory. From Sep. 2016 to Feb. 2018, he was a Postdoctoral Researcher at the Department of Systems Design Engineering, University of Waterloo, Waterloo, Canada. He is currently a Postdoctoral Associate at School of Electrical and Computer Engineering, Cornell University, Ithaca, NY, USA. His research interests include biomedical signal processing, BCI for neurorehabilitation, computational neuroscience, and machine learning.
Ning Jiang, PhD, (S’02–M’09) received the B.S. degree in electrical engineering from Xi’an Jiaotong University, Xi’an, China, in 1998, and the M.Sc. and Ph.D. degrees in Engineering from the University of New Brunswick, Canada, in 2004 and 2009, respectively. He was a Research Assistant Professor with Aalborg University, Denmark from 2009 to 2010, a Marie Curie Fellow with Otto Bock Healthcare GmbH, Germany, from 2010 to 2012, and a Research Scientist with University Medical Center Göttingen, Germany, from 2012 to 2015. Since 2015, he has been an Assistant Professor with the Department of Systems Design Engineering, University of Waterloo, Canada. His research interests include signal processing of electromyography and electroencephalogram, and their applications in neurorehabilitation. He is currently an Associate Editor of the IEEE Journal of Biomedical and Health Informatics, the Brain-Computer Interface, and Frontiers in Neuroscience.