RESEARCH
Serafeim Perdikis, Luca Tonin, Sareh Saeedi, Christoph Schneider, José del R. Millán
The Cybathlon competition has been the first ever international championship for disabled individuals competing with assistive devices1. Held in Zürich, Switzerland in October 2016, it featured the Brain-Computer Interface (BCI) race as one of its most innovative disciplines2. The Cybathlon BCI race aspired to accelerate the development of BCI technology, promote end-user and public awareness, as well as to attract funding agencies and entrepreneurs. The discipline consisted of a virtual race game called “Brain Runners” (Figure 1), where four brain-controlled avatars could be pushed towards the finish line by means of (up to) three mental commands. These commands should be issued by the “pilot” exclusively on suitable color-coded areas (spin on cyan, jump on magenta and slide on yellow “pads”). Pilots should also be able to “idle” and avoid any command delivery on additional white pads. Erroneous commands would slow down the pilot’s course down the race track, which consisted of sixteen consecutive pads (four of each type) placed in random order. Brain Runners has been designed to assess all those skills that are considered crucial to allow BCI in real-world scenarios, while also being attractive to Cybathlon’s live audience. Eleven teams representing BCI research groups from around the world have participated in the competition.
Figure 1. Screenshot of the standard race track of Cybathlon’s Brain Runners game (developed for the BCI Race of the Cybathlon 2016 in cooperation of ETH Zurich and Zurich University of the Arts (ZHdK), Switzerland).
The “Brain Tweakers”, a team of researchers in the CNBI laboratory of EPFL and their two pilots, Eric Anselmo and Numa Poujouly, both suffering from Spinal Cord Injury (SCI, ASIA A) excelled in the Cybathlon BCI race3. Their Cybathlon experience is thus valuable for demonstrating how people with disabilities in their chronic state, unable to use their legs and with severely compromised function of their upper limbs, have been capable of acquiring high-performing brain-control of a demanding application thanks to a training approach grounded on mutual learning of the subject and the BCI4. Importantly, the two pilots acquired their BCI skills in real-world conditions at their homes This culminated in exceptional performances under even more stressful circumstances during the Cybathlon event. The identified key-factor towards the Brain Tweakers success, namely mutual learning which reinstates the significance of subject learning and stresses the critical role of the application, put forward a practical roadmap towards translational MI BCI5.
Both Brain Tweakers pilots qualified for the final, where Poujouly was rewarded with the gold medal, while Anselmo was able to set the competition record in the qualification round. The existence of a significant training effect on race time has been established in either pilot case. These application performances were found to be due to the pilots’ proficiency in all four defined tasks (spin, jump, slide, idle), where the times spent on each pad compared favorably to the lower bounds and lied far away from the upper bounds imposed by the Brain Runners game. Significant increasing trends over the training period of BCI command accuracy were identified for both pilots and all command types, suggesting that the machine learning counterpart of mutual learning was adequately accommodated. In turn, gradually emerging sensorimotor rhythm (SMR) patterns in the β-band, coherent with the motor imagery tasks employed, reflect a significant gradual enhancement of BCI feature separability (i.e., more easily classifiable signals) over the training period and indicate a successful subject learning outcome that fueled the aforementioned BCI performance boost6,7. Hence, successfully coping with application demands should be largely attributed to capacity of the mutual learning methodology to improve, machine and subject-skill performances over time, bringing about the BCI’s high effectiveness and efficiency in translating motor imaginations into avatar actions.
The adopted mutual learning approach targeted both the improvement of the pilots’ MI skills and the optimal calibration of the BCI parameters, but, in contrast to current trends8, has shifted the main burden of learning on the shoulders of the human rather than those of the machine. Specifically, this training regime involved, initially, MI rehearsal without feedback in order to identify MI tasks that spontaneously evoke discriminant brain patterns and train the BCI to recognize them. The pilots then proceeded with closed-loop BCI receiving visual feedback, which aimed the strengthening of pattern separability in an operant conditioning approach. When needed the BCI parameters were also re-estimated to better capture the evolution of brain signal modulation as a result of learning. Emphasizing the focus on the user learning, this was done infrequently, only twice throughout the training period Race training, allowing our pilots to get used to the actual task at hand, occupied the majority of the late training sessions. Accompanied by the precision and parsimony of the applied signal processing and machine learning methodology, this fruitful subject training approach accounts for a large part of the observed performance increase at the neuroimaging, BCI and application levels, ultimately leading to our team’s victory4.
In addition to successful mutual learning, our work revealed that the control paradigm (the mapping between mental and application commands), can be also credited with a large part of our team’s outcomes. Early attempts in our training efforts with the obvious approach of implementing a 3-class BCI failed to provide adequate total command accuracy. Supporting only two commands in order to achieve higher-accuracy for a 2-class-only BCI was clearly also suboptimal given the game demands. Thus, we opted to associate two identified separable MI tasks (hands and feet movements for both pilots) to the spin and jump actions, and explore various solutions to additionally support the slide action. After a fruitful exchange of ideas between the engineers and pilots of our team, as well as the necessary quantified assessment tests, we eventually resorted to a control paradigm that would forward sliding upon reception of a quick spin/jump or jump/spin command sequence, achieved within a configurable timeout (Figure 2). This paradigm proved to be significantly superior to the other evaluated approaches in terms of command accuracy and delivery time for the slide action, while leaving the subject’s performance on the other avatar commands unaffected. The contribution of our end-user’s feedback has been thus also crucial for the final victory.
Figure 2. Mental tasks used to control the game avatar.
In summary, we have shown that severely disabled people can be trained to successfully operate a demanding (support of three control commands and possibility of staying idle) application under real-world and, even, adverse conditions (home use, competitive/stressful event, noisy environments) by means of a non-invasive, self-paced MI BCI (Figure 3). Our work has established the effectiveness and importance of a mutual learning strategy for BCI training that shifts the focus of learning on the subject, as well as a user-centered approach for interface design and personalization, which we strongly believe have been the key elements putting our team ahead of the competition during the Cybathlon event.
Figure 3. Average within-session BCI command accuracy (in percent) for pilots P1 and P2. The dashed black line shows the overall accuracy (average of individual command accuracies) in a session. Adapted from 4.
References
- Riener R. The Cybathlon promotes the development of assistive technology for people with physical disabilities. J Neuroeng Rehabil. 2016;13: 49. doi: 10.1186/s12984-016-0157-2
- Novak D, Sigrist R, Gerig NJ, Wyss D, Bauer R, Götz U, et al. Benchmarking Brain-Computer Interfaces Outside the Laboratory: The Cybathlon 2016. Front Neurosci. 2017;11: 756. doi: 10.3389/fnins.2017.00756
- Perdikis S, Tonin L, del R. Millan J. Brain racers. IEEE Spectrum. 2017;54: 44–51. doi: 10.1109/MSPEC.2017.8012239
- Perdikis S, Tonin L, Saeedi S, Schneider C, Millán JDR. The Cybathlon BCI race: Successful longitudinal mutual learning with two tetraplegic users. PLoS Biol. 2018;16: e2003787. doi: 10.1371/journal.pbio.2003787
- Leeb R, Perdikis S, Tonin L, Biasiucci A, Tavella M, Creatura M, et al. Transferring brain–computer interfaces beyond the laboratory: Successful application control for motor-disabled users. Artif Intell Med. 2013;59: 121–132. doi: 10.1016/j.artmed.2013.08.004
- McFarland DJ, Sarnacki WA, Wolpaw JR. Electroencephalographic (EEG) control of three-dimensional movement. J Neural Eng. 2010;7: 036007. doi: 10.1088/1741-2560/7/3/036007
- Ganguly K, Carmena JM. Emergence of a stable cortical map for neuroprosthetic control. PLoS Biol. 2009;7: e1000153. doi: 10.1371/journal.pbio.1000153
- Statthaler K, Schwarz A, Steyrl D, Kobler R, Höller MK, Brandstetter J, et al. Cybathlon experiences of the Graz BCI racing team Mirage91 in the brain-computer interface discipline. J Neuroeng Rehabil. 2017;14: 129. doi: 10.1186/s12984-017-0344-9
Author Biographies
Serafeim Perdikis received the Ph.D. degree in brain-computer interaction from the Ecole Polytechnique Fédérale de Lausanne (EPFL) in 2014, where he subsequently served as a post-doctoral researcher until May 2015. In June 2015 he joined the Wyss Center for Bio- and Neuro-engineering in Geneva, Switzerland. Since November 2017 he is a research engineer at Mindmaze SA, Switzerland. His main areas of interest are applied machine learning, bio-signal processing, mutual learning and co-adaptation in BCI, as well as translational and rehabilitative applications of BCI. He has been a member of IEEE since 2014.
Luca Tonin received the Ph.D. degree from the École Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland, in 2013. He then pursued 2.5 years of post-doctoral research in the Intelligent Autonomous System Laboratory (IAS-Lab) at the University of Padua, Italy. Since June 2016 he is post-doctoral researcher in the Defitech Chair in Brain–Machine Interface (CNBI) laboratory at EPFL. His primary research interests include bio-signal processing, human–robot interaction, shared control systems and advanced techniques to achieve natural control of BCI driven robotics devices.
Sareh Saeedi received her Ph.D. degree in electrical engineering from École Polytechnique Fédérale de Lausanne (EPFL), Switzerland. Then, she became a researcher at the Center for Neuroprosthetics (CNP) of EPFL. She joined Nissan International S.A. as a research engineer collaborator in 2017. Her research interests include brain–computer interface (BCI) systems, brain correlates of cognitive processes, machine learning, as well as translational and real-world BCI applications.
Christoph Schneider received his Master degrees in technical mathematics and biomedical engineering from TU Wien (TUW), Austria in 2013. He worked for g.tec medical engineering GmbH in the domain of brain-computer interfaces (BCIs) from 2013-2014. Currently, he is a Ph.D. student at CNBI at the École Polytechnique Fédérale de Lausanne (EPFL) specializing in BCIs for covert visual attention. His research interests include neuroscience, closed-loop BCI systems, correlates of visual attention, manipulation of cortical processes, noninvasive brain stimulation, as well as data science and machine learning.
José del R. Millán joined the École Polytechnique Fédérale de Lausanne (EPFL) in 2009 to help establish the Center for Neuroprosthetics. He holds the Defitech Foundation Chair and directs the Brain-Machine Interface Laboratory. He received a PhD in computer science from the Technical University of Catalonia, Barcelona, in 1992. Previously, he was a research scientist at the Joint Research Centre of the European Commission in Ispra (Italy) and a senior researcher at the Idiap Research Institute in Martigny (Switzerland). He has also been a visiting scholar at the Universities of Berkeley and Stanford as well as at the International Computer Science Institute in Berkeley. Dr. Millán has made several seminal contributions to the field of brain-computer interfaces (BCI), especially based on electroencephalogram (EEG) signals. Most of his achievements revolve around the design of brain-controlled robots. He has received several recognitions for these seminal and pioneering achievements, notably the IEEE-SMC Nobert Wiener Award in 2011 and elevation to IEEE Fellow in 2017. During the last years Dr. Millán is prioritizing the translation of BCI to end-users suffering from motor disabilities. As an example of this endeavour, his team won the first Cybathlon BCI race in October 2016. In parallel, he is designing BCI technology to offer new interaction modalities for able-bodied people.
