Ricardo Chavarriaga, Sumit Soman, Zach McKinney, Jose L Contreras-Vidal, Stephen F Bush
Brain-machine interfacing (BMI) systems are a product of multiple integrated technologies, including sensing modules for biosignal acquisition and processing, computational systems for signal decoding and system control, and actuation modules for providing sensory, mechanical, or electrical feedback to the user, and/or for effecting desired physical actions (Figure 1). Such systems commonly comprise a combination of custom-engineered components and modules, in addition to one or more commercial devices or sub-systems.
The heterogeneous (and often ad-hoc) architecture of BMI systems is both reflective of the rich diversity of BMI users, user needs, and use cases (both clinical and non-clinical), and further reflected in the multiplicity of performance and functional outcome measures with which BMI system effectiveness is quantified and reported. This variety is further compounded by the immense complexity and variability of the neuro-physiological mechanisms that interact with BMI systems.
In this landscape of multiform variability, the functional interoperability of BMI systems and constituent technologies is essential to enable user- and use case-specific design and configuration of BMI systems. Moreover, the aggregated analysis (that is, data interoperability) of BMI-associated data – across time points, users, studies, and institutions – is required to maintain scientific, technological, and medical progress towards realizing the collective potential of neuro- and digital health technologies. Such multifaceted interoperability depends critically on the standardization of BMI-related technologies and the data they create. However, due to a host of historical, technological, and institutional factors, this necessary level of neurotechnology standardization does not yet exist.
To address this standardization gap, the IEEE Standards Association (IEEE-SA) initiated an Industry Connections Activity (ICA) on the topic of Neurotechnology for Brain-Machine Interfacing (NT-BMI; IC17-007) in May 2017, after a corresponding workshop at the Systems, Man & Cybernetics (SMC) Conference. This NT-BMI activity is dedicated to the evaluation of existing standards and best practices for BMI system design and usage, as well as to identifying priority areas for new standards to be developed. The NT-BMI members are a multi-stakeholder group, comprising experts and representatives from academia, industry, and regulatory agencies worldwide. In February 2020, the NT-BMI group released an IEEE Standards Roadmap document intended to provide a comprehensive overview of the current practices and future requirements for standardization in the BMI domain.
This article summarizes the main content and key recommendations of the NT-BMI roadmap. Importantly, given the breadth of the neurotechnology field and its applications, the NT-BMI effort is conscious of the need to include input from the neurotechnology, neuroscience, and clinical communities at large. For this reason, the public is invited to provide their feedback and suggestions on the document until September 30 2020, through this web page.
Organization of the Roadmap
As mentioned above, BMI systems are implemented through the integration of multiple elements. Depending on the situation, these elements rely on technologies at different levels of maturity. Correspondingly, the availability of standards may vary considerably depending on the constituent elements. To reflect the nature of BMIs as ‘systems of systems,’ the NT-BMI Standards Roadmap is structured in five functional areas that have been identified by the group: user needs, sensor technology, data representation, storage, & sharing, end effectors, and performance assessment & benchmarking.
At the highest level, the Standards Roadmap is divided into two main parts: first, an executive summary of the main findings and recommendations, followed by a second part detailing the status of standardization for the different BMI functional areas. The Roadmap also presents the findings of a public survey of neurotechnology researchers and developers regarding their views on the status and importance of neurotechnology standardization (See Figure 2), as well as a sampling of expert testimonials highlighting the prospective benefits of such standardization.
State of the art in BMI standardization
BMI sensor technologies encompass a broad spectrum of transducer types, including both invasive and non-invasive modalities. They range from well-established and widely used techniques such as electroencephalography (EEG) or electrocorticography (ECoG) to emerging approaches like microwave and ultrasound imaging, stentrodes, neural lace, and neural dust.
Among the 5 functional areas, sensing technologies are arguably the area with the highest level of standardization, with existing standards addressing a range of widely adopted sensor types. However, even among these standards, there remains extensive leeway for sensor variability with respect to numerous design parameters that affect the performance of the BMI, such as sensor size, shape, and material. This document identifies and discusses these key variables and parameters, along with recommendations for standardization priorities in sensing technologies.
End Effectors – Actuators and Feedback Devices
End effector systems for BMIs can be broadly categorized into exoskeletal devices, prosthetic devices & extracorporeal limbs, powered wheelchairs, neurostimulation devices, and virtual/augmented reality (VR/AR) interfaces (including those for communication). BMI-controlled exoskeletons and prosthetic devices for both upper and lower limbs have been largely pursued as rehabilitative and/or assistive tools for individuals with motor limitations. BCI/BMI powered wheelchairs have also been developed that integrate such devices. Additional use cases include neurostimulation, such as through functional electrical stimulation (FES), transcranial stimulation, deep brain stimulation (DBS), intra-cortical micro-stimulation, among others.
Data Representation, Storage, and Sharing
There have been diverse efforts by BMI research groups worldwide to define data formats for various biosignals used in BMI systems. These measures have taken several forms, including file format specifications, such as EDF/GDF, BCI2000, XDF, MFER, and sdeeg; standards, such as IEEE P1752/P7002/11073, ISO 22077-1:2015, and ANSI/CTA-2057/2058/2059/2061; software frameworks such as Lab Streaming Layer (LSL), OpenBCI, and OpenViBE; and initiative groups such as NeuroData without Borders, NIF, Brain-CODE, Brain Imaging Data Structure (BIDS), and the International Neuroinformatics Coordinating Facility (INC), among others. This Roadmap discusses the need to consolidate and streamline existing efforts towards developing a comprehensive set of standards for data representation, storage, and sharing.
Human factors engineering/usability engineering (HFE/UE) and User-Centered Design (UCD) processes have been shown to yield significant down-stream benefits in product development lifecycles, including higher user satisfaction, better product adoption, reduced net development costs, and early insight regarding future products, features, and markets.
The general processes for HFE/UE/UCD have been articulated in standards such as ISO 9241-210 (Human-Centered Design for Interactive Systems), IEC 62366 (Usability Engineering for Medical Devices), ANSI/AAMI HE75 (Human Factors Engineering—Design of Medical Devices), and ISO 60601-1-6 (Medical Electrical Equipment – collateral standard on usability). Additional pertinent standards are found in the ISO/IEC 25000 series, which defines System and Software Quality Requirements and Evaluation (SQuARE), with specific user needs-related standards including ISO/IEC 25022 (Measurement of quality in use) and ISO/IEC 25066 (Common industry Format for Usability—Evaluation Reports). Despite this wealth of standards defining frameworks for HFE/UE/UCD, there currently exist no standards that translate these high-level processes into BMI technology- or application-specific considerations or recommendations.
Performance Assessment and Benchmarking
Although BMI performance should ultimately be measured under closed-loop conditions in the context of intended use, assessments and benchmarking of its components are typically performed in simulations and off-line analysis, without empirical performance verification. Moreover, a distinction is warranted (yet commonly lacking) between BMI signal quality – determined by transducer characteristics, system configuration parameters (including sensor placement), the electrochemical environment of the sensor-user interface, and the signal conditioning electronics & protocols – and BMI system performance, which describes the accuracy and reliability of system output under some reference condition(s) and task(s). The precise definition of these conditions and tasks at both the functional and cognitive level is essential to obtaining performance benchmarking measures of adequate utility in comparing different BMI systems.
When incompletely characterized or misunderstood, the infidelities and uncertainty at the sensor level can easily propagate through the signal processing, decoding, and control paradigms, thus diminishing BMI system performance significantly. These effects may be difficult or impossible to remove with post-hoc analysis and processing. Thus, effective benchmarking of both BMI signal quality and sensor performance demands standardized protocols at multiple levels, from system configuration to test conditions to user instruction & training.
A wide variety of metrics from machine learning have been used to evaluate the decoding performance including: accuracy, precision, recall/sensitivity, F1-score, or confusion matrices. Additional metrics derived from information theory have also been used to assess performance of continuous decoding of brain signals. The scope and domain-based applicability of benchmarking metrics has been discussed in this roadmap.
The following are the key recommendations that have been identified in the roadmap for the respective focus areas.
- There is no established standard for time synchronization among different systems and modules, since the interfaces and ports to those systems vary widely.
- Consumer-grade sensors should comply with safety and performance standards that are consistent with the requirements of clinical devices, in the light of the prevailing trend towards the use of consumer device data for health and wellness applications.
End Effectors – Actuators and Feedback Devices
- Despite the existence of a standard terminology for prosthetic and orthotic devices, many terms related to the BMI control of these devices require a clearer definition.
- Interconnection between BMI sensing and processing modules and the end-effector requires the definition of standards for data communication.
- Standardization of shared control strategies and architectures will be important to improve the reliability and safety of the BMIs as systems of systems.
Data Representation, Storage, and Sharing
- Efficient data storage and secure interoperability, for both closed and open loop paradigms, has emerged as the need of the hour as far as standardization initiatives for data storage and sharing for BMIs are concerned.
- Aspects related to data security are important when sensitive data is shared across heterogeneous systems. Efficient encryption mechanisms need to be in-built into data representation schemes.
- As highlighted by the current limitations of poor medical data portability between different providers and institutions, there is a need for biosignal acquisition device manufacturers to adopt common data representation formats to facilitate data aggregability, sharing, and interoperability.
- The privacy and security of biodata from BMIs and associated devices (particularly wearable and implantable biosensors) are critical issues that are inadequately addressed by current standards. These concerns must be carefully considered and weighed against the often-competing interests in data shareability and analyzability.
- The definition of users, use cases, and user needs is the necessary foundation of any technological development process for both research and commercial devices, serving as primary inputs for HFE/UE/UCD, risk management, and quality control processes.
- Given that the concept of usability and its evaluation are predicated on a clear and precise definition of the intended users and use cases, the lack of BMI-specific standards in this area presents a significant `standardization gap.’
- User and usability-related standards should expand beyond their current primary focus on usability through the lens of product safety and mitigation of user-related risks, to encompass the totality of usability, including all aspects of effectiveness, efficiency, and user satisfaction.
- Standardization efforts should prioritize and target the aspects of neurotechnology design and development that will have the greatest impact in fulfilling user needs and facilitating clinical validation and commercialization, by providing detailed user needs frameworks that reduce the total time and resources required while maintaining or improving the rigor of device R&D efforts.
- At the clinical implementation level, clear clinical guidelines must be developed regarding the appropriate selection and customization of neurotechnology systems for individual patients/users.
- To the extent possible, HFE/UE/UCD processes for BMI must consider and specify the cognitive aspects of BMI use, including user instructions, feedback, and training.
Performance Assessment and Benchmarking
- Metrics for BMI benchmarking should distinguish between measures of signal quality and system performance, each under well-specified test conditions.
- While consolidation and standardization of existing BMI signal quality measures is merited, a clear priority is to develop standards and protocols for characterizing BMI performance. Importantly, these procedures and metrics should extend beyond the separate evaluation of system sub-components and allow the assessment of the entire BMI system during closed-loop operation under intended use conditions.
- Performance assessment should consider the effects of the human-in-the-loop on system performance. Human factors should be accounted for by way of precise user characterization and instruction, in combination with quantitative metrics of the technical components.
- Importantly, the development of standardized evaluation and reporting criteria for BMI systems and corresponding applications is key to enabling neurotech researchers & developers from different institutions to perform rigorous comparisons of alternate design and methodological approaches to solving a common set of problems.
The consolidated recommendations identified in the NT-BMI Roadmap are as follows.
- Neurotechnology standardization efforts should be invested to educate developers and innovators on the rich benefits of standardization with respect to technological design, quality of research, and the ultimate potential for clinical and commercial development. Accordingly, the standards development process should incorporate and leverage the expressed interests of all neurotechnology stakeholders – including researchers, developers, regulatory and scientific reviewers, clinicians, end users, etc. – via active community engagement by the NT-BMI group and related initiatives.
- BMI safety, security and privacy appear as top priorities for standardization. BMI-specific standards in this domain should build on existing principles, standards, and regulatory guidelines on medical and information technologies.
- To address critical gaps in existing standards and best practices, high-priority areas for new neurotech standards include BMI-related terminology, as well as BMI performance assessment and benchmarking in relevant conditions and applications.
- Existing efforts to improve scientific reproducibility and open science can be leveraged to establish and consolidate standards for data sharing and reporting on neurotechnology developments.
- The neurotechnology and neuroscience communities should consider the possibilities for defining complementary and modular standards that promote translation and scaling between consumer and clinical applications.
- It is important to envision and develop a flexible yet consistent neurotechnology standardization ecosystem that harmonizes between, community-established best practices, soft law, international consensus standards, research reporting guidelines, and government regulation. This may be achieved through implementation of strategies such as regulatory sandboxes and regular update of community guidelines and standards.
- BMI-specific standards should be aligned with existing and emerging standards and regulatory frameworks to address ethical, legal, and societal implications of emerging technologies.
The release of the Standards Roadmap: Neurotechnologies for Brain-Machine Interfacing (NT-BMI) is an early stage of a sustained, continuous standardization effort. In addition to the solicitation of public feedback on the first version of the Roadmap document, discussions of the Roadmap and related topics will continue through a series of events scheduled for 2020, including:
- Workshop“ Global Perspectives on Responsible Artificial Intelligence. Special panel on brain data”, 25-26 June 2020
- IEEE Computer Society Signature Conference on Computers, Software and Applications (COMPSAC 2020), 13-17 July 2020
- Special track on medical technology standards at IEEE International Conference on Human-Machine Systems (ICHMS), 7-9 September 2020
In addition, the work of this IEEE Standards Industry Connections Activity has resulted in the launch of working groups for development of standards in specific areas, as enumerated below:
- IEEE Working Group P2725.1: Standard for Microwave Structural, Vascular or Functional Medical Imaging Device Safety. http://standards.ieee.org/project/2725_1.html
- IEEE Working Group P2794: Reporting Standards for in-vivo Neural Interface Research (RSNIR). Link: http://sagroups.ieee.org/2794/
- IEEE Working Group P2731: Standard for a Unified Terminology for Brain-Computer Interfaces. http://sagroups.ieee.org/2731/
These groups are actively welcoming new participants from all neurotech stakeholder areas to join and contribute to the conception of standards for the new generation of BMIs.
Ricardo Chavarriaga is the head of the Swiss office of the CLAIRE – Confederation of Research Laboratories for AI in Europe; Research associate at Zürich University of Applied Sciences (ZHAW) and executive-in-residence at the Geneva Center for Security Policy (GCSP). Dr. Chavarriaga is chair of the IEEE Standards Association group on Neurotechnologies for Brain-Machine Interfacing and the Publications sub-committee of the IEEE Brain Initiative.
Passionate of responsible development and social implications of technology, Ricardo Chavarriaga is highly interested on the translation of emerging technologies onto applications at service of society. Dr. Chavarriaga has more than 12 years of experience in human-machine interaction, brain-machine interfaces, and artificial intelligence. His work is focused on responsible development of technologies that promote beneficial, humane interaction between human and intelligent machines.
Sumit Soman is a Principal Technical Officer with the Health Informatics group at the Centre for Development of Advanced Computing, Noida, India. He is a Senior Member of the IEEE and holds a Ph. D. from the Department of Electrical Engineering, Indian Institute of Technology Delhi. His research interests include brain computer interfaces and machine learning. He was a recipient of the Director General, CDAC’s Young Innovator Award 2019 and has over 40 conference and journal publications.
Zach McKinney is a post-doctoral research fellow at the BioRobotics Institute of Scuola Superiore Sant’Anna (Pisa), where he manages the development and clinical evaluation of exoskeletal robotic systems for applications in both industry and neural rehabilitation. He concurrently serves as chair of IEEE Working Group P2794, developing a Reporting Standard for in vivo Neural Interface Research (RSNIR), with the aims of promoting the use of rigorous experimental methodologies and comprehensive research reporting for a wide variety of neural interfacing modalities. He has participated in the current IEEE NT-BMI initiative since its launch in 2017 as the leader of the user needs focus area – all towards the creation of a more open, interoperable ecosystem of multimodal human-machine interfacing technologies.
Jose L ‘Pepe’ Contreras-Vidal is Cullen Distinguished Professor and Director of the NSF Research Center for Building Reliable Advances and Innovations in Neurotechnology (IUCRC BRAIN) at the University of Houston. and a Full Affiliate of the Department of Neurosurgery at the Houston Methodist Hospital. He was elected to the rank of Fellow by the Institute of Electrical and Electronic Engineering (IEEE) for his contributions to brain-machine interfaces and wearable exoskeletons, and is a member of the EMBS, SMC, RAS, IEEE Brain Transition Committee, the Systems Council, and the IEEE Roadmaps User Group of the IEEE.
Professor Contreras-Vidal is an associate editor of IEEE Transactions on Human Machine Systems, Brain-Computer Interfaces, and J of Mobile Human Computer Interaction, a Review Editor for Frontiers in Neuroprosthetics, a Topic Editor for Frontiers in Human Neuroscience and Editor-in-Chief of (Springer Nature Switzerland AG) Mobile Brain-Body Imaging and the Neuroscience of Art, Innovation and Creativity. His work has been supported by the NIH, NSF, VA, DARPA, ONR, Foundations, donors, and industry. His research has been highlighted by The Economist, Nature, Science, Science News, Der Spiegel, NSF, Wall Street Journal, SFN, O&P, Scientific American, NPR’s Science Friday, and Neurology Today among others. Pepe’s career development in biomedical engineering has been highlighted in the magazine Science.
Stephen F. Bush is a Senior Scientist at the GE Research Center. He is a past chair of the IEEE Emerging Technical Subcommittee on Nanoscale, Molecular, and Quantum Networking and currently Chair of the IEEE P1906.1.1 (https://standards.ieee.org/project/1906_1_1.html) working group on nanoscale communication networks. He taught Quantum Computation and Communication at Rensselaer Polytechnic Institute and is leading an effort to complete IEEE P1913, Software-Defined Quantum Communication.