Communicated by Distinguished Professor Chin-Teng Lin
Chin-Teng Lin and Alka Rachel John
Humans are easily overwhelmed with tasks that push them beyond their capabilities . Despite their remarkable resilience to diverse working conditions, the work environment must be adapted to afford comfortable interactions with human operator abilities . Modern work environments position human operators at a supervisory level where they have extensive interactions with technology and must integrate multiple streams of information, demanding more cognitive resources  and resulting in a higher workload in the human operators [4, 5].
Such demanding task conditions at least compromise the access to resources , if not wholly depleting resources , impairing human performance. Therefore, increasing workload might result in errors  that will compromise the safety and efficiency of the system . Overload and underload conditions deteriorate performance as the human brain continually pursues cognitive comfort and resource homeostasis . High performance can be maintained by reallocating tasks and adapting system behaviour based on an estimate of the operator’s mental workload. Therefore, an accurate and reliable measurement of the operator’s mental workload is crucial, especially in complex environments such as safety-critical Air Traffic Control (ATC).
An accurate and real-time assessment of workload can be achieved using neurophysiological measures, such as the electroencephalogram (EEG) signal, as the effects of task demand are visible in EEG rhythm variations [11 – 13]. The estimated mental workload can be used to trigger mental workload adaptive strategies in adaptive automation [14 – 17]. Such adaptive automation should conform to the dynamic workload of the operator without having them explicitly state their needs or manually triggering the automation.
Human operators expect adaptive automation to behave like a cognitive empathetic human coworker , stepping in at the right time and assisting on the task that is currently overwhelming them. Nevertheless, the strategies employed by the current adaptive automation remain primitive as researchers fail to exploit the physiological correlates of mental workload in deciding not just “when” but also “what” to adapt. There is a need to develop intelligent adaptive systems that can identify what form of automation to use depending on the type of mental workload experienced by the operator.
We investigated whether the multimodal physiological metrics of mental workload can provide more information about the task contributing to the workload. As shown in Fig.1, we designed multiple objects tracking  and collision prediction tasks, inspired by the real-world tasks that ATC operators routinely perform to ensure safe and efficient air traffic flow. The tracking task emulates the real-world ATC job of keeping track of aircraft, and the conflict detection task of ATC inspired the design of the collision prediction task. Although both these tasks are inspired by the elementary tasks that ATC operators routinely perform in complex work environments, we considered them separately to untangle the differences in the physiological response to workload variations in these tasks.
EEG data were recorded from 24 participants performing tracking and collision prediction tasks with three difficulty levels. The experimental protocol was approved by the University of Technology Sydney Human Research Ethics Expedited Review Committee (ETH19-4197).
Figure 1: (A) shows the experimental design of the tracking task, and (B) shows the design of the collision prediction task. The number of dots shown in these diagrams is just for representation purposes.
The results demonstrate that EEG measures were not just sensitive to the workload variations but also the task type. The tracking task demands the allocation of attentional resources to keep track of one, three or five tracking dots moving randomly among distractor dots. Working memory load is sensitive to increased allocation of attentional resources and is reflected by increases in frontal theta power , and the results show an increase in the frontal theta power with increasing demand in the tracking task.
Tracking dots moving among distractor dots also entails working memory mechanisms related to relevant item maintenance and increases in the memory load, which was reflected by a decrease in the alpha power [21, 22]. The alpha power is also known to decrease with increased memory load [23, 24] and task difficulty . Our findings also substantiate this working memory mechanism as the occipital alpha power decreases with increasing workload levels in the tracking task. Therefore, an increase in tracking load was reflected by the increase in frontal theta power and the decrease in occipital alpha power. No significant changes were observed in the parietal theta, alpha, occipital delta, or theta power with the increasing workload in the tracking task.
Figure 2: (A) EEG processing pipeline for the experiment. (B) Scalp map, dipole locations and PSD at the Frontal and Occipital clusters selected in the tracking task. (C) Scalp map, dipole locations and PSD at the Frontal, Parietal and Occipital clusters selected in the collision prediction task.
In the collision prediction task, anticipating the trajectory of the dots and predicting whether the dots would collide requires attention and internal concentration. Delta power is an indicator of attention or internal concentration in mental tasks, and it has been reported to increase with the increase in workload [25, 26]. We see an increase in the delta power at the occipital sites with increasing workload in the collision prediction task, which validates an increased allocation of attentional resources with increasing levels of workload in the collision prediction task. Additionally, keeping a tab on the trajectory of six, 12 or 18 eight dots adds to the memory load in the participants. Several studies have shown that theta power is correlated with memory load  and working memory capacity .
In the collision prediction task, our results reveal a significant increase in the theta power at the frontal, parietal and occipital clusters, confirming an increase in memory load with increasing workload levels. Furthermore, our results indicate that there is an increase in parietal alpha power with increasing levels of workload. This observed alpha band desynchronisation with the increasing workload is related to relevant item maintenance in the working memory [21, 22, 25, 26] and is known to decrease with increased memory load [23, 24] and task difficulty . So, the increase in workload for the collision prediction was correlated with the increases in frontal theta, parietal theta, occipital delta and theta power and a decrease in parietal alpha power while no significant variation was observed in the occipital alpha power.
Figure 3: Intelligent mental workload adaptive system.
As the neurometrics of workload variations in tracking and collision prediction tasks are different, they might be able to identify the specific aspects that contribute to the increase in workload in complex work environments at any time instant and define the strategies that can be used by the workload adaptive system to mitigate this increase. The differences in neural response to increased workload in the tracking and collision prediction task indicate that these neural measures are sensitive to variations and types of mental workload and their potential utility in not just deciding “when” but also “what” to adapt, aiding the development of intelligent closed-loop mental workload aware systems in ATC, as shown in Fig. 3. This investigation of EEG correlations of workload variation in these basic tasks has applicability to the design of future adaptive systems that integrate neurometrics in deciding the form of automation to mitigate the variations in workload in complex work environments.
- Ahlstrom, Ulf. “An eye for the air traffic controller workload.” In Journal of the Transportation Research Forum, vol. 46, no. 3. 2010.
- Wickens, Christopher D., et al. Engineering psychology and human performance. Psychology Press, 2015.
- Pashler, Harold. “Dual-task interference in simple tasks: data and theory.” Psychological bulletin 116.2 (1994): 220.
- Niosh, N. The changing organization of work and the safety and health of working people. Technical Report 2002-116, National Institute for Occupational Safety and Health (NIOSH), 2002.
- Landsbergis, Paul A. “The changing organization of work and the safety and health of working people: a commentary.” Journal of occupational and environmental medicine 45.1 (2003): 61-72.
- Borragan Pedraz, Guillermo. “Behavioural bases and functional dynamics of cognitive fatigue.” (2016).
- PA Van Dongen, Hans, Gregory Belenky, and James M Krueger. “A local, bottom-up perspective on sleep deprivation and neurobehavioral performance.” Current topics in medicinal chemistry 11.19 (2011): 2414-2422.
- Reason, James. “Human error: models and management.” Bmj 320.7237 (2000): 768-770.
- Xie, Bin, and Gavriel Salvendy. “Prediction of mental workload in single and multiple tasks environments.” International journal of cognitive ergonomics 4.3 (2000): 213-242.
- Hancock, Peter A., and Joel S. Warm. “A dynamic model of stress and sustained attention.” Journal of Human Performance in Extreme Environments 7.1 (2003): 4.
- Borghini, Gianluca, et al. “Measuring neurophysiological signals in aircraft pilots and car drivers for the assessment of mental workload, fatigue and drowsiness.” Neuroscience & Biobehavioral Reviews 44 (2014): 58-75.
- Di Flumeri, Gianluca, Gianluca Borghini, Pietro Aricò, Nicolina Sciaraffa, Paola Lanzi, Simone Pozzi, Valeria Vignali et al. “EEG-based mental workload neurometric to evaluate the impact of different traffic and road conditions in real driving settings.” Frontiers in human neuroscience 12 (2018): 509.
- Lin, Chin-Teng, and Tien-Thong Nguyen Do. “Direct-Sense Brain-Computer Interfaces and Wearable Computers.” IEEE Transactions on Systems, Man, and Cybernetics: Systems (2020).
- Prinzel, Lawrence J., Frederick G. Freeman, Mark W. Scerbo, Peter J. Mikulka, and Alan T. Pope. “A closed-loop system for examining psychophysiological measures for adaptive task allocation.” The International journal of aviation psychology 10, no. 4 (2000): 393-410.
- Schmorrow, Dylan, Kay M. Stanney, Glenn Wilson, and Peter Young. “Augmented cognition in human–system interaction.” Handbook of human factors and ergonomics (2006): 1364-1383.
- Scerbo, Mark W. “Theoretical perspectives on adaptive automation.” (1996): 37-63.
- Kaber, David B., and Mica R. Endsley. “The effects of level of automation and adaptive automation on human performance, situation awareness and workload in a dynamic control task.” Theoretical Issues in Ergonomics Science 5, no. 2 (2004): 113-153.
- Aricò, Pietro, et al. “Human factors and neurophysiological metrics in air traffic control: a critical review.” IEEE reviews in biomedical engineering 10 (2017): 250-263.
- Innes, Reilly J., Nathan J. Evans, Zachary L. Howard, Ami Eidels, and Scott D. Brown. “A broader application of the detection response task to cognitive tasks and online environments.” Human Factors (2019): 0018720820936800.
- Gevins, Alan, and Michael E. Smith. “Neurophysiological measures of working memory and individual differences in cognitive ability and cognitive style.” Cerebral cortex 10.9 (2000): 829-839.
- Capilla, Almudena, et al. “Dissociated α-band modulations in the dorsal and ventral visual pathways in visuospatial attention and perception.” Cerebral Cortex 24.2 (2014): 550-561.
- Puma, Sébastien, et al. “Using theta and alpha band power to assess cognitive workload in multitasking environments.” International Journal of Psychophysiology 123 (2018): 111-120.
- Fairclough, Stephen H., Louise Venables, and Andrew Tattersall. “The influence of task demand and learning on the psychophysiological response.” International Journal of Psychophysiology 56, no. 2 (2005): 171-184.
- Ryu, Kilseop, and Rohae Myung. “Evaluation of mental workload with a combined measure based on physiological indices during a dual task of tracking and mental arithmetic.” International Journal of Industrial Ergonomics 35, no. 11 (2005): 991-1009.
- Sterman, M. B., and C. A. Mann. “Concepts and applications of EEG analysis in aviation performance evaluation.” Biological psychology 40.1-2 (1995): 115-130.
- Wilson, Glenn F. “An analysis of mental workload in pilots during flight using multiple psychophysiological measures.” The International Journal of Aviation Psychology 12, no. 1 (2002): 3-18.
- Jensen, Ole, and Claudia D. Tesche. “Frontal theta activity in humans increases with memory load in a working memory task.” European journal of Neuroscience 15, no. 8 (2002): 1395-1399.
- Sauseng, Paul, Birgit Griesmayr, Roman Freunberger, and Wolfgang Klimesch. “Control mechanisms in working memory: a possible function of EEG theta oscillations.” Neuroscience & Biobehavioral Reviews 34, no. 7 (2010): 1015-1022.