Brain-inspired cognitive model with attention for self-driving cars

Communicated by Dr. Sung-Phil Kim

December 2021

Shitao Chen, Songyi Zhang, Badong Chen and Nanning Zheng

Figure 1

Fig. 1. The framework of CMA model for self-driving cars. The purpose of this method is to establish a cognitive model with attention mechanism, in an end-to-end manner, to complete perceiving, planning, decision-making and controlling simultaneously. (adopted from [16])

As a typical artificial intelligence system, self-driving cars, unlike normal artificial intelligence systems [1], [2], usually concern the safety of people’s lives and property, and have little tolerance of mistakes. With the furthering of research on self-driving technology [3]–[7], the existing computing framework based on the “perception-planning-decision-control” information processing method has increasingly manifested the problems of low computing efficiency, poor environmental adaptability, and insufficient self-learning ability. Our research work mainly refers to the psychological level of human cognition to construct a new type of self-driving method.

The advantage of a human driver is that when operating a car, his subjective consciousness pursues safety. However, the complexity and diversity of the traffic environment can cause deviations between human subjective consciousness and his concrete behavior, which will eventually result in traffic accidents. On the other hand, the characteristic of the machine is that it can accurately perform a given task and has a high consistency during the execution. The core purpose of studying the brain-inspired self-driving method is to combine the safety awareness of human subjective consciousness with the consistency of machine execution to construct a new self-driving method, which is robust and safe.

Human driver’s driving is a process of continuous cognizing and understanding of traffic scenes [8], [9]. Nowadays, there is an unprecedented outpouring of new discoveries and data about the brain mechanisms from neuroscience [10]–[12]. At the same time, the concepts of modern physics and computing have also been widely used in the analysis of human cognitive processes [13]–[15]. These developments provide important enlightenment for us to study and discuss new methods of human-like intelligent driving. In our research, we refer to the cognitive process of human drivers and propose a human-like self-driving method based on cognition process, so that the self- driving system has a mechanism of memorizing, reasoning, and experience updating. It can drive safely in traffic scenes with strong noise, high dynamics and strong randomness. The proposed method effectively solves three basic scientific problems of the human-like self-driving method: 1) How human drivers pay attention to and acquire environmental information; 2) How environmental information is stored and processed in the brain; 3) What is the internal representation behind the driver’s behavior?

Figure 2

Fig. 2. The “Challenger” self-driving car. (a) The “Challenger”, based on CHERY TIGGO platform, with a four-wheel drive system (4WD) and a 4-speed automatic transmission. (b) The computing system and GPS/IMU integrated navigation system on the experimental vehicle. (c) Three cameras mounted on the top of the vehicle. (d) Electronically controlled steering controller driven by a DC motor. (adopted from [16])

Figure 3

Fig. 3. Simulation of the planning and control process with the CMA model. Simulated vehicle trajectories based on diverse lanes. In 3a to 3f, the obstacles were placed at 50, 80, 100, 200, 100, and 100 meters from the self-driving car, respectively, and they occurred at random times to test the planning and control capacity of the models. The lanes were 3.5 meters in width in 3a – 3d, and 4 meters in width in 3e –3f. (adopted from [16])

From the perspective of cognitive construction, the existing information processing framework and technical solutions of autonomous driving cannot well realize the process of human driving cognitive behavior and its information processing mechanism. Therefore, we propose a novel self-driving algorithm model named brain-inspired cognitive model with attention (CMA) [16], which refers to the cognitive process of human drivers and introduces the attention mechanism and vehicle historical behavior sequence into the self-driving algorithm. As shown in Fig.1, corresponding to the three basic scientific problems mentioned above, the proposed model mainly consists of three parts: a convolutional neural network (CNN) for simulating the human visual cortex; a traffic cognitive map for describing the relationship between objects in complex traffic scenes; and a recurrent neural network (RNN) combined with a real-time update of the cognitive map to realize the attention mechanism and long-term memory. The advantage of our proposed model is that it can accurately solve three main problems at the same time: i) detecting the drivable area and lane boundaries of the current lane and adjacent lanes; ii) estimating the distance, position, and attitude of obstacles relative to ego-vehicle; and iii) realizing human-like driving behavior and decision-making process. More importantly, compared with other deep learning or end-to-end methods that directly map the input image to control instructions such as steering wheel angles, our model can accept external navigation and control instructions in the end-to-end driving decision process, which makes the model more interpretable.

As shown in Fig.2, we collected and constructed a large-scale road-vehicle dataset (RVD) using the “Challenger” self-driving car to train and evaluate the proposed model. The data set contains more than 40,000 tagged road image data simultaneously captured by three camera sensors installed on the car. At the same time, the dataset also simultaneously recorded the behavior of the human driver and the status information of the vehicle at each moment.

Figure 4

Fig. 4. Trajectories in the vehicle-following scenario.

The experimental environment is set into two modes, one is a one-way two-lane road, and the other is a one-way single-lane road, which are used to evaluate the performance of self-driving vehicles controlled by the CMA method in two typical driving tasks of lane changing and vehicle following. In the task of lane changing, the lane width was set to 3.5 meters, 4 meters, and 2 meters respectively to test the performance of the algorithm in different road conditions. When the self-driving vehicle was driving, obstacles randomly appeared at different distances in front of the vehicle. During the test, the maximum speed of the vehicle was set to 40Km/h. Based on the lane change task, we conducted a total of 6 sets of comparative experiments. Fig.3 shows the comparison of the self-driving vehicle trajectory generated by the CMA model and by the algorithm proposed by Chen et al. [17] in different lane changing scenarios. In Fig.3, the blue solid line is the trajectory of the self-driving vehicle controlled by the CMA model, the gray dashed line is the trajectory of Chen et al.’s method [17], and the red dashed line is the ideal trajectory. In the task of vehicle following, the road was set as a one-way single-lane free space, and the test road length was 2000 meters. In this scenario, a target vehicle to be followed would be added in front of the test vehicle, and the goal was that the test vehicle could accurately follow the target vehicle. As shown in Fig.4, the red curve in the figure is the trajectory of the target vehicle to be followed, and the green trajectory is the trajectory of self-driving vehicles in 100 vehicle-following tasks under different noise levels.

The experimental results show that our brain-inspired cognition construction method for self-driving driving cars can well model the cognitive and behavioral processes of human drivers. The behaviors generated by self-driving cars controlled by the proposed model is very similar to the behaviors of human drivers, which enables the self-driving vehicles to cope with noisy and highly random traffic scenes, and significantly improves the safety and robustness of the autonomous driving system.


  1. J. Zhang and D. Tao, “Empowering things with intelligence: a survey of the progress, challenges, and opportunities in artificial intelligence of things,” IEEE Internet of Things Journal, vol. 8, no. 10, pp. 7789–7817, 2020.
  2. Y. Lu, “Artificial intelligence: a survey on evolution, models, applications and future trends,” Journal of Management Analytics, vol. 6, no. 1, pp. 1–29, 2019.
  3. B. Paden, M. Čáp, S. Z. Yong, D. Yershov, and E. Frazzoli, “A survey of motion planning and control techniques for self-driving urban vehicles,” IEEE Transactions on Intelligent Vehicles, vol. 1, no. 1, pp. 33–55, 2016.
  4. T. Litman, “Autonomous vehicle implementation predictions: Implications for transport planning,” 2020.
  5. M. Bojarski, D. Del Testa, D. Dworakowski, B. Firner, B. Flepp, P. Goyal, L. D. Jackel, M. Monfort, U. Muller, J. Zhang, et al., “End to end learning for self-driving cars,” arXiv preprint arXiv:1604.07316, 2016.
  6. S. Grigorescu, B. Trasnea, T. Cocias, and G. Macesanu, “A survey of deep learning techniques for autonomous driving,” Journal of Field Robotics, vol. 37, no. 3, pp. 362–386, 2020.
  7. C. Badue, R. Guidolini, R. V. Carneiro, P. Azevedo, V. B. Cardoso, A. Forechi, L. Jesus, R. Berriel, T. M. Paixao, F. Mutz, et al., “Self-driving cars: A survey,” Expert Systems with Applications, vol. 165, p. 113816, 2021.
  8. C. Roesener, F. Fahrenkrog, A. Uhlig, and L. Eckstein, “A scenario-based assessment approach for automated driving by using time series classification of human-driving behaviour,” in Intelligent Transportation Systems (ITSC), 2016 IEEE 19th International Conference on. IEEE, 2016, pp. 1360–1365.
  9. Z. Huang, C. Lv, and J. Wu, “Modeling human driving behavior in highway scenario using inverse reinforcement learning,” arXiv e-prints, pp. arXiv–2010, 2020.
  10. J. Hu, H. Tang, K. C. Tan, and H. Li, “How the brain formulates memory: A spatio-temporal model,” IEEE Computational Intelligence Magazine, vol. 11, no. 2, pp. 56–68, 2016.
  11. M. W. Mathis and A. Mathis, “Deep learning tools for the measurement of animal behavior in neuroscience,” Current opinion in neurobiology, vol. 60, pp. 1–11, 2020.
  12. B. A. Richards, T. P. Lillicrap, P. Beaudoin, Y. Bengio, R. Bogacz, A. Christensen, C. Clopath, R. P. Costa, A. de Berker, S. Ganguli, et al., “A deep learning framework for neuroscience,” Nature neuroscience, vol. 22, no. 11, pp. 1761–1770, 2019.
  13. C. Cinel, D. Valeriani, and R. Poli, “Neurotechnologies for human cognitive augmentation: current state of the art and future prospects,” Frontiers in human neuroscience, vol. 13, p. 13, 2019.
  14. N. Kriegeskorte and P. K. Douglas, “Cognitive computational neuroscience,” Nature neuroscience, vol. 21, no. 9, pp. 1148–1160, 2018.
  15. J. G. Greeno, “Some examples of cognitive task analysis with instructional implications,” in Aptitude, learning, and instruction. Routledge, 2021, pp. 1–22.
  16. S. Chen, S. Zhang, J. Shang, B. Chen, and N. Zheng, “Brain-inspired cognitive model with attention for self-driving cars,” IEEE Transactions on Cognitive and Developmental Systems, vol. 11, no. 1, pp. 13–25, 2017.
  17. C. Chen, A. Seff, A. Kornhauser, and J. Xiao, “Deepdriving: Learning affordance for direct perception in autonomous driving,” in Proceedings of the IEEE International Conference on Computer Vision, 2015, pp. 2722–2730.

Author Biographies

Shitao ChenShitao Chen received the B.E. and the Ph.D. degree in electronic and information engineering from Xi’an Jiaotong University, Xi’an, China, in 2015 and 2021, respectively. He is currently an assistant professor of Xi’an Jiaotong University. He is the executive vice president of Shunan Academy of Artificial Intelligence. His current research interests include computer vision, artificial intelligence, robotics, and autonomous vehicle. He has published over 30 papers in various journals and conference proceedings, while many of them won the best paper awards in summits. He is the member of IEEE and serves as the editor or reviewer of IEEE T-ITS, IV, TCDS, ICRA and other well-known international academic journals and conferences.

Songyi ZhangSongyi Zhang received the B.E. degree in electronic and information engineering in 2017 from Xi’an Jiaotong University, Xi’an, China, where he is currently working toward the Ph.D. degree with the Institute of Artificial Intelligence and Robotics. He is the member of the cognitive and autonomous driving group of the Institute of artificial intelligence and robotics, Xi’an Jiaotong University. His current research interests mainly include autonomous vehicle, computer vision, and robotics. He has published over 10 papers in various journals and conference proceedings.

Badong ChenBadong Chen received the B.S. and M.S. degrees in control theory and engineering from Chongqing University, Chongqing, China, in 1997 and 2003, respectively, and the Ph.D. degree in computer science and technology from Tsinghua University, Beijing, China, in 2008. He was a Post-Doctoral Researcher with Tsinghua University from 2008 to 2010, and a Post-Doctoral Associate with the Computational Neuro Engineering Laboratory, University of Florida, Gainesville, FL, USA, from 2010 to 2012. He visited Nanyang Technological University, Singapore, in 2015, as a Visiting Research Scientist for one month. He is currently a Professor with the Institute of Artificial Intelligence and Robotics (IAIR), Xi’an Jiaotong University, Xi’an, China. He has published two books, four chapters, and over 200 papers in various journals and conference proceedings. His current research interests include signal processing, information theory, machine learning, and their applications to cognitive science and neural engineering. Dr. Chen is a Technical Committee Member of the IEEE SPS Machine Learning for Signal Processing and the IEEE CIS Cognitive and Developmental Systems, and an Associate Editor of the IEEE Transactions on Cognitive and Developmental Systems, the IEEE Transactions on Neural Networks and Learning Systems, and the Journal of the Franklin Institute, and has been on the Editorial Board of Entropy.

Nanning ZhengNanning Zheng received the Graduate degree in electrical engineering and the M.S. degree in information and control engineering from Xi’an Jiaotong University, Xi’an, China, in 1975 and 1981, respectively, and the Ph.D. degree in electrical engineering from Keio University, Yokohama, Japan, in 1985. He is a Distinguished Professor with the Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University. He is the Founder of the Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University. He is currently a Professor and Director of the Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University. His research interests include computer vision, pattern recognition, autonomous vehicle, and brain-inspired computing. He became a member of the Chinese Academy of Engineering in 1999. He is a Council Member of the International Association for Pattern Recognition. He is the President of the Chinese Association of Automation and the Vice-President of the Chinese Society for Cognitive Science.