Capturing Touch for Prosthetic Limbs Through Artificial Skin

RESEARCH
Luke E. Osborn1 and Nitish V. Thakor1,2

Those living with upper limb differences face numerous challenges, including lost limb movement and dexterity as well as missing sensory information during object manipulation. From a user’s perspective, upper limb prostheses still have several issues with control, general discomfort from the socket, and lack of sensory feedback 1. Significant efforts have resulted in sophisticated algorithms for decoding intended prosthesis movements along multiple degrees of freedom that have enabled amputees to regain more dexterous prosthesis control 2. Another seminal advancement is targeted muscle reinnervation surgery 3, which targets nerves to different intact muscle groups such as on the chest to provide a source of well differentiated myoelectric signals for prosthesis control.

Lack of sensory information and feedback has limited the perceptual capability of the amputees. Major advancements were made in 2014 when researchers used implanted stimulating electrodes to provide sensory information back to an upper limb amputee for detecting different objects during grasping 4, conveying pressure information to complete dexterous manipulation of a fragile object 5, and general tactile activation 6.

Touch Complexity

Sensory information, specifically touch, is an extremely complex and multifaceted percept that remains difficult to completely capture. Thousands of receptors in our hands work together to pass tactile information from our fingertips to the spinal cord and into the somatosensory regions of the cortex. For upper limb amputees, the peripheral nerves and feedback to the brain still exist, but these pathways are disrupted at the receptor level in the residual limb. Researchers can take advantage of the remaining intact neural pathways to provide some element of tactile information back to an amputee. One challenge is how to convey specific tactile information by stimulating remaining peripheral nerves either through the skin or directly. However, the disrupted distribution of the receptors in the amputee’s skin and their complex tactile encoding, both individually and as a population, make it hard to replicate complex touch sensations.

Tactile information is captured by various receptors in the skin. Mechanoreceptors are responsible for our ability to perceive sensations such as pressure, texture, vibration, and stretch whereas muscle spindles and Golgi tendons drives our innate ability to perceive position (i.e. proprioception). Thermoceptors convey sensations of temperature while nociceptors enable us to feel mechanical pain, such as a sharp prick or a cut 7. Researchers have been able to provide sensory percepts of pressure 4,5,8, vibrations 8, texture 9, illusory movements 10, and now even pain 11 to upper limb amputees. Extensive knowledge gained from studying skin receptor properties has spurred the development of artificial electronic skin (e-skin) and more specifically the electronic dermis (e-dermis).

Artificial Skin

Researchers have previously developed artificial electronic skins that take advantage of the developments in flexible electronics12,13 to produce e-skins. In one such implementation, the digital mechanoreceptor inspired sensor translates pressure into oscillatory spikes 14, and in another implementation the oscillatory output drives nerve stimulation of an artificial afferent in an invertebrate 15. Most advances in sensors and artificial skins are focused on materials and electronics and typically do not incorporate sensory feedback to a prosthesis or amputee. For upper limb prostheses, one challenge is translating the response of an artificial skin into meaningful sensory information to the user by mimicking the natural sensory encoding of touch.

E-dermis for Perception of Touch and Pain

Using biology as a model, we developed a multilayered electronic dermis (e-dermis) for capturing a range of tactile perceptions at the fingertips of a prosthetic hand. We implemented a neuromorphic model to transform the e-dermis measurements to biologically relevant spiking activity for nerve stimulation, which was then used for transcutaneous electrical nerve stimulation (TENS) to provide sensory feedback (Fig. 1A). A neuromorphic system is one that attempts to mimic components of a neural system through digital signals, in this case representing touch. The idea behind this implementation is to try and capture actual receptor characteristics to convey tactile information to an amputee.

Fig. 1: Multilayered e-dermis for sensory feedback. (A) Touch information capture by the e-dermis is transformed with a neuromorphic model into meaningful spikes for nerve stimulation, which results in sensory perception. (B) Piezoresistive and conductive textiles covered in rubber make up the epidermal and dermal layers of the e-dermis. The e-dermis layering attempts to mimic healthy skin where nociceptors are found in the epidermis and mechanoreceptors in the dermis. Images adapted from 11. Reprinted with permission from AAAS.

The e-dermis mimics the skin and its receptors in several ways: it has an array of sensors (receptors); the sensors are arranged over multiple layers (Fig. 1B); it produces receptor like signals; and it encodes sensor information the manner encoded by nerves. The e-dermis was made up of piezoresistive fabric (Eeonyx), which was placed between intersecting conductive traces (LessEMF) to create pressure sensitive taxels. A 1-mm layer of silicone rubber (Dragon Skin 10, Smooth-On) was added between the epidermal and dermal layers of the e-dermis and a 2-mm rubber layer added protection and compliance to the fingertip e-dermis.

Our goal was to model the skin and its receptors, and to mimic the range of perceptions from light touch to noxious, or painful. To detect pressure and pain, we treated the epidermal (upper) layer of the e-dermis as a nociceptor and the sensing elements in the dermal (bottom) layer as mechanoreceptors. The neuromorphic output from the e-dermis was then used as the stimulation signal for sensory feedback.

To understand the sensory perceptions perceived by an amputee during nerve stimulation. We performed sensory mapping of the phantom hand of one amputee as well as a quantification of the various sensory perceptions, including discomfort resulting from stimulation at a noxious level produced by controlling different stimulation parameters. Additional details can be found in 11.

To evaluate the ability of the amputee wearing the prosthesis to differentiate between tactile pressure and pain, we used 3 objects of varying curvature for a simple prosthesis grasping task (Fig. 2A). The prosthesis was able to reliably detect pain when grasping the sharpest item (Fig, 2B). Indeed, the prosthesis responded with a reflex to drop the object, similar to what happens in biology when we experience pain (i.e. withdrawal reflex). In another experiment where the user’s vision was occluded from the object being grasped, the sharper object was perceived by the user as being more painful (Fig. 2C).

Fig. 2: Perception to pain. (A) Objects used for the grasping task. Each object has a different radius of curvature. (B) The prosthesis perceived the sharp item (object 3) as being painful most of the time, which triggered the automatic pain reflex to drop the object. (C) The amputee also perceived increased pain for the sharp item (object 3). The amputee reported discomfort on an increasing scale up to 10. For context, a pain level of 3 corresponded to discomfort similar of receiving an accidental cut. Images adapted from 11. Reprinted with permission from AAAS.

One question that should be addressed: why pain? Our perception of pain is valuable because it protects our bodies by conveying information on things in our environment that are potentially damaging or harmful. A prosthetic arm doesn’t have this ability. Our recent research investigated how the idea of sensing pain could potentially benefit a prosthesis user. Because a prosthesis doesn’t have the ability to heal itself, we created a prosthesis pain reflex to compliment the sensory information being sent back to the user. In a way, this additional sensation of pain enables the prosthesis itself to become a little more lifelike and “self-aware” in its ability to understand the environment. At the same time, the tactile information being sent back to the user hopefully helps create a more realistic and feature-rich sensation of touch.

The combination of the biologically inspired e-dermis with neuromorphic stimulation models attempts to capture some of the nuanced characteristics of natural receptors, specifically those that convey innocuous and noxious signals. As upper limb prostheses continue to advance we turn to the human body as a template for developing sophisticated sensors and techniques for making these prosthetic devices more lifelike. Recreating the complex sensation of touch requires continued research of how nerve stimulation is perceived by a prosthesis user as well as how we can more accurately convey artificial neural signals that can be perceived as natural sensations.

Acknowledgements

This work was partially funded by the Space@Hopkins funding initiative through Johns Hopkins University. The results of this study were published in the June 2018 edition of Science Robotics:

Osborn, L. E., Dragomir, A., Betthauser, J. L., Hunt, C. L., Nguyen, H. H., Kaliki, R. R., & Thakor, N. V. Prosthesis with neuromorphic multilayered e-dermis perceives touch and pain. Science Robotics, 2018;3(19):eaat3818. DOI: 10.1126/scirobotics.aat3818.

References

  1. Biddiss, E., Beaton, D., and Chau, T. Consumer design priorities for upper limb prosthetics. Disability and Rehabilitation: Assistive Technology. 2007;2(6):346-57. DOI: 10.1080/17483100701714733.
  2. Farina, D. et al. Man/machine interface based on the discharge timings of spinal motor neurons after targeted muscle reinnervation. Nature Biomedical Engineering. 2017;1(2):25. DOI: 10.1038/s41551-016-0025.
  3. Kuiken, T. A. et al. Targeted reinnervation for enhanced prosthetic arm function in a woman with a proximal amputation: a case study. Lancet. 2007;369(9559):371-80. DOI: 10.1016/S0140-6736(07)60193-7.
  4. Raspopovic, S. et al. Restoring natural sensory feedback in real-time bidirectional hand prostheses. Science Translational Medicine. 2014;6(222):222ra19. DOI: 10.1126/scitranslmed.3006820.
  5. Tan, D. W. et al. A neural interface provides long-term stable natural touch perception. Science Translational Medicine. 2014;6(257):257ra138. DOI: 10.1126/scitranslmed.3008669.
  6. Ortiz-Catalan, M., Håkansson, B., and Brånemark, R. An osseointegrated human-machine gateway for long-term sensory feedback and motor control of artificial limbs. Science Translational Medicine. 2014;6(257):257re6. DOI: 10.1126/scitranslmed.3008933.
  7. Abraira, V. and Ginty, D. The sensory neurons of touch. Neuron. 2013;79(4):618-39. DOI: 10.1016/j.neuron.2013.07.051.
  8. Wendelken, S. et al. Restoration of motor control and proprioceptive and cutaneous sensation in humans with prior upper-limb amputation via multiple utah slanted electrode arrays (USEAs) implanted in residual peripheral arm nerves. Journal of Neuroengineering and Rehabilitation. 2017;14(1):121. DOI: 10.1186/s12984-017-0320-4.
  9. Oddo, C. M. et al. Intraneural stimulation elicits discrimination of textural features by artificial fingertip in intact and amputee humans. eLife. 2016;5:e09148. DOI: 10.7554/eLife.09148.
  10. Marasco, P. D. et al. Illusory movement perception improves motor control for prosthetic hands. Science Translational Medicine. 2018;10(432):eaao6990. DOI: 10.1126/scitranslmed.aao6990.
  11. Osborn, L. E. et al. Prosthesis with neuromorphic multilayered e-dermis perceives touch and pain. Science Robotics. 2018;3(19):eaat3818. DOI: 10.1126/scirobotics.aat3818.
  12. Kim, J. et al. Stretchable silicon nanoribbon electronics for skin prosthesis. Nature Communications. 2014;5:5747. DOI: 10.1038/ncomms6747.
  13. Yokota, T. et al. Ultraflexible organic photonic skin. Science Advances. 2016;2(4):e1501856. DOI: 10.1126/sciadv.1501856.
  14. Tee, B. C. K. et al. A skin-inspired organic digital mechanoreceptor. Science. 2015;350(6258):313-6. DOI: 10.1126/science.aaa9306.
  15. Kim, Y. et al. A bioinspired flexible organic artificial afferent nerve. Science. 2018;360(6392):998-1003. DOI: 10.1126/science.aao0098.

Biography

Luke Osborn
Luke Osborn, MSE, is a PhD student in biomedical engineering at Johns Hopkins University. His research in the Neuroengineering and Biomedical Instrumentation Lab focuses on developing tactile sensing technologies, neuromorphic modeling of sensory information, and sensory feedback for upper limb prostheses. He is a student member of the IEEE.

Nitish Thakor
Nitish Thakor, PhD, is a professor of biomedical engineering, electrical and computer engineering, and neurology at Johns Hopkins and directs the Laboratory for Neuroengineering. He is also the director the Singapore Institute for Neurotechnology (SINAPSE) at the National University of Singapore. His research focus is in the field of neuroengineering, including neural diagnostic instrumentation, neural microsystems, neural signal processing, optical imaging of the nervous system, neural control of prostheses, and brain-machine interfaces. He is a recipient of a Research Career Development Award from the National Institutes of Health and a Presidential Young Investigator Award from the US National Science Foundation. He is a founding fellow of the Biomedical Engineering Society and fellow of the IEEE.