Brain Actuated Technologies, Inc

CT99 , 3rd International Cognitive Technology Conference, Conference Proceedings, San Francisco August 11-14,1999, Pages 237-250, Hosted by Michigan State University

Cyberlink - An Interface for Quadriplegic, Traumatic Brain Injured, and Nonverbal Persons

School of Computing, Engineering & Technology, University of Sunderland;
PO Box 299 Sunderland, UK, SR6 OYN; chris.bloor@sunderland.ac.uk

The United States Constitution states that people have the inalienable right to the pursuit of happiness. Also, US residents in government-funded long-term care facilities often have a right to recreation. It has been estimated that there are approximately 4.7 million people, or one in 65 Americans, with traumatic brain injury (TBI) (Cambell 1998). Their condition often impairs their ability to be active and communicate. To withhold recreation from an institutionalized person because of a handicap, is a form of discrimination. Recreational activities need to be complemented by communication in order to express ideas, wants, and needs. This communication is necessary to participate in society. A substantial percentage of the population thus has a potential need for assistive communication, environmental controllers, and recreational devices. Many people who cannot speak depend upon a computer or augmentative device for communication. Current input devices tend to require the use of voice or some form of physical movement, be it limb, facial or eye movement. However, users who have limited use of their eyes, no use of their limbs or face, cannot speak or move their head are severely disadvantaged and often cannot use existing, physically-manipulated assistive technologies. Their needs could be met by mental interfaces that make use of signals from facial movements and mental activity. There have been some early developments of mental interfaces with limited success in Austria (Kalcher 1994) and Australia (Craig et al. 1997,p.8) using EEG waves.

This paper reports a study using a new mental interface that is also supplemented by physical movement. The Cyberlink combines signals from the head potentially to let many handicapped persons be active through the use of video games and control their lighting via an environmental controller, or perhaps to communicate via an on-screen keyboard. The reported study is part of the first author’s Ph.D. research. The author is based in the USA and all practical work is being carried out in New Jersey.

The paper describes the Cyberlink and its associated Cybertrainer software. It then presents the study and its results. Results and observations from the study are then discussed. Plans for future work are then presented.

The Cyberlink is a BrainBody interface that uses BrainBody control technology by combining eye movement, facial muscle, and brain wave bio-potentials detected at the forehead to generate computer inputs that can be used for a variety of tasks. The forehead was chosen as the place to collect signals because it is rich in bio-potentials. Three plastic sensors on a headband detect these signals which are then sent to a Cyberlink interface box, and hence to the serial port of the computer. The signals are amplified, digitized, and translated by a patented decoding algorithm into multiple command signals, allowing hands-free control of the computer (Junker 1995).

Three different control signals are derived from the forehead signals by the Cyberlink interface. The lowest frequency control signal is called the ElectroOcularGraphic or EOG signal. This is the frequency region of the forehead bio-potentials that are responsive primarily to eye movements. The EOG signal is typically used to detect left and right eye motion. The signal can be mapped to left and right cursor motion or on/off switch control.

The second type of control signal is called the ElectroEncephlaGraphic or EEG signal. The Cyberlink interface subdivides this into ten component frequency bands called "Brainfingers". These frequencies reflect internal mental/brainwave activity as well as subtle facial muscle activity. A wide range of facial muscles affects these frequency bands. Users typically learn to control Brainfingers through subtle tensing and relaxing of various muscles including forehead, eye, and jaw muscles. After some experience with the Cyberlink system, most users begin to experiment with more efficient, internal brain-based control methods. Since this frequency region is sensitive to both mental and muscular signals it is called the BrainBody signal.

The third type of control signal is called the ElectroMyoGraphic or EMG signal. The EMG signal primarily reflects facial muscle activity. It is typically used in the Cyberlink system for discrete on/off control of program commands, switch closures, keyboard commands, and the functions of left right mouse buttons (Berg et al. 1998).

A program called Cybertrainer is bundled with the Cyberlink interface hardware. It has several applications to train users and which may be used to help configure the system for particular users. It is also possible to configure the interface for personality type, which affects bio-potentials. Five of these applications are relevant to this study and are described below:

Clicking performs a switch function. It can be based upon a gaze left and right, a jaw clench, raising an eyebrow, or raising one of the ten brainfinger waves to a high enough level to activate a switch. The action that will effect a switch or click action is chosen from the program’s menu. The sensitivity level must then also be adjusted to suite the characteristics of each user.

The typing program is a standard scanning typing program. A keyboard is displayed on screen. The cursor then moves from key to key, dwelling on each key for a set period of time. The user performs their switching action to select the letter or key at the current cursor location.

These are similar to the video game version of Ping-Pong (Brown 1999). In the video game version a control knob is used to move the paddle up and down a vertical axis. The Cybertrainer version uses a brainfinger controlled by signals from the player’s forehead to control vertical axis movement. An intense thought or facial muscle movement sends the paddle from the bottom to the top of the screen. Relaxing moves the paddle back down the screen. The placing of the paddle directly in front of an oncoming ball is scored as a hit. A ball that is not intercepted is known as a miss. There are vertical and horizontal versions of the Ping-Pong programs.

This is a game that requires simultaneous control of movements in the vertical and horizontal axis to navigate a cursor through a maze. Control of an axis is assigned to one of the 10 Brainfingers Movement on an axis can thus be controlled by thought, facial movements, and eye movements. The labyrinth is considered to be complex, comprising a series of over fifty walls and openings to be navigated. A simple maze is also available. Appendix A shows a screen dump of the Labyrinth program.

Intervention by an able bodied person must actively start the music program from a suite of applications that are issued standard with the Cyberlink software. Another intervention is required to select a tune from a menu that appears with a choice of approximately twelve tunes. The Cyberlink user needs no intervention to play the tunes and can do this hands free. The bio-potential signals are used to drive various instruments. Low frequency instruments such as a base drum are assigned to lower frequency Brainfingers, while higher frequency instruments such as a guitar are mapped to higher frequency Brainfingers

The song can be passively listened to for enjoyment or the participant can actively play an instrument. A series of instruments appear on the screen for a brief duration of time. The participant can create a strong signal at the forehead to select that instrument shown. A further signal can be created to select and play a note from a sliding scale of notes that becomes visible when an instrument is selected. (See Appendix A) The music becomes softer as one relaxes and lowers the amplitude of the visible Brainfingers This concept of using gestures to control the amplitude of music was used in an early audible biofeedback device called the Theremin (Paradiso 1997). The ideas of using biofeedback to control music were actually realized by Rosenboom in the 1970s. He created a "Bio-muse" that controlled MIDI musical events by signals collected from the heart, brain, eyes, and muscles (Paradiso 1997). The Cyberlink music device differs from the Theremin and Rosenboom device in its small program size and its adaptability to a laptop sound card as opposed to an elaborate MIDI device or specialized hardware. (Knapp & Lusted 1990, pp. 45-48) have developed an additional music device in which the electrodes are placed upon the scalp and the EMG sensors are connected to other locations than the head.

The participants were all part of a longitudinal study from May 12^th, 1998 to Feb 28^th, 1999. It had already been established, as part of the study, that they were able to use the Cyberlink to navigate the complex labyrinth, and to play Ping-Pong. The purpose of the investigation reported here was to determine whether they could use the Cyberlink independently to play games, communicate, play music, and operate environmental controllers.

We have already reported work in which we demonstrate that navigation of the labyrinth is sufficient to show intention on the part of participants, and that this is a good test of competence in the use of the device (Doherty et al., 1999). Having thus established that participants were able to use the device under supervision (and with intervention) we then wished to investigate whether participants could use the device independently, with minor intervention. Participants are normally checked every fifteen minutes by care staff (for feeding and toiletting needs). The researcher instituted the same regime, also checking what tasks had been performed.

The participants, session procedures and system set up used in the actual study are now described as "Participants."

The eight participants were all volunteers from a variety of group homes, institutions, and private residences. All participants, or their legal guardians, signed consent forms to participate in the study. The participants consisted of six adult men and two adult women. Table 1 gives the distribution of gender and disability. The participants are all experienced Cyberlink users who had successfully completed the Labyrinth and were able to perform a clicking or switching activity on command.

The participant was fitted with the interface headband and the Cybertrainer program started. After selecting the correct participant profile for the Cyberlink, the interface calibration was checked and any necessary adjustment made. Participants then learned how to perform a "click". The interface allows a head bob, eyebrow raise, jaw clench or a particular brainfinger level to be used for this. All participants were able to perform a click and chose to use the eyebrow raise for this. Once participants were able to perform a click, "no-hands" operation was selected and the participant asked to perform a number of tasks.

There are a number of variables in this study which were not possible to control rigorously, for various reasons. These are noted below:

Participants were asked to undertake three tasks when instructed. Each task was chosen from the no hands menu by performing a click as the system scanned the menu options. The first task was to choose and play Ping-Pong, which was the first option on the menu. After each game control returned to the no-hands menu options.

The participants were then asked to perform a communication task using the typing program with the dwell time set at two seconds. Participants were prompted what to type.

The third task was to operate an appliance such as a lamp or radio connected to the computer via the X-10 environmental controller. An option called switch select was available from the no-hands menu. Once in switch select, six boxes could be observed. Navigation of the cursor into a box caused highlighting of that box. A click would then initiate a switching action. The Cyberlink is still under development and minor intervention was required from the experimenter once the participant had successfully chosen a switch. The device has since been modified so that this intervention is no longer required.

It is rare that all activities can be performed in one day. The effects of medicine or fatigue often shorten a test session. Participants also had varying attention spans. Sessions were terminated when a participant indicated that they no longer wished to continue.

A Trogon Pentium laptop computer with a Windows 95 operating system was used. A Cyberlink was used as the interface device. An X10 Environmental Controller (Tandy Corporation) was attached via the serial port. A grounding strap that was worn on the wrist and attached to the back of the Cyberlink prevented static electricity from interfering with the operation of the interface.

The vertical motion of the cursor was controlled by an alpha wave of approximately 4-hertz (brainfinger four), while simultaneous horizontal control was performed using a 2-hertz theta wave (brainfinger two). The first setting rather than the default brainfinger (a beta wave) had to be used for participants with TBI, as they have great difficulty controlling their beta waves.

All participants were able to perform a click and to choose an option from the main menu. Further details are given below for each of the tasks.

All participants played at least one game of Ping-Pong. This was to be expected as this game forms part of the training and setup of the interface. The point here is that participants were able to choose to play Ping-Pong from a menu of options.

All participants, except for participant four correctly chose the typing option and were able to communicate the phases indicated in Table 2. We were not able to determine satisfactorily whether participant four had problems with the click, or whether she preferred to play Ping-Pong rather than type. Ping-Pong was the earliest option, and hence the easiest to choose.

All participants were able to use the environmental controller except for participant four, the reasons for this being given above. Actual tasks completed are given in Table 3 above.

Three participants tested the music program, which became available some time after the start of this study. Details of their reactions to the program are given below. It should be noted that participant 4 is deaf. She was able to get some appreciation of the music being played by tactile feedback, obtained by placing her hands on the speaker. The results of the music inquiry are listed in Table 4 below.

The participants and a mental health professional who assisted in this study made some interesting comments. Participant eight used a system of eye spelling to indicate that he had fun creating the music with brain waves, facial movements and eye movements but did not fully understand the process. He suggested that the sound should be fed into a stereo so that all the subtleties of the sound can be enjoyed. He indicated by eye-spelling to his nurse that he was once in a coma for six weeks and was aware of his surroundings as well as his own physical condition part of the time. The participant had been a classical music conductor, and stated that he would have enjoyed having a Cyberlink to allow him to create music again, play games, and communicate while in his comatose state. He eye spelled that there is an overwhelming feeling of being locked in, and there were vast amounts of time with nothing to do. Psychiatric evaluation of completely paralyzed individuals indicates that boredom or "downtime" is a serious problem and that these patients need some form of active recreation (Knapp & Lusted: 1990, p.45). There were other issues relating to the Cyberlink upon which participant eight commented. He felt if he had an opportunity to use a Cyberlink during his time in a coma, he could have navigated a cursor through a maze (Appendix A) with guidance. This would not only help establish his cognitive state as sentient, but provide relief from boredom while in the coma. The participant states that fun from creating music or playing games can make an intolerable situation a little better. Evidence from the health professional, and from the work of. (Resnick,1996,p. S136) leads us to believe that playing video game Ping-Pong and navigating the Labyrinth maze can help the disabled person to stimulate their mind and develop lost proficiencies. The challenge of improving a score or improving a completion time acts as a positive reinforcement. We also consider that when doctors and staff members who once saw the person as unable to communicate and comprehend, begin to realize that the person has abilities, they reverse the isolation that formally existed. Verbal reinforcement is provided and once again opportunities for nonverbal communication can be explored. This can be considered a first step toward mental health.

Participant one who is a quadriplegic verbal man, indicated that the music program was fun. He verbalized that EMG emanating from eyebrow movements could move a sliding scale that allowed note playing on a certain instrument. The participant found that the volume on the laptop was quite adequate when adjusted for maximum loudness. He had a music background and played the piano before his spinal chord injury. Whilst the music was easy to create he felt he needed extensive practice to fully understand and control it.

Participant four has been deaf since birth. She experienced the music by putting her hands on the speakers, a technique which she has used on other occasions in the group home where she lives. The song called "Postcard from Bali" had a strong beat that vibrated the speakers. The vibration of the speakers caused her to smile and her head bobbed to the beat. The researcher observed some gestures that indicated that the experience with the music made her very happy. The young woman then touched both the happy and good symbols on the communication board at the conclusion of the session. Songs with a less pronounced beat that did not vibrate the speakers did not seem to evoke emotion or head bobbing.

Our initial investigation causes us to consider the music program to be of use for the following reasons. The Cyberlink music program can operate on a standard laptop, only takes a few megabytes and does not need a MIDI interface. It allows deaf people an additional opportunity to experience music through the small vibrating internal laptop speakers. The physical space required by the system allows motor impaired individuals an opportunity to create some music. A laptop and Cyberlink can easily fit on top of a hospital tray. The Cyberlink hardware requirements are only for a minimum of a 80486 DX at 75 megahertz. Many institutions that have such older computer hardware that has been donated or handed down from commercial sources. The researcher has also observed that the participants at the institutions tend to have older PCs with slower processors and smaller hard drives. The digital signal processor hardware is also small. Many of the institutionalized residents stated that they have limited living space and that the Cyberlink is good because it can be stored easily on a shelf.

	Date of Results	Interviewed	Duration of Music
8	February 18,1999	Eye spelled that he had fun using the music program	Indicated after 90 minutes that he had enough and wished to watch Television (NOTE even with adjustments, #music was very soft)
1	February 20,1999	Very verbal and indicated it was fun	Spent 20 minutes composing and verbalized he had other matters to attend to.
4	February 19,1999	Pointed to good and happy on the communication board	Spent 20 minutes with both hands on Speaker,(her idea was to put second hand on speaker) Only songs with a beat got head bobbing, smiling, and looking off in the distance

TABLE 4 - Results of the Music Inquiry

Skilled use of the Cyberlink was established by the study. However, this was only achieved by a change to the Cyberlink and by application of improved understanding of environmental and individual factors that can impede successful use. We will discuss these issues before considering extensions to the use of the Cyberlink.

The participants needed extensive practice and training to produce an acceptable click or switching action. Some participants produced a constant flow of involuntary clicks after the first click had been attempted. This was generally due to a quivering action of the jaw or eyebrow and is termed perseveration. This is highlighted in Table 3 for participants seven and eight. Modifying the Cyberlink software so that signals after the first click was disregarded solved the problem. The length of 'dead' time allowed after a click is now part of the setup routine. It was generally found that a delay of half-a-second was sufficient. This feature was added in August, approximately halfway through the study.

In the earlier part of the study it was found that participants seven and eight could not perform a click satisfactorily under any of settings tried. The signal obtained was always in a high state and could not be varied. Junker (private communication) had experienced similar problems. The cause was traced to static electricity that may develop with wheelchair users in dry atmospheres. A grounding strap, worn on the wrist and attached to the back of the Cyberlink was thus fitted. The signal amplitude then fell to a level that allowed normal clicking.

Participant eight had an Ascentia 950N laptop computer from AST. The amplitude of the BrainBody signal was raised to a high state when the power supply was plugged into the laptop was operated off the mains supply. The signal could not be brought to a low state under any setting. Junker had found that some power supplies might interfere with the normal operation of the Cyberlink interface. The participant thus used the laptop battery supply in all subsequent sessions.

Some participants in the longitudinal study have physical, cognitive, and multi-sensory impairments. It is difficult, at times, to explain the concept of relaxation them. The researcher has tried miming relaxation, deep breaths, or blowing against a piece of paper repeatedly in order to demonstrate the concept, but this is not always successful. It is then necessary to increase the left bias on the cursor movement to compensate for their inability to relax. A familiar movement, such as a head bob, is then used to activate the brainfinger that moves the cursor to the right.

Participant four felt that he might have been able to establish communication while in his comatose state by using the Ping-Pong game. He could have used left and right movement of the paddle to indicate 'yes' and 'no'. There are sensitivity, debounce, and gravity (bias) settings which can be adjusted to filter out signals from involuntary motion and make yes/no communication easier. Simple yes/no questions answered by a person with the Cyberlink could show they are sentient and allow their diagnosis to be upgraded. This is a difficult area medically, and one where diagnosis may not always be 100% correct. The implication of misdiagnosis for the 'comatosed' person are significant. In some cases the person becomes more of an object than a human being. Simple things like a knock on the door when entering a room or hello seem unnecessary, and are not done. No one can be sure if the resident can hear and possibly comprehend what is happening around him or her. What staff do observe is that the person has not communicated with them. This makes staff less likely to notice nonverbal forms of communication such as eye blinks or other movements. The person loses the opportunity for recreation and mental stimulation without the ability to communicate. This leads to the increased likely and severity of depression, loneliness, and anxiety (Resnick, 1996,p. S136). Any new ability to communicate because of the Cyberlink, even in the simplest form, can help improve the person's mental health.

We observed a comatosed young man in an earlier study, reported elsewhere (Doherty, 1999). He used an alpha wave to navigate a cursor through the Labyrinth (Appendix A) under instruction, and was able successfully to navigate the maze seven times out of eighteen attempts with the completion times in the range of 212 to 860 seconds. As indicated earlier, there are certain variables which we cannot control such as medicine or time of visit. The participant often has seizure medicines changed and experiences colds, flu, and aspirating pneumonia. The above observations, combined with expert testimony of medical personnel in his facility provided his doctor with enough data to determine that he is sentient. His diagnosis was upgraded from coma to traumatic brain stem injury. The new diagnosis allowed a higher level of care. The participant has subsequently typed 'HI' using the typing program described previously. We suggest that this device has potential diagnosis value which should be investigated by medical professionals.

We have demonstrated the use of the Cyberlink interface as a communication and recreational device for quadriplegic, traumatic brain-injured, and nonverbal persons. These initial results have been promising in that the majority of participants were able to navigate a maze, play Ping-Pong, communicate by typing and use an environmental controller. The device would appear to have some application in the diagnosis of coma, and as a communication and recreational device for patients who have previously been considered unable to carry out such functions. Further studies are required in this area. Some very preliminary studies of the music program again indicate its validity.

Berg, C., Junker, A., Rothman, A., Leininger, R. (1998) The Cyberlink Interface: Development of A Hands-Free Continuous/Discrete Multi-Channel Computer Interface Small Business Innovation Research Program ( SBIR ) Phase II Final Report Published by Brain Actuated Technologies, Incorporated 139 East Davis Street, Yellow Springs, Ohio 45387 , United States of America

Campbell, C. A. (1998) Spinal Injury Registry Urged, The Record (Passaic Morris Edition of N.J.) September 22 L1-L3

Clarke, S. and Cockton, G., (1988) Linking between Multiple Points in Design Documents, CHI 98 Conference Summary, eds. C-M. Karat et al. ACM, 223-224, 1998.

Cockton G, Clarke S, Gray P and Johnson C, (1996) Literate Development: Weaving Human Context into Design Specifications, in D Benyon and P Palanque (eds) Critical Issues in User Interface Systems Engineering, Springer Verlag, 227 - 248.

Craig, A, Kirkup, L, McIsaac, P, Searle, A. (1997) The Mind as a Reliable Switch: Challenges of Rapidly Controlling Devices Without Prior Learning. Human Computer Interaction: Interact 97

Doherty, E. P., Bloor, C. and Cockton, G. (1999) The "Cyberlink" Brain Body Interface as an Assistive Technology for Traumatically Brain Injured Persons: Results of a Longitudinal Study Accepted for publication in J. Cyberpsychology and Behavior

Kalcher J., Flotzinger D., Gölly S., Neuper G., and Pfurtscheler G. (1994) Brain-Computer Interface (BCI) II, Proc. 4^th Int. Conf. ICCHP 94, Vienna, Austria 171-176

A. Strauss and J. Corbin, (1990) Basics of Qualitative Research - Grounded Theory Procedures and Techniques, Sage Publications

Knapp, B.R, Lusted, H.,(1990) A Bioelectric Controller for Computer Music Applications, Computer Music Journal, M.I.T. (14),1, pp. 42-47

NJ-Law - New Jersey Department of Health, Division of Health Facilities Evaluation and Licensing Regulations 8:39-7.4 Mandatory Resident Activities

Resnick, H., Fries, B., Verbrugge, L., (1997) Windows to their World: The Effect of Sensory Impairments on Social Engagement and Activity Time in Nursing Home Residents. Journal of Gerontology,May 1997, page S-136.

Ability, Issue 29, Summer 1999, Pages 12-13
Published in the UK, Editor in chief , Geoff Busby, = geoffrey.busby@gecm.com

CyberPsychology and Behavior, "The Cyberlink Brain-Body Interface in = Traumatic Brain Injury", Volume 2, Number 3, June 1999, ISSN 1094-9313, pages 249-259

Cyberlink , C.A.T., Cyberlink Actuated Tracker, D.R.E.A.M., Digital Recursive Event Algorithmic Music, Brainfingers, "Bringing the Gift of Hands-Free Access to the World" are trademarks (TM) of Brain Actuated Technologies, Inc.

Featured Research:

University of Sunderland - Eamon P. Doherty M.S.

The following describes a longitudinal study performed with a variety of disabled persons for the purpose of investigating the Cyberlink as an assistive technology to aid people in communicating, recreating, and operating environmental controllers.