EEG/NIRS Brain-Computer Interface Robotic Hand Control
Over summer 2015 I was fortunate to receive funding from the Deutscher Akademischer Austauschdienst (German Academic Exchange Service) to conduct research at the University of Freiburg. I was based in the Department of Neurosurgery and Neuroelectronic Systems, where some incredible research on neural stimulation, neural signals acquisition and processing, and brain-machine interfacing is taking place.
My project was to develop a simple brain-computer interface to control a 3D printed robotic hand that had been laying around the lab for a few years. I worked under Rand Almajidy, a Ph.D. student who is exploring the use of near-infrared spectroscopy (NIRS) and tripolar concentric ring EEG (tEEG) to add new modalities that improved the tried and true EEG BCI. The idea was to actively classify motor imagery states (that is, thinking of moving your hands or feet) while mapping those imagery states to the opening and closing of a finger, allowing for thought control of the robotic hand. I had no previous experience with BCI, and only basic signals processing in my classes, but I was up for the challenge
If you’re not familiar, electroencephalography (EEG) is a method used to record the electrical activity of the brain, usually used to diagnose epileptic seizures. Typically an array of electrodes is placed over the scalp, and sometimes the face, to create a potential map of the brain. While EEG is recorded in real time, it has low spatial resolution, and averaged the neural activity of a large area of the brain below the electrode. Consequently, the signal to noise ratio is quite low.
Tripolar concentric ring EEG (tEEG) is for all purposes the same as a normal EEG , but has three concentric rings (obviously) for recording. This effectively allows the electrode to acquire multiple EEG signals at once, which conveniently lets us exploit common mode rejection to improve SNR.
Near-infrared spectroscopy (NIRS) is the theory underlying pulse oximeters (those digital clips nurses put on your fingertip when you go to the hospital that read out your heart rate and blood oxygen content). It relies on the Beer-Lambert law, which says that absorbance of light by a solution is proportional to the ratio of the intensity of transmitted light and detected light, which in turn is proportional to the concentration of the solution.
In the equation above, A is absorption of light, I is the detected intensity of the transmitted light, I0 is the initial intensity of the light, L is the length of the container the light passes through, ε (epsilon) is the absorptivity of the solute, and c is the concentration of the solute. In a medical context, a modified Beer-Lambert law can be used to determine the concentration of oxygenated hemoglobin [HbO] and deoxygenated hemoglobin [HbR] in the blood. The modified equation is complicated, so I won’t bother with it here, but you can find it in a paper by Kocsis et al. (2006). NIRS utilizes the modified equation, but instead of looking at the intensity of transmitted light, it looks at the intensity of reflected light, since we aren’t putting detectors under the skin.
Together, EEG and NIRS allow us to collect a lot of information about activity in the motor cortex. Neuroscientists have noted that in humans, EEG recordings of the motor cortex show rhythmic activity at 8-12 Hz (the µ rhythm) and 12-30 Hz (the β rhythm) when at rest (no sensory input or motor output). When outputting motor information, there is a decrease in the power of these frequency bands. This lets us distinguish between when a person is sitting still or moving, based on the power of the µ and β bands when compared to a known resting state. Interestingly, this phenomenon occurs to a slightly lesser extent when someone simply thinks about moving, what is called motor imagery. Of course, this information is only useful when coupled with spatiotemporal information – which part of the motor cortex, and therefore the body – is being activated, and when. NIRS imaging gives an idea of where neural activity is occurring by showing where blood oxygen concentrations increase (brain work relies on metabolic energy, after all). The problem with NIRS is that it is slow compared to EEG – a few seconds of delay compared to realtime acquisition. This obviously presents a problem, but if the temporal delay can be resolved, NIRS should provide useful data.
Originally designed by Stephen Oung for his Master’s thesis, the hand serves as a proof of concept of a simple, low-cost, anthropomorphic tool for engineers to build brain-machine interfaces. The movement of the fingers is controlled by and N+1 actuator system, where N is the number of joints in each finger (in this case, N = 3). Simple servo motors serve as the actuators. The flexion and tension of the fingers is controlled by series of “tendons” made of Dyneema polymer. The original design used an Atmega 2560 master microcontroller with a slave SSC-32 board to control the servos. See Stephen’s writeup here for more thorough description of the mechanics, and his thesis.
Because I was attempting to interface the hand with online EEG data, I was able to scrap the Atmega board and interface through my computer’s serial port directly with the SSC-32 for servo control. I’m most comfortable programming in Matlab, so all data analysis and scripting was done in that environment.
The EEG and NIRS data were collected during experiments before I arrived in Freiburg. The data was collected from 8 (t)EEG electrode locations over the motor cortex, and 10 NIRS transmitter/receiver pairs that flanked each electrode location. Electrodes were placed according to the international 10-20 system. The experiments used the BCI2000 µ-rhythm protocol to test and train subjects and record data.
Before classification, the data needed to be thoroughly cleaned up. I opted to use open-source Matlab toolboxes supplemented with some of my own scripts to simplify preprocessing. EEGlab from the Swartz Center for Computational Neuroscience and the BCIlab extension were the core components for automated EEG preprocessing. All EEG data were notch filtered at 60 Hz (the measurements were taken in the U.S.), then bandpassed filtered from 5-35 Hz to isolate the µ and β bands. This was followed by applying a sliding average filter to smooth the data (window length 4, 50% overlap). To spatially filter the data, I ran a black-box function in EEGlab that decomposed the data using Independent Component Analysis, where EMG, ECG, and eye blink artifacts were removed using a blind source separation algorithm. Full disclosure: I didn’t (and still don’t) have much of an idea of how this worked. I trusted the documentation and it’s use in scientific literature. After cleaning the EEG data looked like this:
Even with the cleanup, it was impossible to tell what I was looking at. Before extracting data, I knew to expect the power of the µ and β bands to decrease during motor imagery states. I thought that I might be able to use the a subset of these bands, at a smaller frequency band where the difference between power at rest and power during imagery had the greatest difference. This “characteristic frequency” was identified using a multiple-trial mean and frequency correlation function within the BCI2000 toolbox, and indicated the likeliest spectral range to identify imagery states. To get useful information out of the (t)EEG data, I examined four key features of the spectral data in a +/- 2Hz range surrounding the characteristic frequency: mean power spectral density (PSD), minimum PSD, frequency at which the minimum PSD occurs, and the PSD at the characteristic frequency.
NIRS data was preprocessed in a relatively similar manner: bandpass filtered from 10mHz to 0.5Hz, and run through automated spike artifact and discontinuity rejection. Oxygenated hemoglobin concentrations were generated, and the baseline removed. All NIRS preprocessing was done in NIRSlab, another Matlab toolbox. With NIRS data, it’s not uncommon for a number of channels to be unusable due to poor contact with the scalp or misalignment of the transmitters and receivers. For each dataset, I determined the minimum number of usable channels, and if that number differed between all datasets for a subject, it was reduced by nearest-neighbor averaging. For example, if a subject did five trials, and four channels were usable from the first three, but only two channels usable from the last two trials, the first three trials would have the channel number reduced by averaging data from the closest transmitter/receiver pair. This was a questionable decision in terms of data quality, but I couldn’t think of anything better. The features extracted from the NIRS data included the variance of the [HbO] measurement, the average slope of the [HbO] data, and the mean value of the second derivative of [HbO] within my analysis window.
I dealt with both offline and online datasets. Each offline dataset was composed of 10 trials with 20 distinct and labelled motor imagery states alternating with rest states, while each online dataset also composed 10 trials, with 20 known motor states alternating with rest states.
I used a linear discriminant analysis algorithm (LDA.m) to classify the data into one of four states: right hand imagery, left hand imagery, feet imagery, or both hands imagery. Data was broken into 30 element bins with 50% overlap. I removed a random 80% of samples from each cleaned offline dataset for initial training, and tested the algorithm on the remaining 20%. Typically the probability of a given observation being distinctly in a motor imagery state was low, but different enough from the other three states for a subjective placement when looking at the raw data. For hand control, I needed a binary output, so I set a significance threshold at 0.45, where any observation with this value or higher was sent to 1.
It was obvious after visualizing the outputs of a few trials that four distinct classification states was producing garbage results. There were so many misclassifications that I experimented with reducing the number of possible classification states to three, and then seeing which combination produced optimal results. (Coincidentally, there were only three fingers on the hand, so instead of mapping one imagery state to an “open all fingers” state, I could just use the lack of any distict classification to signal the resting state, and open the hand.) Eliminating left hand motor imagery gave the best classification for the subjects I tested, so I stuck to throwing away the data from the those channels. The best case results had a dataset classified with a false positive rate of only 9% for right hand and both hands imagery (false negative rate ~23% though). Across the board, correct classification of feet imagery states was poor, althought not as poor as left hand imagery. This is likely due to the fact that feet imagery originates at the Cz electrode position, which is flanked by the right and left hand areas of the motor cortex.
The classification to hand movement control scheme was really simple. If an observation was classified distinctly as right hand imagery, the previously closed finger was opened (if one was closed) and the middle finger was closed; for both hands, the ring finger moved, and for feet the pinky finger moved. In cases where one observation was classified into more than one imagery state, nothing would happen, and if no distinct state was found, the subject was assumed to be at rest, and all fingers on the hand would open.
Although I didn’t perform live experiments to test active classification, I fed the online datasets into the algorithm for testing. I expected the accuracy to decline somewhat, and it did, in some cases dramatically, falling to 70% and lower. The video below shows the hand responding to the algorithm when imagery states were maintained for extended periods of time. Towards the end, you’ll see a lot of misclassification, but it gives an idea of the control sequence.
From starting with nearly no knowledge of brain-computer interfacing or machine learning, I think this project turned out really well. I was able to build a fairly accurate LDA classification system from open source software and some basic signals processing and dat analysis. I wanted to explore the reasons why left hand imagery performed so poorly in comparison to other imagery states, but ran out of time before heading back stateside. The program was fantastic, and the people who I worked with were incredible mentors. I’m looking forward to working in Germany again sometime soon!
The code for this project is available here. It’s really poorly written (don’t say I didn’t warn you).
1) Almajidy, Rand, Boudria, Yacine, Hofmann, Ulrich G., Besio, Walter and Mankodiya, Kunal (25-29 Aug. 2015). A Multimodal 2D Brain Computer Interface. Presented at the Engineering in Medicine and Biology Society (EMBC), 2015 37th Annual International Conference of the IEEE, Milan (1067 – 1070). IEEE. DOI 10.1109/EMBC.2015.7318549