This weekend at HackPrinceton, we had a great time building Charles Xavier, our own version of the infamous XMen character!
**Here's some of the details:
**How we built it
Reading brain signals
We initially explored SSVEPs (a technique where looking at a flickering box produces measurable EEG activity at the same frequency of the flickering) [4]. We used canonical correlation analysis (CCA) to identify these frequencies [5], placing electrodes on occipital lobes by measuring the head.
We tested four stimulus frequencies (11, 13, 15, and 17 Hz), recording 15-second intervals for each. After several recorded sessions of 10+ minutes, our offline frequency classifier correctly identified the target in, on average, 9 of 44 analyzed windows (20.5%), below the 25% chance baseline. This was due to substantial electrical interference and signal fluctuations, which were likely due to noise from electrode contact through hair, muscle interference, and variations in placement from our measurements.
We pivoted to jaw-clench detection, using the OpenBCI hardware to capture electromyographic (EMG) activity. Because of the noise produced by muscle activity, we decided to use them as the primary signal [6].
*Object detection
*
EyeTrax, an open-source webcam gaze library is used to isolate objects that the user may want to move towards. We use YOLO and ByteTrack to box the objects in the camera feed, so that the box the user looks at is the object they choose to move toward.
*Eye tracking
*
We use eye-tracking to find the object the user looks towards (if they do not look at the screen they remain stopped). Before moving them though, we confirm by feeding the signal from the electrodes through our ML model to check whether they are saying yes or no to moving. If yes, we send a signal to move the chair!
*Moving the chair
*
Those signals feed a safety state machine that runs 20 times a second, defaulting to a stopped state. It sends an updated wheel speed to an ESP32 every 50 ms, using our custom Python library. The ESP32 turns the signal into 9-byte frames for the hoverboard's two boards. We erased the factory firmware and flashed our own build to be able to control the board's movements (https://github.com/RoboDurden/Hoverboard-Firmware-Hack-Gen2.x).
*Challenges we ran into
*
**Mapping the pinout of the firmware ourselves with a multimeter, probing the pads and reading their voltages one until the hall sensors, the serial lines, and the battery sense lined up. **Noise interfering with the brain recordings, and many false positives as a result until we refined our model. **Aligning the eye-tracker, OpenBCI signals, and ESP32 signals
*Accomplishments that we're proud of
*
We reverse engineered a piece of hardware (the hoverboard's PCB, for which there was no online specs), programming it with our own firmware. This allowed us to control the hoverboard's motors from our ESP32, and we created an extensible Python library to call different functions of the hoverboard from our microcontroller
It was all of our first time working with OpenBCI, and we trialed several brain monitoring approaches, moving from EEG signaling to EMG signaling and troubleshooting high levels of noise and impedance. After significantly improving the signal to noise ratio we trained a custom ML model and set up an API to provide live inference
Setting up the chair was more complicated than we expected, since the hoverboard caused the chair to rock back and forth, adding noise to both our eye tracking mechanism and brain signals. We ended up jerryrigging our chair with a few coat hangers providing a stable base. We also ran into a lack of hardware, setting up a custom switch because we couldn't find one.
Top comments (0)