This project was presented at the 2018 Communication Matters conference and will appear in their journal sometime in 2019. The goal of this project is to enable unique gesture recognition for people with limited control of their motion. Almost all gesture recognition software focuses on the user replicating a predefined gesture, such as a swipe or sign-language. This is not practical for the students that I work with at Beaumont College who have cerebral palsy (CP). We intend to recognise whatever motion a student can comfortably make and then use this to trigger a control. This control will allow the student to interact with his or her communication software to create speech or interact with environmental controls.
Project code and a manual explaining how to install the software and setup the project can be found at my github site here.
Testing with Micro:bit based hardware
The latest HandShake hardware is based on the BBC Micro:bit. One Micro:bit is worn on the wrist and a second is connected to a laptop or communications device. When a gesture is made, the motion is detected by the Micro:bit on the wrist. This Micro: bit uses its radio to signal the Micro:bit connected to the laptop. This Micro:bit then triggers an event on the laptop through the USB cable connected to the laptop. The trigger can also be used as switch to control communications software, such as Smartbox’s Grid software. Thanks to Sensory Software for giving me a licence for this software to allow me to test the hardware with it.
For initial testing, we asked two of the students at Beaumont to operate a light through gesture. Please see the video below.
The advantage of using the Micro:bit is that somebody else makes them! Rather than my having to construct my own kludge, somebody else has kindly made a board which has all the widgets I need. The Micro:bit boards are around £12 each. You need two for this project. Instructions on how to make your own fashionable wrist holder for a Micro:bit from a £3 iPod arm band are here. All of the code and a manual showing how to replicate the system can be found on my GitHub site at: https://github.com/hardwaremonkey/microbit
The system was designed when Grid 2 and Windows 8 were in fashion. To enable the system to trigger Grid 2, we needed to use a Leostick plugged into the communication device, pretending to be a physical keyboard. I could not get Grid 2 to respond to software signals from my script. I needed to get the Leostick to send pretend keystrokes to Grid 2.
With Grid 3 and Windows 10, it looks like the Leostick is no longer required. Grid 3 seems to respond to software commands. The manual and software will be updated to reflect this. The system will still work with the Leostick, it is just an uneccessary extra expense now. I need to do more testing to be sure of this though.
The rest of this article talks of earlier versions of hardware that I made myself.
Testing with XBee based hardware
The photo below shows preliminary testing at Beaumont by one of the students. The pouch on the student’s wrist contains a microcontroller board and an XBee module for wireless data transmission. An accelerometer is attached to the XBee module. As the student moves, data is collected by the microcontroller and sent to a laptop for analysis. When the student shakes his arm within a comfortable range of ‘shakiness’ a bright LED called a blink1 attached to the laptop on the right flashes. The laptop screen shows the accelerometer data. Eventually we hope to do the processing and gesture recognition within the microcontroller on the student’s wrist and have this signal the student’s communication device directly. The accelerometer data display on the laptop allows me to develop the software to do this. See below for more details on the hardware.
Initial research used the Leap Motion. We found that the space that participants could interact with the Leap Motion was too limited for our user group. The code developed for recording and matching gestures will be tested with other technologies though, so the time was not wasted.
To continue with this research I moved to using an accelerometer to measure hand or wrist motion. Initially we will process and pattern match this data in a laptop to recognise a student’s gestures.
To display the real-time accelerometer data I wrote a user interface using the python library pyside. The python library pyqtgraph is used to display the accelerometer data for the x,y, z axis. The YouTube video below shows a recording of an early version of the interface, with data being displayed from the accelerometer real-time and the sample rate being changed. The display is lot more sophistimacated (sic) now.
I will make all of the code available once the project is finished. I use an assembla repository to store the code and would encourage any programmer to set up a git repository. You might think that you have adequate backups of your code…
The hardware has been through a few iterations. Initially I set up two way wireless communication with a pyboard connected to a digital accelerometer. The pyboard is programmed with micropython, so the entire tool chain from the hardware to the user interface is python 3. This code is built on the work of other programmers, who kindly put their code online. The pyboard worked but was a bit bulky. I got a slimmed down system working using an XBee series 1 to directly sample an attached analog accelerometer. This worked well for testing and verified the idea as feasible. Then I went to a talk about the the microbit. The killer feature is that these have board to board wireless communication as well as an accelerometer and microprocessor. This little board looks like it could be the way to go. Plus I can code using micropython again. I needed to change the default range of sensitivity on the onboard accelerometer. After figuring out how to do this I think that I can implement my ideas with this platform. Why didn’t I use this board to start with? It didn’t exist when I started on this project.
Here is a picture showing three iterations of hardware. On the left is an Xbee series 1 attached to an accelerometer kludged together on a breadboard. The board is powered by a single AAA battery using a dc-dc converter to pump up the voltage to 3.3v. In the middle is the same hardware, but soldered on to a prototype board. Best to solder everything down when it is being shaken around! On the right is the microbit board, powered by 2 AAA batteries, which is the board I am now using.
Xbee series 1 on breadboard, Xbee series 1 soldered down, micro:bit
I explored a number of different ideas before the microbit was available. Please see below.
Earlier hardware platforms
A number of smart watches, such as the Pebble, which have accelerometers built in, were trialed. These are designed to pair with a smartphone using Bluetooth Low Energy (BLE) and periodically send and receive data, not to constantly stream accelerometer data to a PC, which is what is required for the initial development work. It would have been nice to get something working reliably with this platform, as the students at Beaumont would be quite happy to wear the latest smartwatch. The only smartwatch I found with a stable and reliable link with a PC was Texas Instruments EZ430-Chronos. This comes with its own receiver dongle, so there is no issue in setting up a reliable link between this watch and a PC. However, the data sampling rate is limited.
I found a few Sparkfun WiTilt accelerometer and gyroscope sensor boards in the lab, left over from a long dead project on tracking people indoors. These are well designed boards with both a wired and an old school Bluetooth interface. Using the Bluetooth interface is a pain as for each iteration of code the device has to be reconnected. Using the wired serial interface allows for faster iterations of code as there is no re-connection to do each time the software is changed. I got this streaming accelerometer data to my laptop. However, this device is no longer manufactured. I emailed Sparkfun who said they had no plans to make anymore. So I started to look at what we can get off the shelf now. Using live hardware allows for others to easily replicate and improve anything that I come up with.
The Pyboard caught my eye. This runs micropython, which allows me to program the board using a version of the Python programming language. As this is the language that I use for my gesture recognition code, I figured this gives me a chance to eventually have all of the pattern matching done on the board. Initially I will take the accelerometer data from the board real time and process it on a laptop. Having the pattern recognition done on the accelerometer hardware will make for a better device that does not need to be constantly paired with a PC. The board will chug away on its own and when it recognises a gesture, send out a signal. That’s the plan anyway.
I’d heard of micropython for a while, but I was brought up on the ethos that with firmware ‘if you can’t do it in C, do it in assembler. If you can’t do it in assembler, it is not worth doing’. Then I listened to a podcast on micropython here and figured it was about time I stopped being such a curmudgeon. There are two types of fool. One says ‘this is old and therefore good’ and the other says ‘this is new and therefore better.’ With hardware design, I get to be both at the same time.
I interfaced the pyboard with an mpu-6050 accelerometer/gyrsocope board. If you want to get one of these, look on eBay where you will find these boards for a few pounds. I modified code from this project on Hackaday which is the site for the discerning electronics enthusiast. I am streaming data from the sensors to my laptop. I need to add some error checking to flag if there are missing data samples and compensate for these and to check the sampling rate is correct. Then write some unit tests, to avoid being a hardware design hypocrite.
As with any new platform, I encountered the usual World of Pain. I managed to install a micropython package over one of the regular python packages on my laptop. I never did figure out how to fix this. As luck would have it, I had a clonezilla image from the night before, which only took 20 nail biting minutes to load. Matt’s top tip – use clonezilla and use it often!