Using Python to detect activity in Sensory Software’s Grid 2

Update: March 2018. This work is being submitted to the Communications Matters conference.

Following on from the eyeBlink post, with the help of Fil at Beaumont, I modified the algorithm I’m using to detect when the Grid 2 or Grid 3 software is being used. The image below shows Sensory Software’s Grid 2 software being used to construct a sentence. The new text appears in the white area at the top of the window. Fil suggested that I change the Python script to just monitor this area at the top of the window. The script now looks for a change in the amount of black text in this area. After the usual software wrangling I think I got it working. The Python script looks at the top 20% of the window and counts the number of black pixels in this area. Every half second it recounts the number of black pixels. If there is a change in the number of black pixels above a threshold, then a trigger is sent to indicate that the Grid software is being actively used. I’m using a threshold of 20 pixels, so there needs to be an increase or decrease of 20 or more black pixels for a change to be detected. This allows you to move your mouse cursor around in the text area at the top of the Grid window without triggering that there has been a change. The activity detection script needs more testing, but preliminary results seem to show it works. Prior to this, I was monitoring the entire Grid window and looking for a change in the whole window above a threshold. This led to false triggers when cells were selected, but not activated. When a cell is selected, the colour of the cell changes, even when it is not activated to produce text. This change in colour was being detected.

Each time we test the script, we find new ways to break it, leading to some ‘try except’ exception handling clauses. The script is designed to run on Windows as Grid 2 and Grid 3 only work on this operating system. I use the win32gui library to interact with Windows and the python imaging library, PIL (known as pillow), to do the image processing.
 
Sensory Software’s Grid 2 Chatterbox grid being used to construct a sentence:
 
Grid 2 communications software by Sensory Software used to create speech.
The latest code and installation details on how to get this running using the BBC microbit to give a flash when the Grid software is being actively used can be found on my github site at:
 
If you have any questions, please ask.

eyeBlink – enabling natural two way conversation with somebody who uses an eyetracker

It can be difficult to tell when a student who uses an eye tracker to operate their communications software is actively using the software. So the temptation is to go and look over their shoulder. The Head Technologist at Beaumont College asked if it is possible to have a light flash to indicate when the communications software is being used. This makes for a more natural two way conversation. You talk to the student and you see the light flashing, so you know that a reply is being composed.

 
After some false starts, I think I have some code that will detect when the software is being used. My script makes an image of the screen each second and looks for a difference. I set a minimum threshold for the difference between the images so that a blinking cursor will not continuously indicate a change.
 
I use a BBC micro:bit to do the flashing as they are cheap, cheerful and reliable. I reckon to have two of these qualities.
 
Please find a picture of Craig, one of the technologists at Beaumont, testing the prototype of the eyeBlink system using a Tobii eyetracker and Sensory Software’s Grid 2 software. You will have to take my word for it that the microbit does indeed flash when the Grid 2 is being used.
Craig testing the eyeBlink hardware at Beaumont College, Lancaster.

CD player for the visually impaired

This blog details a method to allow somebody who is visually impaired to easily listen to their CD collection again. My Mother lost her eyesight through macular degeneration. She has a decent collection of classical music that she built up over a few years. But she can’t see well enough to easily use a CD player anymore. On top of that, her mobility is restricted. I bought her a Roberts Concerto 2 CD player and radio designed for the visually impaired. Please find details and a review here. This is the best that I could find, but it is still fiddly and difficult for somebody without sight to load the CD. It is quite a bulky device, which makes putting it next to an elderly person awkward as it takes up most of chair side table, or the user has to get up and go to where it is placed. Which is a barrier to it ever being used when just getting out of a chair is no longer straight forwards.  I looked at a few potential solutions. I found a portable CD player on eBay and tried that. But again, it takes up a little too much table top space and it is fiddly to load. You don’t realise how poorly controls are laid out on most devices until you try to explain to somebody without vision how to use it. I found some lovely projects where custom built players are built using tags. Audio books are loaded onto a memory card and played using something like a Raspberry Pi single board computer. An NFC coil is used to read a tag placed inside the case of an audiobook or CD and the audio is played from the memory card. Here is an example on Hackaday. I started going down this route. Then had another think. This will add another device to my mother’s chair side table. I will have to:

  1. Build it.
  2. Run off all her CDs to memory card.
  3. Show her how to use it.
  4. Maintain it.

My mother uses a Sovereign USB stick player to listen to her talking books and newspapers. This is a well designed player aimed at the visually impaired. It has decent sound quality. The build cost of a custom device would exceed the cost of a Sovereign and for me to think I would match the sound quality is a tad arrogant. One of the design features of this player is that it will remember the place you were last at on the stick. You can even swap sticks and it remembers the last play position on the previous five sticks you played. Mum already has this next to her and knows how to use it. As a side note, there is now a smaller version of this player available called the sonic which I bought to listen to my podcasts with and loan to Mum when she visits.

So I ran off her CDs to MP3 and put each one on to a cheap USB stick. I bought some small key rings and used these to connect the memory sticks to postcards on which I printed the title of the CD. So far I have run off 10 of these. These 10 sticks and labels fit in an little box on her table next to the player. If the idea works, I will run off some more.
I’ve written this up so that other people in my position have a potential solution to enable others with disability to enjoy their music collections.
Having to pay a couple of dollars each for a cheap USB stick from eBay for each CD may seem a tad pricey, but compared with the time and cost of building a custom device, I think it is money well invested.
The proof of the pudding is in the eating. Please find a photo of Mum at my house, having dozed off while listening to Aled Jones with the bear she gave me about 40 years ago. The bear’s arms are a little saggy now, but we all get a little infirm and need some help as we age.
 
Mum and Bear listening to some music.