Using Python to detect activity in Sensory Software's Grid 2

Update: March 2018. This work is being submitted to the Communications Matters conference.

Following on from the eyeBlink post, with the help of Fil at Beaumont, I modified the algorithm I’m using to detect when the Grid 2 or Grid 3 software is being used. The image below shows Sensory Software’s Grid 2 software being used to construct a sentence. The new text appears in the white area at the top of the window. Fil suggested that I change the Python script to just monitor this area at the top of the window. The script now looks for a change in the amount of black text in this area. After the usual software wrangling I think I got it working. The Python script looks at the top 20% of the window and counts the number of black pixels in this area. Every half second it recounts the number of black pixels. If there is a change in the number of black pixels above a threshold, then a trigger is sent to indicate that the Grid software is being actively used. I’m using a threshold of 20 pixels, so there needs to be an increase or decrease of 20 or more black pixels for a change to be detected. This allows you to move your mouse cursor around in the text area at the top of the Grid window without triggering that there has been a change. The activity detection script needs more testing, but preliminary results seem to show it works. Prior to this, I was monitoring the entire Grid window and looking for a change in the whole window above a threshold. This led to false triggers when cells were selected, but not activated. When a cell is selected, the colour of the cell changes, even when it is not activated to produce text. This change in colour was being detected.

Each time we test the script, we find new ways to break it, leading to some ’try except’ exception handling clauses. The script is designed to run on Windows as Grid 2 and Grid 3 only work on this operating system. I use the win32gui library to interact with Windows and the python imaging library, PIL (known as pillow), to do the image processing.

Sensory Software’s Grid 2 Chatterbox grid being used to construct a sentence:

https://github.com/mattoppenheim/microbit/tree/master/activity_indicator

If you have any questions, please ask.