Python 3, threading and references

Creating a thread

I used threading to enable real-time graphing of data from sensors. One thread collected data from the sensors. The main thread ran the real time graph. I had a few problems getting started. It came down to my incorrect use of brackets when creating the thread.

When we create a thread using the threading library, we need to pass the target to the thread without using brackets. e.g.

thread = threading.Thread(target=ThreadTest)

not

thread = threading.Thread(target=ThreadTest())

Otherwise the target is created in the main thread, which is what we are trying to avoid. Without the brackets, we pass a reference to the target. With the brackets, we have already created the object. I think that this is analogous to passing a pointer in C, but stand to be corrected.

Example

In test1.py I call ThreadTest without using brackets. test_thread starts in the thread and allows test1.py to continue running.

In test2.py, I pass ThreadTest() as the target. In this case the thread does not allow test2.py to continue running.

test1.py

import threading
from thread_test import ThreadTest

thread = threading.Thread(target=ThreadTest)
thread.start()
print('not blocked')

test2.py

import threading
from thread_test import ThreadTest

thread = threading.Thread(target=ThreadTest())
thread.start()
print('not blocked')

test_thread.py

from time import sleep


class ThreadTest():
    def __init__(self):
        print('thread_test started')
        while True:
            sleep(1)
            print('test_thread')

output from test1.py:

thread_test started
not blocked
test_thread
test_thread
test_thread

output from test2.py:

thread_test started
test_thread
test_thread
test_thread

Installing the Eclipse Embedded Systems Register View plugin

I had some ‘issues’ getting the Eclipse Embedded Systems Register View plugin to install in Eclipse Oxygen under Linux Mint 18. When I tried to install the plugin from the Eclipse Marketplace, I repeatedly got an error:

Artifact not found...

Thanks to a Sourceforge issue post here I got the plugin installed. A solution to the problem is towards the bottom of this post. However, I needed to modify the solution slightly. So I’m putting up my work around here.
We need to download the plugin then tell Eclipse where the download is.

Downloading the plugin

We download the plugin using the ‘wget’ command

I had to slightly modify the script given in the Sourceforge post to get this to work. Please find the code below. Make yourself a directory to install the plugin to. Create an executable script with the code below and after you are happy that this is not a nefarious attempt at my making your system into my zombie, run it.

#!/bin/sh

wget http://embsysregview.sourceforge.net/update/site.xml
mkdir -p features plugins
cd ./features

wget http://downloads.sourceforge.net/embsysregview/org.eclipse.cdt.embsysregview_feature_0.2.6.jar
wget http://downloads.sourceforge.net/embsysregview/org.eclipse.cdt.embsysregview.data_feature_0.2.6.r191.jar
cd ../plugins

wget http://downloads.sourceforge.net/embsysregview/org.eclipse.cdt.embsysregview_0.2.6.jar
wget http://downloads.sourceforge.net/embsysregview/org.eclipse.cdt.embsysregview.data_0.2.6.r191.jar

Your directory should now have two subdirectories called ‘plugins’ and ‘features’ each containing two .jar files. There will be a file called site.xml in the parent directory that you created.

Installing the plugin

In Eclipse, go to the ‘Help’ menu. Click on ‘Install New Software’. Next to the ‘Work with’ window click on the button ‘Add’. Next to the window called ‘Name’ click on the button called ‘Local…’ and navigate to the directory where you downloaded the plugin. I called this directory ’embsysregviewer. Please find a screenshot of the menus below.

Now you need to give the repository a name in the ‘Add Repository’ window. I called it ’embsysviewer local’. The name does not matter, so long as it makes sense to you.

Now you should see the Available Software screen fill with goodness:

Select the emsysregview plugin and click on Next. to complete the installation, you need to accept the licence and that you are installing ‘unsigned content’. Eclipse will ask to be restarted. You should now have the Embedded Systems Register View plugin installed.

Getting this to work sunk an hour or so, but it gave me an overview of how to download and install a plugin should I encounter a similar issue in the future.

Eclipse, yotta, C/C++ and the BBC Micro:bit

With the help of the excellent instructions at the link below I set up Eclipse with yotta to compile C code for the BBC Micro:bit under Linux:
http://flames-of-code.netlify.com/blog/microbit-cpp-3/
I get the debugger window to come up, but have not yet used this feature in anger.
The writer, achary, clearly knows more about Eclipse and embedded programming than I do. I got a little stuck at a couple of stages so created this page to pass on my solutions.

The offline C compiler for the BBC Micro:bit is developed at Lancaster University. Installing the yotta compiler and downloading example files is explained here.

The rest of this article assumes you followed the instructions on this installation guide and have cloned the microbit-samples directory. I assume that you have Eclipse installed, either the C/C++ installation or you have installed the C/C++ development environment.

Installing yotta

Instructions for installing yotta can be found in the yotta documentation here. Note that yotta is designed for Python2.7 only. I used pip to install yotta to my user directory, using the –user flag. Then I started getting errors:

‘module’ object has no attribute ‘X509_up_ref’

I faffed around upgrading my cryptography library as mentioned in the yotta documentation. Long story short, the recently installed ‘yotta’ and ‘yt’ commands in ~/.local/bin/yt and ~/.local/bin/yotta both referred to python 3. I probably did something sometime to cause this. To fix the error I changed ~/.local/bin/yt and ~/.local/bin/yotta from:

#!/usr/bin/python3.6

import yotta
yotta.main()

To:

#!/usr/bin/python2.7

import yotta
yotta.main()

Get yotta to work in debug mode

Using yotta with the –debug-build flag allows for easier debugging. By default, the code is compiled in an optimised mode which makes debugging harder. I need all of the help that I can get, so would like to use this flag. The command to run yotta with the –debug-build flag  is:

yotta build --debug-build

However, this will throw an error and the last line of the build will give the erro:

ninja: build stopped: subcommand failed.

I am not sure what a ninja is doing in my system. If I could see him, I would probably already be dead.

To fix this error, the ‘-fomit-frame’ flag needs adding at two places in the file yotta_targets/mbed-gcc/CMake/Platform/mbedOS-GNU-C.cmake in your microbit-samples directory.

The two changes to mbedOS-GNU-C.cmake are:

line 21 from:
set(CMAKE_C_FLAGS_DEBUG_INIT “-g -gdwarf-3”)
to:
set(CMAKE_C_FLAGS_DEBUG_INIT “-g -gdwarf-3 -fomit-frame-pointer”)
line 28 from:
set(CMAKE_ASM_FLAGS_DEBUG_INIT “-g -gdwarf-3”)
to:
set(CMAKE_ASM_FLAGS_DEBUG_INIT “-g -gdwarf-3 -fomit-frame-pointer”)

Then remove the build directory in the microbit-samples directory and rebuild using:

yotta build --debug-build

If you don’t remove the old build directory directory, then the command will still fail. I know this.

Install pyOCD

We use the pyOCD tool to help debug and program the microbit. Details of this tool are on its github page.

‘pyOCD is an Open Source python 2.7 based library for programming and debugging ARM Cortex-M microcontrollers using CMSIS-DAP. Linux, OSX and Windows are supported.’

Note the ‘python 2.7’ bit. Initially I pip installed it, which defaulted to a python 3 install. The install worked, but when I came to try running pyocd, I got a  bunch of assertion errors. So I read the instructions…

To ensure that pyocd is installed using python2.7, I used:

pip2 install --pre -U --user pyocd

–pre # ‘Include pre-release and development versions. By default, pip only finds stable versions.’ from https://pip.pypa.io/en/stable/reference/pip_install/#install-pre
-U # same as –upgrade ‘Upgrade all specified packages to the newest available version.’
–user # libraries go to the user directory. In linux, this removes the need for sudo to install, which is a security issue.

Plug in your microbit.

Now we fire up a gdbserver using the newly installed pyocd. What is a gdbserver? This wikipedia page explains that ‘gdbserver is a computer program that makes it possible to remotely debug other programs.’

sudo /home/bill/.local/bin/pyocd-gdbserver --persist -t nrf51 -bh -r

-t # target (nrf51 is the chipset used on the microbit).
-bh # replace software breakpoints with hardware breakpoints.
-r # halt the target when reset.

On my machine, the command is:

sudo ~/.local/bin/pyocd-gdbserver --persist -t nrf51 -bh -r

I get this output:

INFO:root:DAP SWD MODE initialised
INFO:root:ROM table #0 @ 0xf0000000 cidr=b105100d pidr=2007c4001
INFO:root:[0]<e00ff000: cidr=b105100d, pidr=4000bb471, class=1>
INFO:root:ROM table #1 @ 0xe00ff000 cidr=b105100d pidr=4000bb471
INFO:root:[0]<e000e000:SCS-M0+ cidr=b105e00d, pidr=4000bb008, class=14>
INFO:root:[1]<e0001000:DWT-M0+ cidr=b105e00d, pidr=4000bb00a, class=14>
INFO:root:[2]<e0002000:BPU cidr=b105e00d, pidr=4000bb00b, class=14>
INFO:root:[1]<f0002000: cidr=b105900d, pidr=4000bb9a3, class=9, devtype=13, devid=0>
INFO:root:CPU core is Cortex-M0
INFO:root:4 hardware breakpoints, 0 literal comparators
INFO:root:2 hardware watchpoints
INFO:root:Telnet: server started on port 4444
INFO:root:GDB server started at port:3333

As awac explains, this shows that we have access to 4 hardware breakpoints. The server is started at port 3333. This port will be entered into the Eclipse debugger setup, which is explained below.

Set up Eclipse

http://flames-of-code.netlify.com/blog/microbit-cpp-3/ covers setting up a C/C++ Eclipse project with the microbit-samples code downloaded from the Lancaster University github. I am using Eclipse Oxygen at the time of writing this post.

A new project is set up by using File, New, Makefile Project with Existing Code.

Alter the default build command from ‘make’ to ‘yotta build –debug-build’. Get to this by right clicking on the microbit-samples project and selecting ‘Properties’.

Use control-B to build the code. I get a bunch of depreciation warnings that can be ignored.

dynamic exception specifications are deprecated in C++11 [-Wdeprecated]

Configuring the Eclipse C/C++ debugger for use with yotta

The pyOCD site mentions that the plugin ‘Eclipse Embedded Systems Register View’ should be installed. This took me a little while to figure out how to install, so I created a separate post on how to do this here. This plugin is not yet of use for the microbit. I hope to get CMSIS-SVD configuration files from Nordic for the microbit’s microcontroller to be able to make use of the plugin.

I came a little unstuck when setting up the debug session as I could not find the ‘GDB Hardware Debugging’ option when editing my debug configuration. To get that option we need to install the GNU MCU Eclipse plugin from the Eclipse Marketplace. Go to the Help menu and click on ‘Eclipse Marketplace’. Put ‘gnu mcu’ as the search term to find the plugin. This plugin adds the ‘GDB Hardware Debugging’ option to your run configurations which we use when setting up Eclipse.

This allows me set up for the GDB debugger as shown in the screen shots below. Go to Run, Debug configurations.

Double clicking on the heading ‘GDB Hardware Debugging’ and set up the Main, Debugger and Startup tabs as shown below. The C/C++ application is found in the ‘build’ subdirectory, in the microbit-samples directory at:

microbit-samples/build/bbc-microbit-classic-gcc/source/microbit-samples

The Debugger tab specifies the path to the GDB :

/usr/bin/arm-none-eabi-gdb

Enter port ‘3333’, which we noted earlier when starting the gdb server.

In the Startup tab, click on ‘Load image’ and ‘Use File’. Enter

${workspace_loc:/microbit-samples/build/bbc-microbit-classic-gcc/source/microbit-samples-combined.hex}

Initially I had some errors:

Reset command not defined for device 'Generic TCP/IP'

Looking at this Stackoverflow question, I fixed this by also unchecking Reset & Delay and Halt options in the debugger configuration:

Running the Eclipse debugger

Run the command ‘yt clean’ from the command line in your microbit-samples directory to clean out the last build. Then ‘control-b’ in Eclipse to create a fresh build. I feel I should be able to click on the little bug icon on the menu bar to get to the debugger, but this gives me a ‘launch failed. Binary not found.’ window. So I right click on the project and select ‘Debug As’,’Debug Configurations’, ‘microbit-samples Default’ then click on the ‘Debug’ button on the bottom right. I get offered the chance to go to the debug screen, which is a result.

Using Python to detect activity in Sensory Software’s Grid 2

Update: March 2018. This work is being submitted to the Communications Matters conference.

Following on from the eyeBlink post, with the help of Fil at Beaumont, I modified the algorithm I’m using to detect when the Grid 2 or Grid 3 software is being used. The image below shows Sensory Software’s Grid 2 software being used to construct a sentence. The new text appears in the white area at the top of the window. Fil suggested that I change the Python script to just monitor this area at the top of the window. The script now looks for a change in the amount of black text in this area. After the usual software wrangling I think I got it working. The Python script looks at the top 20% of the window and counts the number of black pixels in this area. Every half second it recounts the number of black pixels. If there is a change in the number of black pixels above a threshold, then a trigger is sent to indicate that the Grid software is being actively used. I’m using a threshold of 20 pixels, so there needs to be an increase or decrease of 20 or more black pixels for a change to be detected. This allows you to move your mouse cursor around in the text area at the top of the Grid window without triggering that there has been a change. The activity detection script needs more testing, but preliminary results seem to show it works. Prior to this, I was monitoring the entire Grid window and looking for a change in the whole window above a threshold. This led to false triggers when cells were selected, but not activated. When a cell is selected, the colour of the cell changes, even when it is not activated to produce text. This change in colour was being detected.

Each time we test the script, we find new ways to break it, leading to some ‘try except’ exception handling clauses. The script is designed to run on Windows as Grid 2 and Grid 3 only work on this operating system. I use the win32gui library to interact with Windows and the python imaging library, PIL (known as pillow), to do the image processing.
 
Sensory Software’s Grid 2 Chatterbox grid being used to construct a sentence:
 
Grid 2 communications software by Sensory Software used to create speech.
The latest code and installation details on how to get this running using the BBC microbit to give a flash when the Grid software is being actively used can be found on my github site at:
 
If you have any questions, please ask.

Running pytest when the test files are in a different directory to the source files

I had a battle to get my testing directory structure to work outside of an IDE. Please find my solution below. Tested on Windows 7 using python 3.6 and Linux Mint using python 3.4, running the code using the command line:

python -m pytest test_compress_files.py

The file I wrote to be tested is called compress_files.py in a directory named \src. The file containing tests to be run using pytest is called test_compress_files.py in a subdirectory \tests, so the full directory path is \src\tests. I needed to add a file called context.py to the \src\tests directory. This file is used in test_compress_files.py to enable access to compress_files.py in the directory above. The __init__.py files are empty.

Directory structure:

\src
__init__.py
compress_files.py

\src\tests
__init__.py
context.py
test_compress_files.py  

compress_files.py contains the script to be tested.

context.py:

import os
import sys
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), '..')))

import compress_files  

The line:

sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__)

comes from the suggestion at the hitch hikers guide to python at:

http://docs.python-guide.org/en/latest/writing/structure/.

This adds the path of the directory above the /src/tests directory to sys.path, which in this case is /src.

test_compress_files.py:

import os
import pytest
from .context import compress_files
from compress_files import *

# tests start here
...

I put this up as an answer to a stackoverflow question here.

Sending parameters to a Jupyter Notebook cell using click

Using libraries such as click and optparse we can send parameters to Python scripts when we run them from the command line. For example, passing a parameter called count with a value of 2 to a script called hello.py:

hello.py --count=2

How can I replicate this functionality in a cell of a Jupyter notebook? I like to run the same code in the notebook so that I can easily copy it to a stand alone script. Using sys.argv to pass parameters to the main function seemed one way to go and works with optparse:

from optparse import OptionParser
import sys


def main():
    parser = OptionParser()
    parser.add_option('-f', '--fake',
                      default='False',
                help='Fake data')
    (options,args) = parser.parse_args()
    print('options:{} args: {}'.format(options, args))
    if options.fake:
        print('Fake detected')
    
def test_args():
    
    print('hello')
    
if __name__ == '__main__':

    sys.argv = ['--fake', 'True' '--help']
    main()

output:

options:{'fake': 'False'} args: ['True--help']

Fake detected

Click seems to be flavor of the month, but I kept on getting a screen full of errors when I tried to run click through a Jupyter notebook cell. If we consider the Click example code:

import click

@click.command()
@click.option('--count', default=1, help='Number of greetings.')
@click.option('--name', prompt='Your name',
            help='The person to greet.')
def hello(count, name):
    """Simple program that greets NAME for a total of COUNT times."""
    for x in range(count):
        click.echo('Hello %s!' % name)

if __name__ == '__main__':
    hello()

If this file is called hello.py, then running from a command line:

hello.py 'Max' --count=3 

Gives this output:

Hello Max!

Hello Max!

Hello Max!

But using the same sys.argv trick that works with optparse produces a screen full of errors when the same code is run from a Jupyter notebook cell. The solution is to put the %%python magic at the start of the cell:

%%python

import sys
import click

@click.command()
@click.option('--count', default=1, help='Number of greetings.')
@click.option('--name', prompt='Your name',
            help='The person to greet.')
def hello(count, name):
    """Simple program that greets NAME for a total of COUNT times."""
    with open('echo.txt', 'w') as fobj:
        for x in range(count):
            click.echo('Hello %s!' % name)

if __name__ == '__main__':
    # first element is the script name, use empty string instead
    sys.argv = ['', '--name', 'Max', '--count', '3']
    hello()

A small tip, but one which cost me an hour or two of pondering. Finally I asked the hive mind of Stackoverflow. Please see this stackoverflow solution.

Bash script to mount and unmount a BBC Micro:bit

I develop using the BBC Micro:bit (which I’ll call a microbit from now on) using Linux. To get a new hex file on to the microbit, the microbit has to be mounted on to the file system. Which may seem obvious. But the microbit has a habit of unmounting itself after being programmed. Or not mounting when it is plugged in. So I wrote a script to make things easier. Because I can. To be more accurate, I copied a script from Stackoverflow and made a few minor modifications. Don’t pretend this isn’t how you have written a lot of your scripts.
Find the code at the bottom of this post. Copy and paste it to a file called microbit_manage.sh. Make the file executable (chmod +x ./microbit_manage.sh) and run using either ‘mount’ or ‘unmount’ as an argument. I’ve got these aliased to ‘mm’ and ‘md’ in my .bashrc.

Example usage:

bill@bill ~ $ microbit_manage.sh mount

found one MICROBIT, device: /dev/sdb
MICROBIT was unmounted
Mounted /dev/sdb at /media/bill/MICROBIT.

bill@bill ~ $ microbit_manage.sh unmount

found one MICROBIT, device: /dev/sdb
MICROBIT was mounted
Unmounted /dev/sdb.

Comments and improvements welcome. The script is:

#!/bin/bash
# microbit_manage.sh
# mount and unmount microbit
# modified from https://askubuntu.com/questions>
# v1.0 matt oppenheim October 2017

if [ $# -eq 0 ] 
then
    echo "no argument supplied, use 'mount' or >
    exit 1
fi

# how many MICRO found in udiksctl dump
RESULTS=$(udisksctl dump | grep IdLabel | grep >

0 )     echo "no $MICRO found in 'udkisksctl du>
        exit 0
        ;;

        # if [[ -z $(mount | grep "$DEVICE") &&>
        if [[ -z $(mount | grep "$DEVICE") ]]
        then
            echo "$DEVICELABEL was unmounted"
            if [ $1 == "mount" ]
            then
                udisksctl mount -b "$DEVICE"
                exit 0
            fi
        else
                echo "$DEVICELABEL was mounted"
                if [ $1 == "unmount" ]
                then
                    udisksctl unmount -b "$DEVI>
                    exit 0
                fi
        fi
        ;;
    esac

Getting the BBC Micro:bit radio to work with the mbed online C/C++ compiler

This blog explains how to get the example programs for working with the non-Bluetooth radio on the BBC Micro:bit to compile correctly using the Mbed online C/C++ compiler.

Short story

Two options:

1 Place the line:

#define MICROBIT_BLE_ENABLED 0

in the MicroBit.h library and forget about the config.json file.

Or

2 Create an mbed_app.json file instead of the config.json file with this content:

{
"macros": [ "MICROBIT_BLE_ENABLED=0" ]
}

Long story

 
The Mbed online compiler and the yotta offline compiler for the BBC Micro:bit are explained at the Lancaster University github site here:

https://lancaster-university.github.io/microbit-docs/

I couldn’t get the example radio programs supplied with the online Mbed C/C++ compiler to work with the BBC Micro:bit. These programs did work with the yotta offline compiler. It took a while to figure out that the config.json file supplied with the examples is being ignored by the Mbed online compiler. The BBC Micro:bit has a custom radio setup which does not work when Bluetooth is enabled. The compiler needs to be told that Bluetooth is disabled. In the examples supplied for both the yotta offline compiler and for the mbed online compiler this is done using a config.json file containing:

{ 
    microbit-dal:{
        bluetooth:{
            enabled: 0 
        } 
     } 
}

The example programs are called simple-radio-rx and simple-radio-tx. For the Mbed online compiler, these can be found at:

https://os.mbed.com/teams/microbit/code/microbit-simple-radio-rx/

https://os.mbed.com/teams/microbit/code/microbit-simple-radio-tx/

For the offline yotta compiler the same programs and config.json files can be found at:

https://github.com/lancaster-university/microbit-samples/tree/master/source/examples

The hex files created using the Mbed online compiler resolutely refused to do anything when I loaded them onto the microbits. I figured out that the the config.json file was being ignored by the Mbed online compiler. To disable the Bluetooth through code, place this line:

#define MICROBIT_BLE_ENABLED 0

in the MicroBit.h library. After doing this, the hex files produced by compiling the example programs using the Mbed online compiler ran correctly.

I posted this on the Mbed questions site and an Mbed moderator said that the issue will be fixed:

https://os.mbed.com/questions/79592/BBC-Microbit-how-to-use-the-radio/?compage=1#c29069

A while later, a helpful guy on Stackoverflow advised me to use an mbed_app.json file instead of the config.json file with this content:

{
    "macros": [ "MICROBIT_BLE_ENABLED=0" ]
}

Setting up yotta and C with the BBC Micro:bit by modifying the examples directory

I set up to program the BBC Micro:bit (which I’ll call the microbit from now on) in C/C++ under Linux. I’ve been using micropython to program the boards up to now, but want to leverage the increased performance that using C can give and some of the C libraries that are available for e.g. encryption.

There is good documentation on the Lancaster University microbit github page:

https://lancaster-university.github.io/microbit-docs/

I installed the offline tools as I spend a lot of time working at sea where you can’t always rely on having an internet connection. I got a demo compiled and loaded by following the instructionso n the Lancaster University github site. Now it was time to write my own code. I had a little trouble getting this going, so here’s what I had to do to get my first program compiled and loaded. I did this by modifying the structure of the examples directory that was created by following the github instructions.

I downloaded the github repository linked above into

~/git/microbit-samples

Initially I made a new directory under:

~/git/microbit-samples/source/my_code

Whenever I ran ‘yt build’, I got a weird error referring to one of the bluetooth example programs. Plus yt build took longer than I thought it ought to. So…..

Move

~/git/microbit-samples/source/examples

out of:

~/git/microbit-samples/source/

Put these folders one level above this folder. Only put the main.cpp file into

~/git/microbit-samples/source/

Then run:

yt clean

and

yt build

I copied the file:

~/git/microbit-samples/build//bbc-microbit-classic-gcc/source/microbit-c-combined.hex

to the microbit. After waiting for the flashing LEDs to stop, it worked!

The microbit unmounts itself each time that you load a new hex file. So using the bash script that I detailed in an earlier blog to quickly mount the microbit is a time saver.

This is a quick way of getting started by building on top of the examples directory that most folk will start with.

disclaimer: I loiter in the same department as the authors of the C/C++ BBC Micro:bit repository.

Enabling the analog to digital converter (ADC) on the BBC Micro:bit using C/C++

To get the example ADC code to work on the Lancaster Github site, change the line:

MicroBitPin P0(MICROBIT_ID_IO_P0, MICROBIT_PIN_P0, PIN_CAPABILITY_BOTH);

To:

MicroBitPin P0(MICROBIT_ID_IO_P0, MICROBIT_PIN_P0, PIN_CAPABILITY_ANALOG);

I tested this using both the online Mbed compiler and the yotta compiler.