micro:bit c toolchain explained, part 2. yotta, file locations

December 2020: I wrote this post when only v1 of the micro:bit existed. I haven't yet checked to see if the contents are still valid for the toolchain used to program v2 of the micro:bit.

This article follows on from part 1 of my attempt to explain the chain of tools used to build your C code into something that can run on the BBC micro:bit. In this part, I will cover the yotta build tool and how the make build tool is used as the top level component.

Here's the final tool chain. Admire the beauty.

Offline tool chain for building an executable from C code for the micro:bit

So what is yotta and why do companies choose names like 'yotta' and 'ninja' for their products when this will make it hard for us to find them on the web?

yotta

Yotta is another build tool. Yotta takes the build file module.json and the configuration files target.json and config.json. Yotta produces the build file for cmake called CMakeLists.txt. If you read part 1 of this explanation of the micro:bit C tool chain, you will be getting used to the idea that one build tool creates the build file for the build tool underneath it. It took me a while to get my head around this. The reason for using these layers of build tools is that there are many other files than the ones that we write to pull into the project to enable our C code to be built to run on the micro:bit hardware.

In your yotta examples directory, which I represent with <yotta_examples> below, have a look in the file <yotta_examples>/yotta_targets/bbc-microbit-classic-gcc/target.json. Almost at the end of the target.json file, you will see where yotta tool is configured to produce a build file for CMake:

"toolchain": "CMake/toolchain.cmake",
  "scripts": {
    "debug": [
      "valinor",
      "--target",
      "NRF51822",
      "$program"

Here ends the description of build tools used to create the hex file that goes onto your micro:bit from the source C code.

make

This section is a bit of an extension/digression. The 'make' tool is not used in the programming stack for micro:bit. I use it to automate building and loading a hex file to a micro:bit.

Make is the Grandad of build tools. I often use this tool for building C projects outside of micro:bit programming. ninja may be faster for large C projects, but make is fast enough for my projects and has proven reliability.

The build file for make tool is called makefile, or sometimes Makefile. Linux does not distinguish between the two names, but Windows will. Typing 'make' at the command line will cause the contents of makefile to be processed by the make tool. You will be used to the idea of a build file being processed by a build tool by now.

I use make to automate building the binary and loading it onto my micro:bit. Here's a listing of a typical makefile for one of my projects:

build_microbit:
	yt build
	# script to mount a microbit
	~/data/infolab2/progs/scripts/microbit_mount.sh mount
	cp ./build/bbc-microbit-classic-gcc/source/microbit-c-combined.hex /media/bill/MICROBIT/

yt build obviously builds the project. The line after the comment checks to see if the micro:bit is mounted and if not, mount it. The final line copies the newly generated hex file to the mount point for the micro:bit, which is specified in the script microbit_mount.sh.

I use a script to mount the micro:bit, as this is unmounted after flashing the new code. I use Linux; this may not be necessary with Windows. Details on the script I use can be found on my blog post. The micro:bit is mounted at /media/bill/MICROBIT. You need to change the mount point in the makefile to the location that you choose to mount your micro:bit.

Automating loading micropython code to the BBC micro:bit in Linux

Introduction

When I edit micropython for a microbit project, I want the code to automagically load onto the microbit each time that I save the code. Then I want to see any messages from the microbit displayed. I worked out a toolchain that does this under Linux.

I summarise the toolchain in the next section, then go through each step in more detail.

For the examples in this post, I am editing a file called 'feeder.py'.

Toolchain

Use your favourite editor for creating your code with. The toolchain to load the code to the microbit kicks off when you save the file.

There are four tools needed to automate the process.

  • A script to locate and mount the microbit. I wrote a blog post about this script here.
  • The uflash utility to write the micropython file to the microbit.
  • screen terminal to display messages from the microbit
  • inotifywait command to monitor the micropython file being edited

uflash

Use pip to install this. Details are in the link given earlier in this post. To load a_file.py to an attached microbit, simply type:

uflash a_file.py

Using virtual environments makes my life easier, so I use one for my microbit programming. There are plenty of blog posts detailing how to set up and use these.

script to attach a microbit

I wrote a blog post on how to create a script that automatically locates and mounts a microbit here. This script is aliased to mm and md in my .bashrc file to mount a microbit and dismount a microbit.

screen

screen is built in to most Linux distros, so I use it here. This connects to the port that is in use by the microbit and displays any output from the microbit.

You need to know what port your microbit is attached to though. On my laptop this is often /dev/ttyACM3, which I use in the example below.

I present a script in the next section to automatically find the port that the microbit is connected to and fire up a screen connection using this.

To fire up the connection manually, if you know the port (in this example /dev/ttyACM3) use:

screen /dev/ttyACM3 115200

Remember to kill screen without leaving it running in the background by using:

control-a k

One potential problem with screen, is that it is easy to 'detach' from screen instead of ending the process, which leaves an instance of screen running invisibly which is attached to the same port as the microbit. When you come to try and run screen or connect to the microbit again, you will see an error like the one below:

can't open device "/dev/ttyACM3" 

To find and kill the errant screen instance:

lsof | grep /dev/ttyACM3

If you get something like:

screen  8610  elm    5u   CHR  166,3      0t0 5599015 /dev/ttyACM3

You still have a screen instance attached to the port. The PID is the second value in the list, in this case 8610. Kill it using e.g.

kill 8610

One way to prevent this potential problem is to replace screen with another terminal display tool such as gtkterm.

script to automate finding and connecting to the microbit

The python3 script below automates finding a connected microbit. The script starts a screen session on the port that the microbit is connected to.

'''
Find pyboard or microbit and open a terminal.
Created on 4 Nov 2015.
From http://wdi.supelec.fr/boulanger/MicroPython/
Usage: find_device [device]
Calls screen on "device". If device is not given, looks for /dev/ttyACM* devices
then /dev/tty.usbmodem* devices (for MacOS).
'''
#!/usr/bin/python
import sys
import os
import fnmatch

BAUD = 115200

def main():
    if len(sys.argv) > 1:
        ttymodems = [sys.argv[1]]
    else:
        ttymodems = fnmatch.filter(os.listdir('/dev'), 'ttyACM*')
        if len(ttymodems) == 0:
            ttymodems = fnmatch.filter(os.listdir('/dev'), 'tty.usbmodem*')
        if len(ttymodems) == 0:
            print('Error: no pyboard found.')
            sys.exit(1)
        ttymodems[0] = '/dev/' + ttymodems[0]
        print('connection at: {}'.format(ttymodems[0]))
    os.system('screen '+ttymodems[0] + ' {}'.format(BAUD))

if __name__ == "__main__":
    main()

inotifywait

inotifywait is a command line tool that can be configured to watch a file for a change, then trigger an action. Here we use it to watch for a change in the micropython file being edited. When this file is saved, inotifywait mounts the microbit and flashes the file to it.

If infotifywait is not already installed, then you need to install inotify-tools.

The command to watch a file called feeder.py for a change, then run the alias mm (to mount the microbit) and to flash the file to the microbit is:

while inotifywait -e modify feeder.py ; do mm && uflash feeder.py ; done

I would have formatted the above line as the other code in this post but found that the syntax highlighter plug-in refuses to show the ampersand symbol correctly.

notes

Some editors have the 'feature' of 'kind of' saving the file you are editing while you are editing it. This creates a new timestamp for the file and activates inotifywait without your saving the file. mu is one of these editors. In this case, I do not use inotify and run the following command from the command line each time I want to flash the updated file. In this example the updated file is called feeder.py:

mm && uflash feeder.py

but why not use a Makefile

Makefiles are typically used to build programming projects, detailing the dependencies of each component. The file is run when 'make' is typed on the command line.

The one-liner command using inotifywait is fine for this simple example where we are editing a single file. Which is most of what I do with the microbit and micropython. Makefile comes into its own for more complex builds involving multiple files. It allows for extra run-time options to be added, e.g. to clean files created by earlier builds.

For completeness, I present a simple Makefile below for this project. Makefile is a useful tool that is worth learning to use for when we have larger projects to build.

Makefile

I created a Makefile containing commands to mount the microbit to the Linux filesystem, then flash a micropython file to the attached microbit.

An example Makefile is below. Note: Use tabs to create the indents after the line 'run:'. Or the World Ends. The supporting file microbit_mount.sh is in the same directory as feeder.py to keep it simple.

MICROPYTHON_FILE=feeder.py

run:
	./microbit_mount.sh mount
	uflash $(MICROPYTHON_FILE)

In this example, microbit_mount.sh is the script to attach a microbit, detailed above.

uflash is the tool used to write the .py file to the microbit. This is detailed above.

To automate the Makefile being run each time that we save new code, we can use the inotifywait command below:

while inotifywait -e close_write feeder.py ; do make; done

micro:bit C toolchain explained, part 1. ninja & cmake.

December 2020: I wrote this post when only v1 of the micro:bit existed. I haven't yet checked to see if the contents are still valid for the toolchain used to program v2 of the micro:bit.

I use the offline build tools from Lancaster University for compiling and building C-code for the BBC Micro:bit board. I used the toolset without really understanding what was going on under the hood. I spent some happy time fixing this. Understanding our tools gives us a better chance of fixing things when they fail. I hope to pass on what I learned here.

So you have written some C code to go onto your micro:bit. Great. Now how does your beautiful prose get converted into the executable (often called a binary) file that can be copied onto the micro:bit?

The components of the toolchain that are unleashed upon your unsuspecting C code are shown below. Let's start at the bottom.

Offline tool chain for building an executable from C code for the micro:bit

The 'file to go to micro:bit' is the end product of the chain of build tools and the Gnu C compiler (called gcc) shown above. This is the file that you copy to your micro:bit and is automatically executed when you power up the micro:bit. Let's talk about gcc.

gcc

gcc stands for the Gnu C Compiler. If we write a simple 'hello world' C program contained in a single file to run on your workstation, telling your C compiler (in our case gcc) how to build the executable can be done in a single command. For example, the command gcc -o hello hello_world.c creates an executable file called 'hello' from a C program in a single file called hello_world.c, using gcc to do the compiling and linking. It gets more complicated when there are multiple files in the project. As we are using the micro:bit, there will be multiple files as many C files need to be brought into your project to enable the code to be built to run on the micro:bit hardware.

gcc is used to do the compiling and linking of your C code and the supporting libraries that are needed to enable the code to run on the micro:bit. gcc comes as standard with most distributions of Linux. But something has to tell gcc what files how to assemble the general mish-mash of interdependent files and libraries. This is what ninja does. ninja is a 'build tool'. Let's discuss what build tools are.

build tools

To understand the rest of the blocks in the toolchain, we need to have a little chat about build tools. A build tool looks for instructions in a text file, which is often called a build file on how to build 'something'. That something could be how to use a compiler to compile and link files written in C, or for indeed another language. That something could also be how to build the instruction text file for a different build tool. This , which will then go on to do 'something else'.

The build file contains a list of rules, such as the flags and controls to run the compiler with. It also contains a list of build statements which say how to build each of the files necessary to produce the output.

There are many different build tools. The micro:bit toolchain uses several. ninja, cmake, yotta and make are all build tools. Each of these tools looks for a text file specific to that tool. Here's a little table for each build tool command and the corresponding build file that it works with.

build toolbuild file
yottayotta_config.json
cmakeCMakeLists.txt
ninjabuild.ninja

A build tool can often be used for building more than one type of language or thing. Let's make build tools clearer by looking at how we use the ninja build tool to build a simple C project.

ninja

In the micro:bit toolchain, the ninja build tool uses a build file called build.ninja to instruct gcc on how to compile and link all of the project's C files. Entering 'ninja' at the command line causes the contents of this file to be processed by the ninja build system. The build.ninja file contains details of the interdependencies of the files that are used to create the binary that we want to load onto the micro:bit.

Let's look at how ninja could be used with a simple three file C project outside of the micro:bit system. Consider a simple example project, split across three files. The three files are called array.c, display.c and display.h. This example runs on your workstation, not on the micro:bit. I'm trying to keep it simple to show what ninja does and what a simple build.ninja file looks like. We will use this example to help explain some of the other build tools as well.

array.c

#include <stdio.h>
#include "display.h"

int main() {
    int v[5] = {1,2,3,4,5};
    display_arr(v, 5);
    return 0;
}

display.h

#ifndef DISPLAY_H
#define DISPLAY_H

void display_arr(int *arr, int len);

#endif

display.c

#include <stdio.h>
#include "display.h"

void display_arr(int *arr, int len) {
    for (int count=0; count<len; count++) {
        printf("arr[%i] = %i\n", count, arr[count]);
    }
}

We can see that the files depend on each other. To build these three files into a single executable we need to tell gcc how each file depends on other files. An example build.ninja file that will compile and link the three example files to create an executable called 'array' is shown below.

build.ninja

cc  = gcc
cflags = -Wall -g
rule compile
    command = $cc $cflags -c $in -o $out

rule link
    command = $cc $in -o $out

build array.o: compile array.c
build display.o: compile display.c
build array: link array.o display.o

default array

Putting all of these files into the same directory and typing ninja will create the following output:

[3/3] gcc array.o display.o -o array

You will find a new file called 'array' in the same folder. This is an executable file. Typing array at the command line gives this unexciting output:

arr[0] = 1
arr[1] = 2
arr[2] = 3
arr[3] = 4
arr[4] = 5

ninja is a relatively recent build tool. Traditionally the make build tool is used to build C programs. I will cover this tool in part 2 and show how this tool can be used to automate loading your code onto the micro:bit. However, with the micro:bit, ninja is used here. I've read that ninja offers about a 10x increase in speed of building the executable for large projects compared with make.

In our example above, I wrote the build.ninja file by hand. But how do we create this build.ninja file for our project? As I mentioned, building an executable for the micro:bit requires a bunch of C files and libraries to be compiled and linked with the source code that we write.

At the top of the ninja website, it states that ninja 'is designed to have its input files generated by a higher-level build system'. In our case, this is done using the CMake build tool. So CMake produces the build.ninja file. Let's have a look at CMake.

CMake

CMake is used to create the build.ninja file. That's right - we are using one build tool to create the build file for another build tool to do something. There is a lot of this going on in the micro:bit build system. CMake can generate the build instruction files for a variety of build tools. Type cmake --help and look at the list of generators at the bottom. CMake uses the Ninja generator to generate the build.ninja file.

Let's use CMake to create a build.ninja file for our previous example C code. In the directory with the C files, create a file called CMakeLists.txt with the content below.

CMakeLists.txt

project(array) 
add_executable(array array.c display.c)

Two lines only! Create a directory called build_ninja and cd into this. Then type CMake .. -G "Ninja". The ".." part will look for the CMakeLists.txt file in the directory above where you are now, then use the contents to create the build.ninja file. You will see a lot of new files are created including a build.ninja file.

The build.ninja file I wrote by hand earlier contains 10 non-blank lines. The new build.ninja file created by CMake contains 95 non-blank lines! Granted, a lot of them are comments, but still.... One of the first lines in the new build.ninja file is include rules.ninja. This file contains an extra 44 non-blank lines. So we have moved from 10 lines to a total of 95 + 44 = 139 lines. I included rules.ninja as a build file in the diagram at the top of this page as this file becomes part of the build.ninja file.

The next obvious question is 'how is the CMakeLists.txt file created for the micro:bit build system'. This is done by yotta.

End dribble

Right, this is enough for one post.

In part 2 we will discuss yotta and the directory structure of the micro:bit C projects.

Additional reading

One good website that helped me figure out the programming toolchain, about 2/3 of the way down the article is https://hackernoon.com/the-first-video-game-on-the-bbc-micro-bit-probably-4175fab44da8.