Converting kicad-nightly build to kicad stable

Summary

Kicad files are text. Using regular expressions we can convert a kicad-nightly file back to a kicad stable file. Not all features were back-converted and some manual editing needed to be done to the back-converted footprints.

Project code is at:
https://github.com/mattoppenheim/convert_kicad_nightly_to_stable

Introduction

I put up some KiCad reference designs for the Jacdac project here.

KiCad comes in two flavours - the stable build, which anyone sane uses and the bleeding edge with bugs and experimental new features version, called the nightly build. I used some footprints that were designed using the KiCad-nightly build. So I put up the reference designs using this flakey version of the software.

Quite rightly, a couple of people objected to this. One made an excellent comment on how to back-convert one of the footprints to the KiCad-stable, using a copy-replace command. All KiCad files are text files, so we can easily edit them.

So I found a way to back-convert the KiCad-nightly footprints to the KiCad-stable build using some copy and replace commands with regular expressions. These regular expressions were too complex to use with a simple command like sed so I wrote a Python script to do this. I even wrote some tests for the Python script to get to grip with pytest.

I didn't have the patience to get all the features back-converted. For instance, I manually deleted arcs and re-drew them.

This scrip is written for the version of the nightly-build that I used, so may or may not work with whatever version of KiCad you use. I'm putting up the code and tests onto GitHub to help anybody else who has the same problem I did.

Jacdac: Using the Black Magic programmer

Summary

Using my home-made Black Magic Probe, I have to push the reset button at the start of each programming cycle, then release the button once I get a correct voltage statement, to get firmware to reliably load onto my Jacdac boards

Introduction

How do we program our Jacdac boards? One solution is to use a home-made Black Magic Probe (BMP) debugger. Instructions on how to make your own from a Blue pill board are here: https://github.com/mmoskal/blackmagic-bluepill

Making your own BMP from a Blue Pill board is a lot cheaper than buying a pre-made BMP, but you have to swallow setting it up yourself.

I had some issues getting my home-made board to work, which I detail here along with the solution.

Home made Black Magic Probe

I was fortunate enough to get the Bluepill-BMP shield that is detailed on GitHub site. Please find a photo of this below. Three components are marked:
Green square - reset button.
Yellow square - board power switch.
Red square - reset signal jumper.

Black Magic Probe board made using a Blue Pill board.

Using the Black Magic Probe programmer

Unless you are externally powering the board that you are programming, it needs to be powered during flashing. Set the board power switch to 'ON' to enable this.

The reset signal is mapped to pin 10 of the 10-pin surface mount connector that your programming cable clips into. The JacConnect socket expects this to be on pin 5. To bridge the reset signal to pin 5, set the jumper as indicated, connecting the HC to XS pins as shown in the red square on the image above. This allows us to use a ribbon connector cable with no butchery and the 'hack-connect/JacConnect' header to program our Jacdac boards. This header is detailed [here].(https://arcade.makecode.com/hardware/dbg)

If you don't short the two pins, you need to do some surgery to your cable to enable the Reset signal to get to pin 5 of the 5-pin JacConnect socket.

Programming a Jacdac board with the Black Magic Probe

Here's where I lost about a week of my life. I repeatedly got errors trying to flash my boards using the instructions here.

The final error message was:

Target voltage: 3.27V
SW-DP scan failed!
built/debug.gdb:5: Error in sourced command file:
Attaching to Remote target failed
(gdb) quit

I followed some methodical troubleshooting to confirm continuity from the pins on the STM32 microcontroller on my target board to the pins on my BMP. Get yourself some fine tip probes. Pomona is a good brand for probes. They don't sponsor me. Be nice if they did.

The magic trick is to hold down the Reset button on your Black Magic Probe board before you hit enter on your keyboard to submit either 'make run BL=1' or 'make run flash' to write the .elf files onto the Jacdac board. The flashing process will stop with a Target voltage message like:

Target voltage: 3.27V

Release the Reset button and the flashing will continue ending with the reassurring message:

*** Flash OK

I flash in two stages - the bootloader first using:

make run BL=1

Then I load the firmware using:

make run

In theory I can load both .elf files in a oner using:

make run ff

But this does not yet work for me.

Victory, tea and biscuits are ours. Or, in my case, an espresso from the machine that somebody thoughtfully donated to the kitchen on the floor where I loiter. Which is leading to a substance abuse situation, however my cycle time home is much reduced.

Using micro:bits to send data to and from an Arduino wirelessly

Summary

We can use microbits to act as a wireless bridge between an Arduino Uno and a PC. One micro:bit connects to the Arduino. The second connects to the PC. Data can be passed to and from the Arduino serial port through the micro:bits.

The problem

A friend has an Arduino based instrument that he wants to communicate with from his PC wirelessly. Why? I don't ask why. He's bigger than I am.

A solution - outline

A micro:bit is connected to the serial pins (TX and RX) on an Arduino Uno through a logic level converter. This micro:bit sends data to and receives data from the serial port on the Arduino. The data received from the Arduino is transmitted to a second micro:bit connected to the PC through a USB port. Data received from the micro:bit connected to the PC is sent to the micro:bit connected to the Arduino.

Some folk call the serial port the UART. This is probably a more accurate name. What ports aren't serial nowadays? The Arduino uses the 'Serial' library to communication with this port, so I'll carry on using the term 'serial' for the port.

The micro:bit connected to the Arduino acts as a wireless bridge to the micro:bit connected to the laptop.

The micro:bit connected to the laptop acts as a wired bridge to the laptop. This micro:bit sends data to a virtual serial port on the laptop and receives information from the laptop on the same serial port. The received data is transmitted to the micro:bit connected to the instrument.

The two micro:bits create a wireless bridge between the Arduino and the laptop, allowing two-way serial communication.

Data flow

Arduino Uno RX, TX pins <-wires-> logic level converter <-wires-> micro:bit <- radio link -> micro:bit connected to PC <- USB cable-> PC

Fritzing diagram showing the Arduino Uno, logic level converter and micro:bit connections

The diagram below shows the connections on the Arduino side. The micro:bit on the PC side is connected to the PC by a USB cable.

Connections between the Arduino Uno, logic level converter and the micro:bit.

Arduino to micro:bit connection

The Arduino Uno used for testing works at 5V, the micro:bit works at 3V. So I use a logic level converter in-between the two boards to allow communication.

I drew a Fritzing diagram showing all the connections. Like many Fritzing diagrams, it looks pretty but is hard to follow. I paid for the software, so I'm using it.

Please see my blog post on the level converter to see how to connect the Arduino to the micro:bit via a logic level converter. The setup may well work without the logic level converter. However, the input pins on the micro:bit are not rated for more than 3.9V. So, applying the 5V signal levels from the Arduino is out of spec, meaning that the micro:bit could fail at any time. Or fail a little bit, just enough to corrupt your data occassionally, leaving you chasing Heisenbugs.

micropython code for the micro:bits

The script for the two micro:bits is almost the same. The UART is set up to use the edge connector pads as RX and TX for the micro:bit connected to the logic level converter. The UART for the micro:bit connected to the PC is left as the default which uses the USB connection to send and receive data to and from the UART.

micropython code for the micro:bit connected to the logic level converter

'''
Forward serial data from pins 0&1 to radio.
Matthew Oppenheim May 2022
'''

import radio
from microbit import display, Image, pin0, pin1, sleep, uart

radio.on()
radio.config(channel=40, group=40, data_rate=radio.RATE_1MBIT)

uart.init(baudrate=115200)
uart.write('Receive data from radio and send to uart.\n')
uart.write('Receive data from uart and send to radio.\n')

# connections to edge connector pads from RX, TX on instrument
uart.init(baudrate=115200, tx=pin1, rx=pin0)

def flash_display():
    ''' Flash a chessboard onto the display. '''
    display.show(Image.CHESSBOARD)
    sleep(200)
    display.clear()

def send_uart(msg_str):
    ''' Send msg_str to serial port. '''
    uart.write(msg_str)

def send_radio(msg_str):
    ''' Send msg_str over radio. '''
    radio.send(msg_str)

# script starts here
flash_display()

while True:
    display.show(Image.DIAMOND)
    # radio.receive() returns a string
    radio_in = radio.receive()
    if radio_in:
        send_uart(radio_in)
# flash display only on instrument side, not laptop
        flash_display()
    if uart.any():
        msg_bytes = uart.read()
        msg_str = str(msg_bytes, 'UTF-8')
        send_radio(msg_str)

micropython code for the micro:bit connected to the PC

'''
Receive data from uart and send to radio.
Matthew Oppenheim May 2022
''''''

import radio
from microbit import display, Image, pin0, pin1, sleep, uart

uart.init(baudrate=115200)
radio.on()
radio.config(channel=40, group=40, data_rate=radio.RATE_1MBIT)
uart.write('Receive data from radio and send to uart.\n')
uart.write('Receive data from uart and send to radio.\n')

def flash_display():
    ''' Flash a chessboard onto the display. '''
    display.show(Image.CHESSBOARD)
    sleep(200)
    display.clear()

def send_uart(msg_str):
    ''' Send msg_str to serial port. '''
    uart.write(msg_str)

def send_radio(msg_str):
    ''' Send msg_str over radio. '''
    radio.send(msg_str)

# script starts here
flash_display()

while True:
    display.show(Image.DIAMOND)
    # radio.receive() returns a string
    radio_in = radio.receive()
    if radio_in:
        send_uart(radio_in)
        #flash_display()
    if uart.any():
        msg_bytes = uart.read()
        msg_str = str(msg_bytes, 'UTF-8')
        send_radio(msg_str)

Testing

I created a test string which mimicks the length of string that my friend wants to send from his instrument to the PC. I created a function which flashes the Arduino LED when the string 'flash' is received. This allows me to test two way communication. The string is streamed at around 100Hz to the PC. I send the string 'flash' from the Arduino serial plotter and see that the LED on the Arduino flashes.

By making the test string a number, I can use the serial plotter tool in the Arduino IDE. So long as the incoming string is not corrupted, I see a straight line. This gives a quick visual check that the data is intact.

I found that so long as the string is 16 characters or less, transmission is robust. Once the string exceeds about 16 characters, there is a marked decrease in reliability. The string is often corrupted. The limiting factor seems to be string length rather than the frequency that the string is sent at. I tested at up to about 100Hz - that is the string is sent about every 10ms. I say 'about' as I didn't use timers and interrupt service routines to try and make this exact. I just put a delay function in the main loop, which is never going to give an exact frequency rate. But it is good enough for the testing presented here.

If you want to get serious about precise timing, have a look at using FreeRTOS and timer callbacks here.

Arduino Uno test code

/*
Send a test string through the serial port.
Check for incoming serial data. 
Flash LED if the incoming serial data starts with "flash".
Matthew Oppenheim May 2022
*/

// String to hold data for the serial port
String g_inputString = "";
bool g_stringComplete = false;
int g_count = 0;
// strings used for testing data transmission
//const String g_test_string = "1,30.2;2,30.4;3,30.64;4,30.64;5,30.64;6,30.64;7,30.64;8,30.64;9,30.64;10,30.64;11,30.64;12,30.64;13,30.64;14,30.64;15,30.64;16,30.64;|A";
//const String g_test_string_1_4 = "1,11.1;2,22.2;3,33.3;4,44.4";
//const String g_test_string_1_2 = "1,11.1;2,22.2|A";
const String g_test_string_1_2 = "123456789012345";

void setup() {
   Serial.begin(115200);
   pinMode(LED_BUILTIN, OUTPUT);
   Serial.println("counter test");
}

void loop() {
  // Check if the serial data string has a newline character at end.
  if (g_stringComplete) {
    // Check if the serial data has a command in it.
    processSerialData(g_inputString);
    // Reset the serial data string and completion flag.
    g_inputString = "";
    g_stringComplete = false;
  }
    g_count = updateCount(g_count);
    //Serial.println(g_count);
    Serial.println(g_test_string_1_2);
    delay(10);
}

void flashLed() {
  digitalWrite(LED_BUILTIN, HIGH);
  delay(200);
  digitalWrite(LED_BUILTIN, LOW);
  delay(200);
}

// increment count up to 255, then reset to 0
int updateCount(int count) {
  if (count > 254) {
    return 0;
  }
  ++count;
  return count;
}

// Check for command in serial_string.
void processSerialData(const String &serialString){
  const String flash_command = "flash";
  // Use startsWith to accommodate different EOL symbols in data.
  if (serialString.startsWith(flash_command)){
    flashLed();
  }
}

// Handle incoming data on serial hardware port
void serialEvent() {
  while (Serial.available()) {
    // Read in character from serial port and add to serial data string.
    char inChar = (char)Serial.read();
    // use += as this modifies left operator without creating new instances
    g_inputString += inChar;
    // If a newline character is received, set the serial data complete flag true.
    if (inChar = '\n') {
      g_stringComplete = true;
    }
  }
}

Other Solutions

My friend tried interacting with his instrument using wifi with a Feather Huzzah connected to the instrument. This works well at home, but failed using the University wifi. We may have a go using our own wifi switch.

A wired solution is always more reliable than wireless. Using a USB cable to stream data from the UART port to the PC will always be more robust than a wireless solution. If you need to access two or more UARTs on your Arduino - for instance the Arduino Mega has four UARTs - we can use a USB to serial converter cable such as this one to connect with each of the UARTs and stream data to the PC. Each of the cables creates a virtual COM port on the PC.

Logic level converter between an Arduino Uno and micro:bit using a FET transistor

Summary

We can use a single FET transistor with two resistors to make a two way logic level converter to interface e.g. Arduino Uno and a micro:bit.

The problem

I need to interface a micro:bit to an Arduino Uno, connecting the RX and TX pins of the Arduino to tabs on the edge connector of the micro:bit. The Arduino Uno has 5V logic levels. The micro:bit has 3V logic levels. The 5V signals from the Arduino Uno may damage the micro:bit.

The setup may well work without the logic level converter. However, the input pins on the micro:bit are not rated for more than 3.9V. So, applying the 5V signal levels from the Arduino is out of spec, meaning that the micro:bit could fail at any time. Or fail a little bit, just enough to corrupt your data occassionally, leaving you chasing Heisenbugs.

A solution

I need a logic level converter. This sits inbetween the Arduino Uno and the micro:bit boards and converts the signal levels so that each board receives the logic levels it works at.

How a logic level converter works

I learned about how logic level converters work from an excellent article in Circuit Cellar by Robert Lacoste. You can read this article for free by opening a free account on the website here. The simplest bi-directional logic level converter in the Lacoste article uses a single field-effect transistor with two 2.2K Ohm resistors. This circuit is simulated below.

Simulation of the FET based logic level converter using Falstad

I use the free online circuit simulation tool Falstad to simulate the device. Go to the site, then click on File, Import from Text. Copy and paste this text in to the text box and click OK.

$ 1 2e-8 21.593987231061412 50 5 50 5e-11
r 256 272 256 352 0 2200
r 464 272 464 352 0 2200
f 368 320 368 368 32 1.5 0.02
w 256 352 256 368 0
w 256 368 352 368 0
w 464 352 464 368 0
w 464 368 384 368 0
w 256 272 256 256 0
w 256 256 368 256 0
w 368 256 368 320 0
R 160 368 96 368 0 2 10000 2.5 2.5 0 0.5
w 160 368 256 368 0
R 256 256 176 256 0 0 40 3 0 0 0.5
R 464 256 560 256 0 0 40 5 0 0 0.5
w 464 256 464 272 0
403 640 352 768 416 0 6_64_0_4099_5_0.003125_-1_2_6_3
w 464 624 464 640 0
R 464 624 560 624 0 0 40 5 0 0 0.5
R 256 624 176 624 0 0 40 3 0 0 0.5
R 496 736 544 736 0 2 10000 2.5 2.5 0 0.5
w 368 624 368 688 0
w 256 624 368 624 0
w 256 640 256 624 0
w 464 736 384 736 0
w 464 720 464 736 0
w 256 736 352 736 0
w 256 720 256 736 0
f 368 688 368 736 32 1.5 0.02
r 464 640 464 720 0 2200
r 256 640 256 720 0 2200
w 496 736 464 736 0
403 32 672 160 736 0 26_64_0_4099_5_0.0015625_-1_2_26_3
207 464 368 528 368 4 High\slevel\slogic
207 256 736 176 736 4 Low\slevel\slogic
403 32 288 160 352 0 11_64_0_4099_5_0.00625_-1_2_11_3
403 512 656 640 720 0 30_64_0_4099_5_0.00625_-1_2_30_3
x 157 222 687 225 4 24 Low-level\slogic\sinput,\shigh-level\slogic\soutput
x 131 587 658 590 4 24 High-level\slogic\sinput,\slow-level\slogic\soutput

You should see this circuit:

One note about Falstad - the app can suck up a lot of the CPU and ramp up the CPU temperature.

The 'scopes shows voltage in green and current in brown at the same time. We are interested in the green voltage levels.

There are two circuits shown. These show the same logic level converter circuit but with the inputs and outputs reversed.

The top circuit shows a 3V signal, simulated by a 10kHz square wave, from the simulated low-level logic device converted to a 5V signal.

The bottom circuit shows a 5V signal input to the right side of the circuit and a 3V signal output from the left.

So the logic level converter simulates correctly for an input signal to either side.

I use 2.2K resistors in the simulated circuit as this is what the boards I received use. Lacoste uses 2.2K resistors in his example.

Explanation of the simulated FET circuit

The FET is called an n-type FET as it uses an n-type semiconductor channel to transmit charge through the component. This channel is incomplete unless the gate has a positive bias compared with the source. Then current can flow along the completed channel. This current flow is shown as animated dots in Falstad. If you hover your cursor over the FET symbol, the gate, drain and source terminals show up as G, D and S. This gate-source bias needs to be above a threshold level for the semiconductor channel to complete and for current to flow. This is a pretty sketchy summary. There are lots of good YouTube videos explaining how a FET works with great animations I wish I'd had access to in the Chibanian when I studied this.

The FET gate is permanently held high by the low level side of the logic converter, in this case 3V. The source and drain of the FET are both weakly held high by 2.2K resistors.

If we slow down the simulation, we see that current flows when the source is held low by the low-level logic driver in the top simulation. Similarly, in the bottom simulation, current flows when the drain is held low by the high-level logic simulation. In both cases, a path to ground is created when the FET starts to conduct. This means that the output level also sees ground.

Have a play with the simulation to get your head around how the device works.

Testing

I bought a logic level converter board from eBay. The board has four logic level converter channels. A photo of the board with four converter channels can be seen below. The top pins are labelled HV1..HV4 - these are the connections for the high logic values. In my use case, these connect to the RX and TX pins on the Arduino Uno. So I only need two of the four channles. The pin labelled HV is connected to the high logic level voltage rail (5V in the case of the Arduino Uno). There is a ground pin connection on both the top and bottom rows, labelled GND. Similarly, the bottom row of connections has LV1..LV4 for each of the four low voltage signals. In my use case, the low level logic channels connect with pads 0 and 1 of the micro:bit's edge connector. I only need two channels. The unused channels are left unconnected.

Logic level converter.

I set the logic level converter up on some breadboard. I hooked a micro:bit to one side and an Arduino Uno to the other. The TX pin from the Arduino goes into the high-level logic side on HV1. Pad 0 of a micro:bit is connected to the low-level logic side on LV1. I connect to pad 0 on the micro:bit using a pin on a Kitronix edge connector that the micro:bit slots into.

A photo showing this arrangement can be seen below.

Testing a logic level converter.

I set up the Arduino Uno to send a signal from the TX pin and scoped the input to the micro:bit on my pocket oscilloscope, a DS213. This may not be a high-spec oscilloscope, but it is good enough for this application. One advantage of the DSO213 'scope is that it is battery powered, so there is no danger of creating ground loops. A close up of the oscilloscope display showing input and output serial data can be seen below:

High level logic input, low level logic output DSO213 display.

The green trace is the low level logic, the yellow the high level trace. I can see that the signal going to the micro:bit is shifted to the expected 3V logic level. The time axis is set to 10us per square. The square waves can be seen to occupy about one 'scope square for each high and low, which gives a baud rate of approximately 1/(10*10^-6) = 100kHz. I set the baud to be 115200, so this all looks about right.

The screen shot shows that the output signals are not the crisp square waves shown on the simulation. This is due to the capacitance in the circuit which was not part of the simulation There is also some ripple and noise on the signals. This is from other real-world artefacts of building circuitry, such as inductance. As the input frequency increases, the capacitance will have an increasing effect on the output signal, causing the edges to become more rounded. We can add capacitors and inductors to our simulated circuits to get more accurate outputs. That is a topic to deal with in another blog one day.

Once I was happy that the setup worked to convert from a high level voltage to a low level voltage, I tested the opposite signal direction. I connected pin 1 from the micro:bit edge connector to a second channel on the low logic level side of the converter board (LV4) and the RX pin of the Arduino Uno to the same channel on the high voltage side (HV4). I generated 3V signals with the micro:bit as the input to the logic level converter. The output was at the 5V logic level of the Arduino Uno. The output signals showed a similar rounding of the signal edges due to capacitance as the high level to low level conversion did.

Low level logic input, high level logic output DSO213 display.

Difference between the converter board and the simulated circuit

The transistors on my boards have 'J1Y' marked on the backs. The only transistor I can find with this marking on the back is the KSA1298Y, which is a bipolar transistor, not a FET. I don't see how the circuit will work with a single bipolar transistor per channel, but I'm open to suggestions. My conclusion is that the transistor is an unidentified FET. The converter board I bought uses 10K Ohm resistors, but this is a minor difference with the 2.2K Ohm resistors used in the simulation. A wide range of resistance values can be used for the resistors in the circuit.

Conclusions

The logic level converter works as expected. The real world implementation has effects from the inherent capacitance present in the circuitry. I'm not sure what model of transistor is used in the circuit boards that I purchased. Using a printed circuit board may reduce the rounding of the signal corners. I'm not sure how well the converter board would work in the MHz range - the rounding may become so severe that the output signals don't reach the necessary logic voltage levels. However, for passing data between serial ports at 115200 baud, the board is adequate.

Downloading YouTube videos using Termux and youtube-dl on Android

Problem

How to download YouTube videos from my Android phone to watch offline.

A solution

Use the youtube-dl command line utility from the Termux app.

Background

I work on ships where watching YouTube videos is difficult. So I want to save videos to watch offline on my Android phone. The Android apps I'd used to do this stopped being able to download YouTube videos. I use youtube-dl from my Linux laptop to download from YouTube. The Termux app on Android gives a Linux like terminal which enables me to install and run youtube-dl.

Setup

Install Termux from the F-Droid app. Don't install Termux from Google Playstore. The version on Playstore is old and not supported. There are many websites giving tutorials on Termux.

Open up Termux. Now we need to install the Python programming language which is used to install and run youtube-dl. From the command line in Termux type:

pkg install python3

Wait a minute or two while python is installed.

Python has a package installation tool called pip. We use this to install youtube-dl.

Type:

pip3 install youtube-dl

Almost there. To download a YouTube video, you copy and paste the YouTube video url and use it with the youtube-dl command like this:

youtube-dl <youtube-video-url>

Marvel as the video is downloaded to your Termux drive. But... how do we view this video? Use the Files file explorer app that comes with Android. The Termux drive shows up on this. Many file browsers don't allow access to the Termux drive. I don't know why either. You can access your newly downloaded video on the Termux drive.

Keeping youtube-dl up to date

Often youtube-dl stops working as YouTube changes something somewhere. Updating youtube-dl fixes this. Update youtube-dl using this command:

pip3 install youtube-dl -U

Extra

There are many options with youtube-dl to can specify the quality of your video and to rename it. You can download only audio which is useful for music videos. You can give a playlist link to youtube-dl and all of the playlist will be downloaded. Have a look for tutorials on youtube-dl if you want to learn more.

The alias I set up in Linux to download videos at the top quality available and rename them is:

alias youtubevid='youtube-dl -f 'best' --ignore-errors --output "%(uploader)s_%(title)s.%(ext)s"'

The alias I use to download audio only at the best quality available and rename the files is:

alias youtubemp3='youtube-dl --extract-audio --audio-format mp3 --ignore-errors --output "%(uploader)s_%(title)s.%(ext)s"'

Kicad – fixing invisible gaps in board outlines on the Edge.Cuts layer

Summary

Kicad appears to tell us that we have gaps in our board outlines when there aren't any. This can be solved by opening up the segments of the board edge slightly at the error points and bridging the gaps with short lines.

The problem

Kicad printed circuit board (PCB) designer tells you that you have a board edge that is incomplete when you run the design rules check (DRC). You can clearly see that you have a lovely, complete outline with no gaps in it. Involuntary sailor-language syndrome occurs. I encountered this issue with versions 5 and 6 of Kicad.

This occurs for me when I inherit a layout that has curves on the board edge. The board edge is drawn on the Edge.Cuts layer on Kicad. I suspect that the cause may be different grids being used at different times in the project. Grid management is one of the many things that needs attention when laying out a PCB.

See the figure below that shows the red pointy arrow of frustration generated by the DRC.

Kicad Edge.Cuts layer showing an error.

The error message in the DRC is shown below. "Error: Board has malformed outline (not a closed shape)". This is not the only error message that can be generated for this issue and the wording may well change between different versions of Kicad.

Kicad DRC error message.

For the screenshots I use a 0.1mm grid and the line thickness on the Edge.Cuts layer is 0.15mm. Only the Edge.Cuts layer is active.

If we select the top part of the curve and then the bottom part of the curve we can see that the arrow points at the join of two curves. The two curves appear to connect with no gap. Please see the two figures below which demonstrate this.

Kicad Edge.Cuts layer, top segment of board edge curve selected.
Kicad Edge.Cuts layer, bottom segment of board edge curve selected.

Take it from me, you can waste hours of your life that you will never get back trying to shift the two curve segments so that they join up so precisely that the Kicad DRC accepts them as a complete outline.

The solution

Open up the two segments so that there is visible gap between them. Keep this gap as small as practical. To do this, click on the edge of one segment and press G for grab, then keep the mouse button down and drag it. See the figure below
.

Kicad Edge.Cuts layer showing two segments with a gap between them.

Now we want to join the two segments with a small 'jumper', using the line tool. To get this line to connect at exactly the end of the segment so that Kicad accepts that it is connected to the curve, we look for the cursor to become a cross with a circle around it. Don't forget to select the Edge.Cuts layer first. For some reason, Kicad defaults to selecting any layer except the Edge.Cuts layer. Which also leads to involuntary sailor-language syndrome.

The figures below show the beginning and end of the short 'jumper' line correctly connecting the separate segments.

Selecting the end of a segment on the Edge.Cuts layer.

Drawing the connecting line between two segments on the Edge.Cuts layer.

Again, note that the cursor becomes a cross in a circle. This shows that the cursor is on the end of the segment.

Run DRC again and rejoice that the red arrow has gone. You may find that the jumper you drew has been adsorbed into one of the segments and no longer appears as a separate entity.

Problems

Adding this jumper may slightly distort your intended board outline. The router bit used to carve out your PCB is probably around 2mm in diameter. By zooming in on the affected region to add your jumper, any small distortion will be smoothed out by the router. In the example screenshots, the small crosses are 0.1mm apart.

Installing kicad nightly-build on Debian 11

The problem

I needed the latest nightly build version of the Kicad PCB design software to open some design files sent to me. I run Debian 11.

How to install kicad nightly-build on Debian

I got started with the information on this site

Create a .list file in the folder /etc/apt/sources.list.d. e.g. /etc/apt/sources.list.d/kicad-nightly.list.

Contents:

deb [arch=amd64] https://debian.sur5r.net/kicad-nightly bullseye main

If you are not running Debian bullseye, change the 'bullseye' to whichever version of Debian that you are running.

sudo aptitude update

There will be an error, complaining of an unsigned key with a name like e.g. E3CA1A89941C42E6.

Fix this by:

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys [missing key name]

sudo aptitude update

sudo aptitude search kicad

You should find kicad-nightly in the list of available files.

sudo aptitude install kicad-nightly

To run:

kicad-nightly

Where I went wrong

I tried to install from an Ubuntu PPA site here.

I ended up with unresolved dependencies, which had other dependencies in a never-ending pit of dependencies.

Remove spikes from a dataset using python

Problem

How to remove random spikes from data.

Summary

Using the pandas library in python we can remove random spikes from data. For this blog, I create a sine wave with random spikes then show the steps used to remove the spikes.

This blog is based on an answer I posted to a Stackoverflow question at:

https://stackoverflow.com/questions/37556487/remove-spikes-from-signal-in-python

A solution

Here's a general method for removing spikes from data. The code is at the end of this post. The variables that need to be tweaked for each data set are in upper case.

I tested this out using bathymetry data. For the sample code, I create a sine wave with random spikes. I call the clipped dataset y_spikey. A sine wave is reasonable example to use as the curves prevent a simple clipping function from being effective.

Input data set

Clip the data

Replace data above HIGH_CUT and below LOW_CUT with np.nan. I call this data set y_clipped.

np.nan are 'not a number' values, which appear as NaN when the data set is printed. The 'np.' portion shows that this data type comes from the (numpy)[numpy.org] library.

Pandas is built on top of numpy so recognises the np.nan data type.

NaN values appear as gaps when graphed. The NaN values are ignored when calculating e.g. averages over ranges of values that include NaN values in them.

Using NaN values instead of zero values leads to less distortion of calculations in the next stages.

The data is clipped in the method def clip_data.

Fit a curve to the clipped data

Calculate a forwards-backwards exponential weighted moving average (FBEWMA) for the clipped data.

There is more about the FBEWMA with links to further explanation here: https://stackoverflow.com/questions/32430566/exponential-smoothing-average

I call this dataset y_ewma_fb.

Why use the FBEWMA instead of a a simple sliding-window averaging function? Sliding an averaging window along the dataset in one direction leads to distortion compared with using the FBEWMA, which uses a sliding window in two directions - one from low to high values (the forwards part of FBEWMA), the other from high to low values (the backwards part of FBEWMA). Filtered data generally has a time shift of half of the filter window length.

With the FBEWMA, there are two filters. One works in an incrementing direction, the other in a decrementing direction. By having two filters, one starting at x=0 and the other starting at x=(maximum value of x), the time shifts are opposite and equal.

The filtered data is then added and the mean used as the output dataset. Doing this removes the time shift associated with using a single filter.

The previous step of clipping the data helps fit the FBEWMA curve to the data that we want to retain. Without clipping, the FBEWMA would have little spikes around the big spikes that we want to remove, making it harder to differentiate the spikes we want to remove from the FBEWMA in the next step.

The variable SPAN adjusts how long the averaging window is and should be adjusted for your data.

y_clipped and y_ewma_fb

This filter is created in the method ewma_fb.

Remove data that is an outlier compared to the FBEWMA curve

Replace the clipped data that is DELTA from the FBEWMA data with np.nan. I call this data set y_remove_outliers.

Using the np.nan data type means that gaps appear on the graph where the clipped data is more than DELTA from the FBEWMA curve.

y_remove_outliers and y_ewma_fb

The code that carries out this stage is in the method remove_outliers.

Interpolate the missing values

Use the pandas. interpolate function to replace the NaN values with data. I call the interpolated dataset y_interpolated. This is your output dataset.

y_spikey and y_interpolated

This stage is carried out by the line

df['y_interpolated'] = df['y_remove_outliers'].interpolate()

Closing comments

When processing a large number of similar datasets, we usually spend some time testing the processing flow (we use the word 'flow' for the set of filters). Different values for the variables such as the lengths of the FBEWMA filters are tested until we get something that 'looks right'.

It could be that several stages of filtering are repeated. We might not like the interpolated data set, product, so pass this through a second set of FBEWMA, removing outliers and interpolation.

Maybe we will apply a smoothing function to the interpolated data to present a more 'pleasant' looking final product. There is no 'one-size fits-all solution.

Data processing is still often led by 'do you like how it looks' rather than rigorous measurable criteria.

Python script

import logging
import numpy as np
import pandas as pd

logging.basicConfig(datefmt='%H:%M:%S',
                    stream=sys.stdout, level=logging.DEBUG,
                    format='%(asctime)s %(message)s')

# Distance away from the FBEWMA that data should be removed.
DELTA = 0.1

# clip data above this value:
HIGH_CLIP = 2.1

# clip data below this value:
LOW_CLIP = -2.1

# random values above this trigger a spike:
RAND_HIGH = 0.98

# random values below this trigger a negative spike:
RAND_LOW = 0.02

# How many samples to run the FBEWMA over.
SPAN = 10

# spike amplitude
SPIKE = 2

def clip_data(unclipped, high_clip, low_clip):
    ''' Clip unclipped between high_clip and low_clip. 
    unclipped contains a single column of unclipped data.'''

    # convert to np.array to access the np.where method
    np_unclipped = np.array(unclipped)
    # clip data above HIGH_CLIP or below LOW_CLIP
    cond_high_clip = (np_unclipped > HIGH_CLIP) | (np_unclipped < LOW_CLIP)
    np_clipped = np.where(cond_high_clip, np.nan, np_unclipped)
    return np_clipped.tolist()

def create_sample_data():
    ''' Create sine wave, amplitude +/-2 with random spikes. '''
    x = np.linspace(0, 2*np.pi, 1000)
    y = 2 * np.sin(x)
    df = pd.DataFrame(list(zip(x,y)), columns=['x', 'y'])
    df['rand'] = np.random.random_sample(len(x),)
    # create random positive and negative spikes
    cond_spike_high = (df['rand'] > RAND_HIGH)
    df['spike_high'] = np.where(cond_spike_high, SPIKE, 0)
    cond_spike_low = (df['rand'] < RAND_LOW)
    df['spike_low'] = np.where(cond_spike_low, -SPIKE, 0)
    df['y_spikey'] = df['y'] + df['spike_high'] + df['spike_low']
    return df

def ewma_fb(df_column, span):
    ''' Apply forwards, backwards exponential weighted moving average (EWMA) to df_column. '''
    # Forwards EWMA.
    fwd = pd.Series.ewm(df_column, span=span).mean()
    # Backwards EWMA.
    bwd = pd.Series.ewm(df_column[::-1],span=10).mean()
    # Add and take the mean of the forwards and backwards EWMA.
    stacked_ewma = np.vstack(( fwd, bwd[::-1] ))
    fb_ewma = np.mean(stacked_ewma, axis=0)
    return fb_ewma

def remove_outliers(spikey, fbewma, delta):
    ''' Remove data from df_spikey that is > delta from fbewma. '''
    np_spikey = np.array(spikey)
    np_fbewma = np.array(fbewma)
    cond_delta = (np.abs(np_spikey-np_fbewma) > delta)
    np_remove_outliers = np.where(cond_delta, np.nan, np_spikey)
    return np_remove_outliers

def main():
    df = create_sample_data()

    df['y_clipped'] = clip_data(df['y_spikey'].tolist(), HIGH_CLIP, LOW_CLIP)
    df['y_ewma_fb'] = ewma_fb(df['y_clipped'], SPAN)
    df['y_remove_outliers'] = remove_outliers(df['y_clipped'].tolist(), df['y_ewma_fb'].tolist(), DELTA)
    df['y_interpolated'] = df['y_remove_outliers'].interpolate()

    ax = df.plot(x='x', y='y_spikey', color='blue', alpha=0.5)
    ax2 = df.plot(x='x', y='y_interpolated', color='black', ax=ax)

main()

Bendlabs single-axis bend sensor

Using the Bendlabs 1-axis bend sensor with an ESP32 board

This post explores using a more tactile and sensitive type of flex sensor than I've used so far. This is part of my ongoing flex sensor assistive technology project.

The flex-sensor is the bendlabs 1-axis flex sensor.

In this post I show how I interfaced this sensor with an Unexpected Maker feather S2 board and started logging data from it. This board uses the ESP32 S2 MCU. The 'S2' variation has the feature that enables the board to appear as a physical keyboard. This will be useful to control communication software as part of the assistive technology project.

A short video showing the test system displaying real-time data can be viewed here:

A photo of the flex sensor from the Bendlabs website is shown below.

Bendlabs 1-axis flex sensor, from the bendlabs website.

A photo showing my grubby hand holding one of the flex-sensors is shown below. I already soldered the wires onto the end connector in this photo. The sensor comes without these.

Bendlabs single-axis flex-sensor.

To use these sensors you can either buy a development kit that Bendlabs sells which has the sensor and a board to plug it into or you can just buy the sensor and solder on your own wires to connect with your own board. I just bought the sensor. I need to be able to coonect this sensor to my own projects. If I can only get the sensor to work with a demo kit, it is not of much use to me.

I soldered on some silicone insulated wires to the end connector. The silicone allows the wire to be more flexible than the regular PVC insulation. This was a little tricky, but not too bad an exercise. I terminated the ends of the wires onto 0.1" header pins. The wires that I soldered onto the flex sensor and the header that they connect with can be seen in the photo below. On the left of the photo are the wires that connect to the Unexpected Maker Feather S2 board. I soldered female headers onto this board. Male or female headers on a development board? I've seen both. I went with female as I could and nobody stopped me.

Bendlabs single-axis flex-sensor with extension wires soldered on to break-out connector.

The pinout for the connector is shown below.

Bendlabs one-axis flex-sensor pinout.

The pinout shows that the signal interface is I2C - these are the SDA and the SCL signals. In addition to these, there are power (VCC) and ground (GND). The part is not 5V tolerant. I used the Unexpected Maker Feather S2 3.3V supply to connect with VCC. There are two other signals to deal with. nDRDY is 'not data ready' - meaning inverted logic on the data ready line. When data is ready, the line will go low. nRST is 'not reset', meaning that when this signal goes low, the sensor enters a reset condition. I connected these signals to two sockets on my board. nDRDY goes to pin 7 and nRST goes to pin 3. In the example code provided, nDRDY goes to pin 4, but I don't have a socket for pin 4, so I chose pin 7 and ajusted the software. More details on this below.

The wire colours that I used and their corresponding signals are:
1 black wire - GND ground
2 red wire - Vcc - 1.62-3.6V NOT 5V TOLERANT
3 green wire - nDRDY
4 blue wire - SDA - needs 10K pull up
5 yellow wire - SCL - needs 10K pull up
6 orange wire - nRST

The board I used has built-in 10K pullup resistors for the dedicated SDA and SCL ports. These pull-up resistors are necessary of I2C communication. I found that out the usual way that I find things out.

The Unexpected Makers Feather S2 board has a Qwiic connector socket which takes care of the power, ground, i2c clock and i2c signal lines. I bought a flexible wire Qwiic connector to attach the Qwiic connector socket on the board with the relevant signal lines on the flex sensor, using the 0.1" pins to connect the two devices.

I used the same colour wires from the flex sensor to the header pins as the Qwiic connector wires have that connect on the opposite side of the header pins.

Unexpected Maker Feather S2 board connected to a Bendlabs single-axis flex-sensor.

code

Demo code from Bendlabs can be found at:

Bendlabs one-axis flex-sensor Github.

This is written to run on the Arduino platform.

This code is aimed at the the Sparkfun Pro nRF52840 Mini. With the hardware abstraction that C allows, the example code compiles for the Unexpected Makers Feather S2 once you install the necessary ESP32 library. Have a search on t'net on how to install the ESP32 library.

The only update that I needed to make to the demo code is that I use pin 7 on my board instead of the pin 4 defined in the code for the nDRDY signal. I adjusted the corresponding line of code:

from

#define ADS_INTERRUPT_PIN  (4)

to

#define ADS_INTERRUPT_PIN  (7)

The quick start guide from Bendlabs says to use the C program 'bend_interrupt_demo' through the Arduino platform. I couldn't get this to work. I used my 'scope on the I2C lines. I could see clock signals on the clock line - SCLK, but no data on the SDA line. I did get the example program 'bend_polled_demo' to work with one correction. Hopefully, this will be corrected in the Github repository by the time that you read this post.

The line:

 int ret_val = ads_read_polled(&sample, &data_type);

should be

 int ret_val = ads_read_polled(sample, &data_type);

in the function:

void loop() {

  float sample[2];
  uint8_t data_type;

  // Read data from the one axis ads sensor
  int ret_val = ads_read_polled(&sample, &data_type);

After flashing the code to the board, I can see real time angle data plotting on the Serial Plotter in the Arduino IDE. See below for a screenshot.

Arduino serial plotter showing flex sensor data.

What next?

The positives of this flex-sensor are:

  • It is more flexible and tactile than the flex sensor I've used so far.
  • Works for positive and negative deflection.
  • More sensitive at detecting flex.

The negatives are:

  • Difficult to make an extension lead for. The soldering is finicky and there are 6 power and signal lines to contend with.
  • Cost. I'm not sure how much of an issue this is. The Bendlabs sensor costs around £40 at the time of writing. This is about double the cost of the other flex sensor I tested.

Bendlabs look to have a business model where they want to be consultants and customise their technology for each product. What I would like is an off the shelf 'plug and play' sensor and a lead with a socket on the end that the sensor plugs into. Hand soldering the leads onto the sensor presents a potential mode of failure, as well as being time consuming.

I will contact Bendlabs to see if they have something like this for sale.

Mounting an Android phone on Debian Linux to synchronise files

In this post I show how to mount the internal storage of an Android phone so that folders can be updated using e.g. FreeFileSync.

An Android phone can be automagically mounted using a file explorer such as Nautilus to copy and paste files. However I cannot use FreeFileSync to synchronise files to a folder on the phone.

This blog post explains a method that worked for me in December 2021 to solve this issue.

I tested this successfully on a Oneplus 6 phone running LineageOS enchilada (based on Android v11) and on a Sony G8411 running Android v9.

Install jmptfs

The tool I use is jmtpfs.

To install this in Debian:

sudo aptitude install jmtpfs

Create a directory to mount the phone's internal storage

We need to create a file to mount the Android phone in.

mkdir ~/tmp/phone

This creates a folder called 'phone' in your home directory in the subdirectory 'tmp'. You can name the folder anything else that you like.

Connect your Android phone to the Debian system

Connect your Android phone using a USB cable. For my phone running Lineage OS based on Android v11, I used the following to allow the internal storage to be accessible by the Debian system:

Go to settings.
Search on USB, select 'USB controlled by'.
On the next screen:
Select USB controlled: by 'This device' and use USB for: File transfer.

For the phone running Android v9, a message box comes up as soon as I connect the phone to the Debian system asking 'Allow access?'. Click on 'allow'.

The menu system in Android changes with each update, so the above instructions may not be accurate for your version of Android.

Mount the phone's internal storage

Back on your Debian terminal mount the phone's internal storage to the directory ~/tmp/phone using:

jmtpfs ~/tmp/phone

You should now be able to use the directory or subdirectories in ~/tmp/phone to sync with or move and delete files through the command line or file explorer.

Mounting under /media

For whatever reason, I could only get the method to work by creating the new folder under my home directory. I couldn't get it to work by creating a new folder under /media, even though I changed the ownership and group name of the directory I created to my username.

Marcel sent a comment to this blog that solves this for him, but I couldn't get the method to work on my system:

1) Change the group and user of the mount point (e.g. /media/phone) to a normal non root user
2) jmtpfs /media/phone (WITHOUT sudo )

Unmount the phone's internal storage

To end the session, exit from anything that is accessing the directory with the phone's internal storage in. Then unmount the directory using:

fusermount -u /tmp/mtp

Then unplug your phone.