skip to contentW3CSVG

Technical Architecture & Algorithms

Overview

HandShake is a real-time gesture recognition system built on two off-the-shelf microcontroller boards. The watch detects intentional movements using accelerometer data and simple algorithms, then wirelessly triggers a receiver that sends signals to AAC software or devices.

This page explains how it actually works under the hood.

System Architecture

┌─────────────────────┐
│  Lilygo T-Watch S3  │
│                     │
│ • Accelerometer     │ ─────────ESP_NOW──────→ ┌──────────────────────┐
│ • ESP32-S3          │      (wireless)         │ Lilygo T-Embed       │
│ • Vibration motor   │                         │                      │
│ • Battery (2000mAh) │                         │ • Vibration motor    │
└─────────────────────┘                         │ • LEDs & speaker     │
         ▲                                       │ • USB/Switch output  │
         │                                       └──────────────────────┘
    User wears this             Device receives signal, outputs to:
                                  • Keyboard (USB HID)

The Watch: Accelerometer Data → Gesture Detection

Accelerometer Hardware

The T-Watch S3 includes an accelerometer that samples acceleration on three axes:

  • X-axis: Left/right movement
  • Y-axis: Up/down movement
  • Z-axis: Forward/backward movement

The accelerometer samples at 100 Hz (100 data points per second). Firmware continuously collects and analyzes this data.

Two Detection Algorithms

The two gestures are project-internal names. They’re not standard ML models—just simple algorithms that work well for people with different movement patterns.

The “Punch” Algorithm

Detects rapid acceleration in any direction.

Algorithm:
1. Calculate total acceleration: sqrt(x² + y² + z²)
2. If acceleration > user's threshold:
   - Trigger detected!
3. Add a minimum inter-shake period (prevents double-triggering from single gesture)

Why it works:

  • Orientation-invariant (doesn’t matter which way the watch is facing)
  • Works for big, jerky movements (common in cerebral palsy)
  • Computationally cheap (runs on tiny microcontroller)

Best for:

  • People who can make large, rapid movements
  • Hand shakes, arm swings, shoulder raises
  • Movements that involve quick acceleration changes

The “Roll” Algorithm

Detects sustained acceleration change on a single axis.

Algorithm:
1. Choose one axis (X, Y, or Z)
2. Calculate rate of change: differentiate acceleration over time
3. If |rate of change| > user's threshold:
   - Trigger detected!
3. Add a minimum inter-shake period (prevents double-triggering from single gesture)

Why it works:

  • Works for subtle, controlled movements such as a gentle wrist roll

Best for:

  • People with smaller or more controlled movements
  • Movements that change speed gradually

Selecting Your Algorithm

During firmware configuration, you choose which algorithm to use. You can change it later by re-flashing.

How to pick:

  1. Start with “punch” (simpler, more generic)
  2. Have the user make their gesture while you watch the T-Embed display
  3. If you get successful triggers, stick with punch
  4. If you get too many false positives or too many misses, try “roll”

Communication: Watch to Receiver

ESP_NOW Protocol

The watch and receiver communicate using ESP_NOW, a proprietary protocol from Espressif (the ESP32 manufacturer).

Why ESP_NOW?

  • Low latency (~3-4ms) - good for real-time interaction
  • Works without WiFi or Bluetooth setup
  • Low power consumption (battery lasts days instead of hours)
  • Both devices use the same ESP32-S3 chip, so no compatibility issues

Signal Flow

User makes gesture
        ↓
Watch accelerometer detects it (< 10ms)
        ↓
Algorithm decides: is this a real gesture?
        ↓
If YES:
  - Vibration motor triggers (haptic feedback)
  - ESP_NOW packet sent to receiver
  - Watch display updates
        ↓
Receiver gets packet (takes ~3ms)
        ↓
Receiver vibrates/beeps/flashes LED
        ↓
Receiver sends keyboard key or switch signal
        ↓
AAC software/device responds
        ↓
Total latency: ~20-30ms (imperceptible to users)

Multiple Pairs

If you have multiple HandShake systems in the same room:

  1. ESP_NOW uses MAC address filtering to ensure each watch only communicates with its paired receiver
  2. Pairs can operate simultaneously without interference
  3. This allows multi-switch control (watch on each wrist, for example)

The Receiver: Signal to Output

T-Embed Hardware

The receiver T-Embed includes:

  • ESP32-S3 microcontroller (same as watch)
  • USB port (for keyboard signals or power)
  • LED ring around the button
  • Small speaker
  • Rotary dial for sensitivity adjustment

Output Options

USB Keyboard (HID)

Gesture detected on watch
        ↓
Receiver gets signal
        ↓
Receiver emulates USB keyboard
        ↓
Sends: "KEY_SPACEBAR" (or configurable key)
        ↓
Computer receives keystroke
        ↓
AAC software responds

This is the default. Works with any AAC software that accepts keyboard input (Grid 3, Predictable, JABtalk, etc.).

Why it works:

  • No special software needed
  • Works across all operating systems
  • If the software accepts keyboard, it works with HandShake

Sensitivity Adjustment

The rotary dial on the T-Embed controls the threshold in real-time.

How It Works

Lower values makes the watch more sensitive to movement.

The threshold is stored in device RAM, so changes are immediate.

When you long-press the button to save, the current dial position is stored in the watch’s persistent memory. Next time you power on, it remembers the setting.

Display Feedback

The T-Embed has a small screen showing:

  • Current dial position
  • Threshold value
  • Status (if a shake was detected)

Calibration Mathematics

During setup, we’re solving for the optimal threshold:

Goal: Maximize true positives, minimize false positives

True positive = user makes gesture, system detects it
False positive = noise triggers gesture incorrectly
False negative = user makes gesture, system misses it

The calibration process is iterative because:
1. Users don't produce identical gestures each time
2. Fatigue, position, and environment affect acceleration
3. We want it to work 80-90% of the time across all conditions

## Power Consumption

During Real World testing the watch never went flat. The Technologist doing the testing reported that the battery life is 'good'. If I do find problems with this I'll look at updating the firmware to further extend battery life. I spent some time doing this prior to the initial deployment and testing.

### T-Embed Receiver

Can run from:
- USB power (plugged into computer)
- Optional internal battery (not included, but can add one)

Recommended: Keep plugged into the AAC device's computer 

## Firmware Architecture

The firmware is written in C++ using the Arduino framework. 

Task based architecture based on a simple scheduler. Using tasks allows for reconfigurable firmware. I've avoided using an RTOS to keep things as simple as I can. If I find I need the extra features of e.g. FreeRTOS or Zephyr then I'll port to one of those.

### Watch Firmware

Main loop (every 10ms):

  • Read accelerometer data
  • Run gesture algorithm
  • Check for triggers
  • Update display if needed
  • Send/receive ESP_NOW packets
  • Check button presses
  • Manage battery

### Receiver Firmware

Main loop (every 10ms):

  • Listen for ESP_NOW packets from watch
  • If gesture received:
    • Flash LED ring
    • Play audio
    • Vibrate
    • Send keyboard/switch signal
  • Check dial position
  • Check button presses
  • Manage display

## Why Accelerometer Instead of ML?

You might wonder: why not use fancy machine learning for gesture recognition?

**Good reasons to use ML**:
- Can learn complex patterns
- Adapts to individual variation automatically
- Sounds impressive

**Good reasons NOT to use ML (in this case)**:
- ML models are resource-intensive 
- Requires training data (users don't want to spend 30 minutes training)
- Black box (hard to debug when it fails)
- Overkill for simple binary decision ("is this a gesture?")
- We need real-time response (<50ms)

The simple algorithms work for this use case because:
- We only need binary output (trigger or not)
- The gesture is intentional (user controls the accelerometer)
- We calibrate per-individual (not one-size-fits-all)
- Threshold can be adjusted 

## Open Source & Customization

The entire firmware is available and can be modified for:
- Different algorithms
- Custom output protocols
- Different sensor configurations
- Integration with other systems

See the configuration site for code repositories and development documentation.

---

**Next:**
- [How to set it up →](./setup.md)
- [Why we built it this way →](./background.md)