skip to contentW3CSVG

HandShake: Gesture Recognition for AAC

HandShake turns any deliberate movement into a switch signal for assistive technology. If someone can make an intentional movement—a hand shake, a foot tap, a head nod—they can now control communication software, environmental controls, gaming systems, or any other switch-adapted device.

Latest Updates

August 2026: The configuration site is live! You can now buy the components directly and set everything up yourself through the web interface. No need to wait for involvement from the developer.

July 2025: Successful testing at National Star in Gloucester confirmed HandShake works in real-world settings. One user is now using it as their primary AAC access method, and three gamers successfully used it as an Xbox adaptive controller.

What is HandShake?

Many people who cannot use standard controllers (buttons, joysticks, eye-tracking) or existing AAC access methods can still make deliberate movements. HandShake lets them use those movements to:

  • Talk: Control speech software like Grid 3, Predictable, JABtalk
  • Play: Use gaming systems like Xbox
  • Interact: Control smart home devices like Alexa-connected lights, fans, curtains

Current Version: v3 (Lilygo Hardware)

HandShake v3 uses two commercially available devices that are well-made, affordable, and easy to source:

Total cost: around $100 for a complete system.

Lilygo T-Watch S3 and Lilygo T-Embed

How It Works

  1. Watch detects movement: The accelerometer on the watch continuously monitors for intentional gestures using one of two algorithms (the “punch” and “roll” algorithms)

  2. Wireless signal sent: When a gesture is detected, the watch sends a radio trigger to the T-Embed receiver using ESP_NOW protocol

  3. Haptic & audio feedback: The watch vibrates and the receiver gives audio feedback to confirm the gesture was recognized

  4. Signal to AAC device: The receiver sends:

    • A keyboard key press (for software control)

  5. Response: The software or device responds to the command

The sensitivity threshold is easily adjustable using the dial on the receiver. You can tune it on the fly to control how soft or hard a gesture is detected.

T-Watch running HandShake firmware

Key Features

Customizable gesture detection: The firmware currently supports two different detection algorithms. You can switch between them or write new ones for different movement patterns. The algorithm that works brilliantly for one person might not work for another. HandShake adapts to the individual, not the other way around.

Multiple units simultaneously: Want dual-switch control? Wear a watch on each wrist. They won’t interfere with each other.

Adjustable sensitivity: Fine-tune the trigger threshold for the individual’s needs. Takes about 30 seconds.

Multi-function receiver button:

  • Long press: Save new sensitivity settings to the watch
  • Double tap: Test the connection to AAC software

Feedback: Vibration motor on the watch, LED ring and audio feedback on the receiver (currently set to a stick click sound requested by a Technologist), so users know when a trigger is sent.

Getting Started

  1. Buy components: Lilygo T-Watch S3, T-Embed, and (optionally) the PCB adapter
  2. Configure online: Visit handshaketec.com to select your settings and download the firmware
  3. Load firmware: Flash the firmware to both devices via USB
  4. Calibrate: Adjust sensitivity dial to match the user’s movement

See the Setup Guide for detailed instructions.

Technical Details

Want to know how the algorithms work? How does gesture recognition happen on a tiny microcontroller? Check out the Technical Deep Dive.

Design Evolution

HandShake didn’t start with Lilygo hardware. It evolved through several iterations—Leap Motion cameras, XBee radio modules, BBC micro:bits—before landing on this current solution. Each iteration taught valuable lessons about what works (and what doesn’t) in the Real World.

If you’re interested in the “why” behind these choices, the user-centered design process, and lessons learned along the way, see Background & Design Evolution.

Research & Validation

HandShake has been evaluated through user testing at specialist colleges with people who have cerebral palsy and other motor conditions. The system is in regular use by participants for both AAC and gaming.

Questions?

This is an active project. If something doesn’t work, let me know. If you build it successfully, I’d love to hear about it.

Next steps: