Training and running a tiny CNN on an ESP32-S3 — and using it to control a motor

End-to-end embedded ML on a $15 microcontroller: camera capture → on-device inference → brushless motor#

The idea: gesture recognition running entirely on an ESP32-S3 with a $2 OV7670 camera module — no external compute. Then use the inference output to drive a brushless motor through an ESC. Full embedded ML pipeline on a $15 microcontroller.

Two parts to it: training the model, and running it on-device against a motor.

The expanded Lab article documents the current binary sketch, its 10,336-byte model, five-label majority vote, and the separate experiments with more classes.

Step 1: the camera#

Step 1 was building camera drivers from scratch using bit-banged GPIO. I wrote that one up separately — the short version is that the OV7670 outputs raw parallel data, the capture code reads the upper four data bits in step with the camera's pixel clock, after configuring the camera's output registers. Result: 160×120 grayscale frames at ~5 FPS, ready to go into a model.

Step 2: collecting training data#

I captured training images directly from the device — same camera, same lighting, same framing it'll see at inference time. That avoided the classic "trained on a phone camera, deployed to an OV7670, surprised when it fails" pattern.

Two classes for the first iteration: open palm and closed fist. Binary classification is the cheapest first step, and once the loop works end-to-end you can scale up to more classes without rebuilding the pipeline.

Step 3: training the model#

→ Binary classification — open palm vs closed fist.

→ 4-layer CNN — small enough to fit on an ESP32-S3.

→ Int8 quantization — the deployed TFLite model is 10.8KB.

The main constraint is the ESP32-S3's tight memory budget — strong enough to overfit easily but not generous enough to forgive a fat architecture. Batch normalisation stabilises training. Aggressive data augmentation (rotation, zoom, translation, brightness, contrast) keeps the model honest against real-world variation. Int8 quantization is what makes the deployed model small enough to ship.

Step 4: motor control#

Added an ESC and a brushless motor on the same board. Inference output drives motor state, with a majority vote across five frame labels to reduce rapid switching from noisy predictions.

The vote makes the output steadier when my hand is between gestures. It also adds a delay when I change gestures, and it does not replace a timeout for camera or inference failures.

The result#

~4 FPS, ~40ms inference time per frame, controlling a motor with no cloud and no external compute — just the ESP32-S3.

The full embedded-ML loop is in place now: camera → frames → on-device CNN → real-time output driving hardware. Adding a new gesture is a matter of training, quantising, and reflashing — not rebuilding the pipeline.

Next: more training images, more gestures, and a wider control surface than "motor on / motor off."

→ More on the lab page: Gesture-controlled motor with on-device CNN.