After getting the OV7670 camera working, I used its images to recognize an open hand or a closed fist on the ESP32-S3. The prediction controls a brushless motor through an ESC. Training happens on the laptop; camera capture, image processing, inference, and the motor decision run on the board.
From a picture to a motor command#
I wanted to try the whole embedded machine-learning process on a small project: collect images, train a model, put it on a microcontroller, and use the result to control something physical.
The first version has two gestures. An open hand starts the motor, and a closed fist stops it. That gave me a simple output to test while I worked through the camera, preprocessing, and model conversion.
OV7670 frame
|
v
Unpack grayscale pixels
|
v
Median filter + contrast adjustment
|
v
TensorFlow Lite Micro inference
|
v
Vote across five predictions
|
v
ESC motor command
The laptop viewer is useful for seeing what the camera and model are doing, but it does not make the motor decision.
Collecting images from the device#
I reused the OV7670 capture code, which produces 160×120 images with 16 brightness levels. The collection tool unpacks the serial stream and saves images into folders named after the gesture.
Using the actual camera meant the training data included the same limited detail and image noise as the deployed input. I still needed variation in position, lighting, and background; using the same camera only removes one source of difference.
The collector also has a burst mode that saves ten frames at a time. That is convenient for gathering examples, but neighbouring frames are often nearly identical. For a stronger evaluation, I would hold out complete recording sessions instead of relying only on a random split of individual images.
Keeping preprocessing consistent#
The board first expands the packed frame into one byte per pixel. It then applies a 3×3 median filter and histogram equalization before copying the image into the model's input tensor.
The median filter reduces isolated pixel noise. Histogram equalization spreads the image's brightness values across the available range. The training scripts apply the same two operations with OpenCV, so the network is trained on images prepared in the same way.
This is an easy place to introduce a mismatch. Changing the desktop preprocessing without changing the firmware can give the model quite different input, even though both sides still use a 160×120 grayscale image.
The model and its memory budget#
The binary gesture sketch includes the converted model as a C byte array in gesture_model.h. The copy used by that sketch is 10,336 bytes, or about 10.1 KiB. My earlier build note rounded a different reported size to 10.8 KB; the embedded header gives the exact size for this version.
The model file is only part of the memory use. The sketch also reserves a 64 KiB tensor arena for TensorFlow Lite Micro, a 9,600-byte packed camera buffer, a 19,200-byte unpacked image, and another image-sized temporary buffer for filtering.
The arena allocation tries PSRAM first and falls back to ordinary byte-addressable heap. At startup, the sketch prints the actual arena use, input dimensions, and output quantization parameters. Those diagnostics help distinguish an allocation problem from a model that loads successfully but predicts poorly.
The interpreter registers only the operations needed by the binary model. Model conversion therefore involves checking more than the file size: the tensor types, output interpretation, and available operators all need to agree with the firmware.
Turning predictions into a decision#
The binary model returns a quantized value. The sketch uses the tensor's scale and zero point to convert it into a probability:
float probability =
(raw_output - output_zero_point) * output_scale;
A threshold of 0.5 assigns an open-hand or fist label. The firmware then keeps the last five labels and counts how many are open hands. Three or more open-hand labels start the motor; otherwise it stops.
This is a majority vote, rather than an average of the probability values. That distinction matters when reading the code or trying to change how responsive it feels.
The sketch waits until it has five predictions before making its first motor decision. Once the history is full, each new frame replaces the oldest label. This reduces rapid changes around the decision boundary, at the cost of a delay when I change gestures.
The ESC connection#
The ESC signal is on GPIO38. The sketch configures a 50 Hz, 14-bit PWM output, sends its minimum setting, and waits three seconds during startup before beginning camera and model setup.
The motor has two configured output levels in this version: minimum and a fixed running level. It is not proportional speed control. Pausing the stream also sends the minimum setting and clears the prediction history.
There are still limits to that control path. In particular, a failed inference does not currently force the motor off, and the camera's blocking capture loop can delay further decisions. The voting helps with noisy classifications, but it does not handle every failure of the input or processing. A timeout that returns the output to minimum would be an important next change.
Seeing what the board is doing#
The serial stream contains the image plus the predicted label, confidence, heap figures, inference duration, and frame rate. The viewer can show all of these together, which makes it easier to tell whether a problem starts in the camera image or later in the model.
In the original motor test I saw roughly 4 FPS, with inference taking about 40 ms per frame. That inference time covers the model invocation; a complete cycle also includes capture, filtering, and serial output.
The confidence shown in the viewer is the model's output for its chosen class. It is useful for debugging, but it is not a measurement of accuracy across different people and rooms.
Experiments with more gestures#
The project now also has separate three-class and six-class training experiments. The six-class version covers a fist and one through five fingers, with a separate firmware sketch and a larger 96 KiB arena.
I keep that separate from the binary motor example because the output changes from one probability to several class scores. Replacing the model header alone is not enough; the firmware has to interpret those outputs correctly too.
The next useful work is collecting more varied sessions, comparing the converted model against the desktop model on the same images, and adding a clear no-hand or uncertain state. I would also improve the motor timeout before giving the gestures more control over the output.