Lip-to-Speech Communication Device

An assistive-device prototype that captures lip movement with a small camera, converts it to text and speech on a host computer, and returns the output to a wearable unit.

Technologies Used

Assistive TechnologyComputer VisionESP32-S3Speech SynthesisPCB DesignAccessibility
⋮⋮

Table of Contents

The problem

This project asks whether silent lip movement can become a practical input method for someone who cannot produce audible speech. The concept is a small camera-based device that watches the mouth, sends the image sequence to a recognition model, and returns synthesized speech with local feedback.

I approached it as an assistive-technology prototype rather than a claim that visual speech recognition is solved. Lighting, camera position, accents, and visually similar mouth shapes all affect accuracy, so the hardware and software were designed to be replaceable as the models improve.

Device hardware

The custom board centers on an ESP32-S3 and an OV5640 camera module. Two LiPo cells in series provide about 7.4 V. A buck converter first reduces that to 5 V efficiently, and a low-dropout regulator then produces a quieter 3.3 V rail for the processor and camera.

That two-stage supply was an intentional tradeoff. A single linear regulator would waste too much energy from the battery, while a switching rail alone could introduce noise near the camera and processor. The ESP32-S3 provides Wi-Fi for the image stream and handles the final output to the attached audio or vibration transducer.

Processing pipeline

The wearable board is responsible for capture, transport, and feedback; a laptop performs the expensive inference:

  1. The OV5640 records the user’s mouth region.
  2. The ESP32-S3 streams frames over Wi-Fi.
  3. A visual-speech model converts the frame sequence into text.
  4. A text-to-speech model renders the recognized phrase.
  5. The result is returned to the device for output through the transducer.

Splitting the system this way kept the wearable hardware small and avoided forcing a large model onto the microcontroller. It also made it possible to test different recognition and speech models without redesigning the PCB.

What I learned

The project made latency an end-to-end problem. Camera capture, wireless transfer, sequence length, inference time, speech synthesis, and playback each add delay. Improving only the model is not enough if the conversation still feels slow. A future revision would focus on tighter camera positioning, confidence-aware correction, and testing with the people the device is intended to support.

The earlier source repository is no longer public, so I removed its broken link.