INSIGHT: Smart Assistive Glass for Visually Impaired
This project was completed for EECS 473 Advanced Embedded System at UM, which also serves as a Major Design Experience (MDE) for Computer Engineering undergraduates. As the sequel of EECS 373, EECS 473 goes deeper into topics including FreeRTOS, Embedded Linux, and PCB design, with the team project as a significant component. Unlike EECS 373, where projects often emphasize technical difficulty for its own sake, EECS 473 requires building a system with real engineering applications—something useful in practice (and, unsurprisingly, still technically challenging).
Motivated by the recent rise of AR/VR wearables such as Meta Ray-Ban and Apple Vision Pro, our team chose to explore a more impact-driven (and timeline-feasible) direction: assistive technology for people with visual impairments. We built a smart-glasses prototype that delivers two core capabilities: obstacle-aware navigation and interactive scene description, providing in-the-moment guidance through audio output and tactile (vibrotactile) feedback. For full technical details and system goals, please refer to our formal proposal.
Concept of Obstacle-aware navigation (left) and Scene description (right). (Figure generated by AI)
In summary, what INSIGHT does is the following:
- Hands-free interaction: the user issues a spoken request, the system interprets intent (navigation v.s. scene description), then returns guidance through speech and vibration.
- Obstacle-aware feedback: live camera frames are processed for obstacle detection, and the wearable receives vibromotor commands for real-time cues.
- Scene description on demand: the user can request a natural-language description of what’s ahead, generated locally by a vision-language model.
- Offline-first by design: no raw audio/video is sent to cloud services; compute stays on the device network (ESP32 ↔ Jetson).
System architecture
INSIGHT is split into two parts: a lightweight wearable smart-glass module and a nearby Jetson Orin Nano base station that runs the heavier AI workloads.
Wearable module (ESP32-S3 + custom PCB)
We designed a custom PCB integrating an ESP32-S3, OV2640 camera, MEMS microphone array, speaker, and vibration motors. Firmware runs under FreeRTOS, scheduling five concurrent tasks: wake-word detection, audio capture/streaming, ~30 fps QVGA video acquisition/streaming, audio playback, and vibromotor control. Because of ESP32 resource limits, we stream JPEG-compressed frames and keep audio/video handling lightweight and robust.
Block diagram of smart-glass module
Base station (Jetson Orin Nano)
The Jetson handles speech and vision pipelines locally and communicates with the glasses through four TCP ports:
- Port 1: audio commands from ESP32 (WAV)
- Port 2: video frames from ESP32 (JPEG)
- Port 3: audio feedback back to ESP32 (MP3)
- Port 4: vibromotor commands as formatted strings
Block diagram of base station
Demo Video and Results
We have created a demo video showing the working pipeline of obstacle aware navigation and scene description feature.
PCB Design and Debug
Two photos of the PCB … As expected, the first PCB came and cannot work properly. It’s been a fun journey figuring out the mistakes I’ve made while designing it. Two takeaways would be
- Double check the pin mapping of the footprint in your library. Some of them are not what you expected (e.g. FPC connector in the reverse order), some are even wrong (e.g. pin 1 on the physical chip is not labelled as pin 1).
- Take power integrity seriously, and be careful with switching regulators. For example, if the battery lifetime is not a big concern, use an LDO instead of a buck converter. As shown in the following picture, the original board with a buck converter never worked (ESP32 brownout when WiFi started). But by replacing it with an LDO, everything magically works fine.
PCB Version 1
After one weeks of debugging and redesigning, we finally got out second version of PCB on time. Fortunately, everything went fine, and there’s no need to jump wires (or even boards) on it. 
PCB Version 2
Project Documentation
More results can be found in our poster (PDF) at the CoE Design Expo, and the final report (PDF) of our course. If you are interested, the implementation details can be found at the following repositories:
Acknowledgements
Special thanks to my fantastic teammates: Haobo Fang, Jinlin (Jack) Li, Yizhe (William) Shen, Zhuoyang Chen, and Ruopu Dong for their collaboration and dedication throughout the project. We’ve spent a busy but unforgettable semester working together. We would also like to thank Dr. Mark Brehob, Matt Smith, James Carl and all of our GSIs, Frederick, Anna, and Alec for their help. This work is funded by Infineon.