Sense: Desktop AI Companion
An ESP32-based AI companion with real-time voice, on-device vision, and proactive assistance — built for neurodivergent users who need gentle, attentive support.
I built an ESP32-based AI companion to address the need for proactive, on-device AI assistance that doesn’t require constant user input. Commercial AI assistants often process data in the cloud, which raises privacy concerns that this project directly addresses by keeping sensitive information local. The project specifically aims to help neurodivergent people by noticing details they might miss, providing a supportive digital presence that complements rather than replaces human interaction. This approach creates a more accessible and responsive AI experience that respects user privacy while offering practical assistance.
The project evolved through several iterations, starting with “xiaozhi,” then moving to an “agentic” approach before settling on the current implementation. The final architecture uses a SenseCAP Watcher device, which costs approximately $30, as the hardware foundation. For voice functionality, I integrated LiveKit, a real-time communication framework, while Hermes Agent handles the system’s orchestration and decision-making processes. The entire system is designed with privacy as a priority, processing data locally on the device whenever possible to minimize external data transmission.
The voice integration using LiveKit proved remarkably robust throughout testing, with minimal latency and clear audio quality even in noisy environments. The on-device processing approach successfully maintained user privacy while still providing useful functionality, as sensitive information never left the ESP32 hardware. The modular architecture proved its worth when iterating between versions, allowing components to be swapped and new approaches tested without rebuilding the entire system.
The vision component remained experimental throughout development, never quite achieving the reliability needed for consistent functionality. I tried several approaches including edge detection and object recognition, but each consumed too many resources for the ESP32’s limited processing power. Balancing performance with the hardware constraints proved particularly challenging, as the 320KB of RAM and 400MHz processor forced constant trade-offs between feature complexity and system stability. Implementing long-term memory required careful consideration of storage limitations, with the 4MB flash memory filling up quickly when storing conversation history and user preferences.
The complete codebase for this ESP32-based AI companion is available at the GitHub repository, containing all firmware, configuration files, and documentation for replication. The LiveKit integration for voice processing, Hermes Agent orchestration code, and experimental vision component are in separate directories for easy reference. A working demo shows voice interaction capabilities and basic vision functionality. For those interested in privacy-first AI approaches or assistive technologies for neurodivergent users, the project documentation includes setup instructions and troubleshooting tips for building your own version or contributing to the codebase.