Voice Assistant on Ray-Ban Stories
Designing the voice assistant for Meta’s 1st generation smart glasses
Date
April 2020 - September 2021
What I did
Conversation design, interaction design, prototyping, user research
Background
At Reality Labs (previously known as Meta’s AR/VR division), I had the opportunity to do conversation design for the Assistant product on Meta's first generation of smart glasses.
The main goal of this project was to bring the Assistant product to life on a 0-to-1 hardware product. My role was designing the handsfree capture experience and the voice interaction model for a smooth launch. 🚀
At launch, we supported taking photos and videos on the glasses with your voice, and quickly followed up with adding basic utility functions, device and media control, calling, and messaging.
Handsfree Capture
My main responsibility on this project was to ensure that voice-enabled capture worked seamlessly and consistently.
Handsfree capture was the most important selling point and differentiator of the glasses: capturing moments from your POV while still staying present in the moment.
How It Works
There is a hardware capture button on the top of the glasses’ right arm, which allows you to take photos and videos. However, since many users are trying to use these glasses to capture when their hands are busy (maybe they are riding a bike, holding their cat up, or have messy hands covered in clay), using your voice to capture allows users to truly stay present in the moment.
Capture Sound Design
Because there is no visual display on these glasses, the user experience relies heavily on voice and audio feedback. With the handsfree capture feature in particular, communicating the state of the camera was very important - both for privacy concerns, and to get great captures! I worked very closely with our sound designers and user researchers to iterate and improve on our capture sound design.
Initially, we used a traditional shutter sound for photos, but early testing revealed users would often end up with blurry photos, especially in low-light conditions. This was because users would hear the shutter, assume the photo had been taken, and then move their head.
But what was actually happening was that the shutter goes off when the camera opens and the capture begins, not when the capture is complete. In sunny outdoor conditions, this delta is negligible, but for many indoor situations, the exposure time could be much longer.
To address this, we explored various alternatives and settled on a solution with two distinct sounds—a start and end earcon, similar to how Night Mode photo capture works on iOS. This is meant to communicate to users, “hey, hold tight until you hear the end of the capture”.
This definitely helped users keep their head still for photos, was more clear about when the camera is open, and ultimately led to better photos.
Offline Assistant
One of the biggest turning points in the project (which actually pushed out our launch date) wasn’t until we had more beta users trying the glasses in their regular life day-to-day, and not just in one-off test sessions. Basically, the glasses were only working well when all conditions were perfect: your glasses and phone were connected with Bluetooth, and your phone had internet.
Given that these are sunglasses, meant for users on the go, we had to shift our design mentality to anticipate and expect real-world scenarios— where unstable Bluetooth connections, unreliable internet, thermal constraints, and low battery were all very real things, not edge cases. To address these common issues, we developed a smaller, offline version of the assistant that is used if your glasses get disconnected from your phone. This ensured that no matter if you’re in your living room playing with your kids, or on a mountain bike in some remote forest, you can still expect a consistent handsfree capture experience.
Handsfree Capture in the real world
The range of user feedback for how people would use handsfree capture was both really exciting and heartwarming. I hoped that this could be a new form of documenting special moments that might have felt like your phone was just getting in the way or ruining the moment.
Parents of young children were particularly excited about capturing more organic moments of their kids without always having their phone out. We even heard from users who previously had issues with traditional cameras due to accessibility issues, and were thrilled they could take photos with their voice.
Post Launch
After our exciting September launch, we soon followed up with adding additional functionality that we knew our users had been asking for: namely calling and messaging contacts with your voice, across Messenger, Whatsapp, and phone contacts. I worked on transitioning this work over to new designers joining the team, as I moved on to the wearables project.