How non-verbal behavior signals enhanced multimodal AI interaction

A dual-phase Wizard-of-Oz study triangulated with semi-structured interviews and surveys to map user behavior in multimodal LLM interactions. Findings directly informed the design space for rapid feature iteration and shaped the future product roadmap.

RoleResearch Scientist

Timeline6 month

MethodsWizard of Oz Study, interviews, surveys, qualitative coding

Team1 Product Manager, 1 Junior Researcher

Context & Problem

Multimodal LLMs are empowering the next generation of voice agents

With the rapid rise of LLMs and wearable smart devices like AR glasses, users now have unprecedented opportunities to interact with on-device assistants through both voice and gesture. While Voice User Interfaces (VUIs) are well-understood, the potential for full-body gestures that contextualize a user’s surroundings remains largely unexplored.

At Meta, we are heavily investing in immersive hardware, from VR headsets like the Oculus Quest 3 to AR glasses such as Meta Ray-Ban, which all support rich multi-modal inputs including facial expressions and body gestures alongside voice. It only makes sense to push further by deepening our understanding of how these smart devices can interpret user intention through the integration of this new data with traditional voice inputs.

Rather than jumping straight into designing specific interaction techniques for our voice agents, I took a step back and advocated to establish the fundamentals first. Before we can innovate on what users interact with, we must understand who they are and how they naturally communicate when given multi-modal tools. Just as we use body language daily to convey intent without words, we needed to de-risk future development by validating how users naturally gesture with voice assistants. This foundational work is essential not only for technical feasibility but also for proving real consumer desirability before scaling our product roadmap.

Research Questions

What we needed to learn

To truly understand the user landscape, I structured this research around three key pillars. The goal was not just to gather data, but to uncover users' natural behaviors before any innovation is introduced and identify the underlying motivations and rationales driving those actions.

1. Opinions: What are users' general attitudes towards interacting with multi-modal VUIs?
2. Preferences: How do users perceive the use of gestures in multimodal VUIs, and what specific preferences guide their choices?
3. Gestures: Which implicit (unconscious) and explicit (conscious) gestures are commonly used when interacting with multi-modal VUIs, and what functions or intents do these gestures serve? :::

::: sec-label "Methods & Process"

A Two-Phase Wizard-of-Oz Study

To investigate future technology without the heavy resource cost of full implementation, I used the Wizard-of-Oz (WoZ) methodology. This approach allowed us to simulate advanced capabilities and answer our core research questions efficiently with timeline and resource constraints. To ensure these findings translated into actionable product insights, I organized internal critique sessions with senior research leadership and cross-functional teams. Additionally, we conducted iterative design workshops where designers and developers tested the generated design space, providing rapid feedback on the co-design experience to accelerate feature iteration.

::: methods-grid

Phase 1: Discovery Wizard-of-Oz Study

I recruited 6 participants in a simulated environment to mock up six daily voice assistant usage scenarios based on common tasks identified through my literature review. This initial phase yielded essential categories of gesture functions and uncovered four key factors influencing how users interact with voice assistants via gestures.

Phase 2: Validation Wizard-of-Oz Study

To validate these findings and refine the initial design space, I conducted a second WoZ study with 12 participants in a fully staged environment designed to mimic real-life situations. I confirmed the four influential factors and built a robust design space featuring four validated categories of gestures specifically for conveying user attention.

Stakeholder Alignment

Throughout both phases, I held weekly sessions with senior research leadership and multifunctional stakeholders. These meetings ensured the study results were not only methodologically sound but also directly applicable to our business objectives.

Internal Design Workshops

I facilitated workshops where cross-functional teams co-designed interaction flows. These sessions focused on rapid prototyping and feedback integration, allowing us to iterate quickly before moving into engineering implementation.

Findings → Implications

Three findings, three product implications

Each key finding was traced directly to a product implication for future development, ensuring the research had measurable impact.

Users recognize the benefits of using gestures with voice assistance.

Finding: 80% of participants confirmed they prefer using gestures alongside voice, particularly in noisy environments where voice quality is insufficient for smooth communication. Four factors influence this preference: system knowledge, social acceptance, habits, and effort. Notably, all participants identified system knowledge and social acceptance as the most critical drivers.

Implications: To promote future products using voice and gestures together, priority should shift from mere feature implementation to helping users build an intuitive mental model of how gestures interact with voice. Simultaneously, providing the public with more information can increase social acceptance for using these combinations in everyday scenarios.

Users can learn and retain gestures with voice assistants.

Finding: A strong learning effect was observed during the study: users who learned to use gestures under noisy conditions continued using them even in quiet scenarios where voice quality was not compromised. This demonstrates that the behavior is not just context-dependent but becomes a retained habit.

Implications: Since gesture usage can be learned and maintained over time, it is crucial to provide a smooth onboarding experience and actively encourage users to adopt gestures from the very beginning of their interaction with a new product. Without this early reinforcement, users are likely to revert to voice-only interactions.

The most intuitive gestures mimic real-life object manipulation.

Finding: Participants spontaneously invented similar gestures to give instructions to voice assistants without any pre-training. These self-developed gestures mimicked interactions with physical objects, such as screwing a bottom cap to adjust volume or writing on paper, that users naturally map digital commands onto familiar real-world actions.

Implications: The system should be designed to accommodate users who will naturally use customized gestures mimicking daily physical interactions. When creating interaction techniques for the new product, designers must translate these everyday physical actions into effective gesture-and-voice design.

Outcome

Actionable design framework and insights driving future development.

Delivered within a strict six-month timeline, this project successfully transitioned from zero to one by investigating user intentions for gesture-voice integration, establishing a foundational understanding of motivation and rationale. The research synthesizes an actionable design framework to accelerate feature iteration while culminating in a peer-reviewed publication accepted at a top-tier HCI venue.

Empirical Insights User preference and attitudes toward future products.
Categorized gestures Explicit gestures organized into 4 functional categories.
Product Implications 3 essential implications for future product development.
Design Framework Streamlined, structured workflow for rapid iteration. :::

::: sec-label "Reflection"

Honest tradeoffs and what I'd do differently

Every project involves compromises in scope, depth, and timing. Here's what I would change if revisiting this work:

::: reflection-list

Accelerate context learning. In the first month, I spent significant time reviewing internal documents and prototyping two interaction concepts before engaging deeply with the team. Going forward, I would prioritize early conversations with senior leadership and project managers to better align research goals with broader company objectives from day one.

Expand participant recruitment and depth. Given the tight timeline, recruiting was challenging. However, there is clear opportunity to target specific age and experience-level cohorts. I would integrate semi-structured interviews alongside surveys to uncover deeper insights into user preferences and motivations.

Leverage team resources more effectively. I spent significant time on solo efforts like literature reviews and building custom software to support the WoZ study. In hindsight, assigning these supporting tasks to research staff would allow me to focus on core questions, implement richer features, and directly influence the product roadmap. :::