XR Input Devices and Interaction Techniques: From Unimodal/Multimodal User Interaction to Intelligent User Interfaces

Overview

Given a headset and no instructions, what do people actually do? They reach, point, speak, and look, often at the same time. The hard part is that there is no single natural vocabulary waiting to be discovered. What people propose depends on how you ask them, what you show them, and what they have used before. We study that dependence directly, because an interaction technique built on the wrong assumption about it will not survive contact with a real user.

We run elicitation and observational studies, then build and measure the techniques they point to. Two results shape the rest: people recall gestures they invented themselves far better than expert-defined or elicitation-derived sets, and showing a referent as text pulls participants into repeating that text back in speech. In multimodal work we time the two channels against each other; the information-bearing gesture stroke lands within about ten milliseconds of speech onset. We then compare implemented controls, in which touch is often faster and more usable than gaze or voice even when people say they prefer gaze.

Designers get evidence in place of intuition: which gesture set to ship, when a second modality earns its cost, and when the simpler control wins. The same measurements feed the next step, interfaces that adapt to the person using them. Knowing that gaming experience reverses which input performs better, and that a more responsive system is not automatically a better one, is what keeps that adaptation honest.