Embodied Referring Expression Comprehension in Human-Robot Interaction
Aman Chadha
Abstract
A large-scale dataset and multimodal model improve embodied interaction comprehension in robots by addressing perspective bias and enhancing multimodal signal integration.
As robots enter human workspaces, there is a crucial need for them to comprehend embodied human instructions, enabling intuitive and fluent human-robot interaction (HRI). However, accurate comprehension is challenging due to a lack of large-scale datasets that capture natural embodied interactions in diverse HRI settings. Existing datasets suffer from perspective bias, single-view collection, inadequate coverage of nonverbal gestures, and a predominant focus on indoor environments. To address these issues, we present the Refer360 dataset, a large-scale dataset of embodied verbal and nonverbal interactions collected across diverse viewpoints in both indoor and outdoor settings. Additionally, we introduce MuRes, a multimodal guided residual module designed to improve embodied referring expression comprehension. MuRes acts as an information bottleneck, extracting salient modality-specific signals and reinforcing them into pre-trained representations to form complementary features for downstream tasks. We conduct extensive experiments on four HRI datasets, including the Refer360 dataset, and demonstrate that current multimodal models fail to capture embodied interactions comprehensively; however, augmenting them with MuRes consistently improves performance. These findings establish Refer360 as a valuable benchmark and exhibit the potential of guided residual learning to advance embodied referring expression comprehension in robots operating within human environments.
Community
The paper introduces Refer360, a comprehensive multimodal dataset for embodied referring expression comprehension in human-robot interaction (HRI), and proposes MuRes, a lightweight guided residual module that selectively reinforces modality-specific features to improve multimodal grounding performance in real-world scenarios.
โก๏ธ ๐๐๐ฒ ๐๐ข๐ ๐ก๐ฅ๐ข๐ ๐ก๐ญ๐ฌ ๐จ๐ ๐ญ๐ก๐ ๐๐๐๐๐ซ๐๐๐ ๐๐๐ง๐๐ก๐ฆ๐๐ซ๐ค + ๐๐ฎ๐๐๐ฌ ๐๐จ๐๐ฎ๐ฅ๐:
๐ง ๐น๐๐๐๐๐๐๐: ๐ญ๐๐๐๐ ๐ฌ๐๐๐๐
๐๐๐
๐น๐ฌ ๐ซ๐๐๐๐๐๐ ๐๐๐๐ ๐ด๐๐๐๐-๐ฝ๐๐๐, ๐ด๐๐๐๐-๐บ๐๐๐๐๐ ๐ด๐๐
๐๐๐๐๐๐๐: Introduces a dataset with synchronized egocentric and exocentric views, RGB, depth, infrared, 3D skeleton, eye gaze, and audio, across indoor and outdoor environments. With 13,990 annotated interactions (3.2M frames), it overcomes biases in existing datasets (e.g., single view, indoor-only, no gesture/gaze integration).
๐ ๐ด๐๐น๐๐: ๐ฎ๐๐๐
๐๐
๐น๐๐๐๐
๐๐๐ ๐ฉ๐๐๐๐๐๐๐๐๐ ๐๐๐ ๐ด๐๐๐๐๐๐๐
๐๐ ๐ญ๐๐๐๐๐: Proposes a novel residual architecture that uses cross-attention to guide modality-specific signals (visual/language) through an information bottleneck, preventing feature dilution during fusion and outperforming both vanilla residuals and attention-only fusion across 4 datasets.
๐ ๐บ๐๐๐๐๐๐๐๐๐๐ ๐ฎ๐๐๐๐ ๐๐๐๐๐๐ ๐ฏ๐น๐ฐ ๐๐๐
๐ฝ๐ธ๐จ ๐ป๐๐๐๐: On Refer360, integrating MuRes into CLIP improved IOU-25 by +3.4%, and on CAESAR-PRO by +4.99%. For broader VQA tasks like ScienceQA and A-OKVQA, MuRes boosted model accuracy by up to +30%, highlighting its generalization ability across task domains.
Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper