BEACON: Language-Conditioned Navigation Affordance Prediction under Occlusion

Xinyu Gao¹ Gang Chen^1,† Javier Alonso-Mora¹

¹TU Delft ^†Corresponding Author

Abstract

Language-conditioned local navigation requires a robot to infer a nearby traversable target location from its current observation and an open-vocabulary, relational instruction. Existing vision-language spatial grounding methods usually rely on vision-language models (VLMs) to reason in image space, producing 2D predictions tied to visible pixels. As a result, they struggle to infer target locations in occluded regions, typically caused by furniture or moving humans. To address this issue, we present BEACON, which predicts an ego-centric Bird's-Eye View (BEV) affordance heatmap over a bounded local region including occluded areas. Given an instruction and surround-view RGB-D observations from four directions around the robot, BEACON predicts the BEV heatmap by injecting spatial cues into a VLM and fusing the VLM's output with depth-derived BEV features. Using an occlusion-aware dataset built in the Habitat simulator, we conduct detailed experimental analysis to validate both our BEV space formulation and the design choices of each module. Our method improves the accuracy averaged across geodesic thresholds by 22.74 percentage points over the state-of-the-art image-space baseline on the validation subset with occluded target locations.

+22.74 ppgeodesic accuracy over the best image-space baseline on the occluded subset

+10.68 ppover direct BEV point prediction fine-tuning

75K / 12Ktrain / validation, occlusion-aware dataset in Habitat 3.0

2.6%structurally invalid rate, one order of magnitude lower than all baselines

Task

The robot receives a free-form, spatially grounded natural-language instruction that refers to a nearby navigation target that can be reached without exploration. However, the target location can be occluded by static objects or by moving humans, and existing image-space methods struggle to infer occluded target locations, because they are tied to visible pixels.

Examples of language-conditioned local navigation under occlusion. The blue boxes mark the robot, the red boxes highlight humans and objects that cause occlusions, and the green boxes indicate target regions.

Method

Given a free-form, spatially grounded natural-language instruction and single-frame surround-view RGB-D observations, BEACON combines an Ego-Aligned Vision-Language Model with a Geometry-Aware Bird's-Eye View Encoder and predicts an affordance heatmap in ego-centric BEV via a Post-Fusion Affordance Decoder, where affordance denotes the score indicating how suitable each location is as a local navigation target; inference takes the argmax as the final predicted target point.

Specifically, Stage 1 adapts the VLM with learned 3D position embeddings and auto-derived ego-centric instruction tuning, converting the annotated target into a coarse direction-and-range answer, for example, "Move towards the FrontRight Region with a Big step." Stage 2 loads the Stage 1 weights, uses the final hidden state of a [NAV] special token as the instruction-conditioned summary embedding, and post-fuses it with Geometry-Aware BEV features to predict the heatmap, supervised as geodesic target regions rather than a single point.

Results

Our method performs best on both the full validation set and the occluded-target subset over different metrics, with 22.74 percentage points higher geodesic accuracy than the best image-space baseline and 10.68 percentage points higher than the direct VLM plus point-head baseline on the occluded subset. The structurally invalid rate is reduced to only 2.6 percent, one order of magnitude lower than all baselines.

Overall quantitative results on local navigation target prediction, comparing image-space baselines and BEACON on the full validation set and occluded-target subset.

Ablation study of key Ego-Aligned VLM and BEV-space design choices.

Qualitative Examples

Successful examples under heavy occlusion.

Failures due to landmark confusion or instruction ambiguity.

MobileOcc: A Human-Aware Semantic Occupancy Dataset for Mobile Robots
Junseo Kim, Guido Dumont, Xinyu Gao, Gang Chen, Holger Caesar, Javier Alonso-Mora. In European Conference on Computer Vision (ECCV), 2026.

abstract pdf web dataset

Dense 3D semantic occupancy perception is critical for mobile robots operating in pedestrian-rich environments, yet it remains underexplored compared to its application in autonomous driving. To address this gap, we present MobileOcc, a semantic occupancy dataset for mobile robots operating in crowded human environments. Our dataset is built using an annotation pipeline that incorporates static object occupancy annotations and a novel mesh optimization framework explicitly designed for human occupancy modeling. It reconstructs deformable human geometry from 2D images and subsequently refines and optimizes it using associated LiDAR point data. Using MobileOcc, we establish benchmarks for two tasks: i) occupancy prediction and ii) pedestrian velocity prediction, using different methods, including monocular, stereo, and panoptic occupancy, with metrics and baseline implementations for reproducible comparison. Beyond occupancy prediction, we further assess our annotation method on 3D human pose estimation datasets, where results demonstrate robust performance across different datasets.

Back to Publications