MobileOcc: A Human-Aware Semantic Occupancy Dataset for Mobile Robots
1TU Delft *Equal Contributions
Abstract
Dense 3D semantic occupancy perception is critical for mobile robots operating in pedestrian-rich environments, yet it remains underexplored compared to its application in autonomous driving. To address this gap, we present MobileOcc, a semantic occupancy dataset for mobile robots operating in crowded human environments. Our dataset is built using an annotation pipeline that incorporates static object occupancy annotations and a novel mesh optimization framework explicitly designed for human occupancy modeling. It reconstructs deformable human geometry from 2D images and subsequently refines and optimizes it using associated LiDAR point data. Using MobileOcc, we establish benchmarks for two tasks: i) occupancy prediction and ii) pedestrian velocity prediction, using different methods, including monocular, stereo, and panoptic occupancy, with metrics and baseline implementations for reproducible comparison. Beyond occupancy prediction, we further assess our annotation method on 3D human pose estimation datasets, where results demonstrate robust performance across different datasets.
In the figure above, we show qualitative results across occupancy resolutions. Top: input image from the UT Campus Object Dataset (CODa). Middle: our semantic occupancy label at fine resolution (0.02 m). Bottom: semantic occupancy at coarse resolution (0.2 m). Gray voxels denote unknown regions, whereas free space is not shown for clarity. Unlike driving-oriented datasets that model humans as rigid boxes, MobileOcc preserves limb-level geometry that is critical for navigation in tight crowd gaps. Below we show the annotation pipeline of our method.
Overview of the MobileOcc pipeline. Given synchronized image streams and LiDAR along a robot trajectory, the pipeline performs data preprocessing, static mapping, human mesh optimization, and 3D occupancy representation generation. Static voxel colors follow the Cityscapes labeling scheme, human instances are assigned distinct colors, and unknown regions are shown in gray. In human mesh optimization, the green and red skeletons represent the ground-truth and the predicted 3D pose of the mesh, respectively.
Our main contributions include:
(1) We introduce MobileOcc, a 3D semantic occupancy prediction dataset built on raw sensor data from the UT Campus Object Dataset (CODa) for mobile robots navigating among near-field pedestrians.
(2) We develop an annotation pipeline that generates occupancy annotations from synchronized RGB images and dense LiDAR data. In particular, it leverages both image data and LiDAR point clouds through a training-free mesh optimization framework to produce accurate, non-rigid representations of human occupancy.
(3) We establish MobileOcc as a benchmark suite by evaluating existing occupancy frameworks on two tasks: i) semantic occupancy prediction and ii) pedestrian velocity prediction, with baselines across monocular, stereo, and panoptic occupancy methods.
Dataset
MobileOcc is annotated on the CODa dataset and output in the nuScenes format for easy integration with existing tools. We evaluate at 0.2 m voxel resolution over a grid spanning x ∈ [0.4, 10.0] m, y ∈ [−4.8, 4.8] m, z ∈ [−1.0, 3.8] m, downsampled to 5 Hz. The training/validation split contains 92,303 / 24,208 frames (79% / 21%), of which 30,457 / 7,165 contain pedestrian annotations. We use 10 semantic classes (one free-space and nine occupied: pedestrian, car, other-structure, pole, road, terrain, truck, two-wheeler, vegetation). Compared to existing driving and off-road occupancy datasets, MobileOcc is the first to provide non-rigid human occupancy with per-instance tracking for mobile robot navigation. We distinguish daytime and nighttime sequences, enabling users to select subsets matching their target deployment conditions.
To validate label quality, we compare our voxel labels against CODa ground truth on 4 sequences (400 frames). Static semantic voxels reach 71.2 mIoU (road / vegetation / terrain) with 96.8% ground-truth point coverage, and pedestrian voxels achieve 84.4% precision and 81.9% recall against 3D bounding-box references, confirming that our labels capture the majority of pedestrian geometry.
Results
We present qualitative results in the following. More quantitative results can be found in the paper.
3D occupancy prediction. We compare stereo (VoxFormer-T), monocular FlashOcc, and Panoptic-FlashOcc baselines under various lighting conditions, including sunny, night, and cloudy scenes.
Qualitative comparison of different baselines. Columns show the stereo (left + right) input, ground-truth occupancy, and predictions from VoxFormer-T, FlashOcc (8f), and Panoptic-FlashOcc (8f).
Panoptic occupancy. Beyond per-voxel semantics, Panoptic-FlashOcc predicts voxel-level panoptic labels, separating individual pedestrian instances within the scene.
Panoptic occupancy prediction using Panoptic-FlashOcc (8f). Columns show the stereo (left + right) input, ground truth, predicted semantic occupancy, and predicted instances.
Pedestrian velocity prediction. Finally, we extend the detection head of Panoptic-FlashOcc to predict per-pedestrian velocities (Panoptic-FlashOcc-vel). Directions are shown using colors and arrows.
Pedestrian velocity prediction using Panoptic-FlashOcc-vel (8f). Each panel shows the stereo-left input, ground-truth velocity, and predicted velocity. The model produces plausible velocities for most pedestrians while exhibiting typical failure cases, such as confusion between forward and backward walking directions.
Related Publications
Flow Policies as Actions of Skill-Level World Models: Learned and Symbolic Abstractions for Long-Horizon Planning
In arXiv preprint, 2026. Under review.
@misc{matoses2026flowskillwm,
title = {Flow Policies as Actions of Skill-Level World Models: Learned and Symbolic Abstractions for Long-Horizon Planning},
author = {Matoses Gimenez, Andreu and Papuc, Andrei-Carlo and Pek, Chris and Alonso-Mora, Javier},
year = {2026},
eprint = {2610.04767},
archiveprefix = {arXiv},
primaryclass = {cs.RO},
url = {https://arxiv.org/abs/2610.04767}
}
Learning Task and Motion Plans from Real Demonstrations with Hybrid Flow Matching
In arXiv preprint, 2026. Under review.
@misc{redondo2026hybridflow,
title = {Learning Task and Motion Plans from Real Demonstrations with Hybrid Flow Matching},
author = {Redondo Garcia, Zuleika and Matoses Gimenez, Andreu and Alonso-Mora, Javier},
year = {2026},
eprint = {2610.04771},
archiveprefix = {arXiv},
primaryclass = {cs.RO},
url = {https://arxiv.org/abs/2610.04771}
}
Building Explicit World Model for Zero-Shot Open-World Object Manipulation
In arXiv, 2026.
BEACON: Language-Conditioned Navigation Affordance Prediction under Occlusion
In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2026.
Cross-Entropy Optimization of Physically Grounded Task and Motion Plans
In IEEE Robotics and Automation Letters (RA-L), 2026.
SADCHER: Scheduling using Attention-based Dynamic Coalitions of Heterogeneous Robots in Real-Time
In IEEE Int. Symposium on Multi-Robot & Multi-Agent Systems (MRS), 2025.
Scenario-based motion planning with bounded probability of collision
In The International Journal of Robotics Research (IJRR), 2025.
TamedPUMA: safe and stable imitation learning with geometric fabrics
In Learning for Dynamics and Control (L4DC), 2025.
SHINE: Social Homology Identification for Navigation in Crowded Environments
In International Journal of Robotics Research (IJRR), 2025.
Overcoming Explicit Environment Representations with Geometric Fabrics
In IEEE Robotics and Automation Letters (RA-L), 2025.
Pushing Through Clutter With Movability Awareness of Blocking Obstacles
In IEEE Int. Conf. on Robotics and Automation (ICRA), 2025.
Globally-Guided Geometric Fabrics for Reactive Mobile Manipulation in Dynamic Environments
In IEEE Robotics and Automation Letters (RA-L), 2025.
Particle-based Instance-aware Semantic Occupancy Mapping in Dynamic Environments
In IEEE Transactions on Robotics (T-RO), 2025.
Evaluating Dynamic Environment Difficulty for Obstacle Avoidance Benchmarking
In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2024.
Topology-Driven Parallel Trajectory Optimization in Dynamic Environments
In IEEE Transactions on Robotics (T-RO), 2024.
Auto-Encoding Bayesian Inverse Games
In 16th Workshop on the Algorithmic Foundations of Robotics (WAFR), 2024.
Safe and stable motion primitives via imitation learning and geometric fabrics
In Robotics: Science and Systems, Workshop on Structural Priors as Inductive Biases for Learning Robot Dynamics, 2024.
Physically Grounded Optimal Realizations of Symbolic Plans
In Robotics: Science and Systems (RSS), Workshop on Frontiers of Optimization for Robotics, 2024.
Reactive grasp and motion planning for adaptive mobile manipulation among obstacles
In Robotics: Science and Systems, Workshop on Frontiers of Optimization for Robotics, 2024.
Multi-Modal MPPI and Active Inference for Reactive Task and Motion Planning
In IEEE Robotics and Automation Letters (RA-L), 2024.
Decentralized Multi-Agent Trajectory Planning in Dynamic Environments with Spatiotemporal Occupancy Grid Maps
In IEEE Int. Conf. on Robotics and Automation (ICRA), 2024.
Contingency Games for Multi-Agent Interaction
In Robotics and Automation Letters (RA-L), 2024.
Multi-Robot Local Motion Planning Using Dynamic Optimization Fabrics
In Proc. IEEE International Symposium on Multi-Robot and Multi-Agent Systems, 2023.
Learning to Play Trajectory Games Against Opponents with Unknown Objectives
In IEEE Robotics and Automation Letters (RA-L), 2023.
RAST: Risk-Aware Spatio-Temporal Safety Corridors for MAV Navigation in Dynamic Uncertain Environments
In IEEE Robotics and Automation Letters (RA-L), 2023.