INTERACT: Intuitive Interaction for Robots among Humans
About the Project
INTERACT aims to equip mobile robots with the ability to navigate and operate safely in human-populated environments. Leveraging advancements in motion planning, multi-robot task assignment, and machine learning, this project seeks to overcome the challenges of modeling intuition and ensuring safety in complex, uncertain settings. By developing intuitive models from past interactions and integrating them into novel optimization methods, INTERACT will enable robots to perform seamless, interaction-aware navigation and task planning. This foundational work paves the way for a new era of automation in both industrial and urban settings, where robots and humans can coexist harmoniously.
INTERACT works mainly on three parts: 1) Perception of the dynamic and open-world environment; 2) Task and motion planning (rename, Andreu); 3) Motion planning and control (rename, Saray).
Perception of the dynamic and open-world environment
The perception system consists of a local mapping layer and a global mapping layer. The local mapping layer focuses on modeling ego-centric local environments under highly dynamic conditions, where moving entities such as pedestrians continuously change the scene structure. To address this challenge, we first proposed a particle-based occupancy representation named the DSP Map, which models dynamic environments from RGB-D point cloud observations. The DSP Map is built upon Random Finite Set (RFS) theory and the Sequential Monte Carlo Probability Hypothesis Density (SMC-PHD) filter, enabling unified probabilistic modeling of dynamic objects with arbitrary shapes. To further improve robustness and reconstruction accuracy, we extended this framework by incorporating semantic information and proposed the Semantic-DSP Map. The method introduces an S2MC-PHD filter that jointly handles uncertainty from both depth observations and semantic predictions, enabling more reliable dynamic environment representation in complex real-world scenes.
The global mapping layer focuses on open-world object recognition and long-term environment representation. We proposed OpenSGA, a framework for constructing 3D scene graphs from posed RGB-D observations. Building upon these scene graphs, we developed a graph alignment framework based on Graph Neural Networks (GNNs) and attention mechanisms to establish object correspondences across different observations. This enables the system to re-recognize previously observed objects by jointly leveraging open-set visual-semantic features and contextual scene relationships.
Task and motion planning
We develop task and motion planning (TAMP) algorithms that enable robotic systems to autonomously decide high-level actions along with the associated motions. Previous research into TAMP algorithms has several simplifications and often does not account for robot dynamics or limitations of low-level controllers. This project uses GPU-based physics simulators to find high-quality and low-cost realizations that can be directly implemented on the real system, as they account for complex interactions with the environment. More information can be found in the related publication: Parallel Realization of Task and Motion Plans.
Two examples are shown below: (1) a pick-and-place problem (move, pick, move, place, move) with a central obstacle that can yield more efficient solutions, and (2) a move-and-push problem where the robot must find the best way to push a stick with an object on top so that it reaches its symbolic goal (inside the box).
(1) Pick and place
Simulation
Cross-entropy evolution and simulated solution for two different obstacles.
Real robot
Real-robot execution.
(2) Move and push
Simulation
Cross-entropy evolution for two stick locations and simulated solution.
Real robot
Real-robot execution.
Motion planning and control (rename, Saray)
We also model interactions between agents with game theory. Details can be found here.
Project Demonstrations
Funding & Partners
This project has received funding from the European Union through ERC, INTERACT, under Grant 101041863. Views and opinions expressed are, however, those of the author(s) only and do not necessarily reflect those of the European Union. Neither the European Union nor the granting authority can be held responsible for them.
Related Publications
Flow Policies as Actions of Skill-Level World Models: Learned and Symbolic Abstractions for Long-Horizon Planning
In arXiv preprint, 2026. Under review.
@misc{matoses2026flowskillwm,
title = {Flow Policies as Actions of Skill-Level World Models: Learned and Symbolic Abstractions for Long-Horizon Planning},
author = {Matoses Gimenez, Andreu and Papuc, Andrei-Carlo and Pek, Chris and Alonso-Mora, Javier},
year = {2026},
eprint = {2610.04767},
archiveprefix = {arXiv},
primaryclass = {cs.RO},
url = {https://arxiv.org/abs/2610.04767}
}
Learning Task and Motion Plans from Real Demonstrations with Hybrid Flow Matching
In arXiv preprint, 2026. Under review.
@misc{redondo2026hybridflow,
title = {Learning Task and Motion Plans from Real Demonstrations with Hybrid Flow Matching},
author = {Redondo Garcia, Zuleika and Matoses Gimenez, Andreu and Alonso-Mora, Javier},
year = {2026},
eprint = {2610.04771},
archiveprefix = {arXiv},
primaryclass = {cs.RO},
url = {https://arxiv.org/abs/2610.04771}
}
Building Explicit World Model for Zero-Shot Open-World Object Manipulation
In arXiv, 2026.
BEACON: Language-Conditioned Navigation Affordance Prediction under Occlusion
In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2026.
MobileOcc: A Human-Aware Semantic Occupancy Dataset for Mobile Robots
In European Conference on Computer Vision (ECCV), 2026.
Cross-Entropy Optimization of Physically Grounded Task and Motion Plans
In IEEE Robotics and Automation Letters (RA-L), 2026.
SADCHER: Scheduling using Attention-based Dynamic Coalitions of Heterogeneous Robots in Real-Time
In IEEE Int. Symposium on Multi-Robot & Multi-Agent Systems (MRS), 2025.
Scenario-based motion planning with bounded probability of collision
In The International Journal of Robotics Research (IJRR), 2025.
TamedPUMA: safe and stable imitation learning with geometric fabrics
In Learning for Dynamics and Control (L4DC), 2025.
SHINE: Social Homology Identification for Navigation in Crowded Environments
In International Journal of Robotics Research (IJRR), 2025.
Overcoming Explicit Environment Representations with Geometric Fabrics
In IEEE Robotics and Automation Letters (RA-L), 2025.
Pushing Through Clutter With Movability Awareness of Blocking Obstacles
In IEEE Int. Conf. on Robotics and Automation (ICRA), 2025.
Globally-Guided Geometric Fabrics for Reactive Mobile Manipulation in Dynamic Environments
In IEEE Robotics and Automation Letters (RA-L), 2025.
Particle-based Instance-aware Semantic Occupancy Mapping in Dynamic Environments
In IEEE Transactions on Robotics (T-RO), 2025.
Evaluating Dynamic Environment Difficulty for Obstacle Avoidance Benchmarking
In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2024.
Topology-Driven Parallel Trajectory Optimization in Dynamic Environments
In IEEE Transactions on Robotics (T-RO), 2024.
Auto-Encoding Bayesian Inverse Games
In 16th Workshop on the Algorithmic Foundations of Robotics (WAFR), 2024.
Safe and stable motion primitives via imitation learning and geometric fabrics
In Robotics: Science and Systems, Workshop on Structural Priors as Inductive Biases for Learning Robot Dynamics, 2024.
Physically Grounded Optimal Realizations of Symbolic Plans
In Robotics: Science and Systems (RSS), Workshop on Frontiers of Optimization for Robotics, 2024.
Reactive grasp and motion planning for adaptive mobile manipulation among obstacles
In Robotics: Science and Systems, Workshop on Frontiers of Optimization for Robotics, 2024.
Multi-Modal MPPI and Active Inference for Reactive Task and Motion Planning
In IEEE Robotics and Automation Letters (RA-L), 2024.
Decentralized Multi-Agent Trajectory Planning in Dynamic Environments with Spatiotemporal Occupancy Grid Maps
In IEEE Int. Conf. on Robotics and Automation (ICRA), 2024.
Contingency Games for Multi-Agent Interaction
In Robotics and Automation Letters (RA-L), 2024.
Multi-Robot Local Motion Planning Using Dynamic Optimization Fabrics
In Proc. IEEE International Symposium on Multi-Robot and Multi-Agent Systems, 2023.
Learning to Play Trajectory Games Against Opponents with Unknown Objectives
In IEEE Robotics and Automation Letters (RA-L), 2023.
RAST: Risk-Aware Spatio-Temporal Safety Corridors for MAV Navigation in Dynamic Uncertain Environments
In IEEE Robotics and Automation Letters (RA-L), 2023.