Multi-Modal MPPI and Active Inference for Reactive Task and Motion Planning
Reactive pick place
Multi-modal pick place
Abstract
Task and Motion Planning (TAMP) has made strides in complex manipulation tasks, yet the execution robustness of the planned solutions remains overlooked. In this work, we propose a method for reactive TAMP to cope with runtime uncertainties and disturbances. We combine an Active Inference planner (AIP) for adaptive high-level action selection and a novel Multi-Modal Model Predictive Path Integral controller (M3P2I) for low-level control. This results in a scheme that simultaneously adapts both high-level actions and low-level motions. The AIP generates alternative symbolic plans, each linked to a cost function for M3P2I. The latter employs a physics simulator for diverse trajectory rollouts, deriving optimal control by weighing the different samples according to their cost. This idea enables blending different robot skills for fluid and reactive plan execution, accommodating plan adjustments at both the high and low levels to cope, for instance, with dynamic obstacles or disturbances that invalidate the current plan. We have tested our approach in simulations and real-world scenarios.
Proposed Scheme
Given symbolic observations $o$ of the environment, the action planner computes $N$ different plan alternatives linked to individual cost functions $C_i$. M3P2I samples control input sequences and uses an importance sampling scheme to approximate the optimal control $u_0^*$.
Simulation Results
Push only using MPPI
Middle to corner -- success
Push only using MPPI
Corner to corner -- fail
Pull only using MPPI
Middle to corner -- fail
Pull only using MPPI
Corner to corner -- fail
Multi-modal motion using M3P2I
Middle to corner -- success
Multi-modal motion using M3P2I
Corner to corner -- success
Reactive pick using MPPI
Pick from top
Multi-modal pick using M3P2I
Top pick and side pick
Multi-modal motion using M3P2I
With collision avoidance
Reactive pick using RL
Oscillation occurs when introducing disturbance
Related Publications
Dynamic Risk-Aware MPPI for Mobile Robots in Crowds via Efficient Monte Carlo Approximations
In IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS), 2025.
Active Disturbance Rejection Control (ADRC) for Trajectory Tracking of a Seagoing USV: Design, Simulation, and Field Experiments
In IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS), 2025.
Sampling-based Model Predictive Control Leveraging Parallelizable Physics Simulations
In IEEE Robotics and Automation Letters (RA-L), 2025.
Biased-MPPI: Informing Sampling-Based Model Predictive Control by Fusing Ancillary Controllers
In IEEE Robotics and Automation Letters (RA-L), 2024.
Interaction-Aware Sampling-Based MPC with Learned Local Goal Predictions
In Proc. IEEE International Symposium on Multi-Robot and Multi-Agent Systems, 2023.
Sampling-Based MPC Using a GPU-parallelizable Physics Simulator as Dynamic Model: an Open Source Implementation with IsaacGym
In Embracing Contacts Workshop at IEEE Int. Conf. on Robotics and Automation (ICRA), 2023.
Multi-Agent Path Integral Control for Interaction-Aware Motion Planning in Urban Canals
In IEEE Int. Conf. on Robotics and Automation (ICRA), 2023.
Regulations Aware Motion Planning for Autonomous Surface Vessels in Urban Canals
In Proc. IEEE Int. Conf. on Robotics and Automation (ICRA), 2022.