AI for Retail: Mobile Manipulation in Dynamic Environments

About the Project

Mobile manipulators are commonly used in close proximity to each other as well as to other objects including humans. This helps improve efficiency and enables these robotic systems to be used for execution of complex tasks. However, this requires obtaining viable trajectories for multiple robots in an ever-changing environment. The AI for Retail (AIR) Lab Delft is a joint TU Delft-Ahold Delhaize industry lab which focuses on building methods to tackle such problems while testing it in a retail grocery store environment. As part of the project, we develop novel methods for trajectory generation for mobile manipulation. In the context of retail environments, it is especially important to generate motions that are safe in the proximity of humans.

Multi-robot motion planning often suffers from high computational costs especially when dealing with large number of robots. The ability of these methods to adapt online in real-time becomes essential. This project introduced multi-robot dynamics fabrics(MRDF) which builds upon dynamic fabrics that are based on local planning using a differential equation. The proposed method can be used for replanning at higher frequencies making it advantageous for complex systems in dynamic environments. Simulation experiments are performed with multiple manipulators in close proximity performing a pick-and-place task. Results highlighted the success of this approach and a significant reduction in computation time making it realistic for real-time local motion planning.

In addition to this, the project also introduced a reactive task and motion planning (TAMP) method which can cope with unforeseen disturbances during runtime. This is done by firstly developing a high-level planner that can sample parallel motion plans to minimise the cost function and computes control input. A behavior tree guides the search which allows real-time high level planning and alternatives can be used in case of disturbances. The main strength of this method is it's ability to reason over the discrete alternatives at the motion planning level. The method was tested for push-pull and object stacking tasks which showed it's ability to deal with corner cases and also solve these tasks with various grasp strategies and severe disturbances.

Project Demonstrations

Funding & Partners

This project has been funded by Ahold Delhaize.

We present a method for sampling-based model predictive control that makes use of a generic physics simulator as the dynamical model. In particular, we propose a Model Predictive Path Integral controller (MPPI), that uses the GPU-parallelizable IsaacGym simulator to compute the forward dynamics of a problem. By doing so, we eliminate the need for manual encoding of robot dynamics and interactions among objects and allow one to effortlessly solve complex navigation and contact-rich tasks. Since no explicit dynamic modeling is required, the method is easily extendable to different objects and robots. We demonstrate the effectiveness of this method in several simulated and real-world settings, among which mobile navigation with collision avoidance, non-prehensile manipulation, and whole-body control for high-dimensional configuration spaces. This method is a powerful and accessible tool to solve a large variety of contact-rich motion planning tasks.

Although autonomous robots have great potential to boost efficiency and throughput across the whole retail chain, they are mostly being deployed in large warehouses and distribution centers. Deploying robots in stores with customers, such as supermarkets, requires substantially more development efforts since they need to safely operate around customers and reliably cope with various uncertainties and disturbances, such as misplaced products. We present our recent efforts in developing a mobile manipulator platform for order picking in realistic supermarket settings. Our robot platform uses state-of-the-art perception and planning algorithms to robustly pick items in the presence of disturbances. In particular, it successfully demonstrates adaptive decision making and rapid replanning. Our robot allows adding new products and teaching new picking maneuvers from demonstrations. We validated our robot in a recreated supermarket in our lab and in a test supermarket of a large Dutch retailer. Our results show how our robot successfully recovers from various disturbances, including misplaced products, errors in picking, and from human interaction. We summarize our lessons learned to bring autonomous robots into real retail environments with customers.

Task and Motion Planning (TAMP) has made strides in complex manipulation tasks, yet the execution robustness of the planned solutions remains overlooked. In this work, we propose a method for reactive TAMP to cope with runtime uncertainties and disturbances. We combine an Active Inference planner (AIP) for adaptive high-level action selection and a novel Multi-Modal Model Predictive Path Integral controller (M3P2I) for low-level control. This results in a scheme that simultaneously adapts both high-level actions and low-level motions. The AIP generates alternative symbolic plans, each linked to a cost function for M3P2I. The latter employs a physics simulator for diverse trajectory rollouts, deriving optimal control by weighing the different samples according to their cost. This idea enables blending different robot skills for fluid and reactive plan execution, accommodating plan adjustments at both the high and low levels to cope, for instance, with dynamic obstacles or disturbances that invalidate the current plan. We have tested our approach in simulations and real-world scenarios.

About the Project

Project Demonstrations

Funding & Partners

Related Publications