Distributed high-level scene reasoning with teams of heterogeneous robots

People

Alvaro Serra-Gomez

Hai Zhu

Prof. Javier Alonso-Mora - Autonomous Multi-Robot Lab (AMR) TU Delft

Funding

This project is funded by the Office of Naval Research Global (ONRG) of the US.

About the Project

In this project, we explore how a team of diverse robots can collaboratively monitor complex environments, such as bustling seaports or major city events. Equipped with varied sensors like cameras and microphones, each robot gathers data from its unique perspective. While some advancements exist in robots reaching consensus on basic features, integrating high-level reasoning with diverse sensing remains a challenge. Our goal is to bridge this gap, utilizing advanced control and recognition techniques. Through our algorithms, robots will not only identify key elements and events but also optimize data acquisition and inter-robot communication, ensuring a consistent scene understanding across the team, all while relying on local sensing and minimal robot-to-robot communications.

Related Publications

Learning scalable and efficient communication policies for multi-robot collision avoidance
Álvaro Serra-Gómez, Hai Zhu, Bruno Brito, Wendelin Böhmer, Javier Alonso-Mora. In Autonomous Robots 47, 1275-1297, 2023.

abstract web pdf

Decentralized multi-robot systems typically perform coordinated motion planning by constantly broadcasting their intentions to avoid collisions. However, the risk of collision between robots varies as they move and communication may not always be needed. This paper presents an efficient communication method that addresses the problem of “when” and “with whom” to communicate in multi-robot collision avoidance scenarios. In this approach, each robot learns to reason about other robots’ states and considers the risk of future collisions before asking for the trajectory plans of other robots. We introduce a new neural architecture for the learned communication policy which allows our method to be scalable. We evaluate and verify the proposed communication strategy in simulation with up to twelve quadrotors, and present results on the zero-shot generalization/robustness capabilities of the policy in different scenarios. We demonstrate that our policy (learned in a simulated environment) can be successfully transferred to real robots.

Active Classification of Moving Targets with Learned Control Policies
A Serra-Gomez, E Montijano, W Boehmer, J. Alonso-Mora. In IEEE Robotics and Automation Letters (RA-L), 2023.

abstract pdf

Abstract: WIn this paper, we consider the problem where a drone has to collect semantic information to classify multiple moving targets. In particular, we address the challenge of computing control inputs that move the drone to informative viewpoints, position and orientation, when the information is extracted using a “black-box” classifier, e.g., a deep learning neural network. These algorithms typically lack of analytical relationships between the viewpoints and their associated out- puts, preventing their use in information-gathering schemes. To fill this gap, we propose a novel attention-based architecture, trained via Reinforcement Learning (RL), that outputs the next viewpoint for the drone favoring the acquisition of evidence from as many unclassified targets as possible while reasoning about their movement, orientation, and occlusions. Then, we use a low-level MPC controller to move the drone to the desired viewpoint taking into account its actual dynamics. We show that our approach not only outperforms a variety of baselines but also generalizes to scenarios unseen during training. Additionally, we show that the network scales to large numbers of targets and generalizes well to different movement dynamics of the targets.

With Whome to Communicate: Learning Efficient Communication for Multi-Robot Collision Avoidance
A. Serra-Gomez, B. Brito, H. Zhu, J. J. Chung, J. Alonso-Mora. In Proc. IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS), 2020.

abstract web pdf video

Abstract: Decentralized multi-robot systems typically perform coordinated motion planning by constantly broadcasting their intentions as a means to cope with the lack of a central system coordinating the efforts of all robots. Especially in complex dynamic environments, the coordination boost allowed by communication is critical to avoid collisions between cooperating robots. However, the risk of collision between a pair of robots fluctuates through their motion and communication is not always needed. Additionally, constant communication makes much of the still valuable information shared in previous time steps redundant. This paper presents an efficient communication method that solves the problem of "when" and with "whom" to communicate in multi-robot collision avoidance scenarios. In this approach, every robot learns to reason about other robots' states and considers the risk of future collisions before asking for the trajectory plans of other robots. We evaluate and verify the proposed communication strategy in simulation with four quadrotors and compare it with three baseline strategies: non-communicating, broadcasting and a distance-based method broadcasting information with quadrotors within a predefined distance.