Distributed high-level scene reasoning with teams of heterogeneous robots


In this project, we explore how a team of diverse robots can collaboratively monitor complex environments, such as bustling seaports or major city events. Equipped with varied sensors like cameras and microphones, each robot gathers data from its unique perspective. While some advancements exist in robots reaching consensus on basic features, integrating high-level reasoning with diverse sensing remains a challenge. Our goal is to bridge this gap, utilizing advanced control and recognition techniques. Through our algorithms, robots will not only identify key elements and events but also optimize data acquisition and inter-robot communication, ensuring a consistent scene understanding across the team, all while relying on local sensing and minimal robot-to-robot communications.

People

Alvaro Serra-Gomez
Hai Zhu
Prof. Javier Alonso-Mora

Funding

This project is founded by the Office of Naval Research Global (ONRG) of the US.

Partners

Office of Naval Research Global (ONRG) of the US.

Publications

Álvaro Serra-Gómez, Hai Zhu, Bruno Brito, Wendelin Böhmer and Javier Alonso-Mora; Learning scalable and efficient communication policies for multi-robot collision avoidance; Autonomous Robots, Aug. 2023
Links: [web], [PDF],
Abstract: Decentralized multi-robot systems typically perform coordinated motion planning by constantly broadcasting their intentions to avoid collisions. However, the risk of collision between robots varies as they move and communication may not always be needed. This paper presents an efficient communication method that addresses the problem of “when” and “with whom” to communicate in multi-robot collision avoidance scenarios. In this approach, each robot learns to reason about other robots’ states and considers the risk of future collisions before asking for the trajectory plans of other robots. We introduce a new neural architecture for the learned communication policy which allows our method to be scalable. We evaluate and verify the proposed communication strategy in simulation with up to twelve quadrotors, and present results on the zero-shot generalization/robustness capabilities of the policy in different scenarios. We demonstrate that our policy (learned in a simulated environment) can be successfully transferred to real robots.
Álvaro Serra-Gómez, Eduardo Montijano, Wendelin Böhmer and Javier Alonso-Mora; Active Classification of Moving Targets With Learned Control Policies; Robotics and Automation Letters (RA-L), IEEE, Jun. 2023
Links: [web], [PDF],
Abstract: In this paper, we consider the problem where a drone has to collect semantic information to classify multiple moving targets. In particular, we address the challenge of computing control inputs that move the drone to informative viewpoints, position and orientation, when the information is extracted using a “black-box” classifier, e.g., a deep learning neural network. These algorithms typically lack of analytical relationships between the viewpoints and their associated outputs, preventing their use in information-gathering schemes. To fill this gap, we propose a novel attention-based architecture, trained via Reinforcement Learning (RL), that outputs the next viewpoint for the drone favoring the acquisition of evidence from as many unclassified targets as possible while reasoning about their movement, orientation, and occlusions. Then, we use a low-level MPC controller to move the drone to the desired viewpoint taking into account its actual dynamics. We show that our approach not only outperforms a variety of baselines but also generalizes to scenarios unseen during training. Additionally, we show that the network scales to large numbers of targets and generalizes well to different movement dynamics of the targets.
Álvaro Serra-Gómez, Bruno Brito, Hai Zhu, Jen Jen Chung and Javier Alonso-Mora; With Whom to Communicate: Learning Efficient Communication for Multi-Robot Collision Avoidance; 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, Oct. 2020
Links: [web], [PDF], [video]
Abstract: Decentralized multi-robot systems typically perform coordinated motion planning by constantly broadcasting their intentions as a means to cope with the lack of a central system coordinating the efforts of all robots. Especially in complex dynamic environments, the coordination boost allowed by communication is critical to avoid collisions between cooperating robots. However, the risk of collision between a pair of robots fluctuates through their motion and communication is not always needed. Additionally, constant communication makes much of the still valuable information shared in previous time steps redundant. This paper presents an efficient communication method that solves the problem of "when" and with "whom" to communicate in multi-robot collision avoidance scenarios. In this approach, every robot learns to reason about other robots' states and considers the risk of future collisions before asking for the trajectory plans of other robots. We evaluate and verify the proposed communication strategy in simulation with four quadrotors and compare it with three baseline strategies: non-communicating, broadcasting and a distance-based method broadcasting information with quadrotors within a predefined distance.