Distributed high-level scene reasoning with teams of heterogeneous robots

project image

People

Alvaro Serra-Gomez - PhD candidate
Prof. Eduardo Montijano - Universidad de Zaragoza
Prof. Javier Alonso-Mora

Funding

This project is funded by the Office of Naval Research Global (ONRG).

More Links
Active Information gathering Risk-aware spatio-temporal(RAST)

About the Project

In this project, we explore how a team of diverse robots can collaboratively monitor complex environments, such as bustling seaports or major city events. Equipped with varied sensors like cameras and microphones, each robot gathers data from its unique perspective. While some advancements exist in robots reaching consensus on basic features, integrating high-level reasoning with diverse sensing remains a challenge. This is essential when deploying a team of robots in dynamic environments for autonomous navigation in applications like search and rescue, multi-view videography and inspection.

One contribution from this project is an interaction and obstacle aware trajectory prediction model which when combined with a model predictive controller(MPC) achieves multi-robot motion planning. This is done by building a neural network model which is trained on a dataset of robot trajectories generated using a simulator. This model is used to predict the planning behavior of robots and help provide robot trajectory predictions in a multi-robot scenario. This is then fed to an MPC framework which is used as the local motion planner. Experiments were conducted using a team of quadrotors which had to fly in a space shared with human obstacles. The quadrotors were able to keep a safe distance to each human obstacle while following the proposed trajectory.

This project also addresses the problem of videography drone teams that have to autonomously capture desired shots of a dynamic target in a complex environment. A two-stage planning pipeline is proposed whereby a high level planner uses a visibility heuristic to choose when each drone should capture which shot. This results in a reference trajectory which is tracked by an online Model Predictive Control (MPC) algorithm which uses a cost function for viewpoint parameters. Demonstrations are then conducted for a videography scenario with a pair of drones assigned to capture shots of a remote controlled car in the presence of obstacles.

Project Demonstrations

Funding & Partners

This project is funded by the Office of Naval Research Global (ONRG).

Distributed multi-target tracking and active perception with mobile camera networks
Sara Casao, Alvaro Serra-Gomez, Ana C. Murillo, Wendelin Böhmer, Javier Alonso-Mora, Eduardo Montijano. In S.I. Collaborative Mobile Smart Cameras, Computer Vision and Image Understanding, 2024.

pdf
Smart cameras are an essential component in surveillance and monitoring applications, and they have been typically deployed in networks of fixed camera locations. The addition of mobile cameras, mounted on robots, can overcome some of the limitations of static networks such as blind spots or back-lightning, allowing the system to gather the best information at each time by active positioning. This work presents a hybrid camera system, with static and mobile cameras, where all the cameras collaborate to observe people moving freely in the environment and efficiently visualize certain attributes from each person. Our solution combines a multi-camera distributed tracking system, to localize with precision all the people, with a control scheme that moves the mobile cameras to the best viewpoints for a specific classification task. The main contribution of this paper is a novel framework that exploits the synergies that result from the cooperation of the tracking and the control modules, obtaining a system closer to the real-world application and capable of high-level scene understanding. The static camera network provides global awareness of the control scheme to move the robots. In exchange, the mobile cameras onboard the robots provide enhanced information about the people on the scene. We perform a thorough analysis of the people monitoring application performance under different conditions thanks to the use of a photo-realistic simulation environment. Our experiments demonstrate the benefits of collaborative mobile cameras with respect to static or individual camera setups.

Learning scalable and efficient communication policies for multi-robot collision avoidance
Álvaro Serra-Gómez, Hai Zhu, Bruno Brito, Wendelin Böhmer, Javier Alonso-Mora. In Autonomous Robots 47, 1275-1297, 2023.

publisher pdf
Decentralized multi-robot systems typically perform coordinated motion planning by constantly broadcasting their intentions to avoid collisions. However, the risk of collision between robots varies as they move and communication may not always be needed. This paper presents an efficient communication method that addresses the problem of “when” and “with whom” to communicate in multi-robot collision avoidance scenarios. In this approach, each robot learns to reason about other robots’ states and considers the risk of future collisions before asking for the trajectory plans of other robots. We introduce a new neural architecture for the learned communication policy which allows our method to be scalable. We evaluate and verify the proposed communication strategy in simulation with up to twelve quadrotors, and present results on the zero-shot generalization/robustness capabilities of the policy in different scenarios. We demonstrate that our policy (learned in a simulated environment) can be successfully transferred to real robots.

Active Classification of Moving Targets with Learned Control Policies
A. Serra-Gomez, E. Montijano, W. Boehmer, J. Alonso-Mora. In IEEE Robotics and Automation Letters (RA-L), 2023.

pdf
WIn this paper, we consider the problem where a drone has to collect semantic information to classify multiple moving targets. In particular, we address the challenge of computing control inputs that move the drone to informative viewpoints, position and orientation, when the information is extracted using a “black-box” classifier, e.g., a deep learning neural network. These algorithms typically lack of analytical relationships between the viewpoints and their associated out- puts, preventing their use in information-gathering schemes. To fill this gap, we propose a novel attention-based architecture, trained via Reinforcement Learning (RL), that outputs the next viewpoint for the drone favoring the acquisition of evidence from as many unclassified targets as possible while reasoning about their movement, orientation, and occlusions. Then, we use a low-level MPC controller to move the drone to the desired viewpoint taking into account its actual dynamics. We show that our approach not only outperforms a variety of baselines but also generalizes to scenarios unseen during training. Additionally, we show that the network scales to large numbers of targets and generalizes well to different movement dynamics of the targets.

RAST: Risk-Aware Spatio-Temporal Safety Corridors for MAV Navigation in Dynamic Uncertain Environments
G. Chen, S. Wu, M. Shi, W. Dong, H. Zhu, J. Alonso-Mora. In IEEE Robotics and Automation Letters (RA-L), 2023.

pdf code
Autonomous navigation of Micro Aerial Vehicles (MAVs) in dynamic and unknown environments is a complex and challenging task. Current works rely on assumptions to solve the problem. The MAV’s pose is precisely known, the dynamic obstacles can be explicitly segmented from static ones, their number is known and fixed, or they can be modeled with given shapes. In this paper, we present a method for MAV navigation in dynamic uncertain environments without making any of these assumptions. The method employs a particle-based dynamic map to represent the local environment and predicts it to the near future. Collision risk is defined based on the predicted maps and a series of risk-aware spatio-temporal (RAST) safety corridors are constructed, which are finally used to optimize a dynamically- feasible collision-free trajectory for the MAV. We compared our method with several state-of-the-art works in 12000 simulation tests in Gazebo with the physical engine enabled. The results show that our method has the highest success rate at different uncertainty levels. Finally, we validated the proposed method in real experiments.

A Framework for Fast Prototyping of Photo-realistic Environments with Multiple Pedestrians
S. Casao, A. Otero, A. Serra-Gomez, AC. Murillo, J. Alonso-Mora, E. Montijano. In , in IEEE Int. Conf. on Robotics and Automation (ICRA), 2023.

pdf
Robotic applications involving people often re- quire advanced perception systems to better understand com- plex real-world scenarios. To address this challenge, photo- realistic and physics simulators are gaining popularity as a means of generating accurate data labeling and designing scenarios for evaluating generalization capabilities, e.g., lighting changes, camera movements or different weather conditions. We develop a photo-realistic framework built on Unreal Engine and AirSim to generate easily scenarios with pedestrians and mobile robots. The framework is capable to generate random and customized trajectories for each person and provides up to 50 ready-to-use people models along with an API for their metadata retrieval. We demonstrate the usefulness of the proposed framework with a use case of multi-target tracking, a popular problem in real pedestrian scenarios. The notable feature variability in the obtained perception data is presented and evaluated.

Wi-Closure: Reliable and Efficient Search of Inter-Robot Loop Closures Using Wireless Sensing
W. Wang, A. Kemmeren, D. Son, J. Alonso-Mora, S. Gil. In IEEE Int. Conf. on Robotics and Automation (ICRA), 2023.

pdf
In this paper we propose a novel algorithm, Wi- Closure, to improve the computational efficiency and robustness of loop closure detection in multi-robot SLAM. Our approach decreases the computational overhead of classical approaches by pruning the search space of potential loop closures, prior to evaluation by a typical multi-robot SLAM pipeline. Wi-Closure achieves this by identifying candidates that are spatially close to each other measured via sensing over the wireless commu- nication signal between robots, even when they are operating in non-line-of-sight or in remote areas of the environment from one another. We demonstrate the validity of our approach in simulation and in hardware experiments. Our results show that using Wi-closure greatly reduces computation time, by 54.1% in simulation and 76.8% in hardware experiments, compared with a multi-robot SLAM baseline. Importantly, this is achieved without sacrificing accuracy. Using Wi-closure reduces absolute trajectory estimation error by 98.0% in simulation and 89.2% in hardware experiments. This improvement is partly due to Wi-Closure’s ability to avoid catastrophic optimization failure that typically occurs with classical approaches in challenging repetitive environments.

Decentralized Probabilistic Multi-Robot Collision Avoidance Using Buffered Uncertainty-Aware Voronoi Cells
H. Zhu, B. Brito, J. Alonso-Mora. In Autonomous Robots (AURO), 2022.

publisher pdf code video
In this paper, we present a decentralized and communication-free collision avoidance approach for multi-robot systems that accounts for both robot localization and sensing uncertainties. The approach relies on the computation of an uncertainty-aware safe region for each robot to navigate among other robots and static obstacles in the environment, under the assumption of Gaussian-distributed uncertainty. In particular, at each time step, we construct a chance-constrained buffered uncertainty-aware Voronoi cell (B-UAVC) for each robot given a specified collision probability threshold. Probabilistic collision avoidance is achieved by constraining the motion of each robot to be within its corresponding B-UAVC, i.e. the collision probability between the robots and obstacles remains below the specified threshold. The proposed approach is decentralized, communication-free, scalable with the number of robots and robust to robots’ localization and sensing uncertainties. We applied the approach to single-integrator, double-integrator, differential-drive robots, and robots with general nonlinear dynamics. Extensive simulations and experiments with a team of ground vehicles, quadrotors, and heterogeneous robot teams are performed to analyze and validate the proposed approach.

Learning Interaction-Aware Trajectory Predictions for Decentralized Multi-Robot Motion Planning in Dynamic Environments
H. Zhu, F. Claramunt, B. Brito, J. Alonso-Mora. In , IEEE Robotics and Automation Letters (RA-L), 2021.

publisher pdf video
This letter presents a data-driven decentralized trajectory optimization approach for multi-robot motion planning in dynamic environments. When navigating in a shared space, each robot needs accurate motion predictions of neighboring robots to achieve predictive collision avoidance. These motion predictions can be obtained among robots by sharing their future planned trajectories with each other via communication. However, such communication may not be available nor reliable in practice. In this letter, we introduce a novel trajectory prediction model based on recurrent neural networks (RNN) that can learn multi-robot motion behaviors from demonstrated trajectories generated using a centralized sequential planner. The learned model can run efficiently online for each robot and provide interaction-aware trajectory predictions of its neighbors based on observations of their history states. We then incorporate the trajectory prediction model into a decentralized model predictive control (MPC) framework for multi-robot collision avoidance. Simulation results show that our decentralized approach can achieve a comparable level of performance to a centralized planner while being communication-free and scalable to a large number of robots. We also validate our approach with a team of quadrotors in real-world experiments.

Multi-robot Task Assignment for Aerial Tracking with Viewpoint Constraints
A. Ray, A. Pierson, H. Zhu, J. Alonso-Mora, D. Rus. In Proc. IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS), 2021.

arxiv pdf
We address the problem of assigning a team of drones to autonomously capture a set desired shots of a dynamic target in the presence of obstacles. We present a two-stage planning pipeline that generates offline an assignment of drone to shots and locally optimizes online the viewpoint. Given desired shot parameters, the high-level planner uses a visibility heuristic to predict good times for capturing each shot and uses an Integer Linear Program to compute drone assignments. An online Model Predictive Control algorithm uses the assignments as reference to capture the shots. The algorithm is validated in hardware with a pair of drones and a remote controlled car.

Online Informative Path Planning for Active Information Gathering of a 3D Surface
H. Zhu, J. J. Chung, N. R. J. Lawrance, R. Siegwart, J. Alonso-Mora. In Proc. IEEE Int. Conf. on Robotics and Automation (ICRA), 2021.

publisher pdf code
This paper presents an online informative path planning approach for active information gathering on three-dimensional surfaces using aerial robots. Most existing works on surface inspection focus on planning a path offline that can provide full coverage of the surface, which inherently assumes the surface information is uniformly distributed hence ignoring potential spatial correlations of the information field. In this paper, we utilize manifold Gaussian processes (mGPs) with geodesic kernel functions for mapping surface information fields and plan informative paths online in a receding horizon manner. Our approach actively plans information-gathering paths based on recent observations that respect dynamic constraints of the vehicle and a total flight time budget. We provide planning results for simulated temperature modeling for simple and complex 3D surface geometries (a cylinder and an aircraft model). We demonstrate that our informative planning method outperforms traditional approaches such as 3D coverage planning and random exploration, both in reconstruction error and information-theoretic metrics. We also show that by taking spatial correlations of the information field into planning using mGPs, the information gathering efficiency is significantly improved.

With Whom to Communicate: Learning Efficient Communication for Multi-Robot Collision Avoidance
A. Serra-Gomez, B. Brito, H. Zhu, J. J. Chung, J. Alonso-Mora. In Proc. IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS), 2020.

arxiv pdf video
Decentralized multi-robot systems typically perform coordinated motion planning by constantly broadcasting their intentions as a means to cope with the lack of a central system coordinating the efforts of all robots. Especially in complex dynamic environments, the coordination boost allowed by communication is critical to avoid collisions between cooperating robots. However, the risk of collision between a pair of robots fluctuates through their motion and communication is not always needed. Additionally, constant communication makes much of the still valuable information shared in previous time steps redundant. This paper presents an efficient communication method that solves the problem of "when" and with "whom" to communicate in multi-robot collision avoidance scenarios. In this approach, every robot learns to reason about other robots' states and considers the risk of future collisions before asking for the trajectory plans of other robots. We evaluate and verify the proposed communication strategy in simulation with four quadrotors and compare it with three baseline strategies: non-communicating, broadcasting and a distance-based method broadcasting information with quadrotors within a predefined distance.

Robust Vision-based Obstacle Avoidance for Micro Aerial Vehicles in Dynamic Environments
J. Liu, H. Zhu, J. Alonso-Mora. In Proc. IEEE Int. Conf. on Robotics and Automation (ICRA), 2020.

arxiv pdf video
In this paper, we present an on-board vision-based approach for avoidance of moving obstacles in dynamic environments. Our approach relies on an efficient obstacle detection and tracking algorithm based on depth image pairs, which provides the estimated position, velocity and size of the obstacles. Robust collision avoidance is achieved by formulating a chance-constrained model predictive controller (CC-MPC) to ensure that the collision probability between the micro aerial vehicle (MAV) and each moving obstacle is below a specified threshold. The method takes into account MAV dynamics, state estimation and obstacle sensing uncertainties. The proposed approach is implemented on a quadrotor equipped with a stereo camera and is tested in a variety of environments, showing effective on-line collision avoidance of moving obstacles.

B-UAVC: Buffered Uncertainty-Aware Voronoi Cells for Probabilistic Multi-Robot Collision Avoidance
H. Zhu, J. Alonso-Mora. In Proc. 2nd IEEE International Symposium on Multi-Robot and Multi-Agent Systems (MRS'19), 2019.

publisher pdf video
This paper presents B-UAVC, a distributed collision avoidance method for multi-robot systems that accounts for uncertainties in robot localization. In particular, Buffered Uncertainty-Aware Voronoi Cells (B-UAVC) are employed to compute regions where the robots can safely navigate. By computing a set of chance constraints, which guarantee that the robot remains within its B-UAVC, the method can be applied to non-holonomic robots. A local trajectory for each robot is then computed by introducing these chance constraints in a receding horizon model predictive controller. The method guarantees, under the assumption of normally distributed position uncertainty, that the collision probability between the robots remains below a specified threshold. We evaluate the proposed method with a team of quadrotors in simulations and in real experiments.