Distributed high-level scene reasoning with teams of heterogeneous robots

About the Project

In this project, we explore how a team of diverse robots can collaboratively monitor complex environments, such as bustling seaports or major city events. Equipped with varied sensors like cameras and microphones, each robot gathers data from its unique perspective. While some advancements exist in robots reaching consensus on basic features, integrating high-level reasoning with diverse sensing remains a challenge. This is essential when deploying a team of robots in dynamic environments for autonomous navigation in applications like search and rescue, multi-view videography and inspection.

One contribution from this project is an interaction and obstacle aware trajectory prediction model which when combined with a model predictive controller(MPC) achieves multi-robot motion planning. This is done by building a neural network model which is trained on a dataset of robot trajectories generated using a simulator. This model is used to predict the planning behavior of robots and help provide robot trajectory predictions in a multi-robot scenario. This is then fed to an MPC framework which is used as the local motion planner. Experiments were conducted using a team of quadrotors which had to fly in a space shared with human obstacles. The quadrotors were able to keep a safe distance to each human obstacle while following the proposed trajectory.

This project also addresses the problem of videography drone teams that have to autonomously capture desired shots of a dynamic target in a complex environment. A two-stage planning pipeline is proposed whereby a high level planner uses a visibility heuristic to choose when each drone should capture which shot. This results in a reference trajectory which is tracked by an online Model Predictive Control (MPC) algorithm which uses a cost function for viewpoint parameters. Demonstrations are then conducted for a videography scenario with a pair of drones assigned to capture shots of a remote controlled car in the presence of obstacles.

Project Demonstrations

Funding & Partners

This project is funded by the Office of Naval Research Global (ONRG).

Smart cameras are an essential component in surveillance and monitoring applications, and they have been typically deployed in networks of fixed camera locations. The addition of mobile cameras, mounted on robots, can overcome some of the limitations of static networks such as blind spots or back-lightning, allowing the system to gather the best information at each time by active positioning. This work presents a hybrid camera system, with static and mobile cameras, where all the cameras collaborate to observe people moving freely in the environment and efficiently visualize certain attributes from each person. Our solution combines a multi-camera distributed tracking system, to localize with precision all the people, with a control scheme that moves the mobile cameras to the best viewpoints for a specific classification task. The main contribution of this paper is a novel framework that exploits the synergies that result from the cooperation of the tracking and the control modules, obtaining a system closer to the real-world application and capable of high-level scene understanding. The static camera network provides global awareness of the control scheme to move the robots. In exchange, the mobile cameras onboard the robots provide enhanced information about the people on the scene. We perform a thorough analysis of the people monitoring application performance under different conditions thanks to the use of a photo-realistic simulation environment. Our experiments demonstrate the benefits of collaborative mobile cameras with respect to static or individual camera setups.

Decentralized multi-robot systems typically perform coordinated motion planning by constantly broadcasting their intentions to avoid collisions. However, the risk of collision between robots varies as they move and communication may not always be needed. This paper presents an efficient communication method that addresses the problem of “when” and “with whom” to communicate in multi-robot collision avoidance scenarios. In this approach, each robot learns to reason about other robots' states and considers the risk of future collisions before asking for the trajectory plans of other robots. We introduce a new neural architecture for the learned communication policy which allows our method to be scalable. We evaluate and verify the proposed communication strategy in simulation with up to twelve quadrotors, and present results on the zero-shot generalization/robustness capabilities of the policy in different scenarios. We demonstrate that our policy (learned in a simulated environment) can be successfully transferred to real robots.

Robotic applications involving people often re- quire advanced perception systems to better understand com- plex real-world scenarios. To address this challenge, photo- realistic and physics simulators are gaining popularity as a means of generating accurate data labeling and designing scenarios for evaluating generalization capabilities, e.g., lighting changes, camera movements or different weather conditions. We develop a photo-realistic framework built on Unreal Engine and AirSim to generate easily scenarios with pedestrians and mobile robots. The framework is capable to generate random and customized trajectories for each person and provides up to 50 ready-to-use people models along with an API for their metadata retrieval. We demonstrate the usefulness of the proposed framework with a use case of multi-target tracking, a popular problem in real pedestrian scenarios. The notable feature variability in the obtained perception data is presented and evaluated.

About the Project

Project Demonstrations

Funding & Partners

Related Publications