Real time Automated Cinematography using Aerial Vehicles

project image

People

Tobias Nageli - PhD candidate, AIT Lab, ETH Zurich
Prof. Javier Alonso-Mora
Prof. Otmar Hilliges - ETH Zurich

Funding

This project is funded by the Netherlands Organisation for Scientific Research (NWO) Applied Sciences with project Veni 15916

More Links
Github: multi-robot collision avoidance Github: MPCC implementation

About the Project

Cinematography and film-making is an application where robotics is getting increased attention. Often times movie directors make use of expensive gear like helicopters and high-end cameras for obtaining certain critical shots. Aerial vehicles can be a suitable alternative which help obtain the desired visuals while sticking to specific constraints outlined by the cameraman. This project focuses on this application and aims at developing a system of automated aerial vehicles which are capable of planning collision free paths in real-time. The generated methods have then been used for a fleet of aerial vehicles across a number of complex scenes to demonstrate the effectiveness of real-time automated drone cinematography.

The first contribution from this project was a model predictive control formulation for a single drone which could plan trajectories based on certain cinematographic constrainsts such as visibility of the actors and their screen positioning. The next step was to then integrate it for multi drones and ensure collision free trajectories. This is done by developing an algorithm that takes high-level plans alongside image-based framing objectives as input from the user and this can be updated in real-time. The algorithm uses this to generate collision free paths for each drone. The real-time nature of this algorithm allows for feedback incorporation while enabling visuals to be captured in cluttered environments with moving actors.

Another aspect of this project focused on human pose estimation using swarm of aerial vehicles to maximise visibility of the human from different viewpoints during long motion sequences and scenarios including jogging or jumping. The proposed method collects images from all drones, detects and labels 2D joint positions. It then estimates the joint positions of the human skeleton and optimizes the relative positions and orientations of the multi-robot swarm. Finally, it computes control inputs for the drones via model-predictive control (MPC) to keep the human visible during motion.

Project Demonstrations

</iframe>

Funding & Partners

This project was carried out in collaboration with ETH Zurich.

Flycon: Real-time Environment-independent Multi-view Human Pose Estimation with Aerial Vehicles
T. Naegeli, S. Oberholzer, S. Pluess, J. Alonso-Mora, O. Hilliges. In , ACM Transactions on Graphics (SIGGRAPH Asia), 2018.

publisher pdf video
We propose a real-time method for the infrastructure-free estimation of articulated human motion. The approach leverages a swarm of camera-equipped flying robots and jointly optimizes the swarm's and skeletal states, which include the 3D joint positions and a set of bones. Our method allows to track the motion of human subjects, for example an athlete, over long time horizons and long distances, in challenging settings and at large scale, where fixed infrastructure approaches are not applicable. The proposed algorithm uses active infra-red markers, runs in real-time and accurately estimates robot and human pose parameters online without the need for accurately calibrated or stationary mounted cameras. Our method i) estimates a global coordinate frame for the MAV swarm, ii) jointly optimizes the human pose and relative camera positions, and iii) estimates the length of the human bones. The entire swarm is then controlled via a model predictive controller to maximize visibility of the subject from multiple viewpoints even under fast motion such as jumping or jogging. We demonstrate our method in a number of difficult scenarios including capture of long locomotion sequences at the scale of a triplex gym, in non-planar terrain, while climbing and in outdoor scenarios.

Real-time Planning for Automated Multi-View Drone Cinematography
T. Naegeli, L. Meier, A. Domahidi, J. Alonso-Mora, O. Hilliges. In ACM Transactions on Graphics SIGGRAPH, vol. 36, no. 4, Article 132, 2017.

publisher pdf video
We propose a method for automated aerial videography in dynamic and cluttered environments. An online receding horizon optimization formulation facilitates the planning process for novices and experts alike. The algorithm takes high-level plans as input, which we dub virtual rails, alongside interactively defined aesthetic framing objectives and jointly solves for 3D quadcopter motion plans and associated velocities. The method generates control inputs subject to constraints of a non-linear quadrotor model and dynamic constraints imposed by actors moving in an a priori unknown way. The output plans are physically feasible, for the horizon length, and we apply the resulting control inputs directly at each time-step, without requiring a separate trajectory tracking algorithm. The online nature of the method enables incorporation of feedback into the planning and control loop, makes the algorithm robust to disturbances. Furthermore, we extend the method to include coordination between multiple drones to enable dynamic multi-view shots, typical for action sequences and live TV coverage. The algorithm runs in real-time on standard hardware and computes motion plans for several drones in the order of milliseconds. Finally, we evaluate the approach qualitatively with a number of challenging shots, involving multiple drones and actors and qualitatively characterize the computational performance experimentally.

Real-time Motion Planning for Aerial videography with Dynamic Obstacle Avoidance and Viewpoint Optimization
T. Naegeli, J. Alonso-Mora, A. Domahidi, D. Rus, O. Hilliges. In , in IEEE Robotics and Automation Letters, vol. 2, no. 3, pp. 1696-1703, 2017.

publisher pdf video
We propose a method for real-time trajectory generation with applications in aerial videography. Taking framing objectives, such as position of targets in the image plane, as input, our method solves for robot trajectories and gimbal controls automatically and adapts plans in real time due to changes in the environment. We contribute a real-time receding horizon planner that autonomously records scenes with moving targets, while optimizing for visibility under occlusion and ensuring collision-free trajectories. A modular cost function, based on the reprojection error of targets, is proposed that allows for flexibility and artistic freedom and is well behaved under numerical optimization. We formulate the minimization problem under constraints as a finite horizon optimal control problem that fulfills aesthetic objectives, adheres to nonlinear model constraints of the filming robot and collision constraints with static and dynamic obstacles and can be solved in real time. We demonstrate the robustness and efficiency of the method with a number of challenging shots filmed in dynamic environments including those with moving obstacles and shots with multiple targets to be filmed simultaneously.