Parallel Autonomy in Automated Vehicles

project image

People

Wilko Schwarting - PhD Candidate MIT
Prof. Javier Alonso-Mora
Sertac Karaman - Massachusetts Institute of Technology (MIT)
Prof. Daniela Rus - Massachusetts Institute of Technology (MIT)

Funding

This project was funded by the Toyota Research Institute (TRI)

About the Project

Autonomous driving systems offer higher safety, fuel economy, mobility and comfort. The main problem in developing such systems arise with how much control should be given to the driver and which situations require a transfer of control to the human. It has been observed that humans are proficient in active control tasks but are not suited to monitoring ones. Several companies have developed different approaches to reach higher levels of autonomy. This project proposes an alternative titled “Parallel Autonomy” which keeps the human in control of the vehicle but a fully autonomous system is running in the background which prevents the human from causing an accident. This has the advantage that if the system is unsure about the correct course of action, it can always follow the human commands and thus will never perform worse than a zero autonomy vehicle.

This project build an entire research platform for this approach and demonstrated its working on an actual vehicle. This required adding certain software and hardware elements to the system. The software side focused on mapping and localisation, object detection and an algorithm to calculate the safe navigation speed for the vehicle at every instance based on the objects surrounding it. These elements were tested on a simulator first to understand its effectiveness. A low level controller was also developed which was used to regulate both the speed and steering angle. On the other hand, additional hardware were also installed. Lidars and a webcam were integrated to sense the environment. The drive-by-wire conversion enables seamless takeover between the human and autonomous system. Experimental results on a Toyota Prius demonstrated the ability of this approach to ensure that the driver cannot leave the path or overspeed.

This project also focused on other safety issues for an autonomous vehicle. A novel method of trajectory generation was developed for autonomous overtaking of a static obstacle. This would result in the car firstly moving slightly in it’s lane to gather information about the road ahead ad then a trajectory is generated to safely execute the maneuver. Another outcome of this project was a proposal to increase the sensing capabilities of an autonomous car using an unmanned aerial vehicle(UAV). The UAV will be able to take off from the car and gather additional data from blind spots which other conventional sensors fail to identify. This would lead to a safer trajectory being planned for the car and identifying potential threats which might be occluded to the driver.

Project Demonstrations

Funding & Partners

Planning and Decision-Making for Autonomous Vehicles
W. Schwarting, J. Alonso-Mora, D. Rus. In Annual Review of Control, Robotics, and Autonomous Systems, vol. 1, pp. 187-210, 2018.

publisher pdf
In this review, we provide an overview of emerging trends and challenges in the field of intelligent and autonomous, or self-driving, vehicles. Recent advances in the field of perception, planning, and decision-making for autonomous vehicles have led to great improvements in functional capabilities, with several prototypes already driving on our roads and streets. Yet challenges remain regarding guaranteed performance and safety under all driving circumstances. For instance, planning methods that provide safe and system-compliant performance in complex, cluttered environments while modeling the uncertain interaction with other traffic participants are required. Furthermore, new paradigms, such as interactive planning and end-to-end learning, open up questions regarding safety and reliability that need to be addressed. In this survey, we emphasize recent approaches for integrated perception and planning and for behavior-aware planning, many of which rely on machine learning. This raises the question of verification and safety, which we also touch upon. Finally, we discuss the state of the art and remaining challenges for managing fleets of autonomous vehicles.

Safe Nonlinear Trajectory Generation for Parallel Autonomy With a Dynamic Vehicle Model
W. Schwarting, J. Alonso-Mora, L. Paull, S. Karaman, D. Rus. In IEEE Transactions on Intelligent Transportation Systems, 2017.

publisher pdf video
High-end vehicles are already equipped with safety systems, such as assistive braking and automatic lane following, enhancing vehicle safety. Yet, these current solutions can only help in low-complexity driving situations. In this paper, we introduce a parallel autonomy, or shared control, framework that computes safe trajectories for an automated vehicle, based on human inputs. We minimize the deviation from the human inputs while ensuring safety via a set of collision avoidance constraints. Our method achieves safe motion even in complex driving scenarios, such as those commonly encountered in an urban setting. We introduce a receding horizon planner formulated as nonlinear model predictive control (NMPC), which includes the analytic descriptions of road boundaries and the configuration and future uncertainties of other road participants. The NMPC operates over both steering and acceleration simultaneously. We introduce a nonslip model suitable for handling complex environments with dynamic obstacles, and a nonlinear combined slip vehicle model including normal load transfer capable of handling static environments. We validate the proposed approach in two complex driving scenarios. First, in an urban environment that includes a left-turn across traffic and passing on a busy street. And second, under snow conditions on a race track with sharp turns and under complex dynamic constraints. We evaluate the performance of the method with various human driving styles. We consequently observe that the method successfully avoids collisions and generates motions with minimal intervention for parallel autonomy. We note that the method can also be applied to generate safe motion for fully autonomous vehicles.

Compositional and Contract-based Verification for Autonomous Driving on Road Networks
L. Liebenwein, W. Schwarting, C.-I. Vasile, J. DeCastro, J. Alonso-Mora, S. Karaman, D. Rus. In , in Proc. of the Int. Symp. on Robotics Research (ISRR), 2017.

publisher pdf video
Recent advances in autonomous driving have raised the problem of safety to the forefront and incentivized research into establishing safety guarantees. In this paper, we propose a safety verification framework as a safety standard for driving controllers with full or shared autonomy based on compositional and contract-based principles. Our framework enables us to synthesize safety guarantees over entire road networks by first building a library of locally verified models, and then composing local models together to verify the entire network. Composition is achieved using assume- guarantee contracts that are synthesized concurrently during verification. Thus, we can reuse local models within and across networks, add additional models to cover local road geometries without re-verifying the entire library, and perform all com- putations in a parallel and distributed way, which enables computational tractability. Furthermore, we employ controller contracts such that any controller satisfying them can be certified safe. We demonstrate the practical effectiveness of our framework by certifying controllers over parts of the Manhattan road network.

Trajectory Optimization for Autonomous Overtaking with Visibility Maximization
H. Andersen, W. Schwarting, F. Naser, Y. H. Eng, M. H. Ang Jr, D. Rus, J. Alonso-Mora. In proc. IEEE Intelligent Transportation Systems Conference, 2017.

publisher pdf video
In this paper we present a trajectory generation method for autonomous overtaking of static obstacles in a dynamic urban environment. In these settings, blind spots can arise from perception limitations. For example, the autonomous car may have to move slightly into the opposite lane in order to cleanly see in front of a car ahead. Once it has gathered enough information about the road ahead, then the autonomous car can safely overtake. We generate safe trajectories by solving, in real-time, a non-linear constrained optimization, formulated as a Receding Horizon planner. The planner is guided by a high-level state machine, which determines when the overtake maneuver should begin. Our main contribution is a method that can maximize visibility, prioritizes safety and respects the boundaries of the road while executing the maneuver. We present experimental results in simulation with data collected during real driving.

Foresight: Remote Sensing For Autonomous Vehicles Using a Small Unmanned Aerial Vehicle
A. Wallar, B. Araki, R. Chang, J. Alonso-Mora, D. Rus. In Proc. of the Conf. on Field and Service Robotics (FSR), 2017.

publisher pdf video
A large number of traffic accidents, especially those involving vulnerable road users such as pedestrians and cyclists, are due to blind spots for the driver, for example when a vehicle takes a turn with poor visibility or when a pedestrian crosses from behind a parked vehicle. In these accidents, the consequences for the vulnerable road users are dramatic. Autonomous cars have the potential to drastically reduce traffic accidents thanks to high-performance sensing and reasoning. However, their perception capabilities are still limited to the field of view of their sensors. We propose to extend the perception capabilities of a vehicle, autonomous or human-driven, with a small Unmanned Aerial Vehicle (UAV) capable of taking off from the car, flying around corners to gather additional data from blind spots and landing back on the car after a mission. We present a holistic framework to detect blind spots in the map that is built by the car, plan an informative path for the drone, and detect potential threats occluded to the car. We have tested our approach with an autonomous car equipped with a drone.

A Parallel Autonomy Research Platform
F. Naser, D. Dorhout, S. Proulx, S. D. Pendleton, H. Andersen, W. Schwarting, L. Paull, J. Alonso-Mora, M. H. Ang Jr., S. Karaman, R. Tedrake, J. Leonard, D. Rus. In Proc. of the IEEE Symposium on Intelligent Vehicles (IV), 2017.

publisher pdf video
We present the development of a full-scale “parallel autonomy” research platform including software and hardware. In the parallel autonomy paradigm, the control of the vehicle is shared; the human is still in control of the vehicle, but the autonomy system is always running in the background to prevent accidents. Our holistic approach includes: (1) a drive-by-wire conversion method only based on reverse engineering mounting of relatively inexpensive sensors onto the vehicle implementation of a localization and mapping system, (4) obstacle detection and (5) a shared controller as well as (6) integration with an advanced autonomy simulation system (Drake) for rapid development and testing. The system can operate in three modes: (a) manual driving, (b) full autonomy, where the system is in complete control of the vehicle and (c) parallel autonomy, where the shared controller is implemented. We present results from extensive testing of a full-scale vehicle on closed tracks that demonstrate these capabilities.

Parallel Autonomy in Automated Vehicles: Safe Motion Generation with Minimal Intervention
W. Schwarting, J. Alonso-Mora, L. Paull, S. Karaman, D. Rus. In Proc. of the IEEE Int. Conf. on Robotics and Automation (ICRA), 2017.

publisher pdf video
Current state-of-the-art vehicle safety systems, such as assistive braking or automatic lane following, are still only able to help in relatively simple driving situations. We introduce a Parallel Autonomy shared-control framework that produces safe trajectories based on human inputs even in much more complex driving scenarios, such as those commonly encountered in an urban setting. We minimize the deviation from the human inputs while ensuring safety via a set of collision avoidance constraints. We develop a receding horizon planner formulated as a Non-linear Model Predictive Control (NMPC) including analytic descriptions of road boundaries, and the configurations and future uncertainties of other traffic participants, and directly supplying them to the optimizer without linearization. The NMPC operates over both steering and acceleration simultaneously. Furthermore, the proposed receding horizon planner also applies to fully autonomous vehicles. We validate the proposed approach through simulations in a wide variety of complex driving scenarios such as left-turns across traffic, passing on busy streets, and under dynamic constraints in sharp turns on a race track.