Experience with increasingly autonomous systems in aviation and other complex domains has shown that performance breakdowns tend to occur at transition points and in off-nominal conditions, rather than during routine operations. In particular, operators experience ‘automation surprises’ during transitions between levels of automation, in cases where the system acts in unanticipated or unexplained ways, and when it transfers control to the operator without adequate advance warning [1]. Further, a lack of transparency regarding the capabilities, limitations, and strategies of highly automated systems has been associated with trust miscalibration and the failure to intervene when necessary. Keeping operators involved, informed, and engaged on a continuing basis has proven critical for handling unexpected events and preventing such events from turning into accidents.
