Safer autonomous systems are all about trust

The coming of autonomous systems doesn’t just mean self-driving cars. Advances in artificial intelligence will soon mean that we, for example, have drones delivering medicines, crew-less ships navigating safely through busy sea lanes, and all kinds of robots assisting us.

by KU Leuven, M-Group

As long as these autonomous systems stay out of sight, or out of reach, they are readily accepted by people. The rapid and powerful movements of assembly-line robots can be a little ominous, but while these machines are at a distance or inside protective cages we are at ease. However, in the near future we’ll be interacting with “cobots” – robots intended to assist humans in a shared workspace. For this to happen smoothly we need to ensure that the cobots will never accidently harm us. This question of safety when interacting with humans is paramount. No one worries about a factory full of autonomous machines that are assembling cars. But if these cars are self-driving, then the question of their safety is raised immediately. People lack trust in autonomous machines and are much less prepared to tolerate a mistake made by one. So even though the widespread introduction of autonomous vehicles would almost eliminate the more-than 20,000 deaths on European roads each year, it will not happen until we can provide the assurance that these systems will be safe and perform as intended. And this is true for just about every autonomous system that brings humans and automated machines into contact.

Trust matters

If deployed tomorrow, existing self-driving cars would have many fewer accidents than those driven by humans. But this doesn’t mean that people are ready to hand-over the steering wheel. We tolerate many thousands of deaths on the road every year, but the first crash involving two full-autonomous vehicles that results in a fatality will be/are now headline news all over the world. And then what? Will there be a public outcry? Will gangs come with pitchforks to smash the machines? Will self-driving cars be like the Hindenburg disaster and airships? Autonomous vehicles, indeed all autonomous systems, need to be made safe enough so that people trust them. The destination, therefore, is clear; the route, however, is a difficult one. The Safer Autonomous Systems ITN project is designed to get us to our destination, safely. 

Not based on evolution

Until now, safety assurance has been integrated into the design processes, based on safety standards and demonstrating compliance during the system’s test phases. However, existing standards are developed primarily for human-in-the-loop systems, where a human can step in and take over at any time. They do not extend to autonomous systems, where behavior is based on pre-defined responses to a particular situation. What’s more, current assurance approaches generally assume that once the system is deployed, it will not learn or evolve. On the one hand, advances in machine learning mean that autonomous systems can be given the potential to learn from their mistakes, and the mistakes of all the systems they are connected to, making their abilities to operate safely infinitely better than previous generations. On the other, machine learning means more uncertainty about how the system will decide to react to a particular circumstance in the future, making safety assurance a hard task, which can only be accomplished by a highly-skilled, interdisciplinary workforce.

Machine learning means more uncertainty about how the system will react to a particular circumstance, making safety assurance a hard task.

Are you ready yet, to take a seat on an autonomously controlled airplane? If you hesitate to say “yes”, then you are tacitly acknowledging the need for a training and research program such as the Safer Autonomous Systems ITN.

Safe under all conditions

The main objective of the Safer Autonomous Systems (SAS) project is to identify ways that we can establish people’s trust in autonomous systems by making these systems demonstrably safer. In order to achieve this objective we have identified three challenges to be addressed by the early-stage researchers (ESRs) in their 15 individual research projects. This simply-stated objective, and the interdisciplinary needs required for its realization, is of such complexity that we see a large training network involving some of Europe’s flagship companies – such as Bosch, Airbus and Jaguar Land Rover – together with leading European universities – like KU Leuven and the University of York – as the best way to tackle these challenges, which are briefly described as follows:

Increased autonomy, by definition, means a significant reduction of the time during which a human is involved in the system’s decision making, thereby reducing the residual control afforded to humans. Studies have shown that it may take minutes for a non-actively involved human operator (e.g. a passenger in a self-driving car) to take over control in case of an emergency. Moreover, just putting a self-driving car to a full stop on a busy high-way by removing its power (so-called fail-stop behavior) is definitely not a safe action. In contrast, an autonomous system should be fail-operational (perhaps with reduced functionality) under all circumstances, monitor its own safety and make its own decision about a sensible and safe reaction. The challenge therefore is to design autonomous systems in such a way that they remain safe under all conditions, even in the case of component failures.

Virtual model-based testing

Testing is the most intuitive way to reveal unsafe behavior. However, autonomous systems must operate in a near infinite range of situations. When we test autonomous systems, we must therefore systematically determine which range and diversity of situations should be simulated and tested. We need to test them on roads, in the rain, and with people in the way. We need to test them when they’re in intermittent supervisor contact and when they’ve got an unbalanced wheel. And we need to test them in all the possible combinations of those cases. Testing autonomous systems in the field is clearly too costly and too time consuming and might even be harmful for the system or its environment. Hence, virtual model-based testing is the only viable option. However, breakthrough solutions are required to guarantee the rigor of our virtual testing and to optimize its overall coverage.

The main objective of the Safer Autonomous Systems (SAS) project is to identify ways that we can establish people’s trust in autonomous systems by making these systems demonstrably safer.

Three sub-objectives to achieve trust

More autonomy is possible only through new technologies, e.g., machine learning, for which no accepted safety-assurance strategies currently existLegacy experience as well as established standards and regulations are lacking. Implicitly or explicitly, current safety-assurance practices and safety standards assume that the behavior of the system is known at the design stage and can be assessed for its safety prior to system deployment. As autonomous systems might learn and evolve over time, this is no longer possible. This means that meeting the current safety standards for autonomous systems is either impossible to do or completely insufficient to assure safety throughout the life time of the system.

To achieve the main Scientific/Technical (S/T) objective of trust in autonomous systems by overcoming the 3 challenges we have decided on three sub-objectives that will be the aims of the project’s 3 research Work Packages (WPs):

Objective 1: To integrate guaranteed safe behavior directly into the architecture/design of the autonomous system.

Objective 2: To prove by model-based safety-analysis techniques that the behavior of an autonomous system remains safe under all possible conditions.

Objective 3: To ensure that the safety-assurance strategies that combine the architectural/design measures with the evidence allow us to have trust in the autonomous system, which is very likely to be learning and evolving.

For more information on the Safer Autonomous Systems ITN project, contact prof. dr. ir. Davy Pissoort or prof. dr. ing. Jeroen Boydens.

3 Replies to “Safer autonomous systems are all about trust”

Comments are closed.