As medical facilities and airlines increasingly integrate artificial intelligence (AI) into their operations, a recent study from The Ohio State University warns that an AI system's performance alone isn't enough to guarantee safety in high-stakes environments. The research underlines the imperative need to evaluate both the algorithms and the human operators in tandem, to get a clear picture of the impact AI has on human decision-making. This joint approach could potentially reveal the extent of risks associated with errors in these safety-critical settings.
The study, conducted by the engineering department at Ohio State, suggests that solely relying on AI technology – expecting it to function faultlessly – is a risky approach. In fact, it is suggested that teams should be designed to be capable to still operate effectively even when AI performance falters. Dane Morey, a research scientist at Ohio State's Department of Integrated Systems Engineering, emphasized the critical nature of the human-machine collaboration. "An AI algorithm can never be perfect. So if you want an AI algorithm that's ready for safety-critical systems, that means something about the team, about the people and AI together, has to be able to cope with a poor-performing AI algorithm," Morey told the Ohio State News Service.
In a study setting that replicated a remote patient-monitoring scenario, 450 nursing students and 12 licensed nurses from Ohio State were observed using AI-assisted technologies. Their task was to assess patient cases and determine the urgency of care needed. The findings indicated a significant improvement in performance – between 50% and 60% – when the AI provided accurate predictions about medical emergencies. However, this performance boost flipped to a dramatic drop, over 100% degradation in decision-making, when the AI predictions were inaccurate.
The Cognitive Systems Engineering Lab at Ohio State, directed by associate professor Mike Rayo, is working to bridge the identified gap in AI deployment within critical medical and defense contexts. Their Joint Activity Testing research program, developed with the assistance from faculty emeritus David Woods, interrogates how teams handle a spectrum of AI performances – from high to critically flawed. "The point is this is not about making really good safety-critical system technology. It’s the joint human-machine capabilities that matter in a safety-critical system," Morey stated in the study, recently published in npj Digital Medicine, according to the Ohio State News.
