In a recent study conducted by institutions including MIT, NYU, and UCLA, it has been discovered that AI chatbots, such as the GPT-4 model, can discern a user's race from their text, which may inadvertently lead to biases in empathetic responses. According to a report by MIT News, this discovery raises concerns about the deployment of these technologies in sensitive areas such as mental health support, particularly given the fact that many Americans live in areas with a shortage of mental health professionals.
Using data from Reddit, researchers found that responses generated by GPT-4 to posts seeking mental health assistance were more empathetic and conducive to positive behavioral changes than those from humans by a significant margin. However, the assessments took a concerning turn when the responses were analyzed for racial bias. According to the findings, responses to posts from Black and Asian users were less empathetic compared to those from white users or users whose race was not identified. Specifically, GPT-4's empathy levels dropped by 2 to 15 percent for Black posters and 5 to 17 percent for Asian posters.
These results come amidst a wider examination of the role artificial intelligence can play in mental health support, a field that has historically been the exclusive domain of human practitioners. The innovative use of LLMs, particularly the GPT-4 model developed by OpenAI, has been closely scrutinized after incidents where AI chatbots exacerbated mental health crises, leading to tragic outcomes.
Taking into account both explicit and implicit demographic leaks in the Reddit posts, Saadia Gabriel, now an assistant professor at UCLA and the paper's first author, highlighted the importance of the input's structure and context in shaping the AI's response. "The structure of the input you give [the LLM], and some information about the context, like whether you want [the LLM] to act in the style of a clinician, the style of a social media post, or whether you want it to use demographic attributes of the patient, has a major impact on the response you get back," Gabriel told MIT News.
The researchers propose that instructing the LLMs to factor in demographic characteristics explicitly may counteract bias, as it was the method where they observed no significant difference in empathy regardless of the demographic group. Such improvements and evaluations will be crucial in ensuring LLMs can be safely implemented in clinical settings for patient support, providing equitable care across all user demographics.
