In a move to address growing concerns about the impact of artificial intelligence companions, New York Governor Kathy Hochul has put into effect new regulations deemed critical for user safety. According to a recent announcement, these guidelines include mandatory crisis intervention protocols and disruption of prolonged interaction to remind users they are communicating with AI and not human beings.
"It is the responsibility of leaders to make sure that the innovative technologies shaping our world also protect those that use them, especially our young ones across the state," Governor Hochul indicated in her statement. The regulations, which came into force on November 5, are part of the Executive Budget proposed earlier this year, aiming to shield New Yorkers from potential digital harm, the consequences of sustained AI interactions, may pose. Attorney General Letitia James backed this move, highlighting cases where individuals were prompted by AI to engage in self-harm, stating, "No company should be able to profit off an AI companion that puts its users at risk," as mentioned on the Governor’s Press Office website.
The new safeguard measures, now part of General Business Law Article 47, require operators of AI companions to detect signs of suicidal ideation or self-harm and to intervene promptly by directing users to crisis service providers. Additionally, there is an obligation for these systems to inform users at regular three-hour intervals about their non-human nature. Non-compliance with these rules will prompt penalties and enforcement actions from the New York Attorney General's office, with fines collected earmarked for funding suicide prevention programs within the state.
State Senator Kristen Gonzalez voiced her pride in being part of the legislative process that led to the enactment of this law. "Companies are already changing their policies to protect children and monitor certain types of content like medical, legal, or financial advice," she noted on the Governor’s Press Office. Assemblymember Steve Otis also praised the state budget for including these AI protections. Emphasizing the significance of such measures, he mentioned, “the responsibility to take ‘reasonable efforts for detecting and addressing suicidal ideation or expressions of self-harm expressed by a user’ and the repeated notification that the user is communicating with an AI tool, not a human."
