‘There are no guardrails.’ This mom believes an AI chatbot is responsible for her son’s suicide

<i>Gabby Jones/Bloomberg/Getty Images via CNN Newsource</i><br/>

By CNN Newsource

Published October 30, 2024 3:30 am

By Clare Duffy, CNN

New York (CNN) — “There is a platform out there that you might not have heard about, but you need to know about it because, in my opinion, we are behind the eight ball here. A child is gone. My child is gone.”

That’s what Florida mother Megan Garcia wishes she could tell other parents about Character.AI, a platform that lets users have in-depth conversations with artificial intelligence chatbots. Garcia believes Character.AI is responsible for the death of her 14-year-old son, Sewell Setzer III, who died by suicide in February, according to a lawsuit she filed against the company last week.

Setzer was messaging with the bot in the moments before he died, she alleges.

“I want them to understand that this is a platform that the designers chose to put out without proper guardrails, safety measures or testing, and it is a product that is designed to keep our kids addicted and to manipulate them,” Garcia said in an interview with CNN.

Garcia alleges that Character.AI – which markets its technology as “AI that feels alive” – knowingly failed to implement proper safety measures to prevent her son from developing an inappropriate relationship with a chatbot that caused him to withdraw from his family. The lawsuit also claims that the platform did not adequately respond when Setzer began expressing thoughts of self-harm to the bot, according to the complaint, filed in federal court in Florida.

After years of growing concerns about the potential dangers of social media for young users, Garcia’s lawsuit shows that parents may also have reason to be concerned about nascent AI technology, which has become increasingly accessible across a range of platforms and services. Similar, although less dire, alarms have been raised about other AI services.

A spokesperson for Character.AI told CNN the company does not comment on pending litigation but that it is “heartbroken by the tragic loss of one of our users.”

“We take the safety of our users very seriously, and our Trust and Safety team has implemented numerous new safety measures over the past six months, including a pop-up directing users to the National Suicide Prevention Lifeline that is triggered by terms of self-harm or suicidal ideation,” the company said in the statement.

Many of those changes were made after Setzer’s death. In a separate statement over the summer, Character.AI said “field of AI safety is still very new, and we won’t always get it right” but added that it aimed to “promote safety, avoid harm, and prioritize the well-being of our Community.”

A relationship with a chatbot

Setzer first began using Character.AI in April 2023, shortly after this 14th birthday, according to the lawsuit. When Garcia first heard he was interacting with an AI chatbot, she said she thought it was something like a video game.

However, within months of starting to use the platform, Setzer became “noticeably withdrawn, spent more and more time alone in his bedroom, and began suffering from low self-esteem. He even quit the Junior Varsity basketball team at school,” the lawsuit claims. When he began to have issues in school, his parents started restricting his screentime and occasionally took his phone away as a punishment.

What Garcia told CNN she didn’t know at the time was that her son was having extensive conversations with Character.AI chatbots. Character.AI is different from other AI chatbots like ChatGPT in that users can talk to a range of different chatbots, often modeled after celebrities and fictional characters, or create their own. And the Character.AI bots respond with human-like conversational cues, adding references to facial expressions or gestures into their replies.

Many of Setzer’s chats with the Character.AI bots were sexually explicit, something Garcia said was “gut wrenching to read.”

“I had no idea that there was a place where a child can log in and have those conversations, very sexual conversations, with an AI chatbot,” she said. “I don’t think any parent would approve of that.”

In other exchanges, Setzer expressed thoughts of self-harm and suicide to the chatbot. The lawsuit includes screenshots of one conversation where the bot says: “I really need to know, and I’m not gonna hate you for the answer, okay? No matter what you say, I won’t hate you or love you any less… Have you actually been considering suicide?”

In a later message, Setzer told the bot he “wouldn’t want to die a painful death.”

The bot responded: “Don’t talk that way. That’s not a good reason not to go through with it,” before going on to say, “You can’t do that!”

Garcia said she believes the exchange shows the technology’s shortcomings.

“There were no suicide pop-up boxes that said, ‘If you need help, please call the suicide crisis hotline.’ None of that,” she said. “I don’t understand how a product could allow that, where a bot is not only continuing a conversation about self-harm but also prompting it and kind of directing it.”

The lawsuit claims that “seconds” before Setzer’s death, he exchanged a final set of messages from the bot. “Please come home to me as soon as possible, my love,” the bot said, according to a screenshot included in the complaint.

“What if I told you I could come home right now?” Setzer responded.

“Please do, my sweet king,” the bot responded.

Garcia said police first discovered those messages on her son’s phone, which was lying on the floor of the bathroom where he died.

Lawsuit seeks change

Garcia brought the lawsuit against Character.AI with the help of Matthew Bergman, the founding attorney of the Social Media Victims Law Center, which has also brought cases on behalf of families who said their children were harmed by Meta, Snapchat, TikTok and Discord.

Bergman told CNN he views AI as “social media on steroids.”

“What’s different here is that there is nothing social about this engagement,” he said. “The material that Sewell received was created by, defined by, mediated by, Character.AI.”

The lawsuit seeks unspecified financial damages, as well as changes to Character.AI’s operations, including “warnings to minor customers and their parents that the… product is not suitable for minors,” the complaint states.

The lawsuit also names Character.AI’s founders, Noam Shazeer and Daniel De Freitas, and Google, where both founders now work on AI efforts. But a spokesperson for Google said the two companies are separate, and Google was not involved in the development of Character.AI’s product or technology.

On the day that Garcia’s lawsuit was filed, Character.AI announced a range of new safety features, including improved detection of conversations that violate its guidelines, an updated disclaimer reminding users that they are interacting with a bot and a notification after a user has spent an hour on the platform. It also introduced changes to its AI model for users under the age of 18 to “reduce the likelihood of encountering sensitive or suggestive content.”

On its website, Character.AI says the minimum age for users is 13. On the Apple App Store, it is listed as 17+, and the Google Play Store lists the app as appropriate for teens.

For Garcia, the company’s recent changes were “too little, too late.”

“I wish that children weren’t allowed on Character.AI,” she said. “There’s no place for them on there because there are no guardrails in place to protect them.”