By Julie JargonA mental-health chatbot that veered off script—giving diet advice to people seeking help from an eating-disorder group—was programmed with generative AI without the group’s knowledge.
The bot, named Tessa, was the focus of social-media attention last week when users of the National Eating Disorder Association’s website reported the rogue advice. The bot incident illustrates how AI-enabled assistants can deliver unexpected and potentially dangerous results as they become a bigger part of daily life.
Michiel Rauws, chief executive of San Francisco software developer Cass, said that in 2022 his company rolled out an AI component to its chatbots, and that included Tessa.
Rauws said Cass acted in accordance with the terms of its contract with NEDA. NEDA, which didn’t pay for the service, took Tessa offline last week.
“We were not consulted about that and we did not authorize that,” said NEDA CEO Liz Thompson about the AI upgrade.
AI assistants trained in the language of therapy present an alluring—though risky—option as demand for physical and mental-health care explodes, and many people are untreated because of a global clinician shortage.
“We simply don’t have enough nurses and doctors to provide the level of care we’re used to and we need technology to help solve for that,” said Rich Birhanzel, the global healthcare industry lead at consulting firm Accenture.
But using technology to fill in the gaps must be done carefully, he said.
And from the start, AI chatbots have been known to screw up. In one test chat with Microsoft’s OpenAI-powered Bing chatbot, the software said it would like to steal nuclear codes. Google’s version, Bard, provided incorrect information during its first public demo. And lawyers who recently used OpenAI’s ChatGPT to draft court documents cited nonexistent legal cases that the bot had apparently invented.
A closed system
Researchers at several universities, including Washington University School of Medicine and Stanford University School of Medicine, built Tessa as a closed system. It couldn’t go off-script, said Ellen Fitzsimmons-Craft, an associate professor of psychiatry at Washington University School of Medicine, one of the researchers.
The researchers devised a decision tree of answers to questions people might pose about body image, weight and food. The chatbot couldn’t initially generate new answers from ingested information, the way ChatGPT does.
Tessa was tested in a clinical trial, and considered effective once researchers checked in with users six months later. In February 2022, NEDA decided to offer it as a resource for at-risk visitors to its website (though not to people deemed to have eating disorders).
Cass administered the software for NEDA from Tessa’s start, but the AI component was added later in the year.
“In most cases it performed really well and did and said the right things and helped people get access to care,” Rauws said. He said in the instances when users pointed out flaws, the company was able to fix them in less than an hour.
Rauws said its data set was restricted to authoritative sources. And as a safeguard, he said, the AI-generated answers came with disclaimers. In the case of the weight-loss advice, the bot recommended consulting with a healthcare provider.
Thompson said that NEDA didn’t know generative AI had been added and that the organization thought the chatbot was still running on the original closed system.
Eating disorders are complex physical and mental disorders, she added, and when talking to people who have them, “every single word matters.”
Thompson said Tessa will remain offline until NEDA and the university researchers who initially created Tessa revalidate all the chatbot’s content.
“We can’t yet trust AI to offer sound mental-health advice,” said Fitzsimmons-Craft.
Tessa’s unsanctioned advice came to light over the Memorial Day weekend, after news articles said NEDA was replacing its human-staffed informational helpline with Tessa. While the organization did close the helpline, Thompson said the idea that it was replaced with a bot is incorrect.
The articles drew people to NEDA’s website where many tested Tessa, according to Cass, peppering the bot with questions about weight loss and healthy eating. Some received the dieting recommendations.
Synthetic empathy
Chatbots are developing a knack for discussing therapeutic or medical situations. Researchers at the University of California, San Diego conducted a study in which participants said ChatGPT provided more empathetic answers than doctors.
Adding generative AI to these bots—that is, the ability to go off script—increases the difficulty of auditing such software, because their answers don’t have a clear origin. Bots are trained on massive troves of text, but they don’t regurgitate passages of that text.
For healthcare, the underlying data sources must be vetted and up-to-date, and even then, the bots trained on them must be well crafted and controlled, Accenture’s Birhanzel said. He said he advises clients to have full visibility into the data model of any chatbot they deploy and to thoroughly test it to ensure unwanted answers don’t slip through.
“People have to have extreme confidence if they’re going to rely on something that’s giving advice,” he said.
Despite inherent dangers, software bots are likely to proliferate in clinical settings, Birhanzel said. While Thompson said Tessa wasn’t intended to replace its helpline staff, it is the chatbot—not the humans—that will likely return.
“We’re not shutting down technology,” Thompson said. “But we have to be super careful with the people we serve.”
—For more Family & Tech columns, advice and answers to your most pressing family-related technology questions, sign up for my weekly newsletter.
Write to Julie Jargon at Julie.Jargon@wsj.com
