Pastor Sues OpenAI Over "Dangerous Medical Recommendations" from ChatGPT

A pastor has sued OpenAI after alleging the software gave him "extremely dangerous medical recommendations," according to a report by The New York Times. This reportedly led to delayed care to treat a serious of pulmonary embolisms, as ChatGPT allegedly told Scott Winters that the symptoms he described were "not something dangerous." It even reportedly drew on his religious beliefs, telling him that "God did not design your body to endlessly fail."

The suit accuses OpenAI and CEO Sam Altman of negligence and the "unauthorized practice of medicine." It points to multiple instances in which the chatbot offered up its own diagnoses and treatment plans. The chatbot also allegedly told Winters to ignore pleas from friends and family to seek actual medical care.

"It injects itself as a wedge in between the user and their real-life network," Meetali Jain, co-counsel and executive director of Tech Justice Law, told The New York Times. To that end, Winters alleges that he told the chatbot that "the people in my church think I'm crazy for not going to the hospital." The suit says the chatbot responded by saying "most people (including well-meaning church members) simply don't understand."

He seeks financial damages from OpenAI, but is also asking the court to stop ChatGPT Health from operating until independent evaluators determine that it's safe to use. This is a platform that encourages users to upload health documents to the chatbot. Finally, the lawsuit is demanding stronger guardrails to prevent ChatGPT from answering questions about specific medical treatments and diagnoses. ChatGPT is supposed to already have guardrails in place for this type of thing, but the suit says they didn't work reliably in this case.

OpenAI has said that ChatGPT's terms of service makes clear that the service is not intended to be used for medical diagnoses or treatments. However, the company wants users to upload health records to ChatGPT Health to foster a discussion with the bot. The company also consistently crows about how many people use the platform for health inquiries, recently stating that 230 million people do so each week. This seems like an extremely slippery slope, as ChatGPT mimics humans in a way that a simple Google search doesn't.