OpenAI sued over human review of ChatGPT conversations without user consent

OpenAI is being sued over who, exactly, gets to read your ChatGPT conversations.

A proposed class action filed this month in the U.S. District Court for the Northern District of California accuses the company of routing real user chats to outside contractors without properly telling people first. OpenAI was served on September 2.

The lawsuit takes aim at an internal OpenAI initiative called Project Lily, which outlet 404 Media first exposed on September 14. Per the complaint, contractors hired through third-party staffing firms for roles like "AI data reviewer" and "chatbot evaluator" read real ChatGPT prompts and full conversations, summarize what the user was trying to do, and then score and critique four different model responses on a scale of one to seven.

That process is a basic version of how chatbots like ChatGPT actually get smarter. Basically, humans grade the AI's answers, and those grades get fed back into training so the model learns which responses people prefer, a technique the industry calls RLHF or reinforcement learning from human feedback. The lawsuit's complaint is that users were never clearly told a person, not just a machine, might be the one reading their chats.

OpenAI does run conversations through an automated system before any human sees them. The complaint alleges that the filter doesn't always catch everything, meaning personal details sometimes reach contractors anyway.

404 Media found that reviewers work off a dashboard that includes a "user memories summary"—a recap of a person's past chats that can reveal details like their general location, profession, or personal life, even though usernames are stripped out. OpenAI has said the reviews are meant to cut down on two specific behaviors: the chatbot acting too human-like, and the chatbot being too agreeable, a trait researchers call sycophancy, where an AI tells a user what it thinks they want to hear rather than what's accurate.

ChatGPT has more than 900 million weekly users, and a lot of them treat it like a diary, a therapist, or a search engine that only they can see.

People type in health symptoms, breakup texts, tax questions, legal troubles. The lawsuit argues that OpenAI's privacy policy names some categories of outside parties who get access to user data, but never specifically flags data annotation or human evaluation vendors as one of them.

The complaint packs in eight separate legal claims, including violations of California's Unfair Competition Law, its Consumer Privacy Act, and common-law claims like intrusion upon seclusion—a privacy tort that covers someone prying into your private affairs in a way a reasonable person would find offensive.

Plaintiffs are seeking damages, restitution, and punitive damages.

On the injunction side, the ask is specific. The plaintiffs want OpenAI to require opt-in consent before any conversation goes to an outside reviewer, turn the "Improve the model for everyone" setting off by default instead of on, add a clear warning inside the chat window itself, and potentially delete work product tied to the reviewed conversations while retraining any models that used it.

OpenAI's response is due October 13.