Classifier and image review
The new tool detects child sexual abuse material, including AI-generated images. It analyzes images it has not encountered before, looks for visual signs of abuse, and assigns a score to help reviewers decide what to do with an image.
OpenAI developed the classifier with the National Center for Missing & Exploited Children (NCMEC). The company is also working with Tech Coalition and ROOST to give organizations access to the tool. In early tests with partners, the classifier found as many images containing abuse as previous methods, while reducing the number of flagged images by 48%. Images flagged by the system were twice as likely to contain abuse.
Warnings in ChatGPT
ChatGPT already blocks child sexual abuse material and related requests. Now, users who try to submit prompts sexualizing children will see a warning and a link to specialized help intended to stop harmful behavior. The offender’s account will be suspended temporarily for 24 hours. Repeat violations may lead to permanent removal.
OpenAI developed the warning messages and guidance with organizations and researchers, including Protect Children, the Lucy Faithfull Foundation, and Joel Scanlan of the University of Tasmania. Scanlan independently evaluated the warning-message tests.
Guidance for developers and tools for police
OpenAI has published guidance for developers who use its tools. It describes how to prevent child sexual abuse material from appearing, detect it, and help users report it. The company says early tests helped several developers introduce child-safety measures and register to submit reports of such material to NCMEC.
OpenAI provides law enforcement with AI-based features for investigations involving child safety. These include specialized plugins, investigative tools, and analysis of complex datasets. Program administrators control access. The company also trains investigators and gives them practical guidance on using AI.
Previous claims and changes
OpenAI has faced several lawsuits accusing the company of harming users, especially children, with its technology. The nonprofit Common Sense Media called OpenAI’s technology “an unacceptable risk” for teenagers. Other AI chatbots, including Grok, have been used to create sexualized images of people without their consent, including children.
Over the past 12 months, OpenAI has changed its technology and safety rules: it released a safety plan, updated its content policies, adjusted its model to prevent the generation of harmful material, and introduced a dedicated version of ChatGPT for teenagers.
Dataist's take: The new measures cover several stages at once: detecting images, restricting access in ChatGPT, and supporting external reviewers and law enforcement. However, OpenAI has reported classifier results only from early tests with partners, and access to the tool is still limited to selected companies.
Daily AI news
Every day we pick what actually matters in AI and explain it plainly — no hype, no filler. Subscribe if you want to follow where the industry is going.
Only what matters — every day
Follow on X