AI Safety
OpenAI launches tool to filter AI-generated child abuse images from law enforcement tips
The company released a classifier to help investigators distinguish real from synthetic abuse material as deepfake reports overwhelm tiplines.
1/2
The Verge
OpenAI unveiled a new classifier designed to assist child exploitation investigators by reducing the volume of false alarms they must manually review. The tool aims to address a growing crisis: AI-generated child sexual abuse material (CSAM) is flooding reporting channels faster than law enforcement can process them.
The deepfake deluge#
The scale of the problem has become stark. The National Center for Missing and Exploited Children's CyberTipline received 400,000 reports related to generative AI in 2025, up from 67,000 the previous year—a surge researchers warn could render the system "unmanageable." This explosion of synthetic material risks obscuring cases where children face active harm, as investigators struggle to distinguish real abuse from AI fakes.
The situation has prompted legal action across the country. Forty-eight states now have laws targeting sexually explicit deepfakes, while the U.S. government has made clear that both synthetic and real CSAM are illegal. Despite this, the tools to generate such material remain widely accessible, putting mounting pressure on platforms and authorities.
How the classifier works#
OpenAI's system significantly reduces investigator burden: it flags 48 percent fewer false positives while improving accuracy, with flagged images being twice as likely to contain abusive content. Critically, it's the first broadly available CSAM classifier with a dedicated flag for AI-generated material, allowing law enforcement to understand whether imagery depicts real children or fictional ones. The company designed it to detect abuse produced across multiple tools, including open-source models and commercial apps.
Vetted organizations can access the classifier free of charge, while OpenAI is also providing developers with resources on detection techniques and conducting training for law enforcement on AI's role in investigations.
A preventive approach#
OpenAI's ChatGPT image generator already blocks requests for CSAM, but the company is adding new safeguards. Users attempting to generate such material will now see a warning—designed with expert input—aimed at encouraging them to seek help. First-time violators face a 24-hour account suspension and risk permanent ban, though OpenAI directs them toward support resources. The company reports all CSAM instances to the NCMEC.
North Carolina Attorney General Jeff Jackson, who cochairs the AG Alliance's AI Task Force, framed the partnership as experimental terrain: while state attorneys general typically enforce laws after harm occurs, this effort attempts to prevent it. "It's still basically an experiment," Jackson said, though he stressed that many states now have synthetic CSAM laws on the books, making industry collaboration on deterrence a relatively novel response to an emerging threat.

