MobiGyaan
  • Telecom News
  • Smartphone News
  • OS News
  • Apps
  • Reviews
  • Guides
MobiGyaan
  • Telecom News
  • Smartphone News
  • OS News
  • Apps
  • Reviews
  • Guides
MobiGyaan
Home Apps

OpenAI introduces CSAM detection tool, new ChatGPT warnings and child safety measures

ChatGPT to introduce warnings and 24-hour access pauses for flagged harmful behaviour

by Divyang Makwana
November 16, 2026
in Apps, Other Stuff
Share on WhatsAppShare on Twitter

Add MobiGyaan as a preferred source on GoogleOpenAI has announced a new set of child safety measures aimed at preventing the misuse of AI systems for child sexual exploitation and abuse. The initiatives include a CSAM detection classifier available through the OpenAI API, new warnings in ChatGPT to interrupt harmful behaviour, safety guidance for developers, and AI-powered tools and training for law enforcement investigators.

The measures build on OpenAI’s Child Safety Blueprint and its existing efforts to detect, block and report child sexual abuse material (CSAM). The company is also working with child protection organisations, technology providers and law enforcement agencies to strengthen responses to online child exploitation.

OpenAI CSam Detection Child Safety

OpenAI Introduces CSAM Detection Classifier

One of the key announcements is a new image classifier designed to identify potential CSAM by examining visual indicators of abuse. The tool assigns scores to images to help human reviewers prioritise potentially abusive material, including AI-generated content. It can also help identify previously unseen images.

OpenAI says the classifier is the first broadly available CSAM detection tool with a specific signal indicating whether material may be AI-generated. It was trained, evaluated and developed for real-world deployment with assistance from the National Center for Missing and Exploited Children (NCMEC), which operates the CyberTipline, a centralised reporting platform for online child exploitation.

The classifier is available at no cost through the OpenAI API to vetted organisations of different sizes. OpenAI is also working with the Tech Coalition and ROOST to share access to the tool. Additionally, it supports Thorn’s Safer CSAM detection system and participates in the Tech Coalition’s Lantern programme, which enables the sharing of signals related to online child sexual exploitation and abuse.

CSAM Classifier Evaluation Results

OpenAI tested the classifier internally and with early-access partners, using their feedback to identify weaknesses and improve its performance.

In an evaluation involving user-uploaded images, OpenAI reported that the new classifier:

  • Flagged an estimated 48% fewer images for review while identifying roughly the same number of abusive images as its existing tool.
  • Produced flags that were about twice as likely to contain abusive material.

Early-access partners also reported results from their evaluations:

  • fal: Reviewers confirmed 72% of high-confidence flags as novel CSAM in tests involving human-reviewed production images.
  • invideo AI: One in four flagged cases resulted in a ban.
  • SafetyKit: Detected 90% of cases involving drawings or animations that sexualise children.

OpenAI has also published a CSAM classifier model card and an intake form for organisations interested in accessing the tool. The company plans to continue improving the classifier based on feedback from participating organisations.

ChatGPT To Display Warnings Against Harmful Behaviour

OpenAI is introducing new warnings in ChatGPT to interrupt attempts to find abusive material. These warnings will explain the harm caused by sexualising children, including through AI-generated content, and direct users towards help.

When an attempt to find abusive material is flagged, ChatGPT will display a warning explaining the consequences of such behaviour. The company will also remove the flagged conversation and pause the user’s access to ChatGPT for 24 hours, while keeping specialist support available.

Repeated violations may result in longer restrictions, account closure or reports to authorities.

OpenAI said the approach was developed with input from Protect Children, the Lucy Faithfull Foundation, Dr. Caoilte Ó Ciardha, a researcher and Honorary Reader in Forensic Psychology at the University of Kent, and Joel Scanlan of the University of Tasmania, who independently evaluated a trial of the deterrent messages.

OpenAI Publishes CSAM Safety Guidance For Developers

OpenAI has also published new CSAM guidance to help developers prevent abuse, detect harmful content and respond to suspected cases. The guidance was developed with input from the Tech Coalition, Thorn, NCMEC and the Internet Watch Foundation.

It covers several areas, including:

  • Establishing clear content policies and user reporting mechanisms.
  • Using detection tools to review and report suspected abuse.
  • Using safety identifiers to connect misuse to individual users.
  • Providing appropriate support for teams reviewing sensitive material.

Developers can use the guidance alongside OpenAI’s image classifier and Moderation API, ROOST’s self-hosted review console Coop, and Thorn’s Safer CSAM detection system.

OpenAI has also worked with Anthropic and Google to jointly commission additional guidance on responding to disclosures of child sexual exploitation and abuse in AI systems. Published by the Tech Coalition, it provides recommendations for developers handling such disclosures.

The company said it has seen an increase in developers accessing child safety tools and registering with NCMEC to report CSAM.

AI Tools & Training For Law Enforcement Investigators

Another part of OpenAI’s initiative focuses on supporting specialist child-safety law enforcement units with AI tools and training.

As discussed publicly with Thorn, OpenAI said AI can help investigators organise and prioritise information while reducing unnecessary exposure to abusive material.

The support includes dedicated plugins, investigative tools and complex data analysis capabilities, with access managed by programme administrators.

The first investigative tool is designed to help officers:

  • Review lengthy chat logs and legal process returns.
  • Identify potential connections between information.
  • Create timelines with findings linked to their original source material.

Initial training will include an overview and demonstration of the tools, along with a practical field guide for investigators. Access will depend on agency authorisation and programme requirements.

OpenAI also plans to propose additional training at regional, national and international conferences.

OpenAI Prohibit CSAM

OpenAI’s Continued Child Safety Efforts

OpenAI said it will continue improving its safeguards and working with child protection organisations, developers, researchers and law enforcement agencies to prevent child sexual exploitation and strengthen responses when abuse occurs.

The latest measures bring together image detection, user-facing interventions, developer guidance and investigative support as part of the company’s broader approach to child safety.

Tags: AIChatGPTOpenAI
SendTweetShare
Previous Post

LAVA Probuds Xplore 13° TWS earbuds launched in India at ₹969

  • About Us
  • Advertise
  • Privacy Policy
  • Contact Us

© 2008-2026 - MobiGyaan. All Rights Reserved.

  • About Us
  • Advertise
  • Contact Us
  • Disclaimer
  • Privacy Policy
  • Search
  • Submit App
  • Team MobiGyaan
  • Work with us

© 2008-2026 - MobiGyaan. All Rights Reserved.