OpenAI steps up child safety with new image classifier and ChatGPT warnings | Tech News | #childsafety | #kids | #chldern | #parents | #schoolsafey


OpenAI is expanding its child-safety measures beyond existing content restrictions. The US-based artificial intelligence company has introduced an AI image classifier to detect child sexual abuse material (CSAM), new warnings and temporary account restrictions in ChatGPT alongside the guidance for developers building AI platforms. The company also said that it is providing AI-powered tools and training to child-safety investigators, while expanding partnerships with organisations to improve the detection and reporting of online child exploitation. The move comes amid growing concerns over the misuse of generative AI to create child sexual abuse material.AI image classifierAccording to the company, the new image classifier is designed to identify potential CSAM by examining visual characteristics rather than relying only on matching images against databases of previously identified abusive material. This is intended to help safety teams detect new or modified images, including those generated using AI, which may not have an existing digital fingerprint.  The classifier produces risk scores that help reviewers prioritise images for further examination. OpenAI said this is the first broadly available CSAM classifier with a specific signal to indicate whether material may have been generated using AI. The tool was developed with assistance from the National Center for Missing and Exploited Children (NCMEC) in the US, which operates the CyberTipline, a central reporting platform for online child exploitation. OpenAI said early internal evaluations found that the classifier could reduce the number of images requiring review by 48 per cent while identifying roughly the same number of abusive images as its existing system. Images flagged by the new classifier were also about twice as likely to contain abuse. The company has made the tool available to vetted organisations through its API at no cost.ChatGPT to introduce warnings and temporary restrictionsAlongside the image classifier, OpenAI is introducing interventions within ChatGPT to discourage attempts to access or generate abusive material involving children. The company already prohibits CSAM-related requests and uses safeguards to prevent its models from responding to such prompts. The new measures add warnings that explain the harm caused by sexualising children, including when the material is AI-generated. When such behaviour is flagged, ChatGPT will remove the conversation and temporarily pause the user’s access to the service for 24 hours. The warning will also direct users towards specialist support services that can help them address harmful behaviour. OpenAI said repeated violations could result in longer access restrictions, account closure and reports to law enforcement authorities.New guidance for developers building AI platformsOpenAI has also published guidance for developers on preventing, detecting and responding to CSAM on their platforms. The recommendations cover setting clear rules for users, providing ways to report suspected abuse, using content detection tools and establishing processes to review and report flagged material. According to the company, the guidance was developed with inputs from the Tech Coalition, Thorn, NCMEC and the Internet Watch Foundation. The company has also worked with Anthropic and Google to commission guidance on responding to disclosures of child sexual exploitation and abuse in AI systems. The recommendations, published through the Tech Coalition, are intended to help developers handle such incidents and establish appropriate response procedures. AI tools for child-safety investigatorsAnother part of the announcement focuses on supporting law enforcement agencies investigating online child exploitation. OpenAI is providing specialist child-safety units in the US with AI tools and training to help them organise information and examine large volumes of investigative material. The first tool will help investigators review lengthy chat logs and legal process records, identify possible connections between individuals or cases, and prepare timelines with findings linked to their original sources. The company said the tools will be available subject to agency authorisation and programme requirements. OpenAI said AI could help investigators prioritise information and reduce unnecessary exposure to abusive material during investigations. Initial training will include demonstrations and a field guide to help law enforcement teams understand how the tools can be used in their work. Partnerships with child-safety organisationsOpenAI said it is also working with organisations across the technology and child-protection sectors to improve the detection and reporting of online abuse. It has joined the Tech Coalition’s Lantern programme, which allows participating companies to share signals related to online child sexual exploitation and abuse. The company is also working with NCMEC, Thorn and other organisations to make detection tools and reporting resources more accessible. OpenAI said tackling online child exploitation requires coordination between technology platforms, child-protection groups, developers and law enforcement rather than relying on individual companies to address the problem. 

————————————————
Source link