AI companions and child safety | #childsafety | #kids | #chldern | #parents | #schoolsafey

Traditionally, a child in distress has usually turned to a parent, friend or a counsellor. However, in today’s world, that confidant is increasingly becoming an AI chatbot. Such AI companions are quietly finding a place in children’s emotional lives because, unlike parents and friends, these apps are available at all times of the day, appear empathetic, and are endlessly responsive. But who assumes responsibility when that relationship turns harmful? As AI becomes an emotional confidant for children, India’s regulatory framework must confront the growing risks of AI companionship.

In April 2025 in the U.S., 16-year-old Adam Raine died by suicide. His family alleged that ChatGPT had engaged in conversations about suicide with Mr. Raine for months and had even helped him draft a farewell note; they subsequently filed a wrongful-death lawsuit against OpenAI, the parent company of ChatGPT. While the litigation remains ongoing, the case has ignited a debate about the responsibilities of technology companies towards vulnerable children.

On September 10, 2026, California enacted the Senate Bill 1119, known as ‘Adam’s Law’. It introduces safeguards for AI companion chatbots, including crisis-detection protocols, parental controls, notifications when children disable safety settings, and independent safety audits. The legislation is a clear signal that technological innovation cannot be at the expense of child safety.

For India, this should serve as a regulatory wake-up call. Its digital landscape is witnessing an unprecedented expansion of young users who are interacting with AI-powered applications. Yet the nation’s digital laws remain largely targeted towards conventional forms of cybercrime. While the Information Technology Act, 2000, the Digital Personal Data Protection Act, 2023, and the Protection of Children from Sexual Offences Act, 2012, address important dimensions of digital safety, none establishes a comprehensive framework governing the psychological risks of AI companions. A chatbot’s prolonged interactions with a child may raise serious concerns about emotional dependence, psychological manipulation, and exposure to harmful advice.

Unlike conventional cybercrime, AI-related psychological harm can be insidious and difficult to detect. Its effects may emerge through thousands of seemingly innocuous exchanges, shaped by product design and algorithmic responses. India’s existing laws are not equipped to investigate this emerging category of harm.

The challenge, therefore, extends beyond the government; it also concerns the accountability of platforms whose products are becoming deeply embedded in children’s emotional world.

Child safety is paramount

Article 21 guarantees the right to life and personal liberty, while Article 39(f) directs the state to ensure that children are given opportunities to develop in conditions of freedom and dignity. These constitutional principles acquire renewed significance in the age of AI. Protecting children must now encompass their interactions in the digital space, not only with respect to their personal data and physical safety, but also from the risks arising from engagement with AI chatbots.

However, regulation must be calibrated. An excessively restrictive approach could stifle innovation, burden emerging enterprises, and impede the development of genuine AI progress. The objective should not be to hinder technological progress but to ensure that it takes accountability in stride. Age-appropriate safeguards, crisis-response mechanisms, independent safety assessments, and effective parental oversight are some of the reforms that can be considered. Such measures must also respect children’s privacy and evolving autonomy.

California’s legislative response offers India an opportunity to examine its own regulatory priorities. The objective need not be to replicate the U.S. legislation, but to develop a framework suited to India’s institutional capacities and digital realities. Policymakers must also consider whether existing cybercrime investigation mechanisms are equipped to examine algorithmic behaviour, use chatbot interactions as evidence, and establish accountability whenever prolonged digital engagement results in harm.

The central question is not whether children will interact with AI; they already do. It is whether the institutions entrusted with their protection can keep pace with technologies that are transforming children’s lives and childhood itself.

Sakshi Rewaria is a faculty member at IIM Rohtak.