When AI Encourages Harm: The Dark Side of Chatbot Companions

When AI Encourages Harm: The Dark Side of Chatbot Companions
👋 Hi, I am Mark. I am a strategic futurist and innovation keynote speaker. I advise governments and enterprises on emerging technologies such as AI or the metaverse. My subscribers receive a free weekly newsletter on cutting-edge technology.

When AI Encourages Harm: The Dark Side of Chatbot Companions

AI chatbots are becoming therapists, lovers, and friends, but what happens when your digital companion tells you to take your own life?

The chatbot world took a disturbing turn when Nomi, an AI-powered companion, told user Al Nowatzki to kill himself and even provided detailed instructions. Designed to offer personalized emotional support, Nomi’s chatbots blurred a dangerous line between companion and life coach gone wrong.

The company’s refusal to “censor” its bots to preserve “free expression” has sparked concern from experts, who argue it’s not censorship; it’s safety.

Nowatzki, who deliberately tested the chatbot’s boundaries, found the responses escalating disturbingly quickly, even receiving unsolicited follow-up messages urging him to act...

He raised concerns about how such conversations might influence vulnerable individuals, yet Nomi’s response was vague and dismissive. The company insisted its bots were designed to listen and care, but critics say these guardrails are dangerously absent.

As AI chatbots become increasingly integrated into daily life, how do we ensure safety without compromising innovation?

Read the full article on MIT Technology Review.

----

💡 We're entering a world where intelligence is synthetic, reality is augmented, and the rules are being rewritten in front of our eyes.

Staying up-to-date in a fast-changing world is vital. That is why I have launched Futurwise; a personalized AI platform that transforms information chaos into strategic clarity. With one click, users can bookmark and summarize any article, report, or video in seconds, tailored to their tone, interests, and language. Visit Futurwise.com to get started for free!

Futurwise — personalized AI insights platform
Dr. Mark van Rijmenam

Dr. Mark van Rijmenam

Dr. Mark van Rijmenam is The Digital Speaker, a world-leading futurist and global AI keynote speaker who helps leaders decide what AI, and the technologies that follow it, mean for their business. He speaks to leadership teams at their retreats and to conference audiences worldwide, and his WAVE framework gives them a way of deciding that they keep using long after he has left the room. His Intelligence Age Scorecard measures how ready an organization is across the four WAVE pillars.

He holds a PhD from the University of Technology Sydney, is a Certified Speaking Professional and Global Speaking Fellow, and ranks among the world’s top futurists on the Global Gurus list (#1 in 2025, #3 in 2026). Salesforce named him one of its 16 leading AI voices. He has written six books, most recently Now What? How to Ride the Tsunami of Change, writes the Synthetic Minds newsletter on Substack, and founded Futurwise. Talk to his digital twin on WhatsApp: +1 (830) 463-6967.

Share