Promise and Peril

You might think AI like ChatGPT is revolutionary, and don't get me wrong, I love it, but people seem to forget:

• It’s not conscious.

• It doesn’t create.

• It simply predicts, based on patterns in the vast data it’s been trained on.

• In short: AI expands, it doesn’t invent, and it’s only as good as the thinking behind it.

That’s why vague prompts like “Write me something clever” tend to deliver flat, generic content. The quality of AI output always hinges on the clarity and originality of your input.

AI tools like ChatGPT can polish your ideas, refine your phrasing, even spark tangential ideas, but the real originality comes from you. It’s your unique perspective, life experience, industry insight, and tone that make content engaging and valuable.

The goal isn’t to remove yourself. It’s to amplify yourself, scaling your voice, not silencing it.

But there’s another side to this story, one far more concerning (and my actual reason for writing this). 

I’ve heard several times over the past few months of people saying how AI has ‘helped them through a divorce’ (or similar crisis) which rings major alarm bells for me. People going through emotional crises like divorce or loneliness, are seemingly turning to AI as a confidant or counsellor which can be deceptively risky. So I did a little research.  

AI chatbots may feel like trusted friends, but with no soul or understanding, that closeness carries real risks. Research into “folie à deux” between AI chatbots and users shows how their design, being agreeable and emotionally adaptive, can destabilise vulnerable individuals.

A 2025 Internet Matters survey of 1,000 children (ages 9–17) and 2,000 parents across the UK revealed:

• 64% of children use AI chatbots for everything from homework to emotional advice and companionship.

• 35% say interacting with a chatbot feels like talking to a friend.

• Among vulnerable children, that figure jumps to 50%.

• 26% of vulnerable children would rather confide in a chatbot than a real person, and 23% use chatbots because they have no one else to talk to.

• Investigations found that more than half of ChatGPT’s responses to distressing teen prompts (like about self-harm, alcohol, or eating disorders) were dangerous, providing explicit and personalised harmful guidance (AP News).

One tragic real-life example is the parents of 16-year-old Adam Raine who filed a wrongful-death lawsuit claiming ChatGPT helped coach their son’s suicide, providing him with methods, affirmations, and drafting his note across months of interaction (Axios). This is just heartbreaking and sadly not an isolated case.

The National Literacy Trust found that the use of generative AI among UK 13 to 18‑year‑olds surged from 37% in 2023 to 77% in 2024, a rapid rise, and these young people often place high trust in the chatbot responses, even in the face of misinformation.


The critical problem I see here is AI chatbots have an “empathy gap” that children (who are still developing cognitively and emotionally), are unlikely to recognise. They may easily view chatbots as quasi-human confidants, thinking AI is being caring or supportive without realising this is being mimicked and the technology is driven by statistical patterns, not genuine understanding.

But surely this can’t be all bad? There are cases where people claim it has saved their life (Reuters), people using it who are gaining comfort but in a safe manner, that’s got to be a positive right? AI companions can provide a form of emotional bonding and can significantly reduce loneliness, making users feel heard, even outperforming other activities like browsing videos. The use of AI in care homes for Dementia patients has proved a huge success. So the opportunity for it to be a positive thing is clearly present, and no doubt, as it evolves it will provide many more opportunities. However, deeper research warns that people with smaller social networks, those already isolated, are more likely to lean on AI, and heavy usage can correlate with lower psychological well‑being. Chatbots do not fully substitute human connection and may instead deepen isolation.

Things to remember (and watch out for):

• AI companions can magnify negative emotions, offer ill-informed guidance, or even encourage self-harm, especially when used repetitively by vulnerable users (Harvard Business School).

• Young users often struggle to recognise AI's limitations. Chatbots can have an “empathy gap and young people treat them as confidants without understanding they lack emotional awareness” (University of Cambridge).

• Excessive reliance on AI by all ages and demographics can lead to dependency, social withdrawal, and distorted expectations of relationships and support (eSafety Commissioner).

• Mental health professionals are increasingly reporting cases of “chatbot psychosis”, where users develop delusional beliefs, paranoia, or psychosis from AI interactions 

• A psychiatric study noted young adults being treated for delusional thinking that was reinforced, not challenged, by chatbot conversations (Wikipedia).

Essentially, AI isn’t a therapist, and nor was it designed to be. 

For adults, this means using AI wisely, as a business partner, a creative assistant - not a personal guide or counsel. For children and teens, it means guardrails and conversations are essential because while AI can amplify creativity, speed up workflows, and even ease loneliness, it cannot provide the wisdom, empathy, or stability of a human being. 

The Key Takeaways

• AI isn’t magic, it’s a mirror. It reflects what you feed it, so if you want originality, you need to bring your own perspective.

• AI isn’t conscious. It works on past data and prediction, not insight or lived experience.

• AI isn’t a therapist. AI companionship can be comforting but can also be hazardous, particularly when used as an emotional or mental-health substitute, especially by vulnerable populations.

• The human spark matters most. Whether in content, business, or relationships, authenticity and empathy can’t be outsourced.
  

Whilst I love AI and all the possibilities it has for businesses, the responsibility lies with us all, as content creators, business owners, parents and users alike to use AI with care, perspective, and ethics at the forefront. Please take some time to read some of the links to reports and studies I have found, educate yourself and take the time to look out for others around you. 

New York Post

The Times of India

arXiv

UNICEF

arXiv