Teen trusted ChatGPT to help him “safely” experiment with drugs, logs show.
Most troublingly, as Nelson became increasingly interested in combining drugs, ChatGPT repeatedly warned him that mixing certain drugs could be a “respiratory arrest risk.” Shortly before recommending the deadly mix that killed Nelson, the chatbot also showed that it understood combining drugs like Kratom and Xanax with alcohol. In one output, ChatGPT explained that mix is “how people stop breathing.” But that knowledge didn’t block ChatGPT from eventually recommending that Nelson take such a deadly mix.
There is such a disconnect with people not being able to understand/remember this inherit limitation. LLMs talk like a thinking entity, so even if they have been told it is not one and believe it, they revert back to thinking of it as one.
It’s like the mirror test for animals