The term AI researchers use for the AI’s unreliability is “hallucination“:
He recently asked both LaMDA and ChatGPT to chat with him as if it were Mark Twain. When he asked LaMDA, it soon described a meeting between Twain and Levi Strauss, and said the writer had worked for the bluejeans mogul while living in San Francisco in the mid-1800s. It seemed true. But it was not. Twain and Strauss lived in San Francisco at the same time, but they never worked together.
Scientists call that problem “hallucination.” Much like a good storyteller, chatbots have a way of taking what they have learned and reshaping it into something new — with no regard for whether it is true.
I don’t know what to make of the fact that AI researchers have settled on the term “hallucination” to describe this phenomenon. I find it interesting.
Actually, I find it intriguing. “Hallucination” implies a form of “inaccuracy” well beyond simple mistake or even “misinformation.” There’s a nightmarish, dystopian quality to the word.
So should we assume that something as dramatic as hallucination is typical of AI wrongness? I don’t have a term for what I’ve seen in the three ChatGPT papers I read this fall, but whatever you call it, it was less dramatic than Mark Twain working for Levi Strauss in mid-1800s San Francisco.
We will see. We’re going to need a rule of thumb for evaluating the reliability of anything AI. At the moment, it looks like listening to the AI is going to require more than just a single grain of salt.
Artificial intelligence: other posts
And see:
Does the AI cheat, part 1
Does the AI cheat, part 2
Does the AI cheat, part 3
Is the AI a bull**** artist?
Catherine & Katharine 
The chatbots are powered mainly by large language models, which predict what token (word, punctuation, or grammatical modifier) comes next based on huge training sets. If they have been trained on enough academic papers, they “know” what sequences of tokens are likely, mimicking the citation format and author-journal-title words relationships they have seen. The models are very good at producing stuff similar to what is in the training set, but not copied from it.
“Hallucination” is not a bad word for the phenomenon, as it is similar to the creation of visual or auditory illusions by random firing of portions of the brain with no sensory input, just one network triggering another.
What is amazing is that “understanding” does not seem to be necessary for writing things that seem coherent and that convince people who don’t know the underlying truth. I suspect that AI-generated conspiracy theories and disinformation are going to be a big thing in the next couple of election cycles.
Incidentally, students definitely have fabulated citations in the past. I had to clear some junk out of my Google Scholar entries, where students in some college had cited papers that I had never written (and my name is unique, so they weren’t by someone else of the same name).
LikeLike