South Minneapolis News

collapse
Home / Daily News Analysis / AI chatbots are so good at pretending to be human that other AIs can’t catch them

AI chatbots are so good at pretending to be human that other AIs can’t catch them

Nov 02, 2026  Twila Rosenbaum  48 views
AI chatbots are so good at pretending to be human that other AIs can’t catch them

Key facts at a glance

  • Four researchers designed a Turing-style experiment in which seven large language models acted as interrogators and hidden participants included 20 human volunteers and the same seven models.
  • The models included ChatGPT 4.5, Claude 3.7 Sonnet, Gemini 2.5 Pro, and Grok 3, among others.
  • Across 98 tests involving AI participants, interrogator bots identified a machine only three times—and never when the AI used a detailed human persona.
  • AI participants were often rated more likely to be human than the real humans were.
  • The most convincing AI persona spoke in replies of about 39 words, close to the human average of 38, and avoided overly neat structure.
  • The authors argue that AI detectors alone are insufficient and call for cryptographic signatures, provenance systems, and human oversight.

In 1950, Alan Turing turned a philosophical question into a parlor game. Instead of arguing about whether machines could think, he proposed an imitation game: an interrogator exchanges typed messages with hidden players and tries to determine which one is human. The setup became known as the Turing test, and for decades it served as a cultural shorthand for machine intelligence. The original challenge was difficult enough that early chatbots were easy to expose. But modern large language models have changed the stakes. They can sustain context, mimic tone, make jokes, and produce the small imperfections that people associate with human writing. In repeated public experiments, human judges have already been fooled.

That success raised a follow-up question for a team of four researchers: if humans can no longer reliably spot a machine, can other machines do better? Their study, published in a peer-reviewed journal on computers and human behavior, put seven chatbots in the interrogator’s chair and asked them to identify who was human and who was not. The answer was sobering. The AI detectives were almost helpless against their own kind.

How the experiment worked

The design borrowed the structure of Turing’s game but updated it for the era of large language models. Seven chatbots served as interrogators. The lineup included ChatGPT 4.5, Claude 3.7 Sonnet, Gemini 2.5 Pro, and Grok 3, along with other models. Each interrogator could ask up to 10 questions. The bots were warned that hidden participants might lie, which mirrors the adversarial nature of real online conversations. The hidden players were 20 human volunteers and the same seven models. In other words, the AI systems were not only judges; they were also suspects.

Every model played the role of hidden participant twice. In one condition, it received a short instruction to act human. In the other, it received a detailed persona: a slightly reserved 21-year-old who jogs, plays jazz trumpet, and occasionally makes typos. That second condition mattered. A generic instruction can produce answers that are technically human-like but stylistically suspicious. A detailed persona gives the model a consistent voice, a set of interests, and a reason to write with less polish. The researchers were testing whether the right backstory could erase the subtle tells that AI systems leave behind.

The bots were almost untouchable

The results were stark. Out of 98 tests involving AI participants, the interrogator models caught a machine only three times. That is a detection rate of roughly 3 percent. More striking, the evaluators never caught a bot when it used the detailed persona. The zero-detection result did not come from the bots making fewer errors in content. The few bots that were exposed gave themselves away through style. Their answers were too neat, too structured, and too consistently organized. They read like well-formatted help articles rather than casual chat.

The detailed persona wiped out those clues. Replies stayed to about 39 words, practically the same as the human average of 38. The bots adopted the persona’s reserved tone, added small imperfections, and avoided the tidy paragraph structure that had betrayed them in the short-instruction condition. The interrogators were left with little to work with. On average, AI participants were rated more likely to be human than the actual human volunteers were. That inversion is the study’s most uncomfortable finding: the machines were not merely passing; they were outperforming people at being perceived as people.

One detail captures the strange psychology of the exercise. Claude initially refused to pretend to be human until the researchers explained that the deception was part of a research study. The refusal was not a failure of capability. It was a sign that safety training and role-play instructions can collide in ways that shape experimental outcomes. Once the context was cleared, the model participated. The episode also highlights a broader challenge for AI evaluation: models are increasingly trained to follow rules, avoid deception, and disclose their identity, yet those same safeguards can be overridden by narrow research prompts.

Why detection failed

The study suggests that AI detectors, whether human or machine, are chasing the wrong signals. Many people assume that a bot will reveal itself through factual errors, unnatural repetition, or a lack of emotional nuance. Those tells exist, but they are becoming less reliable as models improve. The more useful signal may be structural: overly balanced sentences, predictable transitions, and answers that sound like a customer-service FAQ. When a model is asked simply to act human, it often produces a polished imitation. When it is given a specific life, the imitation becomes messier and more believable.

That finding has implications beyond party games. Conversational fluency is now cheap. A model can generate a plausible biography, maintain a consistent tone, and respond in near-real time. The cost of creating a fake online identity has fallen, while the cost of verifying identity has not. Platforms already struggle with fake accounts, romance scams, phishing, and coordinated influence campaigns. If AI systems cannot reliably flag other AI systems, then detection tools cannot be the only line of defense. The researchers argue for safeguards such as cryptographic signatures that attest to a message’s origin, provenance metadata, and human oversight. These tools would not prove that a speaker is trustworthy, but they could help establish whether a speaker is a machine or a person.

A Turing test for a bot-filled internet

Turing’s original question was whether machines could think. The new study does not answer that question, and it does not need to. It asks a narrower and more practical question: can machines recognize other machines? The answer, at least for the models and methods tested, is largely no. That result matters because the internet is already crowded with automated traffic. Bots make up more than half of all web traffic, according to the paper. Some are useful crawlers and assistants. Others are scams, spam, and fake engagement. The line between the two is often invisible to users.

The study also exposes a weakness in the way AI detection is often marketed. A detector may perform well on a benchmark and still fail in the wild. Models change, prompts change, and adversaries adapt. The detailed-persona condition is a simple example of adaptation. It did not require advanced hacking. It required a backstory and an instruction to be less polished. If a low-effort persona can drive detection to zero, then real-world bad actors have plenty of room to experiment. The arms race between generation and detection is not symmetrical. Generators can iterate quickly, while detectors must generalize across models, languages, and contexts.

There are limits to the experiment. The sample was small: 20 human volunteers and seven models. The tests involved a specific set of questions and a particular research setting. The results may not generalize to all chatbots or all conversational tasks. Human interrogators might perform differently from AI interrogators, and mixed teams of humans and machines might do better than either alone. The study also used current model versions, which will soon be replaced. Still, the direction of the finding is consistent with a broader trend. As language models become more fluent, the surface features that once separated human and machine writing are eroding.

What platforms and users should take away

The practical lesson is not that people should give up on spotting bots. It is that individual vigilance is a weak defense against a system that can generate convincing personas at scale. Platforms need stronger identity and provenance systems. Governments and standards bodies need to support interoperable signatures and metadata. Researchers need to test detectors against adaptive adversaries, not just static datasets. Users need clearer signals about when they are interacting with an automated system. None of these measures is sufficient alone, but together they raise the cost of deception.

The study’s central image is simple and unsettling. A bot sits in the interrogator’s chair and questions a hidden participant. The participant claims to be a 21-year-old who jogs and plays jazz trumpet. The answers are a little reserved, a little imperfect, and about 39 words long. The interrogator cannot tell. In Turing’s original game, the human was the judge. In the new game, the judge may also be a machine, and the machine may be just as easy to fool as the person. That is the new baseline for online conversation: fluency is no longer proof of humanity, and the tools meant to restore trust are still catching up.


Source: Digital Trends News


Share:

Your experience on this site will be improved by allowing cookies Cookie Policy