The latest Turing Test reveals the conclusion of the Human or not game.

The latest Turing Test reveals the conclusion of the Human or not game.

AI21 Labs’ “Human or Not?” Experiment Challenges Assumptions About AI

Human or Not

AI21 Labs, a leading AI research organization, recently conducted a groundbreaking social experiment called “Human or Not?” that has sparked intriguing discussions about the capabilities of AI chatbots and their potential impact on human jobs. The experiment involved over 2 million participants who engaged in more than 15 million conversations through AI21 Labs’ website. At the end of each chat, participants had to guess whether their conversation partner was a human or an AI bot. Surprisingly, nearly one-third of the participants guessed wrong.

The inspiration for this experiment came from Alan Turing’s famous evaluation of a machine’s ability to exhibit intelligence indistinguishable from that of a human. Turing, a renowned mathematician, predicted in 1950 that computers would eventually be able to play the “imitation game” so convincingly that an average interrogator would have no more than a 70% chance of correctly identifying the machine after 5 minutes of questioning. This prediction laid the foundation for what is now known as the Turing Test.

AI21 Labs’ “Human or Not?” experiment aimed to test Turing’s prediction and shed light on the current state of AI in early 2023. The results of the experiment were both fascinating and thought-provoking. Overall, participants guessed correctly 68% of the time, demonstrating that distinguishing between humans and AI chatbots is not always easy. When paired with an AI chatbot, participants guessed correctly only about 60% of the time, while they achieved a 73% accuracy rate when conversing with another human.

While the experiment wasn’t a perfect Turing Test, it clearly showcased how AI models can convincingly mimic human conversation, challenging our assumptions about the limitations of AI. This has significant implications for AI ethics and raises questions about the future implementation of AI in our online world.

During the experiment, human participants employed various strategies to identify AI bots, such as asking personal questions, discussing current events, and assessing the level of politeness in the responses. However, the AI chatbots managed to confuse players with their human-like behaviors, including the use of slang, making typos, displaying rudeness, and demonstrating an understanding of the context of the conversation.

Amos Meron, the creative product lead at AI21 Labs during the experiment, stated that the goal of “Human or Not?” was to provide a deeper understanding of AI’s capabilities and its potential role in our future. The experiment aimed to move beyond viewing AI solely as a productivity tool and instead consider AI as future members of our online world. In a time when people are questioning how AI should be implemented, experiments like this help shape the conversation.

As someone who had the opportunity to participate in the experiment, I was consistently paired with humans and successfully guessed their identity each time. The distinction became clear to me when my conversation partners used internet slang, refused to answer questions, or demonstrated a lack of knowledge. Some players attempted to imitate AI chatbots by using phrases like “as an AI language model,” but their attempts were often imperfect, and human participants easily saw through them.

AI21 Labs’ “Human or Not?” experiment has undoubtedly challenged our preconceived notions about AI and its ability to replicate human conversation. With AI chatbots becoming increasingly popular, it is crucial to explore the ethical implications and consider how AI should be integrated into our future. This experiment serves as a reminder that AI is rapidly advancing and has the potential to reshape various aspects of our lives.