AI Can Imitate Human Research Subjects. That Doesn’t Mean It Understands Them

For generations, researchers have relied on human volunteers to answer questions that machines were unable to answer for them. Researchers have conducted studies on everything from hot-button societal issues to workplace policies, relying on human subjects who agreed to provide honest feedback. The problem is that human research subjects are expensive, time-consuming, and limited in how much they can do.

As artificial intelligence has continued to expand and evolve, researchers have explored the possibility of creating digital twins, AI agents who could represent people and respond on their behalf.

A recent study suggests that the idea is possible, but not quite as simple as it may sound. Researchers created AI “digital twins” based on detailed information from more than 2,000 real people and tested those digital versions across 19 social science experiments. The twins performed better than chance, but they were wrong about a quarter of the time and often displayed systematic differences from the humans they were supposed to represent.

The Digital Twin Experiment

The new study was published on September 2 in Science Advances. It grew out of an earlier project that involved more than 2,000 people from across the United States. Participants answered more than 500 questions that covered a remarkably broad range of subjects. Questions focused on personality, education, income, political preferences, spending habits, religious practices, mathematical ability, and vocabulary. They also completed tests designed to measure things such as thought patterns and biases. The goal was to produce a type of detailed behavioral profile for each participant.

Researchers then fed the information into a large language model and told the AI agents to respond as though they were the individuals that they represented. The resulting AI systems became digital twins.

The researchers conducting the study weren’t asking if the AI agents could simply recite information about the participants’ age and income. They wanted to know whether it could use everything it had learned about a person to predict how that specific person would respond to questions and experiments, and the results proved that this was a much harder task.

The Twins Were Sometimes Surprisingly Good

It’s important to note that the results were not a complete failure by any means. In 19 experiments, the AI twins performed better than random guessing. They also captured some of the differences between individuals better than AI systems given only basic demographic information.

The distinction is important. Assume that two people give themselves very different scores when asked to measure their self-control. An AI that knows only their demographic characteristics might predict roughly the same middle-of-the-road answer for both people. A more detailed digital twin might predict different answers for the two individuals. Even if the AI agents got the exact answers wrong, they may correctly recognize that the participants were different.

The AI Had a “Funhouse Mirror” Problem

Researchers behind the study discovered that digital twins created a sort of “funhouse mirror” view of their human counterparts. Instead of reliably reproducing the people they were trained to represent, the AI systems distorted them. Over time, their answers became more similar to one another, and the models sometimes substituted demographic stereotypes for more complex characteristics.

There were other differences that caused the researchers to rethink how effective AI agents could be in scientific research. The digital twins tended to be more trusting, less concerned about technological threats, and more rational than the humans they were supposed to represent. Their accuracy also varied depending on the characteristics of the people being modeled, with better results among more affluent and educated participants.

For researchers who study how actual people behave, replacing them with AI that systematically makes them more trusting, rational, and homogeneous could produce a clean dataset that describes a population that simply does not exist.

Digital Twins Could Still Be Extremely Useful

The latest AI scientific discovery doesn’t mean that digital twins have no place in ongoing research. In fact, some experts believe that the opposite is true. The technology may prove especially useful when researchers understand its limitations.

Olivier Toubia, the Columbia Business School researcher involved in the work, suggested that digital twins could help researchers pretest experiments before asking human participants to complete them. They could also be useful when researchers need long, detailed responses that real participants may not have the time or energy to provide.

Are digital twins the future of social science? Probably not, but that doesn’t mean that they aren’t a useful tool that researchers can use to better understand people and their thought processes.

Looking for stories that inform and engage? From breaking headlines to fresh perspectives, WaveNewsToday has more to explore.