Judging Faces Without Words
Imagine sitting on a city bus full of strangers. Look around at the people riding with you. If you needed help, how would you choose whom to trust? By scanning people's faces, your brain is already making interpretations about their personalities by judging certain facial features.
For decades, neuroscientists have been studying how facial bias ultimately influences the way people behave, but the experiments have been limited by the words used to describe people. Studies typically ask participants to rate faces on a fixed set of rating scales that are provided by the experimenter on words like "competent" or "trustworthy." Now, a team of researchers at Caltech has developed a way to study trait judgment of faces without using any word labels.
"We asked people to directly interpret the similarity in personality between faces," explains Yue Xu (PhD '26), who led the study and is first author on a paper about it published online October 19 in Nature Human Behaviour. "It's a more natural way of thinking about people's personalities without being cued to think about specific trait words."
In the past, studies aiming to link our perception of faces with personality traits typically involved study subjects who looked at faces on a computer monitor, then made a quick judgment by answering questions like, "How trustworthy does this person look to you on a scale from 1 to 5?" The results of prior work have linked such snap opinions to decisions people make in real life about politicians, hiring considerations, and more. But they also raise questions about what happens if you speak a different language, if you are only given a particular set of words concocted by the experimenter, or if you're unable to use language at all.
The new study is the latest effort by Ralph Adolphs (PhD '93), Bren Professor of Psychology, Neuroscience, and Biology at Caltech and senior author of the paper, to reduce bias in face perception studies. Instead of rating faces on trait words, Adolphs, Xu, and the team asked more than 3,000 study participants in two different experiments to look at faces on a computer screen without giving them any rating scale. Instead, participants were instructed to arrange the faces based on how similar or different they appeared to be based on personality traits; similar faces were placed close to one another, and different faces were placed further apart.
The first experiment focused on a set of predominantly white faces to allow the researchers to do a direct comparison with a previous study that used those same faces. The second experiment featured Black and Asian faces, in addition to white faces, to examine how the first experiment's findings extend to individuals of different races.
Using computational methods to assess how participants had arranged faces on their screen, the researchers were able to identify five key trait dimensions present across both experiments: approachability, warmth, competence, morality, and youthfulness. As expected by the researchers, the second experiment also showed prominent effects of stereotypes associated with race.
"It's important to note that what we are studying is people's bias and stereotypes, because they actually don't know about these people outside of this task," says Xu, who was a Chen Graduate Fellow during her time at Caltech. "Our findings don't say anything about the true personality of the actual individuals represented by these faces but tell us what people's first impressions are when they look at these faces."
Next, the team compared their five key dimensions with historical data from prior experiments that did use ratings on words such as sociable and caring, and found they matched up well.
"When I took this task myself, it felt extremely difficult to do without language ratings, as is usually done," says Adolphs, who is also an affiliated faculty member of the Tianqiao and Chrissy Chen Institute for Neuroscience. "So, I was amazed that it worked at all. People gave consistent results that they could not have produced if they weren't doing the experiment properly."
Not only does the new method help reduce study bias by removing trait words, but it also makes language unnecessary, opening future experiments to more people from different cultures or backgrounds or those who are nonverbal.
"One obvious application would be that you could do the task with participants who cannot use the English language, perhaps because they are aphasic or do not speak English," Adolphs says. "I could imagine extending it also to infants or nonhuman primates. It opens up a whole new field of research since we now have a model of how we judge people's faces that does not depend on language."
Xu is continuing her research as a postdoctoral fellow at UC Riverside, and the Adolphs lab has plans to look at brain activity to help answer questions such as where these faces are represented in the brain and at what point language enters the judgment process.
In one new project, Adolphs next aims to record single-neuron activity in patients with epilepsy who have electrodes temporarily implanted in their brains to monitor seizures, work that will be performed in collaboration with electrophysiologist and neuroscientist Ueli Rutishauser (PhD '08), who is a faculty associate in biology and biological engineering at Caltech, an affiliated faculty member of the Tianqiao and Chrissy Chen Institute for Neuroscience, and a faculty member at Cedars-Sinai, where he directs the Center for Neural Science and Medicine. Such recordings allow for detailed investigations of these questions as subjects look at faces—with more precise timing than is possible with standard neuroimaging methods like functional magnetic resonance imaging.
The Nature Human Behaviour paper is titled "Trait judgments of faces using a task without word labels." Umit Keles, a computational scientist in the Adolphs lab, and former Adolphs lab member Chujun Lin (PhD '19), now an assistant professor of psychology at Columbia University, are co-authors of the study. The work was supported by funding from the National Institute of Mental Health, the Simons Foundation Collaboration on the Global Brain, and the Japan Science and Technology Agency.
Ralph Adolphs
Yue Xu (PhD '26)
