Computers Predict People's Tastes in Art

Do you like the thick brush strokes and soft color palettes of an impressionist painting such as those by Claude Monet? Or do you prefer the bold colors and abstract shapes of a Rothko? Individual art tastes have a certain mystique to them, but now a new Caltech study shows that a simple computer program can accurately predict which paintings a person will like.

The new study, appearing in the journal Nature Human Behaviour, utilized Amazon's crowdsourcing platform Mechanical Turk to enlist more than 1,500 volunteers to rate paintings in the genres of impressionism, cubism, abstract, and color field. The volunteers' answers were fed into a computer program and then, after this training period, the computer could predict the volunteers' art preferences much better than would happen by chance.

"I used to think the evaluation of art was personal and subjective, so I was surprised by this result," says lead author Kiyohito Iigaya, a postdoctoral scholar who works in the laboratory of Caltech professor of psychology John O'Doherty, an affiliated member of the Tianqiao and Chrissy Chen Institute for Neuroscience at Caltech.

The findings not only demonstrated that computers can make these predictions but also led to a new understanding about how people judge art.

"The main point is that we are gaining an insight into the mechanism that people use to make aesthetic judgments," says O'Doherty. "That is, that people appear to use elementary image features and combine over them. That's a first step to understanding how the process works."

In the study, the team programmed the computer to break a painting's visual attributes down into what they called low-level features—traits like contrast, saturation, and hue—as well as high-level features, which require human judgment and include traits such as whether the painting is dynamic or still.

"The computer program then estimates how much a specific feature is taken into account when making a decision about how much to like a particular piece of art," explains Iigaya. "Both the low- and high-level features are combined together when making these decisions. Once the computer has estimated that, then it can successfully predict a person's liking for another previously unseen piece of art."

The researchers also discovered that the volunteers tended to cluster into three general categories: those who like paintings with real-life objects, such as an impressionist painting; those who like colorful abstract paintings, such as a Rothko; and those who like complex paintings, such as Picasso's cubist portraits. The majority of people fell into the first "real-life object" category. "Many people liked the impressionism paintings," says Iigaya.

What is more, the researchers found that they could also train a deep convolutional neural network (DCNN) to learn to predict the volunteer's art preferences with a similar level of accuracy. A DCNN is a type of machine-learning program, in which a computer is fed a series of training images so that it can learn to classify objects, such as cats versus dogs. These neural networks have units that are connected to each other like neurons in a brain. By changing the strength of the connection of one unit to another, the network can "learn."

In this case, the deep-learning approach did not include any of the selected low- or high-level visual features used in the first part of the study, so the computer had to "decide" what features to analyze on its own.

"In deep-neural-network models, we do not actually know exactly how the network is solving a particular task because the models learn by themselves much like real brains do," explains Iigaya. "It can be very mysterious, but when we looked inside the neural network, we were able to tell that it was constructing the same feature categories we selected ourselves." These results hint at the possibility that features used for determining aesthetic preference might emerge naturally in a brain-like architecture.

"We are now actively looking at whether this is indeed the case by looking at people's brains while they make these same types of decisions," says O'Doherty.

In another part of the study, the researchers also demonstrated that their simple computer program, which had already been trained on art preferences, could accurately predict which photos volunteers would like. They showed the volunteers photographs of swimming pools, food, and other scenes, and saw similar results to those involving paintings. Additionally, the researchers showed that reversing the order also worked: after first training volunteers on photos, they could use the program to accurately predict the subjects' art preferences.

While the computer program was successful at predicting the volunteers' art preferences, the researchers say there is still more to learn about the nuances that go into any one individual's taste.

"There are aspects of preferences unique for a given individual that we have not succeeded in explaining using this method," says O'Doherty. "This more idiosyncratic component may relate to semantic features, or the meaning of a painting, past experiences, and other individual personal traits that might influence valuation. It still may be possible to identify and learn about those features in a computer model, but to do so will involve a more detailed study of each individual's preferences in a way that may not generalize across individuals as we found here."

The study, titled, "Aesthetic preference for art can be predicted from a mixture of low- and high-level visual features," was funded by the National Institute of Mental Health (through Caltech's Conte Center for the Neurobiology of Social Decision Making), the National Institute on Drug Abuse, the Japan Society for Promotion of Science, the Swartz Foundation, the Suntory Foundation, and the William H. and Helen Lang Summer Undergraduate Research Fellowship. Other Caltech authors include Sanghyun Yi, Iman A. Wahle (BS '20), and Koranis Tanwisuth, who is now a graduate student at UC Berkeley.