Caltech Neurobiologists Discover Individuals Who "Hear" Movement

PASADENA, Calif.-- Individuals with synesthesia perceive the world in a different way from the rest of us. Because their senses are cross-activated, some synesthetes perceive numbers or letters as having colors or days of the week as possessing personalities, even as they function normally in the world. Now, researchers at the California Institute of Technology have discovered a type of synesthesia in which individuals hear sounds, such as tapping, beeping, or whirring, when they see things move or flash. Surprisingly, the scientists say, auditory synesthesia may not be unusual--and may simply represent an enhanced form of how the brain normally processes visual information.

Psychologists previously reported visual, tactile, and taste synesthesias, but auditory synesthesia had never been identified. Caltech lecturer in computation and neural systems Melissa Saenz discovered the phenomenon quite by accident.

"While I was running an experiment at the Caltech Brain Imaging Center, a group of students happened to pass by on a tour, and I volunteered to explain what I was doing," explains Saenz, who, along with Christof Koch, the Lois and Victor Troendle Professor of Cognitive and Behavioral Biology at Caltech and professor of computation and neural systems, reports the finding in the August 5 issue of the journal Current Biology.

"As part of the experiment, a moving display was running on my computer screen with dots rapidly expanding out, somewhat like the opening scene of Star Wars. Out of the blue, one of the students asked, "Does anyone else hear something when you look at that?" After talking to him further, I realized that his experience had all the characteristics of a synesthesia: an automatic sensory cross-activation that he had experienced all of his life," says Saenz.

A search of the synesthesia literature revealed that auditory synesthesia--of any kind--had never been reported. Intrigued, Saenz began to look for other individuals with the same ability, using the original movie seen by the student as a test. "I queried a few hundred people and three more individuals turned up," she says. "Having that specific example made it easy to find more people. That movie just happens to be quite "noisy" to the synesthetes and was a great screening tool. When asked if it made a sound, one of the individuals responded, "how could it not?" I would have been less successful had I just generally asked, "Do you hear sounds when you see things move or flash?" because in the real environment, things that move often really do make a sound," for example, a buzzing bee.

This may be why auditory synesthesia hadn't been detected by neurobiologists. "People with auditory synesthesia may be even less likely than people with other synesthetic associations to fully realize that their experience is unusual. These individuals have an enhanced soundtrack in life, rather than a dramatically different experience, compared to others," says Saenz. However, when asked, all of the synesthetes could name examples of daily visual events that caused sounds that they logically knew to be only in their minds, such as seeing a fluttering butterfly or watching television with the sound turned off.

Saenz and Koch found that the four synesthetes outperformed a group of nonsynesthetes on a simple test involving rhythmic patterns of flashes similar to visual Morse code. Normally, such patterns are easier to identify with sound (beeps) than with vision (flashes), so the researchers predicted that synesthetes would have an advantage with visual patterns because they actually heard a sound every time they saw a flash.

In the test, the subjects saw a series of flashes and had to guess if a second sequence, played afterward, represented the same temporal pattern or not. As a baseline measurement, a similar test was given using sequences of beeps. Both the synesthetes and the control group performed equally well when given beeps. However, with visual flashes synesthetes were much more accurate, responding correctly more than 75 percent of the time, compared to around 50 percent--the level predicted by chance--in the control group. "Synesthetes had an advantage because they not only saw but also heard the visual patterns," Saenz says.

Saenz and Koch suspect that as much as 1 percent of the population may experience auditory synesthesia. In fact, she and Koch think that the brain may normally transfer visual sensory information over to the auditory cortex, to create a prediction of the associated sound. "This translation might result in actual sound perception in synesthetes, perhaps due to stronger than normal connections, says Saenz, who has begun brain imaging experiments to study this connectivity in synesthetes and nonsynesthetes.

"We might find that motion processing centers of the visual cortex are more interconnected with auditory brain regions than previously thought, even in the 'normal' brain," Saenz says. "At this point, very little is known about how the auditory and visual processing systems of the brain work together. Understanding this interaction is important because in normal experience, our senses work together all the time."

The work was supported by the Mind Science Foundation, the Gordon and Betty Moore Foundation, the Mathers Foundation, and the National Institute of Mental Health.

View the video used to identify auditory synesthetes, in a quiet location, at http://www.klab.caltech.edu/~saenz/movingdots.html.