The McGurk effect: when we hear with our eyes

Visual and auditory information are key in understanding speech. When we talk to someone, we don't just listen to what he says, but we also look at how he says it by moving his lips.

The ability of human speech is based on the integration of visual and auditory information, something that is evidenced by the fact that we can experience a curious illusory phenomenon: the McGurk effect.

It can be said that this peculiar phenomenon occurs when we hear with our eyes, causing what we hear to change depending on what we see. Let's find out what this interesting visual-auditory effect consists of.

• Related article: "17 curiosities about human perception"

What is the McGurk effect?

We tend to believe that our senses function independently: when we hear, we only hear; and when we see, we only see. Based on this belief, it would be reasonable to think that a visual stimulus is not capable of distorting our way of perceiving sound. However, the reality is that it can, since our perceptual experiences are the product of a complex process of information mixing, the same mixture that gives rise to a peculiar phenomenon: the effect McGurk.

Surely on more than one occasion you have had a conversation in an extremely noisy environment. It might have been in a nightclub, on the terrace of a bar on a busy street, or in a high school classroom. When there is high background noise, it is difficult for us to understand what the person tells us that we have just front and, in order to understand something, we use the old and instinctive trick of observing his mouth while she talks.

In these cases, visual and auditory information are not analyzed separately, but rather are combined. The human brain has a region called the superior temporal sulcus, which is specialized in combining both types of information, In the examples we have given, it would be in charge of combining the phonemes that our interlocutor pronounces with the movement of their lips.

Due to this ability to combine multimodal information, the superior temporal sulcus is the neurological stage where the illusory McGurk effect occurs, which would be nothing more than the result of an error in the decoding of the message when two different sensory modalities interact, making what we see does not coincide with what we hear.

If we do a quick search on YouTube, we can find more than one video where this phenomenon is exposed in a practical way. This link directs us to a good example of this phenomenon:

In this specific case, the person in the video is saying / ba / all the time, however, depending on how he moves his lips you can hear either / ba / or / pa /.

This effect can also occur with other combinations of syllables. For example, it can be achieved with the combination / ka / (visual) and / pa / (auditory), which gives rise to the perception of / ta /. Another example would be seeing someone making lip movements that correspond to the syllable / ga / but while the syllable / ba / is being spoken it will be perceived as / da /.

The way in which we hear the same sound varies drastically depending on whether we look at the way the person speaking to us moves his lips. This not only affects the perception of simple sets of sounds such as syllables, but it has also been proven that It works with complete sentences, although you have probably witnessed it yourself in some of the situations we have mentioned before.

One of the first findings related to the McGurk effect and the interaction between sensory modalities is that having the possibility of seeing how our interlocutor's lips move considerably improves the volume of what we hear.

It has been seen that it gives us the sensation of hearing phonemes up to 15 decibels louder when we have the emitter within our visual field. This occurs even when the acoustic conditions are not adverse, such as being in a room without any sound or in a quiet place.

• Maybe you iterate: "Parts of the human brain (and functions)"

History of its discovery

This phenomenon was first described in 1976 in an article by British cognitive psychologist Harry McGurk together with his colleague John MacDonald titled "Hearing lips and seeing voices". His original study was to consist of investigating the patterns of imitation of a group of children who were developing the ability to speak and the experiment involved presenting them with several videos of people pronouncing different syllables.

However, a playback error occurred. The technician in charge of mounting the video made a mistake and caused the image and sound to not be synchronized, causing a recording of a person to be seen saying something that did not correspond to the sound that He heard.

At the time of playing the video, McGurk and MacDonald heard a third phoneme instead of the one spoken with the lips and the one that was emitted. It was serendipity that wanted these two researchers to discover this peculiar auditory illusion.

• Related article: "Neuropsychology: what is it and what is its object of study?"

Its importance in the study of human speech

The discovery of this effect is considered a proof that the visual and auditory systems have evolved together to allow, among other things, better speech processing. Our visual system helps us to discriminate sounds that are difficult to differentiate, an advantage that deaf people carry out when they read lips.

Being able to see how our interlocutor moves his lips increases confidence about the message perceived through the auditory system. That is, if two independent systems point to the same solution, in this case the same message, we trust that message more than if we only receive it through one channel.

It is worth mentioning that the McGurk effect does not occur automatically. For it to occur, it is necessary that we pay attention to our interlocutor and, when distracting stimuli are incorporated, both visual and auditory, this illusion is attenuated.

In fact, this proves that the effect is not due to poor reception of visual or auditory information, but to an error in the integration of these two sensory modalities.

Another fact that gives force to the idea that the visual system supports the auditory is that, when we see a person who speaks to us but we cannot hear what he is saying to us at all, not only our visual cortex is activated, but also the auditory one, even if we are not listening to anything.

• You may be interested in: "Optic nerve: parts, course and related diseases"

The McGurk effect and brain dysfunctions

It has been seen that brain injuries and impaired reading skills, in addition to manifesting mental disorders, it influences the probability of the McGurk effect occurring.

People who have had a callosotomy show the McGurk effect more slowly. It seems that children with specific language disorder they present the McGurk effect in a weaker way compared to those children who do not have language acquisition disorders or reading and writing difficulties.

Laterality also influences, seeing that right-handed people are more likely to experience this effect.

• Related article: "Brain hemispheres: myths and realities"

The McGurk effect in different languages

Regardless of which language is spoken, its speakers depend, to some degree, on visual information during speech perception. Nevertheless, the intensity of the McGurk effect has been seen to vary from language to languageIt is seen that in languages ​​such as Spanish, Turkish, Italian, Dutch, English and German, their speakers experience this effect more strongly than Chinese and Japanese speakers.

That Asian language speakers show the McGurk effect less frequently may be due to the cultural practice of avoiding eye contact. Added to this, Chinese and Japanese in particular are two languages ​​with very syllabic linguistic structures, usually consonant + vowel type and consonant + vowel + consonant, which makes them especially adept at detecting syllables regardless of how their lips move. interlocutor.