Although it is widely accepted that synthetic voices that imitate the voices of real people (so-called “deepfakes”) can be quite convincing, the question remains what our brains really think of these attempts at imitation. To answer this question, researchers at the University of Zurich put a number of volunteers in fMRI scanners and were able to observe how their brains reacted to real and synthetic voices. The perhaps somewhat surprising discovery is that the human brain shows differences in two brain regions depending on whether it is hearing a real or a fake voice. This means that at some level we are aware of the fact that we are listening to a deepfake.
The detailed results of (Claudia Roswandowitz) and colleagues are published in Communication biologyFor the study, 25 volunteers were asked to accept or reject the voice samples they heard as natural or synthesized and to perform an identity comparison with the supposed speaker. The natural voices came from four male (German) speakers, whose voices were also used to train the synthesis model. Not only was the performance in the identity comparison with the synthesized voices worse, the resulting fMRI scans also showed very different brain activity depending on whether it was the natural or the synthesized voice.
One of these regions was the auditory cortex, which clearly indicates that there were acoustic differences between the natural and the fake voice, the other was the nucleus accumbens (NAcc). This part of the basal forebrain is involved in cognitive processing of e.g. motivation, reward and reinforcement learning, which plays a key role in social, maternal and addictive behavior. Overall, the deepfake voices are characterized by acoustic imperfections and do not trigger the same feeling of recognition (and thus reward) as natural voices.
Until deepfake voices can be significantly improved, we seem to be safe for now.
