Philosophers have long debated how the mind combines sensations to form the world we perceive. Now researchers are using modern technology to study processes in the brain and provide insight into how we process sensory information about space and time to form complex experiences.
Professor David Paul Melcher, of the Department of Psychology and Cognitive Sciences at the University of Trento in Italy, is using brain scanners and behavioural methods to try to evaluate the different ways that auditory, tactile and visual information interact with each other to form perceptions of space and time, and particularly how these fit together to form rich experiences.
‘The goal is to look at how perception puts together phenomena: how we start from the very passive view of perception, where people respond almost like a reflex to brief stimulation, to the active construction of an entire experience over time and space,’ said Prof. Melcher, whose COPEST project is funded by a five-year European Research Council (ERC) grant.
An example would be a mosquito on your hand. It would be registered by the brain with respect to its location on your retina for vision, with respect to your head for hearing, and with respect to your hand for touch. The time at which the information about the mosquito from the different senses reaches the brain also differs dramatically.
‘The main thing is trying to develop the idea of a link between the way that information is combined across different spatial reference frames and different senses, and the duration over which different time information is combined,’ Prof. Melcher said.
A noise or a moving image
Studying how the brain reacts to different types of stimuli, such as a sound or an image, could help to resolve questions about how the brain stores and uses or combines the sense data in order to create a unified perceptual experience.
‘This has been debated a lot in the philosophical literature, that either you recreate the world in your head, or that the world is out there and in effect you put the information together as needed,’ Prof. Melcher said.
“‘Either you recreate the world in your head, or ... the world is out there and in effect you put the information together as needed.’
Modern scanning technology such as functional magnetic resonance imaging (fMRI) and magnetoencephalography (MEG) can play a major role in shining light on problems that debate alone cannot resolve.
Volunteers in the brain scanner might be exposed to a stimulus – perhaps a noise or a moving image – and then asked to make judgments about the experience.
The researchers monitor and assess the changes in the brain resulting from that particular combination of sensory stimuli, and compare these to the mental state the brain was in before the experience began.
‘So we are trying to run experiments where we create environments in which you shift from trial to trial between these spatial and temporal reference frames, and we are using neuro-imaging to see how individual areas of the brain change and also the way that connectivity between areas changes,’ Prof. Melcher said.
Space-time conundrum
The initial plan was to study how the brain deals with space and time separately, and then to look at them together. But the experiments keep showing interactions between these apparently different reference frames.
‘In some of our temporal experiments we’ve found an effect of space and in some of our spatial experiments an effect of time,’ Prof. Melcher said.
The project, which runs until 2017, is looking at the idea that space and time could appear to mix because in fact much of the brain is hierarchically organised, with different levels of representation.
This could help to explain the relationship between the brain’s rapid response to urgent incoming stimulation, where data is sampled locally and briefly, and higher-level information processing, such as comprehension of complex language.
‘The idea is that higher levels of representation – allowing you to understand things like events, narratives, meaningful actions, stable objects, navigation and so on – combine information over longer periods of time and over multiple spatial reference frames,’ Prof. Melcher said. ‘In order to do something such as understanding a sentence, you need to have a more abstract representation that combines information over longer periods of time, but where you lose, for example, the sound of a particular syllable,’ he added.
Such a hierarchy would help to explain how we process sense data and could help to improve technologies intended to help those with difficulties in sensory processing, such as people who have had strokes or with some forms of autism.
‘A main goal of doing this research is to push along this science to the point where we can actually have a positive impact on people’s lives – everyday people, and of course, particularly patient groups. That’s one of the things that’s most exciting,’ Prof. Melcher said.
A better understanding of the mid- and higher-level perceptual interface with information would also help to improve technologies for creating virtual experiences. These could in turn be used to take the research further.
'If we have the ability to present a compelling virtual experience, then that gives us control over events that we would not have using real objects,’ Prof. Melcher said. ‘So it definitely goes both ways – it’s not a coincidence that research funding agencies on the one hand and companies on the other are spending a lot of energy and money on developing this type of science and new technologies.'
Recommended for you
