A new computer program that translates visual images into sounds and music is enabling blind people to see faces and other objects through their ears, and the results are inspiring brain experts to change their view of how the senses work.
The EyeMusic sensory substitution device (SSD), developed by researchers at the Hebrew University of Jerusalem in Israel, produces sounds based on what its camera can see. Blind people trained to understand the sounds can perceive images without using their eyes, and are able to describe faces, understand other people’s emotions, identify body postures, read words and even select the only red apple from a plate of fruit.
It works by representing the height of objects by the pitch of the sound – tall objects are high-pitch; low objects are lower pitch.
The width of an object is represented by the duration of the sound; colour is represented by different musical instruments including the violin, trumpet and organ and different colour shades are conveyed through volume manipulations.
The result might sound a little grating but, with the right training, users can develop a fluent understanding of the tones, allowing them to form images in their minds. The work is opening up the possibility of using these devices in rehabilitation programmes to improve blind people’s independence and quality of life.
For the researchers, one of the most striking findings from their work has been how the brain reacts to this new sensory input. Although the ears are doing all the work in detecting signals from the outside world, it is the brain’s visual centres that light up when viewed using functional MRI (fMRI) scanning – a technology that shows which brain areas are most active while performing a specific task.
This has contributed to the conclusion that humans see with their brains rather than their eyes and that the brain is much more flexible than previously imagined.
‘Our work with SSDs highlights that the brain is a flexible task-machine rather than a sensory machine as previously conceived,’ said Professor Amir Amedi who leads the BRAINVISIONREHAB project, which developed the EyeMusic device and was funded by the EU’s European Research Council.
‘The exact same areas found to be in use in the sighted population when perceiving body shapes through vision, for instance, were activated in the congenitally blind population when perceiving the same body shapes via audition, using an SSD.’
Blind people have successfully used the EyeMusic device to describe faces. Video courtesy of BRAINVISIONREHAB
Using hearing instead of sight is not the only sensory substitution that Prof. Amedi’s team has been working on. Their EyeCane is a device that vibrates when a blind person waves it in front of them, providing information about the distance between the user and objects around them. This allows blind people to navigate their way around a room or through a maze, ultimately building a 3D map of their environment in much the same way as a bat navigates using sound.
‘We are working to bring this device to the blind community as soon as possible,’ said Prof. Amedi. ‘It is both relatively cheap to manufacture and incredibly intuitive to learn.’
The group is also working on combining the EyeMusic and the EyeCane together to further enrich the amount of information that blind users can perceive through sensory substitution devices.
As researchers delve deeper into the science of the senses, they are discovering just how adaptable our brains are to dealing with different sensory inputs. Prof. Amedi’s lab has been studying what happens in the brain when people read Braille – a tactile writing system used by people who are blind.
‘The brain is a flexible task-machine rather than a sensory machine.’
Prof. Amir Amadi, Hebrew University of Jerusalem, Israel
It was already known that in people who can see, a region of the brain called the Visual Word Form Area develops when they learn to read with their eyes. Using fMRI, researchers found that the same region is active when blind people read Braille.
However, as part of the EU-funded METABRAILLE project, Dr Marcin Szwed, at the Jagiellonian University in Krakow, Poland, is studying what happens when people who can see read Braille.
Dr Szwed decided to teach sighted people to read Braille over a nine-month period and scanned their brains throughout the process.
‘We found that when subjects read the tactile alphabet the most active area of the brain is the visual cortex rather than the areas associated with touch,’ he explained.
To confirm their finding, the team used magnets to temporarily inhibit the visual cortex. With their visual centres scrambled, the subjects’ tactile reading was impaired.
Dr Szwed said these studies give further insights into the plasticity of the brain, showing that areas once thought of as being dedicated to a single set of tasks – such as touch or sight – can be recruited to other jobs. It is also clear that there is a significant communication between these areas when the brain is performing several related processes at once.
This, he suggests, could help to explain why humans are so good at learning complex tasks such as driving or learning to play musical instruments. Our brains are much more flexible than we think, meaning it may be time to update how we view it.
‘If you look at medical textbooks, the pictures of the brain make it look like each brain area has a fixed role,’ said Dr Szwed. ‘What I hope to achieve is to have that picture changed.’
Some materials are special not for what they contain, but for what they don’t contain. Such is the case with metal-organic frameworks (MOFs) – ultra-porous structures that are being developed for a variety of future applications from fire-proofing to drug-delivery.
Tiny pieces of plastic, now ubiquitous in the marine environment, have long been a cause of concern for their ability to absorb toxic substances and potentially penetrate the food chain. Now scientists are beginning to understand the level of threat posed to life, by gauging the extent of marine accumulation and tracking the movement of these contaminants.
Artificial intelligence (AI) and cyber security should be priorities in future EU industrial research policy in order to reinvigorate industry and recover jobs that have been lost abroad, according to Professor Jürgen Rüttgers, a former research minister in Germany.
The world’s largest radio telescope, known as the Square Kilometre Array (SKA) and situated over two continents, will be able to detect the first stars and galaxies emerging from the ‘murk’ at the beginning of the universe and much more besides, according to Professor Phil Diamond, Director General of SKA. He spoke to Horizon at the opening of the Shared Sky art exhibition in Brussels, Belgium on 16 April, where indigenous artists from SKA host nations South Africa and Australia use traditional painting and folk art to explore the themes of astronomy, spirituality and a borderless sky.
Are metal organic frameworks the hole-y grail of nanomaterials?
Tiny plastic particles could impact human health.
A new report on how to reinvigorate Europe's industrial sector recommends prioritising AI and cyber security research.