A new computer program that translates visual images into sounds and music is enabling blind people to see faces and other objects through their ears, and the results are inspiring brain experts to change their view of how the senses work.
The EyeMusic sensory substitution device (SSD), developed by researchers at the Hebrew University of Jerusalem in Israel, produces sounds based on what its camera can see. Blind people trained to understand the sounds can perceive images without using their eyes, and are able to describe faces, understand other people’s emotions, identify body postures, read words and even select the only red apple from a plate of fruit.
It works by representing the height of objects by the pitch of the sound – tall objects are high-pitch; low objects are lower pitch.
The width of an object is represented by the duration of the sound; colour is represented by different musical instruments including the violin, trumpet and organ and different colour shades are conveyed through volume manipulations.
The result might sound a little grating but, with the right training, users can develop a fluent understanding of the tones, allowing them to form images in their minds. The work is opening up the possibility of using these devices in rehabilitation programmes to improve blind people’s independence and quality of life.
For the researchers, one of the most striking findings from their work has been how the brain reacts to this new sensory input. Although the ears are doing all the work in detecting signals from the outside world, it is the brain’s visual centres that light up when viewed using functional MRI (fMRI) scanning – a technology that shows which brain areas are most active while performing a specific task.
This has contributed to the conclusion that humans see with their brains rather than their eyes and that the brain is much more flexible than previously imagined.
‘Our work with SSDs highlights that the brain is a flexible task-machine rather than a sensory machine as previously conceived,’ said Professor Amir Amedi who leads the BRAINVISIONREHAB project, which developed the EyeMusic device and was funded by the EU’s European Research Council.
‘The exact same areas found to be in use in the sighted population when perceiving body shapes through vision, for instance, were activated in the congenitally blind population when perceiving the same body shapes via audition, using an SSD.’
Blind people have successfully used the EyeMusic device to describe faces. Video courtesy of BRAINVISIONREHAB
Using hearing instead of sight is not the only sensory substitution that Prof. Amedi’s team has been working on. Their EyeCane is a device that vibrates when a blind person waves it in front of them, providing information about the distance between the user and objects around them. This allows blind people to navigate their way around a room or through a maze, ultimately building a 3D map of their environment in much the same way as a bat navigates using sound.
‘We are working to bring this device to the blind community as soon as possible,’ said Prof. Amedi. ‘It is both relatively cheap to manufacture and incredibly intuitive to learn.’
The group is also working on combining the EyeMusic and the EyeCane together to further enrich the amount of information that blind users can perceive through sensory substitution devices.
As researchers delve deeper into the science of the senses, they are discovering just how adaptable our brains are to dealing with different sensory inputs. Prof. Amedi’s lab has been studying what happens in the brain when people read Braille – a tactile writing system used by people who are blind.
‘The brain is a flexible task-machine rather than a sensory machine.’
Prof. Amir Amadi, Hebrew University of Jerusalem, Israel
It was already known that in people who can see, a region of the brain called the Visual Word Form Area develops when they learn to read with their eyes. Using fMRI, researchers found that the same region is active when blind people read Braille.
However, as part of the EU-funded METABRAILLE project, Dr Marcin Szwed, at the Jagiellonian University in Krakow, Poland, is studying what happens when people who can see read Braille.
Dr Szwed decided to teach sighted people to read Braille over a nine-month period and scanned their brains throughout the process.
‘We found that when subjects read the tactile alphabet the most active area of the brain is the visual cortex rather than the areas associated with touch,’ he explained.
To confirm their finding, the team used magnets to temporarily inhibit the visual cortex. With their visual centres scrambled, the subjects’ tactile reading was impaired.
Dr Szwed said these studies give further insights into the plasticity of the brain, showing that areas once thought of as being dedicated to a single set of tasks – such as touch or sight – can be recruited to other jobs. It is also clear that there is a significant communication between these areas when the brain is performing several related processes at once.
This, he suggests, could help to explain why humans are so good at learning complex tasks such as driving or learning to play musical instruments. Our brains are much more flexible than we think, meaning it may be time to update how we view it.
‘If you look at medical textbooks, the pictures of the brain make it look like each brain area has a fixed role,’ said Dr Szwed. ‘What I hope to achieve is to have that picture changed.’
When an outbreak strikes, speed is critical. Health workers must act quickly not only to contain and treat an emerging or re-emerging disease, but also to use this window to evaluate potential treatments and vaccines. And the challenge becomes even greater in sub-Saharan Africa when you’re trying to develop new approaches in the face of multiple emerging diseases.
Understanding the progression from the stepping reflex to independent walking could help find new therapies for children with cerebral palsy (CP) – a movement disability caused by brain damage before, during or shortly after birth.
Nature provides people with everything from food and water to timber, textiles, medicinal resources and pollination of crops. Now, a new approach aims to measure exactly what a specific ecosystem supplies in order to incentivise decision-makers and businesses to help combat biodiversity loss.
Europe’s position on privacy, regulation and competition could be a key way to attract entrepreneurs who share those values but there is still some work to do in encouraging ambition, according to Nicklas Bergman, a Swedish entrepreneur and technology investor. Over the past two years, he and other entrepreneurs have advised the European Commission on the design of the European Innovation Council (EIC), an initiative to support companies, researchers and entrepreneurs hoping to start their own business or scale up their projects internationally. The second phase of the pilot was launched on 18 March 2019.
To protect species, we need to speak the language of business, say experts.
He has advised the EU on its new European Innovation Council.
Species loss needs urgent international action, says Prof. Georgina Mace.