A German computer scientist has hit on a new way of modelling objects for films and video games that could create much richer images than current methods achieve, and cost less.
At the moment, actors wear light markers – usually placed at or near joints – and execute a series of movements while surrounded by a circle of cameras. A computer then creates a digital skeleton that moves in the same way as the actor, and can be manipulated electronically. This was how boy wizard Harry Potter managed to defeat his enemy Voldemort in Harry Potter and the Deathly Hallows, and how the blue humanoids in Avatar achieved their acrobatic flourishes.
However, the technique, called motion capture, only succeeds in creating lifelike movement: the shape and surfaces then need to be completed by a graphic artist, and the surfaces still don’t look real. So researcher Christian Theobalt, based in Saarbrücken, Germany, is using sophisticated algorithms to create digital models of moving characters with much less work.
Theobalt’s method – known as performance capture – uses textures and edges instead of light markers, providing models that already feature shape, movements and surface appearance. That yields far richer images with textures and moving surfaces. It could be applied in video games, as well as in fields such as medicine and engineering.
‘The actual challenge is that in the real world, things are much more complex.’
Christian Theobalt head of the Graphics, Vision and Video research group, the Max-Planck-Institute for Informatics, Germany
‘We have to find a replacement for markers in natural images,’ said Theobalt, who heads the Graphics, Vision and Video research group at the Max-Planck-Institute for Informatics, Germany. ‘The complexity is orders of magnitude higher.’
The complex interaction of light and objects in the real world produces drastic changes in the appearance of surfaces. That makes it hard to maintain a grasp on them, and so far, performance capture only works in special, restricted circumstances and just for scenes such as a single person in simple clothing.
Under a five-year project funded by a European Research Council (ERC) grant, Theobalt is aiming to extend the technology to digitally capture multiple objects with complex surface textures under arbitrary lighting. The objects will also move and interact in unpredictable ways. He wants this to be done quickly, in detail and using just a few cameras, some of which may be moving.
‘So far, we are still just in the studio, where the lighting can be controlled,’ said Theobalt. ‘The actual challenge is that in the real world, things are much more complex. We are rethinking the way we do dynamic scenery construction.’
Grasping moving objects is an area where computers still lag the human brain by a long way. Typically, designers create virtual images in two parts. First comes the modelling stage, where a mathematical model is created for the shapes, textures, and movements that will be needed in a scene. Second, they use graphic visualization technology to twist, stretch, and move the model. This is done using an algorithm that ‘renders’ the model by simulating the way light interacts with its surface as it moves, making it appear real.
Christian Theobalt who heads the Graphics, Vision and Video research group at the Max-Planck-Institute for Informatics, Germany. Image courtesy of Christian Theobalt.
The tools for this second stage have leapt ahead in recent years, and computer graphic images are now common in visual media. Doctors and engineers also use them to display high quality results from scanners and other imaging devices. But model generation has not kept up.
Models used to create computer-generated scenes in movies must often be crafted manually using software, a process that often takes months of work for just a few seconds of footage.
‘The ultimate long-term goal of performance capture is to bridge this gap,’ Theobalt said.
The technology could be valuable: the global movies and entertainment market is expected to be worth EUR 69 billion by the end of 2016, according to Marketline, a UK-based research company, and Europe has many graphics and visual technology firms competing for a share of that. In addition, doctors could assess the healing process using computer models of treated parts of a patient’s body. Engineers and physicists could measure complex deforming surfaces. And new search tools could also be built for image databases.
‘Sports doctors said that if they can make such measurements, they can derive the underlying muscle movements and forces,’ said Theobalt. Despite the complexity of the work, he said, ‘In major areas we will make large progress.’
Concrete has become our building material of choice for countless structures such as bridges, towers and dams. But it also has a huge environmental footprint mostly due to carbon dioxide emissions from the production of cement – one of its main constituents. Researchers are now experimenting with root vegetables and recycled plastic in concrete to see whether this can make it stronger – and more sustainable – and even power streetlights or air pollution sensors.
Retrofitting Europe’s buildings for energy efficiency is not enough to slash the carbon footprint of the construction sector and cut emissions in time to meet the Paris climate agreement goals, according to Dr Catherine De Wolf, assistant professor of design and construction management at TU Delft in the Netherlands.
Artificial intelligence (AI) used by governments and the corporate sector to detect and extinguish online extreme speech often misses important cultural nuance, but bringing in independent factcheckers as intermediaries could help step up the fight against online vitriol, according to Sahana Udupa, professor of media anthropology at Ludwig Maximilian University of Munich, Germany.
In the summer of 2014 a strange building began to take shape just outside MoMA PS1, a contemporary art centre in New York City. It looked like someone had started building an igloo and then got carried away, so that the ice-white bricks rose into huge towers. It was a captivating sight, but the truly impressive thing about this building was not so much its looks but the fact that it had been grown.
Independent factcheckers can bring context to AI tools, says media anthropologist.
Live mycelium networks, capable of information processing, could be used as building materials.
Dr Kate Rychert studies ocean plate structures.