Meet Neuralangelo, the Nvidia AI that turns 2D video into 3D scenes

Just when you thought the wild world of AI couldn't surprise any more, Nvidia unveils its new Neuralangelo AI, which can turn 2D video clips into 3D structures and scenes. It literally generates detailed replicas of buildings, sculptures and real objects from video clips taken on a mobile or camera. 

The idea behind Neuralangelo, which 'sculpts' life-like 3D scenes from video, like a modern-day Michelangelo (geddit), is that these 3D scenes can then be imported into other software. The Neuralangelo assets can be edited for use in video games, arch-viz, web 3 projects, well, anything really, including aiding digital artists to define light sources.

This comes on the back of Nvidia's AI for conversing with game characters and the new text-to-3D generative Ai that's still in development from researchers at HiFA. In a Neuralangelo demo (below) Nvidia showed how its AI could recreate complex objects, such as Michelangelo’s David, as well as something as everyday as a flatbed truck.

Nvidia has released a detailed explanation of how Neuralangelo works, sharing how it can translate intricate details in a video, such as glass and smooth marble, into 3D. The pitch is this AI will help speed-up workflows of creatives and unlock 3D virtual worlds and assets to new industries not replace jobs.

"The 3D reconstruction capabilities Neuralangelo offers will be a huge benefit to creators, helping them recreate the real world in the digital world," said Ming-Yu Liu, senior director of research and co-author on the paper, in a press statement. "This tool will eventually enable developers to import detailed objects – whether small statues or massive buildings – into virtual environments for video games or industrial digital twins."

The best AI generators are the ones that can fit into, and enhance, a workflow rather than replace jobs. New generative AI models like Adobe Firefly, for example, can fit into a concept art workflow and speed the process, not replace the artists doing the work. Read Ken Coleman's tutorial on how to use Adobe Firefly or my interview with Chris Colman from Final Frontier, who is using AI everyday.