Enter the GPU – The uncanny valley
After watching the videos in this morning’s next-gen lighting article, I wasn’t nearly as enlightened as I thought I’d be. Many PC gamers would probably feel the same, because it’s something we’ve experienced before. I’m sure that every one will be impressed with what Nvidia’s GPU the GeForce GTX TITAN is capable of though. We’ve seen some really awesome real-time face rendering technology and amazing looking animation over the years, but we’ve never really seen anything like this. This is the human head we hope to see in all of our video games.
NVIDIA Co-founder and CEO Jen-Hsun Huang demonstrated their new Face Works technology running on the GTX TITAN at the annual GPU Technology Conference explaining exactly what the significance of this technology is, the wonder is not only how uncanny the realism is, but the capability of the technology to shrink the size of the data to something more usable in video games.
To create the demo Nvidia partnered with Dr. Paul Debevec of the Institute for Creative Technology (ICT) at the University of Southern California where Debevec and his team have been building next generation systems that can capture facial data to within a tenth of a millimeter without all the extra crap like special makeup or specialized cameras. ICT developed an astonishing ‘light stage’ technology which uses photographic techniques to capture the complete 3D shape of an actor’s face as well as the most important elements to represent the human skin as it really is. From light reflection and the transmission of light through the skin to how light reflects off the oils in the skin, and nearly microscopic lines and bumps on the skin’s surface.
Debevec’s team had an actor model several dozen of basic facial expressions, and after capturing the data they ran it through a compression engine. The results of which is a pared down set of data that can be “mixed and matched to generate a wide range of expressions.” However, all the detail gathered by Debevec’s team is nothing if it can’t be displayed on a screen interactively. This is where Nvidia’s Face Works technology comes in. To be able to process and display every single detail, like light transmission through the ears and nose complex skin shaders have to execute over 8, 000 instructions per pixel to allow realistic rendering of the face in any lighting situation. These are the stats from the Nvidia blog:
“At full HD resolution that’s 82 billion floating point operations per second (FLOPs) per frame – or 4.9 trillion FLOPs per second at 60 frames per second. And all that doesn’t include the 161 filtered texture fetches per pixel.”
However amazing all the detail and the technology allowing the capturing of the data, it’s just not possible to put this into something like a video game, purely because of the size. The data by Debevec’s team containing only a few expressions exceeds 32 gigabytes. Needless to say it’s not something your average GPU would be able to handle. Nvidia’s Face Works technology is critical in making this data usable, as it reduces the size of this data, to a mere 300 megabytes. And how is that done? With sorcery. Or if maybe like this:
“The trick is to take advantage of similarities in the textures for each of the actor’s expressions. Each texture is divided into an eight by eight grid of tiles, and those tiles that are recognized as being nearly identical to the same tile in other expressions are discarded.”