[Eurographics 2020 - CVPR 2020] State-of-the-Art on Neural Rendering
Today I would like to share a recent talk presented at Eurographic 2020 on the topic of neural rendering. Neural rendering is a very recent field at the intersection of machine learning and computer graphics that aims at employing machine learning models in order to generate photorealistic images. It is a very promising field of research that is evolving at a fast pace and in parallel with traditional computer graphics methods.
Traditional computer graphics requires huge efforts to create 3D scenes (including geometry, material reflectance and texture, lighting conditions…) as well as computing power to render these scenes into photorealistic images. In particular, examples of such computations include complex light transport simulations and global illumination algorithms that require to simulate how light behaves in our real-world (i.e., simulate how light waves bounce on all the surfaces that compose a scene). Instead, neural rendering tries to make machine learning models (e.g., convolutional neural networks) directly learn these effects from observed data (e.g., from rendered images or from 3D information). The advantage of such solution is that once a neural network has be trained to solve the problem, the inference time becomes very fast (often real-time). As a result expensive algorithms, such as light transport simulations become cheap to compute. Unfortunately, although a lot of progress has been made recently, neural renderers still do not give a full control on the scene parameters (unlike the traditional computer graphics pipeline). Neural rendering is therefore an active area of research.
The presentation deals with the following topics :
Semantic photo synthesis : how to create photorealistic pictures from coarse semantic descriptions
Novel-view synthesis : how to render an object from a novel viewpoint when we only have access to observations from a few specific viewpoints
Free-viewpoint videos : neural rendering for performance capture, i.e. how to render photorealistic animated humans in virtual environments
Relighting : how to change the illumination in renderings, photographs with the help of machine learning
Face and body reenactment : how to transfer the movements of a face/body (e.g facial expression) to another person in images and videos
I would recommend anyone who is interested in the latest research in computer vision and graphics to watch this talk!
[Update] A longer version of this talk was presented at CVPR 2020 during the Neural Rendering tutorial.