Image-Based Relighting Framework

Image-based relighting [1] is a famous algorithm in visual effects to render an object in an arbitrary environment. The main idea is take pictures of the object illuminated from several known directions (one picture corresponds to one direction). The set of obtained pictures is called a reflectance field and corresponds to how the object reflects light from a particular viewpoint. Then by calculating a linear combination of these input images it is possible to create a rendering of the object in any arbitrary environment.

Figure : (Left) Input images : plant illuminated from 253 directions (only 2 directions shown. source of the images [2]). (Middle) Mapping of the directions on a latitude-longitude map of the new environment and computation of the Voronoi diagram. (Right) Relit plant in the Grace Cathedral environment.

Conventional image-based relighting requires a light-stage i.e a dome of light sources that can illuminate an object from known directions. However it is a very expensive device that requires a laboratory environment to be used. Hence my MSc thesis focused on implementing an image-based relighting algorithm that works with pictures taken in a regular room. The main challenge comes from the number of light sources available. Indeed a light stage such as light stage 6 has 253 illumination conditions regularly placed over an hemisphere, hence creating a dense sampling of the upper hemisphere whereas a regular room has only a few light sources available. Besides having a sparse lighting basis, the light sources available in a regular room are often area light sources (e.g a window) instead of point light sources. As a result certain lights tend to dominate over the others. Finally the geometry of the room is important as light bounces on every diffuse surface in the room (e.g the walls) creating indirect light sources (global illumination). All these effects have to be taken into account in the algorithm.

The software I implemented during my MSc thesis includes image-based relighting algorithms using data captured with a light stage [2], a regular room [4] and a free-form light stage [3] for ground truth comparison. It was implemented in C++ using OpenCV library for the computer vision component and Qt library for the user interface.

Figure : User interface for three different relighting algorithms

A few result of the office room relighting algorithm are shown below. All the environment maps are available on Paul Debevec’s website [5]. More details are available in my MSc thesis. The Image-based relighting framework is also available on my github (LGPL license).

Figure : From left to right : relighting of a decoration bird in the Eucalyptus Grove environment and in the Uffizi Gallery. Relighting of a decoration egg in St. Peter's Basilica and the Uffizi Gallery.

---

References

[1] Acquiring the Reflectance Field of a Human Face, Paul Debevec, Tim Hawkins, Chris Tchou, Haarm-Pieter Duiker, Westley Sarokin, and Mark Sagar, SIGGRAPH 2000 Conference Proceedings

[2] USC Institute for Creative Technologies “Light Stage Data Gallery”, http://gl.ict.usc.edu/Data/LightStage/, accessed February 2016

[3] The free-form light stage, Vincent Masselus, Philip Dutré, Frederik Anrys In Proc. EGWR, EGRW ’02, 247–256.

[4] Image-Based Relighting using Room Lighting Basis, Antoine Toisoul, Abhijeet Ghosh, CVMP 2016

[5] Light Probe Image Gallery . Paul Debevec http://www.pauldebevec.com/Probes/, accessed February 2016

My MSc thesis for this project is available here (low resolution due to file size constraint).