Pop-up Light Field: An Interactive Image-Based Modeling and Rendering System

Conventional Light Field	Pop-up Light Field

We present an image-based modeling and rendering system, which we call pop-up light field, that models a sparse light field using a set of coherent layers. In our system, the user specifies how many coherent layers should be modeled or “popped up” according to the scene complexity. A coherent layer is defined as a collection of corresponding planar regions in the light field images. A coherent layer can be rendered free of aliasing, all by itself or against other background layers. To construct coherent layers, we introduce a Bayesian approach, coherence matting, to estimate alpha matting around segmented layer boundaries by incorporating a coherence prior to maintain coherence across images.

Publications

Harry Shum, Jian Sun, Shuntaro Yamazaki, Li Yin and Chi-Keung Tang,
“Pop-up Light Field: An Interactive Image-Based Modeling and Rendering System”,
ACM Transactions on Graphics, vol.23(2), pp.143-162,
Apr. 2004

Supplementary Material

Demo video

This video is a compilation of the main results of this project. (Watch this at YouTube.com)

Figures in Paper

Concept of Popup Light Fields: Rendering using the 5x5 Tsukuba light field data set is shown in the top left. Aliasing is clearly visible near the front objects in the bottom left image because the input light field is sparse. The top row shows that the pop-up light field splits the scene gradually into 4 coherent layers, and achieves an anti-aliased rendering as shown in the bottom right image.

Coherent Layers: Light field (with images I1 and I2) can be represented by a set of coherent layers (L1 and L2). A coherent layer is a collection of layered images in the light field.

User Interface: The UI for Pop-up light field construction.

Results

Result of pop-up light field rendering of the Plaza sequence rendered from a novel viewpoint (in the position midway between the 11th and 12th frames). The input consists of only 16 images. We have used 16 layers to model the pop-up light field.

Pokemon 9x9: Comparison of conventional light field rendering and pop-up light field rendering.