Coplanar Shadowgrams


Real object	Reconstructed shape

Acquiring 3D models of intricate objects (like tree branches, bicycles and insects) is a hard problem due to severe self-occlusions, repeated thin structures and surface discontinuities. In theory, a shape-from-silhouettes (SFS) approach can overcome these difficulties and use many views to reconstruct visual hulls that are close to the actual shapes. In practice, however, SFS is highly sensitive to errors in silhouette contours and the calibration of the imaging system, and therefore not suitable for obtaining reliable shapes with a large number of views. We present a practical approach to SFS using a novel technique called coplanar shadowgram imaging, that allows us to use dozens to even hundreds of views for visual hull reconstruction. Here, a point light source is moved around an object and the shadows (silhouettes) cast onto a single background plane are observed. We characterize this imaging system in terms of image projection, reconstruction ambiguity, epipolar geometry, and shape and source recovery. The coplanarity of the shadowgrams yields novel geometric properties that are not possible in traditional multi-view camera-based imaging systems. These properties allow us to derive a robust and automatic algorithm to recover the visual hull of an object and the 3D positions of light source simultaneously, regardless of the complexity of the object. We demonstrate the acquisition of several intricate shapes with severe occlusions and thin structures, using 50 to 120 views.

Publications

Shuntaro Yamazaki, Srinivasa G. Narasimhan, Simon Baker and Takeo Kanade,
“The theory and practice of coplanar shadowgram imaging for acquiring visual hulls of intricate objects”,
International Journal of Computer Vision,
To appear.
[Paper] [BibTex]

Shuntaro Yamazaki, Srinivasa G. Narasimhan, Simon Baker and Takeo Kanade,
“Coplanar shadowgrams for acquiring visual hulls of intricate objects”,
Proc. International Conference of Computer Vision,
Oct. 2007.
[Paper] [Poster] [Video] [Presentation] [BibTex]

Supplementary Material

ICCV 2007 Video

This video is a compilation of the main results of this project. (Watch this at YouTube.com)

Shadowgram Acquisition

The left picture shows the setup used to capture coplanar shadowgrams. The setup includes a digital camera, a single point light source, and a rearprojection screen. The object is placed close to the screen to cover a large field of view. Two or more spheres are used to estimate the initial light source positions.

This video demonstrates the acquisition of shadowgrams. The point source is moved within a half space with respect to a rear-projection screen, while the object, the camera and the screen all remain stationary. In order to cover the entire surface area of an intricate shape, we must capture a sufficiently large number of shadowgrams.

A point source illuminates the object and its shadow cast on a planar rear-projection screen represents the silhouette of the object. Coplanar shadowgrams from multiple viewpoints are obtained by translating the light source. Note that the relative transformation between the object and the screen remains fixed across different views.

This video shows a sequence shadowgrams acquired using our experimental setup. The light source positions are estimated using the shadowgrams of at least two spheres. The acquired shadowgrams can then be used to recover the visual hull of the object.

Example shadowgrams obtained using the setup.

Sensitivity of Visual Hull Reconstruction

The top row shows the visual hulls reconstructed using the light source positions estimated by our method. As the number of silhouettes increases, the visual hull gets closer to the actual shape. The bottom row shows the reconstructions obtained from slightly erroneous source positions. As the number of views increases, the error worsens significantly.

Optimization of Light Source Positions

This video shows an optimization of light source positions using epipolar constraints and silhouette consistency. The ground truth and estimated source positions are presented respectively in red and yellow.

As the light source positions are estimated accurately, silhouette consistency is improved. An acquired silhouette and that generated from a reconstructed visual hull are shown respectively in green and yellow. The match between the acquired and re-projected silhouettes increases until it becomes almost perfect.

This video is the trace the progression of our optimization algorithm for this object starting from the erroneous reconstruction.

Reconstruction from Simulated Shadowgrams

This video is a CG rendering of the visual hull model acquired by coplanar shadowgrams. A 3D mesh model of a seaweed object is used to generate 49 coplanar shadowgrams by our simulator. This object has many thin sub-branches with numerous occlusions, which are successfully reconstructed by our method.

A visual hull of a coral object is reconstructed using 84 simulated coplanar shadowgrams generated from a 3D mesh model.

A visual hull of a bicycle object is reconstructed using 61 simulated coplanar shadowgrams generated from the 3D model.

A visual hull of a spider object is reconstructed using 76 simulated coplanar shadowgrams generated from the 3D model.

Reconstruction from Real Shadowgrams

A visual hull of a wreath object is reconstructed using 122 real coplanar shadowgrams acquired by our experiment setup.

A visual hull of a polygon-ball object is reconstructed using 45 real coplanar shadowgrams acquired by our experiment setup.

A visual hull of a palmtree object is reconstructed using 56 real coplanar shadowgrams acquired by our experiment setup.

A visual hull of an octopus object is reconstructed using 53 real coplanar shadowgrams acquired by our experiment setup.

Example Application to Computer Graphics

This video is a CG animation rendered using the 3D shape models acquired by coplanar shadowgrams.

Miscellaneous

Our acquisition technique has been extended to a mask-based light field camera which enables to reconstruct the visual hull of dynamic objects. See also the project page at Brown University.

Another version of our project page at Carnegie Mellon University is here.