Temporal Dithering of Illumination for Fast Active Vision

[Click to open a movie] [Click to open a movie] [Click to open a movie] [Click to open a movie]

Active vision techniques use programmable light sources, such as projectors, whose intensities can be controlled over space and time. We present a broad framework for fast active vision using Digital Light Processing (DLP) projectors. The digital micromirror array (DMD) in a DLP projector is capable of switching mirrors "on" and "off" at high speeds (106/s). An off-the-shelf DLP projector, however, effectively operates at much lower rates (30-60Hz) by emitting smaller intensities that are integrated over time by a sensor (eye or camera) to produce the desired brightness value. Our key idea is to exploit this temporal dithering of illumination, as observed by a high-speed camera. The dithering encodes each brightness value uniquely and may be used in conjunction with virtually any active vision technique. We apply our approach to five well-known problems: (a) structured light-based range finding, (b) photometric stereo, (c) illumination de-multiplexing, (d) high frequency preserving motion-blur and (e) separation of direct and global scene components, achieving significant speedups in performance. In all our methods, the projector receives a single image as input whereas the camera acquires a sequence of frames.

Publications

Srinivasa G. Narasimhan, Sanjeev J. Koppal, and Shuntaro Yamazaki,
Temporal Dithering of Illumination for Fast Active Vision”,
Proc. European Conference on Computer Vision, pp.830-844,
Oct. 2008.
[Paper] [Movie] [Presentation] [BibTex]

Supplementary Material

CVPR 2009 Video Review

This video is a compilation of the main results of this project for shape acquisition.

Fast Structured Light Projection

This figure shows the acquisition setup for structured light based 3D reconstruction. The PHOTRON high speed camera is placed vertically above the Infocus DLP projector. A vertical plane is placed behind the scene (statue) for calibration.

Unlike conventional structure light projection, we project a single static pattern composed of stripes in distinct colors. The color are chosen so that each stripe is modulated temporally in distinct way by the DLP.

Temporal Dithering of Illumination

A sequence of frames captured by the high speed camera at 3kHz illustrates the fast modulation of illumination incident on the scene. The DLP projector decomposes each stripe into a sequence of binary patterns, which in turn can be used to identify the stripe at each pixel in acquired images assuming a static scene.

The correspondences between projector and camera pixels are estimated by finding the best match of the intensity profiles in temporal domain. The intensity profiles of a projector pixels are captured from the right edge of the acquired images where stripe pattern is projected on a plane located at a known distance. This alleviates the synchronization issue between the projector and the camera.

Shape Recovery

The shape of a statue object is recovered using 25 continuous images acquired by a high-speed camera at 3kHz. The background region is eliminated by thresholding out the pixels which have large error in the stripe matching.

We reconstructed a human hand using 20 continuous images acquired by a high-speed camera at 3kHz. In the following results, each data set has six movies: the acquired images replayed at actual and 1/300 speeds, and the recovered shapes visualized by mesh rendering from front and side views at actual and 1/10 speeds, respectively.
  • Acquired images: actual and 1/300 speeds.
  • Reconstructed shape (front view): actual and 1/10 speeds.
  • Reconstructed shape (side view): actual and 1/10 speeds.
The shape of a hand spinning a pen is also reconstructed using 10 continuous images acquired at 3kHz.
  • Acquired images: actual and 1/300 speeds.
  • Reconstructed shape (front view): actual and 1/10 speeds.
  • Reconstructed shape (side view): actual and 1/10 speeds.
Another object is the face of a speaking person which is reconstructed from 20 continuous images acquired at 3kHz.
  • Acquired images: actual and 1/300 speeds.
  • Reconstructed shape (front view): actual and 1/10 speeds.
  • Reconstructed shape (side view): actual and 1/10 speeds.
We reconstructed a person putting his tongue out using 10 continuous images acquired at 3kHz.
  • Acquired images: actual and 1/300 speeds.
  • Reconstructed shape (front view) actual and 1/10 speeds.
  • Reconstructed shape (side view): actual and 1/10 speeds.
A cloth waving is reconstructed using 10 continuous images acquired at 3kHz.
  • Acquired images: actual and 1/300 speeds.
  • Reconstructed shape (front view): actual and 1/10 speeds.
  • Reconstructed shape (side view): actual and 1/10 speeds.

De-Multiplexing High-speed Illuminations

We capture the appearances of a dynamic scene illuminated from multiple lighting directions using multiple DLP projectors. We draw upon the idea of illumination de-multiplexing, where the images of the scene are simultaneously captured from multiple source directions and de-multiplexed in software to obtain the desired images under each lighting direction. A mirror sphere is placed in the scene to measure the dithering intensities from the three projectors.

De-multiplexing illumination from three projectors to create appearances under each lighting direction. The scene consists of a wiry polyhedral ball falling vertically, and illuminated by three DLP projectors simultaneously. We can separate the illumination and obtain three sequences of images illuminated by either 1st, 2nd, or 3rd projector.

Fast Photometric Stereo

Three DLP projectors simultaneously illuminate a fast moving flag and the camera operates at 3kHz. The projectors and camera are far enough away from the scene to assume orthographic viewing and distant lighting. The surface normal at a point on the object is computed by photometric stereo using three de-multiplexed illumination images.

High-speed Separation of Direct and Global Images

The DLP projector and the camera are co-located using a beam splitter. A single checker pattern with two intensities 113 and 116 are input to the projector.

The DLP projector emits complementary checker patterns onto the scene (movie) that are used to separate the direct and global components (movie).

Motion Deblurring

When the photograph of a dancer is acquired under a DLP illumination, we see multiple copies of the body that moves simultaneously in multiple directions.

When a brick with the writing “ECCV08” falling vertically is illuminated by a fluorescent source, the resulting motion-blur appears like a smear across the image. On the other hand, when the scene is illuminated using a DLP projector, we see distinct copies of the text that are translated downward. Thus, the DLP illumination preserves more high frequencies in the motion-blurred image.

Miscellaneous

See how DLP® technology works at the official page by Texas Instruments.

See also another project page at Carnegie Mellon University.