Image-based rendering using parameterized image varieties

This paper addresses the problem of characterizing the set of all images of a rigid set of m points and n lines observed by a weak perspective or paraperspective camera. By taking explicitly into account the Euclidean constraints associated with calibrated cameras, we show that the corresponding image space can be represented by a six-dimensional variety embedded in R2(m+n) and parameterized by the image positions of three reference points. The coefficients defining this parameterized image variety (or PIV for short) can be estimated from a sample of images of a scene via linear and non-linear least squares. The PIV provides an integrated framework for using both point and line features to synthesize new images from a set of pre-recorded pictures (image-based rendering). The proposed technique does not perform any explicit three-dimensional scene reconstruction but it supports hidden-surface elimination, texture mapping and interactive image synthesis at frame rate on ordinary PCs. It has been implemented and extensively tested on real data sets.