Computer Vision-ECCV 2002: 7th European Conference on Computer Vision, Copenhagen, Denmark, May 28-31, 2002 - Proceedings, Part 3

Lecture Notes in Computer Science Edited by G. Goos, J. Hartmanis, and J. van Leeuwen 2352 3 Berlin Heidelberg New Y...

Author: Anders Heyden | Gunnar Sparr | Mads Nielsen | Peters Johansen

12 downloads 1069 Views 19MB Size Report

This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!

Report copyright / DMCA form

DOWNLOAD PDF

Lecture Notes in Computer Science Edited by G. Goos, J. Hartmanis, and J. van Leeuwen

2352

3

Berlin Heidelberg New York Barcelona Hong Kong London Milan Paris Tokyo

Anders Heyden Gunnar Sparr Mads Nielsen Peter Johansen (Eds.)

Computer Vision – ECCV 2002 7th European Conference on Computer Vision Copenhagen, Denmark, May 28-31, 2002 Proceedings, Part III

13

Series Editors Gerhard Goos, Karlsruhe University, Germany Juris Hartmanis, Cornell University, NY, USA Jan van Leeuwen, Utrecht University, The Netherlands Volume Editors Anders Heyden Gunnar Sparr Lund University, Centre for Mathematical Sciences Box 118, 22100 Lund, Sweden E-mail: {Anders.Heyden,Gunnar.Sparr}@math.lth.se Mads Nielsen The IT University of Copenhagen Glentevej 67-69, 2400 Copenhagen NW, Denmark E-mail: [email protected] Peter Johansen University of Copenhagen Universitetsparken 1, 2100 Copenhagen, Denmark E-mail: [email protected]

Cataloging-in-Publication Data applied for Die Deutsche Bibliothek - CIP-Einheitsaufnahme Computer vision : proceedings / ECCV 2002, 7th European Conference on Computer Vision, Copenhagen, Denmark, May 28 - 31, 2002. Anders Heyden ... (ed.). - Berlin ; Heidelberg ; New York ; Barcelona ; Hong Kong ; London ; Milan ; Paris ; Tokyo : Springer Pt. 3 . - 2002 (Lecture notes in computer science ; Vol. 2352) ISBN 3-540-43746-0 CR Subject Classification (1998): I.4, I.3.5, I.5, I.2.9-10 ISSN 0302-9743 ISBN 3-540-43746-0 Springer-Verlag Berlin Heidelberg New York This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, re-use of illustrations, recitation, broadcasting, reproduction on microfilms or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer-Verlag. Violations are liable for prosecution under the German Copyright Law. Springer-Verlag Berlin Heidelberg New York a member of BertelsmannSpringer Science+Business Media GmbH http://www.springer.de © Springer-Verlag Berlin Heidelberg 2002 Printed in Germany Typesetting: Camera-ready by author, data conversion by PTP-Berlin, Stefan Sossna e.K. Printed on acid-free paper SPIN: 10870041 06/3142 543210

Preface

Premiering in 1990 in Antibes, France, the European Conference on Computer Vision, ECCV, has been held biennially at venues all around Europe. These conferences have been very successful, making ECCV a major event to the computer vision community. ECCV 2002 was the seventh in the series. The privilege of organizing it was shared by three universities: The IT University of Copenhagen, the University of Copenhagen, and Lund University, with the conference venue in Copenhagen. These universities lie ¨ geographically close in the vivid Oresund region, which lies partly in Denmark and partly in Sweden, with the newly built bridge (opened summer 2000) crossing the sound that formerly divided the countries. We are very happy to report that this year’s conference attracted more papers than ever before, with around 600 submissions. Still, together with the conference board, we decided to keep the tradition of holding ECCV as a single track conference. Each paper was anonymously refereed by three different reviewers. For the final selection, for the first time for ECCV, a system with area chairs was used. These met with the program chairs in Lund for two days in February 2002 to select what became 45 oral presentations and 181 posters. Also at this meeting the selection was made without knowledge of the authors’ identity. The high-quality of the scientific program of ECCV 2002 would not have been possible without the dedicated cooperation of the 15 area chairs, the 53 program committee members, and all the other scientists, who reviewed the papers. A truly impressive effort was made. The spirit of this process reflects the enthusiasm in the research field, and you will find several papers in these proceedings that define the state of the art in the field. Bjarne Ersbøll as Industrial Relations Chair organized the exhibitions at the conference. Magnus Oskarsson, Sven Spanne, and Nicolas Guilbert helped to make the review process and the preparation of the proceedings function smoothly. Ole Fogh Olsen gave us valuable advice on editing the proceedings. Camilla Jørgensen competently headed the scientific secretariat. Erik Dam and Dan Witzner were responsible for the ECCV 2002 homepage. David Vernon, who chaired ECCV 2000 in Dublin, was extremely helpful during all stages of our preparation for the conference. We would like to thank all these people, as well as numerous others who helped in various respects. A special thanks goes to Søren Skovsgaard at the Congress Consultants, for professional help with all practical matters. We would also like to thank Rachid Deriche and Theo Papadopoulo for making their web-based conference administration system available and adjusting it to ECCV. This was indispensable in handling the large number of submissions and the thorough review and selection procedure. Finally, we wish to thank the IT University of Copenhagen and its president Mads Tofte for supporting the conference all the way from planning to realization. March 2002

Anders Heyden Gunnar Sparr Mads Nielsen Peter Johansen

Organization

Conference Chair Peter Johansen

Copenhagen University, Denmark

Conference Board Hans Burkhardt Bernard Buxton Roberto Cipolla Jan-Olof Eklundh Olivier Faugeras Bernd Neumann Giulio Sandini David Vernon

University of Freiburg, Germany University College London, UK University of Cambridge, UK Royal Institute of Technology, Sweden INRIA, Sophia Antipolis, France University of Hamburg, Germany University of Genova, Italy Trinity College, Dublin, Ireland

Program Chairs Anders Heyden Gunnar Sparr

Lund University, Sweden Lund University, Sweden

Area Chairs Ronen Basri Michael Black Andrew Blake Rachid Deriche Jan-Olof Eklundh Lars Kai Hansen Steve Maybank Theodore Papadopoulo Cordelia Schmid Amnon Shashua Stefano Soatto Bill Triggs Luc van Gool Joachim Weichert Andrew Zisserman

Weizmann Institute, Israel Brown University, USA Microsoft Research, UK INRIA, Sophia Antipolis, France Royal Institute of Technology, Sweden Denmark Technical University, Denmark University of Reading, UK INRIA, Sophia Antipolis, France INRIA, Rhône-Alpes, France The Hebrew University of Jerusalem, Israel University of California, Los Angeles, USA INRIA, Rhône-Alpes, France K.U. Leuven, Belgium & ETH, Zürich, Switzerland Saarland University, Germany University of Oxford, UK

Organization

VII

Program Committee Luis Alvarez Padmanabhan Anandan Helder Araujo Serge Belongie Marie-Odile Berger Aaron Bobick Terry Boult Francois Chaumette Laurent Cohen Tim Cootes Kostas Daniilidis Larry Davis Frank Ferrie Andrew Fitzgibbon David J. Fleet David Forsyth Pascal Fua Richard Hartley Vaclav Hlavac Michal Irani Allan Jepson Peter Johansen Fredrik Kahl Sing Bing Kang Ron Kimmel Kyros Kutulakos Tony Lindeberg Jim Little Peter Meer David Murray Nassir Navab Mads Nielsen Patrick Perez Pietro Perona Marc Pollefeys Long Quan Ian Reid Nicolas Rougon José Santos-Victor Guillermo Sapiro Yoichi Sato Bernt Schiele Arnold Smeulders

University of Las Palmas, Spain Microsoft Research, USA University of Coimbra, Portugal University of California, San Diego, USA INRIA, Lorraine, France Georgia Tech, USA Leheigh University, USA INRIA, Rennes, France Université Paris IX Dauphine, France University of Manchester, UK University of Pennsylvania, USA University of Maryland, USA McGill University, USA University of Oxford, UK Xerox Palo Alto Research Center, USA University of California, Berkeley, USA EPFL, Switzerland Australian National University, Australia Czech Technical University, Czech Republic Weizmann Institute, Israel University of Toronto, Canada Copenhagen University, Denmark Lund University, Sweden Microsoft Research, USA Technion, Israel University of Rochester, USA Royal Institute of Technology, Sweden University of Brittish Columbia, Canada Rutgers University, USA University of Oxford, UK Siemens, USA IT-University of Copenhagen, Denmark Microsoft Research, UK California Insititute of Technology, USA K.U. Leuven, Belgium Hong Kong University of Science and Technology, Hong Kong University of Oxford, UK Institut National des Télécommunications, France Instituto Superior Técnico, Lisbon, Portugal University of Minnesota, USA IIS, University of Tokyo, Japan ETH, Zürich, Switzerland University of Amsterdam, The Netherlands

VIII

Organization

Gerald Sommer Peter Sturm Tomas Svoboda Chris Taylor Phil Torr Panos Trahanias Laurent Younes Alan Yuille Josiane Zerubia Kalle Åström

University of Kiel, Germany INRIA, Rhône-Alpes, France Swiss Federal Institute of Technology, Switzerland University of Manchester, UK Microsoft Research, UK University of Crete, Greece CMLA, ENS de Cachan, France Smith-Kettlewell Eye Research Institute, USA INRIA, Sophia Antipolis, France Lund University, Sweden

Additional Referees Henrik Aanaes Manoj Aggarwal Motilal Agrawal Aya Aner Adnan Ansar Mirko Appel Tal Arbel Okan Arikan Akira Asano Shai Avidan Simon Baker David Bargeron Christian Barillot Kobus Barnard Adrien Bartoli Benedicte Bascle Pierre-Louis Bazin Isabelle Begin Stephen Benoit Alex Berg James Bergen Jim Bergen Marcelo Bertamlmio Rikard Berthilsson Christophe Biernacki Armin Biess Alessandro Bissacco Laure Blanc-Feraud Ilya Blayvas Eran Borenstein Patrick Bouthemy Richard Bowden

Jeffrey E. Boyd Edmond Boyer Yuri Boykov Chen Brestel Lars Bretzner Alexander Brook Michael Brown Alfred Bruckstein Thomas Buelow Joachim Buhmann Hans Burkhardt Bernard Buxton Nikos Canterakis Yaron Caspi Alessandro Chiuso Roberto Cipolla Dorin Comaniciu Kurt Cornelis Antonio Criminisi Thomas E. Davis Nando de Freitas Fernando de la Torre Daniel DeMenthon Xavier Descombes Hagio Djambazian Gianfranco Doretto Alessandro Duci Gregory Dudek Ramani Duraiswami Pinar Duygulu Michael Eckmann Alyosha Efros

Michael Elad Ahmed Elgammal Ronan Fablet Ayman Farahat Olivier Faugeras Paulo Favaro Xiaolin Feng Vittorio Ferrari Frank Ferrie Mario Figueireda Margaret Fleck Michel Gangnet Xiang Gao D. Geiger Yakup Genc Bogdan Georgescu J.-M. Geusebroek Christopher Geyer Peter Giblin Gerard Giraudon Roman Goldenberg Shaogang Gong Hayit Greenspan Lewis Griffin Jens Guehring Yanlin Guo Daniela Hall Tal Hassner Horst Haussecker Ralf Hebrich Yacov Hel-Or Lorna Herda

Organization

Shinsaku Hiura Jesse Hoey Stephen Hsu Du Huynh Naoyuki Ichimura Slobodan Ilic Sergey Ioffe Michael Isard Volkan Isler David Jacobs Bernd Jaehne Ian Jermyn Hailin Jin Marie-Pierre Jolly Stiliyan-N. Kalitzin Behrooz Kamgar-Parsi Kenichi Kanatani Danny Keren Erwan Kerrien Charles Kervrann Renato Keshet Ali Khamene Shamim Khan Nahum Kiryati Reinhard Koch Ullrich Koethe Esther B. Koller-Meier John Krumm Hannes Kruppa Murat Kunt Prasun Lala Michael Langer Ivan Laptev Jean-Pierre Le Cadre Bastian Leibe Ricahrd Lengagne Vincent Lepetit Thomas Leung Maxime Lhuillier Weiliang Li David Liebowitz Georg Lindgren David Lowe John MacCormick Henrik Malm

Roberto Manduchi Petros Maragos Eric Marchand Jiri Matas Bogdan Matei Esther B. Meier Jason Meltzer Etienne Mémin Rudolf Mester Ross J. Micheals Anurag Mittal Hiroshi Mo William Moran Greg Mori Yael Moses Jane Mulligan Don Murray Masahide Naemura Kenji Nagao Mirko Navara Shree Nayar Oscar Nestares Bernd Neumann Jeffrey Ng Tat Hieu Nguyen Peter Nillius David Nister Alison Noble Tom O’Donnell Takayuki Okatani Nuria Olivier Ole Fogh Olsen Magnus Oskarsson Nikos Paragios Ioannis Patras Josef Pauli Shmuel Peleg Robert Pless Swaminathan Rahul Deva Ramanan Lionel Reveret Dario Ringach Ruth Rosenholtz Volker Roth Payam Saisan

Garbis Salgian Frank Sauer Peter Savadjiev Silvio Savarese Harpreet Sawhney Frederik Schaffalitzky Yoav Schechner Chrostoph Schnoerr Stephan Scholze Ali Shahrokri Doron Shaked Eitan Sharon Eli Shechtman Jamie Sherrah Akinobu Shimizu Ilan Shimshoni Kaleem Siddiqi Hedvig Sidenbladh Robert Sim Denis Simakov Philippe Simard Eero Simoncelli Nir Sochen Yang Song Andreas Soupliotis Sven Spanne Martin Spengler Alon Spira Thomas Strömberg Richard Szeliski Hai Tao Huseyin Tek Seth Teller Paul Thompson Jan Tops Benjamin J. Tordoff Kentaro Toyama Tinne Tuytelaars Shimon Ullman Richard Unger Raquel Urtasun Sven Utcke Luca Vacchetti Anton van den Hengel Geert Van Meerbergen

IX

X

Organization

Pierre Vandergheynst Zhizhou Wang Baba Vemuri Frank Verbiest Maarten Vergauwen Jaco Vermaak Mike Werman David Vernon Thomas Vetter

Rene Vidal Michel Vidal-Naquet Marta Wilczkowiak Ramesh Visvanathan Dan Witzner Hansen Julia Vogel Lior Wolf Bob Woodham Robert J. Woodham

Chenyang Xu Yaser Yacoob Anthony Yezzi Ramin Zabih Hugo Zaragoza Lihi Zelnik-Manor Ying Zhu Assaf Zomet

Table of Contents, Part III

Shape 3D Statistical Shape Models Using Direct Optimisation of Description Length . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . R.H. Davies, C.J. Twining, T.F. Cootes, J.C. Waterton, C.J. Taylor

3

Approximate Thin Plate Spline Mappings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 G. Donato, S. Belongie DEFORMOTION: Deforming Motion, Shape Average and the Joint Registration and Segmentation of Images . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 S. Soatto, A.J. Yezzi Region Matching with Missing Parts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 A. Duci, A.J. Yezzi, S. Mitter, S. Soatto

Stereoscopic Vision I What Energy Functions Can Be Minimized via Graph Cuts? . . . . . . . . . . . . . . . . . 65 V. Kolmogorov, R. Zabih Multi-camera Scene Reconstruction via Graph Cuts . . . . . . . . . . . . . . . . . . . . . . . . 82 V. Kolmogorov, R. Zabih A Markov Chain Monte Carlo Approach to Stereovision . . . . . . . . . . . . . . . . . . . . . 97 J. S´en´egas A Probabilistic Theory of Occupancy and Emptiness . . . . . . . . . . . . . . . . . . . . . . . 112 R. Bhotika, D.J. Fleet, K.N. Kutulakos

Texture Shading and Colour / Grouping and Segmentation / Object Recognition Texture Similarity Measure Using Kullback-Leibler Divergence between Gamma Distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133 J.R. Mathiassen, A. Skavhaug, K. Bø All the Images of an Outdoor Scene . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148 S.G. Narasimhan, C. Wang, S.K. Nayar Recovery of Reflectances and Varying Illuminants from Multiple Views . . . . . . . . 163 Q.-T. Luong, P. Fua, Y. Leclerc

XII

Table of Contents, Part III

Composite Texture Descriptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180 A. Zalesny, V. Ferrari, G. Caenen, D. Auf der Maur, L. Van Gool Constructing Illumination Image Basis from Object Motion . . . . . . . . . . . . . . . . . . 195 A. Nakashima, A. Maki, K. Fukui Diffuse-Specular Separation and Depth Recovery from Image Sequences . . . . . . . 210 S. Lin, Y. Li, S.B. Kang, X. Tong, H.-Y. Shum Shape from Texture without Boundaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225 D.A. Forsyth Statistical Modeling of Texture Sketch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 240 Y.N. Wu, S.C. Zhu, C.-e. Guo Classifying Images of Materials: Achieving Viewpoint and Illumination Independence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255 M. Varma, A. Zisserman Estimation of Multiple Illuminants from a Single Image of Arbitrary Known Geometry . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272 Y. Wang, D. Samaras The Effect of Illuminant Rotation on Texture Filters: Lissajous’s Ellipses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 289 M. Chantler, M. Schmidt, M. Petrou, G. McGunnigle On Affine Invariant Clustering and Automatic Cast Listing in Movies . . . . . . . . . . 304 A. Fitzgibbon, A. Zisserman Factorial Markov Random Fields . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 321 J. Kim, R. Zabih Evaluation and Selection of Models for Motion Segmentation . . . . . . . . . . . . . . . . 335 K. Kanatani Surface Extraction from Volumetric Images Using Deformable Meshes: A Comparative Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 350 J. Tohka DREAM2 S: Deformable Regions Driven by an Eulerian Accurate Minimization Method for Image and Video Segmentation (Application to Face Detection in Color Video Sequences) . . . . . . . . . . . . . . . . . . 365 S. Jehan-Besson, M. Barlaud, G. Aubert Neuro-Fuzzy Shadow Filter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 381 B.P.L. Lo, G.-Z. Yang Parsing Images into Region and Curve Processes . . . . . . . . . . . . . . . . . . . . . . . . . . 393 Z. Tu, S.-C. Zhu

Table of Contents, Part III

XIII

Yet Another Survey on Image Segmentation: Region and Boundary Information Integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 408 J. Freixenet, X. Mu˜noz, D. Raba, J. Mart´ı, X. Cuf´ı Perceptual Grouping from Motion Cues Using Tensor Voting in 4-D . . . . . . . . . . . 423 M. Nicolescu, G. Medioni Deformable Model with Non-euclidean Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . 438 B. Taton, J.-O. Lachaud Finding Deformable Shapes Using Loopy Belief Propagation . . . . . . . . . . . . . . . . . 453 J.M. Coughlan, S.J. Ferreira Probabilistic and Voting Approaches to Cue Integration for Figure-Ground Segmentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 469 E. Hayman, J.-O. Eklundh Bayesian Estimation of Layers from Multiple Images . . . . . . . . . . . . . . . . . . . . . . . 487 Y. Wexler, A. Fitzgibbon, A. Zisserman A Stochastic Algorithm for 3D Scene Segmentation and Reconstruction . . . . . . . . 502 F. Han, Z. Tu, S.-C. Zhu Normalized Gradient Vector Diffusion and Image Segmentation . . . . . . . . . . . . . . 517 Z. Yu, C. Bajaj Spectral Partitioning with Indefinite Kernels Using the Nystr¨om Extension . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 531 S. Belongie, C. Fowlkes, F. Chung, J. Malik A Framework for High-Level Feedback to Adaptive, Per-Pixel, Mixture-of-Gaussian Background Models . . . . . . . . . . . . . . . . . . . . . . . . 543 M. Harville Multivariate Saddle Point Detection for Statistical Clustering . . . . . . . . . . . . . . . . . 561 D. Comaniciu, V. Ramesh, A. Del Bue Parametric Distributional Clustering for Image Segmentation . . . . . . . . . . . . . . . . . 577 L. Hermes, T. Z¨oller, J.M. Buhmann Probabalistic Models and Informative Subspaces for Audiovisual Correspondence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 592 J.W. Fisher, T. Darrell Volterra Filtering of Noisy Images of Curves . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 604 J. August Image Segmentation by Flexible Models Based on Robust Regularized Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 621 M. Rivera, J. Gee

XIV

Table of Contents, Part III

Principal Component Analysis over Continuous Subspaces and Intersection of Half-Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 635 A. Levin, A. Shashua On Pencils of Tangent Planes and the Recognition of Smooth 3D Shapes from Silhouettes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 651 S. Lazebnik, A. Sethi, C. Schmid, D. Kriegman, J. Ponce, M. Hebert Estimating Human Body Configurations Using Shape Context Matching . . . . . . . 666 G. Mori, J. Malik Probabilistic Human Recognition from Video . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 681 S. Zhou, R. Chellappa SoftPOSIT: Simultaneous Pose and Correspondence Determination . . . . . . . . . . . . 698 P. David, D. DeMenthon, R. Duraiswami, H. Samet A Pseudo-Metric for Weighted Point Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 715 P. Giannopoulos, R.C. Veltkamp Shock-Based Indexing into Large Shape Databases . . . . . . . . . . . . . . . . . . . . . . . . . 731 T.B. Sebastian, P.N. Klein, B.B. Kimia EigenSegments: A Spatio-Temporal Decomposition of an Ensemble of Images . . 747 S. Avidan On the Representation and Matching of Qualitative Shape at Multiple Scales . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 759 A. Shokoufandeh, S. Dickinson, C. J¨onsson, L. Bretzner, T. Lindeberg Combining Simple Discriminators for Object Discrimination . . . . . . . . . . . . . . . . . 776 S. Mahamud, M. Hebert, J. Lafferty Probabilistic Search for Object Segmentation and Recognition . . . . . . . . . . . . . . . . 791 U. Hillenbrand, G. Hirzinger Real-Time Interactive Path Extraction with On-the-Fly Adaptation of the External Forces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 807 O. G´erard, T. Deschamps, M. Greff, L.D. Cohen Matching and Embedding through Edit-Union of Trees . . . . . . . . . . . . . . . . . . . . . 822 A. Torsello, E.R. Hancock A Comparison of Search Strategies for Geometric Branch and Bound Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 837 T. M. Breuel Face Recognition from Long-Term Observations . . . . . . . . . . . . . . . . . . . . . . . . . . . 851 G. Shakhnarovich, J.W. Fisher, T. Darrell

Table of Contents, Part III

XV

Stereoscopic Vision II Helmholtz Stereopsis: Exploiting Reciprocity for Surface Reconstruction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 869 T. Zickler, P.N. Belhumeur, D.J. Kriegman Minimal Surfaces for Stereo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 885 C. Buehler, S.J. Gortler, M.F. Cohen, L. McMillan Finding the Largest Unambiguous Component of Stereo Matching . . . . . . . . . . . . 900 ˇara R. S´

Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 915

Table of Contents, Part I

Active and Real-Time Vision Tracking with the EM Contour Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A.E.C. Pece, A.D. Worrall

3

M2Tracker: A Multi-view Approach to Segmenting and Tracking People in a Cluttered Scene Using Region-Based Stereo . . . . . . . . . . . . . . . . . . . . . 18 A. Mittal, L.S. Davis

Image Features Analytical Image Models and Their Applications . . . . . . . . . . . . . . . . . . . . . . . . . . 37 A. Srivastava, X. Liu, U. Grenander Time-Recursive Velocity-Adapted Spatio-Temporal Scale-Space Filters . . . . . . . . 52 T. Lindeberg Combining Appearance and Topology for Wide Baseline Matching . . . . . . . . . . . . 68 D. Tell, S. Carlsson Guided Sampling and Consensus for Motion Estimation . . . . . . . . . . . . . . . . . . . . . 82 B. Tordoff, D.W. Murray

Image Features / Visual Motion Fast Anisotropic Gauss Filtering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 J.-M. Geusebroek, A.W.M. Smeulders, J. van de Weijer Adaptive Rest Condition Potentials: Second Order Edge-Preserving Regularization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 M. Rivera, J.L. Marroquin An Affine Invariant Interest Point Detector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128 K. Mikolajczyk, C. Schmid Understanding and Modeling the Evolution of Critical Points under Gaussian Blurring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143 A. Kuijper, L. Florack Image Processing Done Right . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158 J.J. Koenderink, A.J. van Doorn Multimodal Data Representations with Parameterized Local Structures . . . . . . . . . 173 Y. Zhu, D. Comaniciu, S. Schwartz, V. Ramesh

Table of Contents, Part I

XVII

The Relevance of Non-generic Events in Scale Space Models . . . . . . . . . . . . . . . . 190 A. Kuijper, L. Florack The Localized Consistency Principle for Image Matching under Non-uniform Illumination Variation and Affine Distortion . . . . . . . . . . . . . . . . . . . 205 B. Wang, K.K. Sung, T.K. Ng Resolution Selection Using Generalized Entropies of Multiresolution Histograms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220 E. Hadjidemetriou, M.D. Grossberg, S.K. Nayar Robust Computer Vision through Kernel Density Estimation . . . . . . . . . . . . . . . . . 236 H. Chen, P. Meer Constrained Flows of Matrix-Valued Functions: Application to Diffusion Tensor Regularization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 251 C. Chefd’hotel, D. Tschumperl´e, R. Deriche, O. Faugeras A Hierarchical Framework for Spectral Correspondence . . . . . . . . . . . . . . . . . . . . . 266 M. Carcassoni, E.R. Hancock Phase-Based Local Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 282 G. Carneiro, A.D. Jepson What Is the Role of Independence for Visual Recognition? . . . . . . . . . . . . . . . . . . . 297 N. Vasconcelos, G. Carneiro A Probabilistic Multi-scale Model for Contour Completion Based on Image Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 312 X. Ren, J. Malik Toward a Full Probability Model of Edges in Natural Images . . . . . . . . . . . . . . . . . 328 K.S. Pedersen, A.B. Lee Fast Difference Schemes for Edge Enhancing Beltrami Flow . . . . . . . . . . . . . . . . . 343 R. Malladi, I. Ravve A Fast Radial Symmetry Transform for Detecting Points of Interest . . . . . . . . . . . . 358 G. Loy, A. Zelinsky Image Features Based on a New Approach to 2D Rotation Invariant Quadrature Filters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 369 M. Felsberg, G. Sommer Representing Edge Models via Local Principal Component Analysis . . . . . . . . . . . 384 P.S. Huggins, S.W. Zucker Regularized Shock Filters and Complex Diffusion . . . . . . . . . . . . . . . . . . . . . . . . . 399 G. Gilboa, N.A. Sochen, Y.Y. Zeevi

XVIII

Table of Contents, Part I

Multi-view Matching for Unordered Image Sets, or “How Do I Organize My Holiday Snaps?” . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 414 F. Schaffalitzky, A. Zisserman Parameter Estimates for a Pencil of Lines: Bounds and Estimators . . . . . . . . . . . . . 432 G. Speyer, M. Werman Multilinear Analysis of Image Ensembles: TensorFaces . . . . . . . . . . . . . . . . . . . . . 447 M.A.O. Vasilescu, D. Terzopoulos ‘Dynamism of a Dog on a Leash’ or Behavior Classification by Eigen-Decomposition of Periodic Motions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 461 R. Goldenberg, R. Kimmel, E. Rivlin, M. Rudzsky Automatic Detection and Tracking of Human Motion with a View-Based Representation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 476 R. Fablet, M.J. Black Using Robust Estimation Algorithms for Tracking Explicit Curves . . . . . . . . . . . . . 492 J.-P. Tarel, S.-S. Ieng, P. Charbonnier On the Motion and Appearance of Specularities in Image Sequences . . . . . . . . . . . 508 R. Swaminathan, S.B. Kang, R. Szeliski, A. Criminisi, S.K. Nayar Multiple Hypothesis Tracking for Automatic Optical Motion Capture . . . . . . . . . . 524 M. Ringer, J. Lasenby Single Axis Geometry by Fitting Conics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 537 G. Jiang, H.-t. Tsui, L. Quan, A. Zisserman Computing the Physical Parameters of Rigid-Body Motion from Video . . . . . . . . . 551 K.S. Bhat, S.M. Seitz, J. Popovi´c, P.K. Khosla Building Roadmaps of Local Minima of Visual Models . . . . . . . . . . . . . . . . . . . . . 566 C. Sminchisescu, B. Triggs A Generative Method for Textured Motion: Analysis and Synthesis . . . . . . . . . . . . 583 Y. Wang, S.-C. Zhu Is Super-Resolution with Optical Flow Feasible? . . . . . . . . . . . . . . . . . . . . . . . . . . . 599 W.Y. Zhao, H.S. Sawhney New View Generation with a Bi-centric Camera . . . . . . . . . . . . . . . . . . . . . . . . . . . 614 D. Weinshall, M.-S. Lee, T. Brodsky, M. Trajkovic, D. Feldman Recognizing and Tracking Human Action . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 629 J. Sullivan, S. Carlsson

Table of Contents, Part I

XIX

Towards Improved Observation Models for Visual Tracking: Selective Adaptation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 645 J. Vermaak, P. P´erez, M. Gangnet, A. Blake Color-Based Probabilistic Tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 661 P. P´erez, C. Hue, J. Vermaak, M. Gangnet Dense Motion Analysis in Fluid Imagery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 676 ´ M´emin, P. P´erez T. Corpetti, E. A Layered Motion Representation with Occlusion and Compact Spatial Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 692 A.D. Jepson, D.J. Fleet, M.J. Black Incremental Singular Value Decomposition of Uncertain Data with Missing Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 707 M. Brand Symmetrical Dense Optical Flow Estimation with Occlusions Detection . . . . . . . . 721 L. Alvarez, R. Deriche, T. Papadopoulo, J. S´anchez Audio-Video Sensor Fusion with Probabilistic Graphical Models . . . . . . . . . . . . . . 736 M.J. Beal, H. Attias, N. Jojic

Visual Motion Increasing Space-Time Resolution in Video . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 753 E. Shechtman, Y. Caspi, M. Irani Hyperdynamics Importance Sampling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 769 C. Sminchisescu, B. Triggs Implicit Probabilistic Models of Human Motion for Synthesis and Tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 784 H. Sidenbladh, M.J. Black, L. Sigal Space-Time Tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 801 L. Torresani, C. Bregler

Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 813

Table of Contents, Part II

Surface Geometry A Variational Approach to Recovering a Manifold from Sample Points . . . . . . . . . J. Gomes, A. Mojsilovic

3

A Variational Approach to Shape from Defocus . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 H. Jin, P. Favaro Shadow Graphs and Surface Reconstruction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 Y. Yu, J.T. Chang Specularities Reduce Ambiguity of Uncalibrated Photometric Stereo . . . . . . . . . . . 46 ˇara O. Drbohlav, R. S´

Grouping and Segmentation Pairwise Clustering with Matrix Factorisation and the EM Algorithm . . . . . . . . . . 63 A. Robles-Kelly, E.R. Hancock Shape Priors for Level Set Representations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78 M. Rousson, N. Paragios Nonlinear Shape Statistics in Mumford–Shah Based Segmentation . . . . . . . . . . . . 93 D. Cremers, T. Kohlberger, C. Schn¨orr Class-Specific, Top-Down Segmentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109 E. Borenstein, S. Ullman

Structure from Motion / Stereoscopic Vision / Surface Geometry / Shape Quasi-Dense Reconstruction from Image Sequence . . . . . . . . . . . . . . . . . . . . . . . . . 125 M. Lhuillier, L. Quan Properties of the Catadioptric Fundamental Matrix . . . . . . . . . . . . . . . . . . . . . . . . . 140 C. Geyer, K. Daniilidis Building Architectural Models from Many Views Using Map Constraints . . . . . . . 155 D.P. Robertson, R. Cipolla Motion – Stereo Integration for Depth Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . 170 C. Strecha, L. Van Gool

Table of Contents, Part II

XXI

Lens Distortion Recovery for Accurate Sequential Structure and Motion Recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186 K. Cornelis, M. Pollefeys, L. Van Gool Generalized Rank Conditions in Multiple View Geometry with Applications to Dynamical Scenes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201 K. Huang, R. Fossum, Y. Ma Dense Structure-from-Motion: An Approach Based on Segment Matching . . . . . . 217 F. Ernst, P. Wilinski, K. van Overveld Maximizing Rigidity: Optimal Matching under Scaled-Orthography . . . . . . . . . . . 232 J. Maciel, J. Costeira Dramatic Improvements to Feature Based Stereo . . . . . . . . . . . . . . . . . . . . . . . . . . . 247 V.N. Smelyansky, R.D. Morris, F.O. Kuehnel, D.A. Maluf, P. Cheeseman Motion Curves for Parametric Shape and Motion Estimation . . . . . . . . . . . . . . . . . 262 P.-L. Bazin, J.-M. V´ezien Bayesian Self-Calibration of a Moving Camera . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277 G. Qian, R. Chellappa Balanced Recovery of 3D Structure and Camera Motion from Uncalibrated Image Sequences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 294 B. Georgescu, P. Meer Linear Multi View Reconstruction with Missing Data . . . . . . . . . . . . . . . . . . . . . . . 309 C. Rother, S. Carlsson Model-Based Silhouette Extraction for Accurate People Tracking . . . . . . . . . . . . . 325 R. Plaenkers, P. Fua On the Non-linear Optimization of Projective Motion Using Minimal Parameters . 340 A. Bartoli Structure from Many Perspective Images with Occlusions . . . . . . . . . . . . . . . . . . . 355 D. Martinec, T. Pajdla Sequence-to-Sequence Self Calibration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 370 L. Wolf, A. Zomet Structure from Planar Motions with Small Baselines . . . . . . . . . . . . . . . . . . . . . . . . 383 R. Vidal, J. Oliensis Revisiting Single-View Shape Tensors: Theory and Applications . . . . . . . . . . . . . . 399 A. Levin, A. Shashua

XXII

Table of Contents, Part II

Tracking and Rendering Using Dynamic Textures on Geometric Structure from Motion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 415 D. Cobzas, M. Jagersand Sensitivity of Calibration to Principal Point Position . . . . . . . . . . . . . . . . . . . . . . . . 433 R.I. Hartley, R. Kaucic Critical Curves and Surfaces for Euclidean Reconstruction . . . . . . . . . . . . . . . . . . . 447 F. Kahl, R. Hartley View Synthesis with Occlusion Reasoning Using Quasi-Sparse Feature Correspondences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 463 D. Jelinek, C.J. Taylor Eye Gaze Correction with Stereovision for Video-Teleconferencing . . . . . . . . . . . . 479 R. Yang, Z. Zhang Wavelet-Based Correlation for Stereopsis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 495 M. Clerc Stereo Matching Using Belief Propagation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 510 J. Sun, H.-Y. Shum, N.-N. Zheng Symmetric Sub-pixel Stereo Matching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 525 R. Szeliski, D. Scharstein New Techniques for Automated Architectural Reconstruction from Photographs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 541 T. Werner, A. Zisserman Stereo Matching with Segmentation-Based Cooperation . . . . . . . . . . . . . . . . . . . . . 556 Y. Zhang, C. Kambhamettu Coarse Registration of Surface Patches with Local Symmetries . . . . . . . . . . . . . . . 572 J. Vanden Wyngaerd, L. Van Gool Multiview Registration of 3D Scenes by Minimizing Error between Coordinate Frames . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 587 G.C. Sharp, S.W. Lee, D.K. Wehe Recovering Surfaces from the Restoring Force . . . . . . . . . . . . . . . . . . . . . . . . . . . . 598 G. Kamberov, G. Kamberova Interpolating Sporadic Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 613 L. Noakes, R. Kozera Highlight Removal Using Shape-from-Shading . . . . . . . . . . . . . . . . . . . . . . . . . . . . 626 H. Ragheb, E.R. Hancock

Table of Contents, Part II

XXIII

A Reflective Symmetry Descriptor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 642 M. Kazhdan, B. Chazelle, D. Dobkin, A. Finkelstein, T. Funkhouser Gait Sequence Analysis Using Frieze Patterns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 657 Y. Liu, R. Collins, Y. Tsin Feature-Preserving Medial Axis Noise Removal . . . . . . . . . . . . . . . . . . . . . . . . . . . 672 R. Tam, W. Heidrich Hierarchical Shape Modeling for Automatic Face Localization . . . . . . . . . . . . . . . . 687 C. Liu, H.-Y. Shum, C. Zhang Using Dirichlet Free Form Deformation to Fit Deformable Models to Noisy 3-D Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 704 S. Ilic, P. Fua Transitions of the 3D Medial Axis under a One-Parameter Family of Deformations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 718 P. Giblin, B.B. Kimia Learning Shape from Defocus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 735 P. Favaro, S. Soatto A Rectilinearity Measurement for Polygons . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 746 ˇ c, P.L. Rosin J. Zuni´ Local Analysis for 3D Reconstruction of Specular Surfaces – Part II . . . . . . . . . . . 759 S. Savarese, P. Perona Matching Distance Functions: A Shape-to-Area Variational Approach for Global-to-Local Registration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 775 N. Paragios, M. Rousson, V. Ramesh Shape from Shading and Viscosity Solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 790 E. Prados, O. Faugeras, E. Rouy Model Acquisition by Registration of Multiple Acoustic Range Views . . . . . . . . . . 805 A. Fusiello, U. Castellani, L. Ronchetti, V. Murino

Structure from Motion General Trajectory Triangulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 823 J.Y. Kaminski, M. Teicher Surviving Dominant Planes in Uncalibrated Structure and Motion Recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 837 M. Pollefeys, F. Verbiest, L. Van Gool A Bayesian Estimation of Building Shape Using MCMC . . . . . . . . . . . . . . . . . . . . 852 A.R. Dick, P.H.S. Torr, R. Cipolla

XXIV

Table of Contents, Part II

Structure and Motion for Dynamic Scenes – The Case of Points Moving in Planes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 867 P. Sturm What Does the Scene Look Like from a Scene Point? . . . . . . . . . . . . . . . . . . . . . . . 883 M. Irani, T. Hassner, P. Anandan

Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 899

3D Statistical Shape Models Using Direct Optimisation of Description Length Rhodri H. Davies1 , Carole J. Twining1 , Tim F. Cootes1 , John C. Waterton2 , and Chris J. Taylor1 1

Division of Imaging Science, Stopford Building, Oxford Road, University of Manchester, Manchester, M13 9PT, UK [email protected] 2 AstraZeneca, Alderley Park, Macclesﬁeld, Cheshire, SK10 4TG, UK

Abstract. We describe an automatic method for building optimal 3D statistical shape models from sets of training shapes. Although shape models show considerable promise as a basis for segmenting and interpreting images, a major drawback of the approach is the need to establish a dense correspondence across a training set of example shapes. It is important to establish the correct correspondence, otherwise poor models can result. In 2D, this can be achieved using manual ‘landmarks’, but in 3D this becomes impractical. We show it is possible to establish correspondences automatically, by casting the correspondence problem as one of ﬁnding the ‘optimal’ parameterisation of each shape in the training set. We describe an explicit representation of surface parameterisation, that ensures the resulting correspondences are legal, and show how this representation can be manipulated to minimise the description length of the training set using the model. This results in compact models with good generalisation properties. Results are reported for two sets of biomedical shapes, showing signiﬁcant improvement in model properties compared to those obtained using a uniform surface parameterisation.

1

Introduction

Statistical models of shape show considerable promise as a basis for segmenting and interpreting images in 2D [5]. The basic idea is to establish, from a training set, the pattern of ‘legal’ variation in the shapes and spatial relationships of structures for a given class of images. Statistical analysis is used to give an eﬃcient parameterisation of this variability, providing a compact representation of shape and allowing shape constraints to be applied eﬀectively during image interpretation [6]. A key step in building a model involves establishing a dense correspondence between shape boundaries/surfaces over a reasonably large set of training images. It is important to establish the ‘correct’ correspondences, otherwise an ineﬃcient parameterisation of shape can result, leading to diﬃculty in deﬁning shape constraints eﬀectively. In 2D, correspondence is often established using manually deﬁned ‘landmarks’ but this is a time-consuming, error-prone A. Heyden et al. (Eds.): ECCV 2002, LNCS 2352, pp. 3–20, 2002. c Springer-Verlag Berlin Heidelberg 2002

4

R.H. Davies et al.

and subjective process. In principle, the method extends to 3D, but in practice, manual landmarking becomes impractical. In this paper we show how an ‘optimal’ model can be built by automatically deﬁning correspondences across a training set of 3D shapes. Several previous attempts have been made to build 3D statistical shape models [3,4,10,12,13,14,21,25]. The problem of establishing dense correspondence over a set of training shapes can be posed as that of deﬁning a parameterisation for each of the training set, assuming correspondence between equivalently parameterised points. Kelemen et. al [14] and Hill et. al [12] use diﬀerent arbitrary parameterisations of the training shapes. Christiensen et al. [7], Szekely and Lavalle [23] and Rueckert et al. [21] describe methods for warping the space in which the shapes are embedded. Models can then be built from the resulting deformation ﬁeld [10,13,21]. Brett and Taylor [3,4] and Wang et. al [25] use shape ‘features’ (e.g. regions of high curvature) to establish point correspondences. The correspondences found using the methods described above are not, in any obvious sense, the correct ones. We show in section 2 (ﬁg. 1) that unsatisfactory models can result if correspondences are chosen inappropriately. We start from the position that the correct correspondences are, by deﬁnition, those that build the ‘best’ model. We deﬁne the ‘best’ model as that with optimal compactness, speciﬁcity and generalisation ability. We have shown elsewhere [8] that a model with these properties can be obtained using an objective function based on the minimum description length principle [18]. We have also described a method that uses the objective function to build models in 2D that are better than the best models we could build using manual landmarks [9]. The representation of the parameterisation did not, however, extend to surfaces in 3D. In this paper we describe the derivation of the objective function. We also describe a novel method of representing and manipulating the parameterisation of a surface allowing the construction of shape models in 3D. The method ensures that only valid correspondences can be represented. In the remainder of the paper, we establish notation, and outline the modelbuilding problem. We then provide a summary of the derivation of the minimum description length objective function. We show how a set of surface parameterisations can be represented explicitly and manipulated to build an optimal model. Finally, we present qualitative and quantitative results of applying the method to surfaces obtained from 3D MR images of brain ventricles and rat kidneys.

2

Statistical Shape Models

A 2d (3d) statistical shape model is built from a training set of example outlines (surfaces), aligned to a common coordinate frame. Each shape, Si , (i = 1, . . . ns ), can (without loss of generality) be represented by a set of n points regularly sampled on the shape, as deﬁned by some parameterisation φi . This allows each shape Si to be represented by an np -dimensional shape vector xi , formed by concatenating the coordinates of its sample points. Using Principal Component

3D Statistical Shape Models

5

analysis, each shape vector can be expressed using a linear model of the form: ¯ + Pbi = x ¯+ pm bm (1) xi = x i , m

where x ¯ is the mean shape vector, P = {pm } are the eigenvectors of the covariance matrix (with corresponding eigenvalues {λm }) that describe a set of orthogonal modes of shape variation and b = {bm } are shape parameters that control the modes of variation. Since our training shapes are continuous, we are interested in the limit np → ∞. This leads to an inﬁnitely large covariance matrix, but we note that there can only be, at most, ns − 1 eigenvalues that are not identically zero (although they may be computationally zero). This means that in the summation above, the index m only takes values in the range 1 to ns − 1. To calculate the non-zero eigenvalues, we consider the np × ns data matrix ¯) : i = 1, . . . ns }. The np × np W constructed from the set of vectors {(xi − x covariance matrix is given by D = np1ns WWT with eigenvectors and eigenvalues {pm , λm } thus: Dpm = λm pm . (2) If we deﬁne {pm , λm , } to be the eigenvectors and eigenvalues of the ns × ns matrix, D = np1ns WT W then: D pm = λm pm From (2) : Dpm = λm pm 1 ⇒ WWT pm = λm pm , np ns

(3)

pre-multiplying by WT : m

m

⇒ D (WT p ) = λm (WT p ) Similarly: D(Wpm ) = λm (Wpm ).

(4)

Therefore, for all λm = 0, we can assign indices such that: λm = λm and pm = Wpm .

(5)

Thus the ns − 1 eigenvalues of D, which are not identically zero, can be obtained directly from D , and the eigenvectors are a weighted sum of the training shapes. As shown in [15], in the limit np → ∞ the ij th element of D is given by the inner product of shapes i and j: ¯ ¯ Dij = dt (Si (φi (t)) − S(t)) · (Sj (φj (t)) − S(t)) (6) ns where S¯ = n1s i=1 Si is the mean shape and Si (φi ) is a continuous representation of Si parameterised by φi . The integral can be evaluated numerically.

6

R.H. Davies et al.

Fig. 1. The ﬁrst mode of variation (±3σ) of two shape models built from the training set of hand outlines but parameterised diﬀerently. Model A was parameterised using manual ‘landmarks’ and model B was parameterised using arc-length parameterisation. The ﬁgure demonstrates that model B can represent invalid shape instances.

New examples of the class of shapes can be generated by choosing values of {bm } within the range found in the training set. The utility of the linear model of shape shown in (1) depends on the appropriateness of the set of parameterisations {φi } that are chosen. An inappropriate choice can result in the need for a large set of modes (and corresponding shape parameters) to approximate the training shapes to a given accuracy and may lead to ‘legal’ values of {bm } generating ‘illegal’ shape instances. For example, ﬁgure 1 shows two 2D models generated from a set of 17 hand outlines. Model A uses a set of parameterisations of the outlines that cause ‘natural’ landmarks such as the tips of the ﬁngers to correspond. Model B uses one such correspondence but then uses a simple path length parameterisation to position the other sample points. The variances of the three most signiﬁcant modes of models A and B are (1.06, 0.58, 0.30) and (2.19, 0.78, 0.54) respectively. This suggests that model A is more compact than model B. All the example shapes generated by model A using values of {bm } within the range found in the training set are ‘legal’ examples of hands, whilst model B generates implausible examples and is thus of limited utility for imposing shape constraints when the model is used for image search, see ﬁg. 1. The set of parameterisations used for model A were obtained by marking ‘natural’ landmarks manually on each training example, then using simple path length parameterisation to sample a ﬁxed number of equally spaced points between them. This manual mark-up is a time-consuming and subjective process. In principle, the modelling approach extends to 3D, but in practice, manual landmarking becomes impractical. We propose to overcome this problem by automatically deﬁning correspondences between a training set of example shapes.

3

An Information Theoretic Objective Function

We wish to deﬁne a criterion for selecting the set of parameterisations {φi } that are used to construct a statistical shape model from a set of training boundaries {Si }. Our aim is to choose {φi } so as to obtain the ‘best possible’ model. Ideally, we would like a model with optimal:

3D Statistical Shape Models

7

Generalisation Ability: the model can describe any instance of the object not just those seen in the training set; Speciﬁcity: the model can only represent valid instances of the object; Compactness: the variation is explained with few parameters. To achieve this, we follow the principle of Occam’s razor which can be paraphrased: ‘simple descriptions generalise best’. As a quantitative measure of ‘simplicity’, we choose to apply The Minimum Description Length (MDL) Principle [18,19]. The MDL principle is based on the idea of transmitting a data set as an encoded message, where the code is based on some pre-arranged set of parametric statistical models. The full transmission then has to include not only the encoded data values, but also the coded model parameter values. Thus MDL balances the model complexity, expressed in terms of the cost of sending the model parameters, against the quality of ﬁt between the model and the data, expressed in terms of the coding length of the data. Comparison of Description Lengths calculated using models from diﬀerent classes can be used as a way of solving the Model Selection Problem [11]. However, our emphasis here is not on selecting the class of model, but on using the Description Length for a single class of model as an objective function for optimisation of correspondence between the shapes. We will use the simple two-part coding formulation of MDL. Although this does not give us a coding which is of the absolute minimum length [20], it does however give us a functional form which is computationally simple to evaluate, hence suitable to be used as an objective function for numerical optimisation. 3.1

The Model

Our training set of ns shapes is sampled according to the parameterisations {φi } to give a set of np -dimensional shape vectors {xi }. We choose to model this set of shape vectors using a multivariate Gaussian model. The initial step in constructing such a model is to change to a coordinate system whose axes are aligned with the principal axes of the data set. This corresponds to the orientation of the linear model deﬁned earlier (1): xi = x ¯+

n s −1

pm bm i .

(7)

m=1

The ns − 1 mutually-orthogonal eigenvectors {pm } span the subspace which contains the training set, and, by appropriate scaling, can be transformed into an orthonormal basis set for this subspace. We will also order these vectors in terms of non-decreasing eigenvalue to give us our ﬁnal orthonormal basis set {˜ pm }. To transmit a shape xi using this model, we ﬁrst have to transmit the mean shape x ¯, then the deviations from the mean shape, which can be written thus: ˜ m · (xi − x ¯). yim ≡ p

(8)

8

R.H. Davies et al.

¯ is We will assume that the code length for the transmission of the mean shape x a constant for a given training set and number of sample points. Furthermore, the code length for the transmission of the set of ns − 1, np -dimensional basis vectors {˜ pm } is also constant for a given training set and number of sample points and need not be considered further. 3.2

The Description Length

˜ m , we now have to transmit the set of values Y m ≡ {yim : For each direction p i = 1 to ns }. Since we have aligned our coordinate axes with the principal axes of the data, and aligned our origin with the mean shape, each direction can now be modelled using a one-dimensional, centred Gaussian. In the Appendix, we derive an expression for the Description Length of one-dimensional, bounded and quantised data, coded using a centred Gaussian model. To utilise this result, we ﬁrst have to calculate a strict upper-bound R on the range of our data and also estimate a suitable value for the data quantisation parameter ∆. Suppose that, for our original shape data, we know that the coordinates of our sample points are strictly bounded thus: −

r r ≤ xiα ≤ for all α = 1 to np , i = 1 to ns . 2 2

(9)

Then, the strict bound for the coordinates {yim } is given by: √ R = r np , so that |yim | ≤ R for all i, m.

(10)

The data quantisation parameter ∆ can be determined by quantising the coordinates of our original sample points. Comparison of the original shape and the quantised shape then allows a maximum permissible value of ∆ to be determined. For example, for boundaries obtained from voxelated images, ∆ will typically be of the order of the voxel size. This also determines our lower bound on the modelled variance σmin ≡ 2∆. The parameters R and ∆ are held constant for a given training set, hence we need not consider the Description Length for the transmission of these values. Our original data values Y m are now replaced by their quantised1 values Yˆ m . The variance of the quantized data is then calculated thus: (σ m )2 =

ns 1 (ˆ y m )2 . ns i=1 i

(11)

The Description Length Dm for each direction is then given by (see Appendix): • If σ m ≥ σmin : • If σ m < σmin but the range of Yˆ m ≥ ∆: • Else: 1

Dm = D(1) (σ m , ns , R, ∆) Dm = D(2) (σ m , ns , R, ∆) Dm = 0.

We will use a ˆ to denote the quantised value of a continuum variable a.

3D Statistical Shape Models

3.3

9

The Objective Function

Let us deﬁne ng to be the number of directions for which the ﬁrst of the above criteria holds, and nmin the number which satisfy the second. Then, since the directions are ordered in terms of non-increasing eigenvalue/variance, the total Description Length for our training set, and our Objective function, can be written thus: F=

ng p=1

ng +nmin

D(1) (σ p , ns , R, ∆) +

D(2) (σ q , ns , R, ∆).

(12)

q=ng +1

We now consider the form of this objective function. For the linear model deﬁned earlier (1): ns 1 (y m )2 (13) n p λm = ns i=1 i In the limit ∆ → 0, the quantised values in Yˆ approach their continuum values, so that: (14) σ m → n p λm . If we also consider the limit where ns is suﬃciently large, it can be seen that the functions D(1) and D(2) can be written in the form: D(1) (σ m , ns , R, ∆) ≈ f (R, ∆, ns ) + (ns − 2) ln σ m D(2) (σ m , ns , R, ∆) ≈ f (R, ∆, ns ) + (ns − 2) ln σmin +

(ns + 3) 2

m

σ σmin

2

(15) −1

where f is some function which depends only on R, ∆ and ns . So, in this dual m limit, the part of the objective function which depends on the {σ } contains terms similar to the determinant of the covariance matrix (that is ln λm ) used by Kotcheﬀ and Taylor [15]. However, our objective function is well-deﬁned, even in the limit λm → 0, where in fact such a direction makes no contribution to the objective function. Whereas in the form used previously, without the addition of artiﬁcial correction terms, it would have an inﬁnitely large contribution.

4

Manipulating the Parameterisation of a Surface

In order to build a statistical shape model, we need to sample a number of corresponding points on each shape. As demonstrated in section 2, the choice of correspondences determines the quality of the model. We have chosen to cast this correspondence problem as that of deﬁning the parameterisation φi , of each shape so as to minimise the value of F in (12). We propose to sove for {φi } using numerical optimisation, which requires that we formulate an explicit (and ideally, compact) representation of the φi that can be manipulated. We would also like the computational complexity to be minimal (i.e. it should only involve the evaluation of elementary functions).

10

R.H. Davies et al.

Each surface in our training set is originally represented as a triangular mesh that is topologically equivalent to a sphere. We obtain an initial parameterisation by mapping each surface mesh to a unit sphere, where the mapping must be such that there is no folding or tearing. Each mapped mesh can then be represented thus: (16) Si = Si (θ, ψ), Si R3 where Si is the set of original positions of the mesh vertices for the ith surface in Euclidean space, and (θ, ψ) are the spherical polar coordinates of each mapped vertex. Various approaches have been described to achieve such mappings [1,2, 24]. Since we intend to optimise the parameterisations, the ﬁnal result should signiﬁcantly not depend on this initial mapping. We have used the method described by Brechbulher [2] to produce the results reported below. Changes in parameterisation of a given surface correspond to altering the positions of the mapped vertices on the sphere. That is: Si → Si , θ → θ , ψ → ψ where

Si (θ, ψ) =

Si (θ , ψ )

and

θ =

(17) φθi (θ, ψ),

ψ =

φψ i (θ, ψ).

Note that we have separate parameterisation functions φi = (φθi , φψ i ) for each surface. Valid parameterisation functions φi correspond to exact homeomorphic mappings of the sphere. That is, mappings that are continuous, one-to-one and onto. In the following sections, we present a number of such mappings. 4.1

Symmetric Theta Transformations

Consider an arbitrary point P on the unit sphere. For simplicity, assume that the spherical polar co-ordinates (θ, ψ) on the sphere have been redeﬁned so that P corresponds to the point θ = 0. Let us ﬁrst consider a rotationally symmetric mapping that reparameterises the θ coordinate: θ → f (θ).

(18)

For the mapping to be homeomorphic and continuous with the identity, f must be a diﬀerentiable non-decreasing monotonic function over the range 0 < θ < π, with f (0) = 0, f (π) = π. Any such monotonic function f can be rewritten in terms of the cumulative distribution function of some density function ρ(θ), deﬁned over the range 0 ≤ θ ≤ π. As our normalised density function, we take a constant term plus a wrapped Cauchy distribution. The wrapped Cauchy [16] is a normalisable, uni-modal distribution for circular data, of variable width, which has an analytic indeﬁnite integral:

1 − α2 1 1+A (19) ρ(θ) = N 1 + α2 − 2α cos θ where N = π [1 + A]. Hence: θ f (θ) = π 0

1 (1 + α2 ) cos θ − 2α ds ρ(s) = θ + A arccos 1+A 1 + α2 − 2α cos θ

(20)

3D Statistical Shape Models

11

Fig. 2. Left: Original sphere. Right: Sphere after asymmetric θ transformation.

where α (α ≡ e−a , a ∈ R) is the width, and A (A ≥ 0) is the amplitude of the Cauchy. The constant term is included so that f (θ) = θ when A = 0. i.e. the parameterisation is unchanged when the Cauchy has zero magnitude. 4.2

Asymmetric Theta Transformations

We can also perform non-symmetric transformations: θ → f (θ, ψ).

(21)

Deﬁne ψ to be the ψ coordinates redeﬁned so that a point (Q = P ) corresponds to ψ = 0. An asymmetric transformation around the point P , towards a point Q can be achieved using (20) and making the amplitude A a smooth function of the ψ coordinate: A → A(ψ ). (22) One such way to do this is to use the wrapped Cauchy distribution to obtain: A(ψ ) = A0

1 − β2 1 − β2 − 2 1 + β − 2β cos ψ (1 + β)2

(23)

where β (β ≡ e−b ) is the width of the subsidiary Cauchy. We have chosen the formulation such that A(ψ ) has a minimum value of zero. An example of an asymmetric transformation is shown in ﬁgure 2.

12

R.H. Davies et al.

4.3

Shear and Twist

We also consider transformations of the ψ coordinate. This is equivalent to shearing and twisting the sphere about the axis deﬁned by the point P . So, for example, we could consider a reparameterisation of the form: ψ → ψ + g(θ) where

g(θ) =

B 2π

1+

γ2

1 − γ2 − 2γ cos (θ − θ0 )

(24)

(25)

where B is the amplitude, γ (γ ≡ e−c , c ∈ R) is the width and θ0 is the position of the centre. This transformation is continuous with the identity at B = 0 (i.e. the transformation has no aﬀect when B = 0). It can also be localised about θ = θ0 in the limit of zero width. An example of such a transformation is shown in Figure 3. Figure 4 shows an example of applying a combination of all the transformations described above.

5

Optimising the Parameterisations

We now wish to manipulate {φi } so as to minimise F in (12). We have found that, for the objects modelled in this paper, the symmetric theta transformations, alone, provide a suﬃcient group of reparameterisations. To manipulate the parameterisations, we ﬁx the position P and width a of each of the Cauchy kernels and vary its magnitude A. To select the positions P of the kernels, we (approximately) uniformly sample the sphere and centre the kernels at these sample points. It is not strictly possible to position an arbitrary number of equidistant points on the sphere, but a good

Fig. 3. Left: Original sphere. Right: Sphere after shear transformation.

3D Statistical Shape Models

13

Fig. 4. Left: Original sphere. Right: Sphere after combined transformation.

approximation can be obtained by recursive subdivision of a polyhedron (initially an octohedron) and projecting the points onto the sphere surface. At each level of recursion, each triangle is divided into 4 smaller triangles by placing 3 new vertices halfway along each edge - this gives 12 × 4k−2 new kernels on the k th level of recursion. We choose to use a multiresolution approach to the optimisation. The basic idea is to begin with broad Cauchies and to iteratively reﬁne the parameterisation by introducing additional, narrower Cauchies between the existing ones. We have found, by cross validation, that the best value for the width parameter, is ak = 1/2k−2 , i.e. the widths are halved at each level of recursion. A local optimisation algorithm is employed to ﬁnd the optimum magnitude of each kernel. At each level of recursion, we have 12 × 4k−2 kernels for each shape, creating a 12ns × 4k−2 -dimensional conﬁguration space. This is generally too diﬃcult to optimise robustly and reliably. We have found, however, that optimising the magnitude of a single kernel in turn on each shape gives better results and is substantially quicker. We used the Nelder-Mead simplex algorithm [17] to obtain the results reported in section 6.

6

Results

We present qualitative and quantitative results of applying our method to a training set of 16 rat kidneys and 8 anterior horns of brain ventricles. In each case the shapes were segmented by hand from a set of 3D magnetic resonance images. The algorithm was run for three levels of recursion, giving a total of 66 kernels per shape. We compare the results to models built by uniformly sampling the surface. In ﬁgures 5 and 6, we show qualitative results by displaying the variation captured by the ﬁrst three modes of each model (bm varied by ±2[standard deviatiations seen across the training set]). We show quantitative results in

14

R.H. Davies et al.

Fig. 5. The ﬁrst three modes of variation ±2λm of the automatically produced model of the brain ventricle

Fig. 6. The ﬁrst three modes of variation ±2λm of the automatically produced model of the rat kidneys

3D Statistical Shape Models

15

Table 1. A quantitative comparison of each model showing the variance explained by each mode. F is the value of the objective function and VT is the total variance.

Mode 1 2 3 4 5 6 VT F

Kidneys

Automatic 3.06 2.75 1.35 0.94 0.76 0.46 10.51 1396

Uniform 3.23 3.01 1.70 1.08 0.87 0.57 11.98 1417

Mode 1 2 3 4 5 6 VT F

Ventricles

Automatic 11.98 1.41 1.16 0.69 0.36 0.22 15.90 366

Uniform 12.58 1.77 1.49 0.79 0.45 0.24 17.46 371

Fig. 7. A plot showing the cumulative variance described by each mode of the model. This is a measure of the compactness of each model.

table 1, tabulating the variance explained by each mode, the total variance and the value of F . In both cases F is substantially better for the optimised model. Figure 7 shows the cumulative variance plotted against the number of modes used for each model; this measures the compactness of the model. The plots show that, for the entire range of modes, the optimised models are substantially more compact than those obtained by uniformly-sampling the surface. To test the generalisation ability of the models, we performed leaveone-out tests on each model. In ﬁgure 8 we show the approximation error for representing an unseen shape as a function of the number of modes used in the reconstruction. The optimised models perform substantially better than the uniformly sampled models whatever the number of modes used, demonstrating superior generalisation ability.

7

Discussion and Conclusions

We have described a method for building 3D statistical shape models by automatically establishing optimal correspondences between sets of shapes. We have shown that the method produces models that are more compact than those

16

R.H. Davies et al.

Fig. 8. Leave one out tests on the models. The plot shows the number of modes used against the mean squared approximation error. This measures the ability of each model to generalise to unseen examples.

based on uniformly-sampled shapes and have substantially better generalisation ability. We have described a novel method of reparameterising closed surfaces. The method guarantees that the resulting reparameterisation is homeomorphic - an essential constraint. With a suitable choice of boundary conditions, the representation can also be applied to reparameterise open surfaces. Although we have only reported results on a relatively small set of example shapes, we believe the method scales easily to deal with sets of 40-50 shapes. The local optimisation algorithm employed will, however, be substantially slower. We are currently looking for more robust and faster methods of ﬁnding the minimum. The MDL objective function and optimisation approach provides a uniﬁed framework for dealing with all aspects of model building. For example, we plan to investigate including the alignment of the shapes in the optimisation.

Appendix: Description Length for One-Dimensional Centred Gaussian Models In this Appendix, we show how to construct an expression for the description length required to send a one-dimensional, centred data set using a Gaussian model. Our data model is the family of centred Gaussian distributions: 1 y2 (26) ρ(y; σ) = √ exp − 2 . 2σ σ 2π Following the two-part coding scheme [19], the total description length is computed as the sum of two parts; the description length for sending the value and accuracy of the parameter σ, and the description length for coding the data according to this model. To calculate these description lengths, we use the fundamental result that the ideal-coding codeword length for a discrete event A, encoded using a statistical model with associated event probabilities P (A) is given by the Shannon Coding codeword length [22]:

3D Statistical Shape Models

L(A; P ) = − log2 P (A) bits, or: − ln P (A) nats.2

17

(27)

We take our centred data set to be Y = {yi : i = 1 to ns }, where the data is known to lie within a strictly bounded region. To reduce the continuum values {yi } to a set of discrete events, we quantize the data values using a parameter ∆, so that Y → Yˆ = {ˆ yi : i = 1 to ns } 3 , where for any quantized value yˆ from any possible data set: − R < yˆ < R and yˆ = m∆, m ZZ.

(28)

Coding the Parameters We will assume that the parameter σ is described to an accuracy δ. We will also assume that our now quantized parameter σ ˆ is bounded, hence has the allowed values: ˆ ≤ σmax . (29) σ ˆ = nδ, n IN and σmin ≤ σ Given the absence of any prior knowledge, we assume a ﬂat distribution for σ ˆ over this range, which gives us the codeword length: σmax − σmin . (30) Lσˆ = ln δ Note that our receiver cannot decrypt the value of σ ˆ without knowing the value of δ. If we assume that the quantization parameter δ is of the form: δ = 2±k , k IN

(31)

then it can easily be seen that it can be coded directly with a codeword length: Lδ = 1 + | log2 δ| bits ≈ 1 + | ln δ| nats

(32)

where the additional bit/nat codes for the sign in the exponent of δ. The total code length for transmitting the parameters is then given by Lσˆ + Lδ . Coding the Data For our Gaussian data model, the probability P (ˆ y ) associated with a bin centred at yˆ is: yˆ+∆/2 2 ∆ P (ˆ y) = y /2ˆ (33) dk ρ(k; σ ˆ ) ≈ √ exp −ˆ σ2 . σ ˆ 2π yˆ−∆/2

2

3

In what follows, we will restrict ourselves to natural logarithms, and codeword lengths in units of nats. However, expressions can be easily converted to the more familiar binary lengths by noting that 1 bit ≡ ln 2 nats. We will use α ˆ to denote the quantized value of the continuum value α.

18

R.H. Davies et al.

It can be shown numerically that this is a very good approximation (mean fractional error ≤ 1.0% ± 0.8%) for all values σ ˆ ≥ 2∆, hence we will take σmin = 2∆. The code length for the data is then: Ldata = −

ns

ln P (ˆ yi ) = −ns ln ∆ +

i=1

ns ns 1 ln(2πˆ σ2 ) + 2 yˆ2 2 2ˆ σ i=1 i

(34)

The variance of the quantized data is: σ2 =

ns 1 yˆ2 and σmax = R. ns i=1 i

(35)

In general σ diﬀers from the nearest quantized value σ ˆ thus: δ . (36) 2 So, averaging over an ensemble of data sets, and assuming a ﬂat distribution for dσ over this range, we ﬁnd: δ2 δ2 1 1 δ2 , 2 ≈ 2 1 + 2 , ln σ d2σ = ˆ 2 ≈ ln σ 2 − . (37) 12 σ ˆ σ 4σ 12σ 2 σ ˆ = σ + dσ , |dσ | ≤

By substituting these expressions into (34) and using (35) gives the following expression for the Description Length of the data: ns δ 2 ns ns ln(2πσ 2 ) + + . (38) 2 2 12σ 2 The total Description Length for the parameters and the data is then: ns ns ns δ 2 σmax − σmin . (39) +| ln δ|−ns ln ∆+ ln(2πσ 2 )+ + L(1) = 1+ln δ 2 2 12σ 2 Ldata = −ns ln ∆ +

By diﬀerentiating w.r.t. δ and setting the derivative to zero, we ﬁnd that the optimum parameter accuracy δ is:

12 ∗ δ (σ, ns ) = min 1, σ (40) ns which then allows us to write the above expression as: L(1) = D(1) (σ, ns , R, ∆).

(41)

In the case where σ < σmin , but the quantized data occupies more than one bin, we will model the data using a Gaussian of width σmin and a quantization parameter δ = δ ∗ (σmin , ns ). An analogous derivation to that given above then gives us the Description Length: ns σmax − σmin 2 ) + | ln δ| − ns ln ∆ + ln(2πσmin L(2) = 1 + ln δ 2 ns δ 2 ns σ 2 δ2 − + (42) 1 + ≡ D(2) (σ, ns , R, ∆). 2 2 2 24σmin 2σmin 4σmin

3D Statistical Shape Models

19

The only remaining case is where all the quantized data lies in one bin at the origin. This requires no further information to describe it fully, hence has a description length of zero. Acknowledgements. The authors would like to thank Alan Brett and Johann Kim for their contribution to this work. Rhodri Davies would like to acknowledge the BBSRC and AstraZeneca4 for their ﬁnancial support.

References 1. Angenent, S., S. Haker, A. Tannenbaum and R. Kikinis: “On the laplace-beltrami operator and brain surface ﬂattening”. IEEE Trans. Medical Imaging, 18: p. 700711. 1999 2. Brechbulher, C., G. Gerig and O. Kubler, “Parameterisation of closed surfaces for 3-D shape description”, in Computer Vision and Image Understanding, 61(2), p154-170, 1995 3. Brett A.D. and C.J. Taylor, “A Method of automated landmark generation for automated 3D PDM construction”, in Image and Vision Computing, 18(9), p. 739-748, 2000 4. Brett A.D. and C.J. Taylor, “Construction of 3D Shape Models of Femoral Articular Cartialge Using Harmonic Maps”, in MICCAI’00, p. 1205-1214, 2000 5. Cootes, T., A. Hill, C. Taylor, and J. Haslam, “The use of Active shape models for locating structures in medical images”. Image and Vision Computing, 12: p. 355-366. 1994. 6. Cootes, T., C. Taylor, D. Cooper and J. Graham, “Active shape models - their training and application”. Computer Vision and Image Understanding, 61: p. 3859. 1995. 7. Christensen, G. E., S. C. Joshi, M.I. Miller, “Volumetric Transformation of Brain Anatomy”, in IEEE Trans. Medical Imaging, 16, p864-877, 1997 8. Davies, Rh. H., C.J. Twining, T.F. Cootes, J.C. Waterton and C.J. Taylor, “A Minimum Description Length Approach to Statistical Shape Modelling”, in IEEE trans. Medical Imaging, To appear 9. Davies, Rh. H., T.F. Cootes, C.J. Twining and C.J. Taylor, “An Information Theoretic Approach to Statistical Shape Modelling”, in British Machine Vision Conference - BMVC’01, p 3-12, 2001 10. Fluete, M., S. Lavalee, “Building a Complete Surface Model from Sparse Data Using Statistical Shape Models: Application to Computer Assisted Knee Surgery”, in MICCAI’98, p878-887, 1998 11. M. H. Hansen and B. Yu, “Model Selection and the Principle of Minimum Description Length”, Technical Memorandum, Bell Labs, Murray Hill, N.J. 1998. 12. Hill,A., A. Thornham and C.J. Taylor, “Model based interpretation of 3D medical images”, in British Machine Vision Conference, BMVC’93, p339-348, 1993 13. Joshi et. al , “Gaussian Random Fields on Sub-Manifolds for Characterizing Brain Surfaces”, in IPMI’97, p381-386, 1997 14. Kelemen, A., G. Szekely, and G. Gerig, “Elastic model-based segmentation of 3-D neuroradiological data sets”. IEEE Transactions On Medical Imaging, 18(10): p. 828-839. 1999. 4

AstraZeneca, Alderley Park, Macclesﬁeld, Cheshire, UK

20

R.H. Davies et al.

15. Kotcheﬀ, A.C.W. and C.J. Taylor, ”Automatic Construction of Eigenshape Models by Direct Optimisation”. Medical Image Analysis, 2: p. 303-314. 1998. 16. Mardia, K. V., Statistics of Directional Data, Academic Press,London,1972 17. Press, W.H., S. A. Teukolsky, W. T. Vetterling, B. P. Flannery, Numerical Recipes in C, Cambridge University Press; 1993 18. Rissanen, J. R. ”A universal prior for integers and estimation by minimum description length”, Annals of Statistics, vol. 11, no. 2, pp. 416-431, 1983. 19. Rissanen, J. R. Stochastic Complexity in Statistical Inquiry, World Scientiﬁc Series in Computer Science, Vol. 15, World Scientiﬁc Publishing Co., Singapore, 1989. 20. Rissanen, J. R. “Fisher Information and Stochastic Complexity”, IEEE Transactions on Information Theory, vol. 42, no. 1, pp. 40-47, 1996. 21. Rueckert, D., Frangi, F., Schnabel, J.A., “Automatic construction of 3D statistical deformation models using non-rigid registration”, MICCAI’01 p77-84. 2001. 22. Shannon, C. E., “A mathematical theory of communication,”, Bell System Technical Journal, vol. 27, pp. 379-423 and 623-656, 1948. 23. Szeleky, G., S. Lavalee, “Matching 3-D Anatomical Surface with Non-Rigid Deformations using Octree-Splines”, in International Journal of Computer Vision, 18(2), p171-186, 1996 24. D. Tosun and J.L. Prince, “Hemispherical map for the human brain cortex,” Proc. SPIE Medical Imaging 2001, p 290-300. 2001. 25. Wang, Y., B. S. Peterson, and L. H. Staib. “Shape-based 3D surface correspondence using geodesics and local geometry”. CVPR 2000, v. 2: p. 644-51. 2000.

Approximate Thin Plate Spline Mappings Gianluca Donato1 and Serge Belongie2 1

Digital Persona, Inc., Redwood City, CA 94063 [email protected] 2 U.C. San Diego, La Jolla, CA 92093-0114 [email protected]

Abstract. The thin plate spline (TPS) is an eﬀective tool for modeling coordinate transformations that has been applied successfully in several computer vision applications. Unfortunately the solution requires the inversion of a p × p matrix, where p is the number of points in the data set, thus making it impractical for large scale applications. As it turns out, a surprisingly good approximate solution is often possible using only a small subset of corresponding points. We begin by discussing the obvious approach of using the subsampled set to estimate a transformation that is then applied to all the points, and we show the drawbacks of this method. We then proceed to borrow a technique from the machine learning community for function approximation using radial basis functions (RBFs) and adapt it to the task at hand. Using this method, we demonstrate a signiﬁcant improvement over the naive method. One drawback of this method, however, is that is does not allow for principal warp analysis, a technique for studying shape deformations introduced by Bookstein based on the eigenvectors of the p × p bending energy matrix. To address this, we describe a third approximation method based on a classic matrix completion technique that allows for principal warp analysis as a by-product. By means of experiments on real and synthetic data, we demonstrate the pros and cons of these diﬀerent approximations so as to allow the reader to make an informed decision suited to his or her application.

1

Introduction

The thin plate spline (TPS) is a commonly used basis function for representing coordinate mappings from R2 to R2 . Bookstein [3] and Davis et al. [5], for example, have studied its application to the problem of modeling changes in biological forms. The thin plate spline is the 2D generalization of the cubic spline. In its regularized form the TPS model includes the aﬃne model as a special case. One drawback of the TPS model is that its solution requires the inversion of a large, dense matrix of size p × p, where p is the number of points in the data set. Our goal in this paper is to present and compare three approximation methods that address this computational problem through the use of a subset of corresponding points. In doing so, we highlight connections to related approaches in the area of Gaussian RBF networks that are relevant to the TPS mapping A. Heyden et al. (Eds.): ECCV 2002, LNCS 2352, pp. 21–31, 2002. c Springer-Verlag Berlin Heidelberg 2002

22

G. Donato and S. Belongie

problem. Finally, we discuss a novel application of the Nystr¨ om approximation [1] to the TPS mapping problem. Our experimental results suggest that the present work should be particularly useful in applications such as shape matching and correspondence recovery (e.g. [2,7,4]) as well as in graphics applications such as morphing.

2

Review of Thin Plate Splines

Let vi denote the target function values at locations (xi , yi ) in the plane, with i = 1, 2, . . . , p. In particular, we will set vi equal to the target coordinates (xi ,yi ) in turn to obtain one continuous transformation for each coordinate. We assume that the locations (xi , yi ) are all diﬀerent and are not collinear. The TPS interpolant f (x, y) minimizes the bending energy 2 2 2 (fxx + 2fxy + fyy )dxdy If = R2

and has the form f (x, y) = a1 + ax x + ay y +

p

wi U ((xi , yi ) − (x, y))

i=1

where U (r) = r2 log r. In order for f (x, y) to have square integrable second derivatives, we require that p

wi = 0

i=1

p i=1

w i xi =

p

and wi yi = 0 .

i=1

Together with the interpolation conditions, f (xi , yi ) = vi , this yields a linear system for the TPS coeﬃcients: K P w v = (1) a o PT O where Kij = U ((xi , yi ) − (xj , yj )), the ith row of P is (1, xi , yi ), O is a 3 × 3 matrix of zeros, o is a 3 × 1 column vector of zeros, w and v are column vectors formed from wi and vi , respectively, and a is the column vector with elements a1 , ax , ay . We will denote the (p + 3) × (p + 3) matrix of this system by L; as discussed e.g. in [7], L is nonsingular. If we denote the upper left p × p block of L−1 by L−1 p , then it can be shown that T If ∝ v T L−1 p v = w Kw .

Approximate Thin Plate Spline Mappings

23

When there is noise in the speciﬁed values vi , one may wish to relax the exact interpolation requirement by means of regularization. This is accomplished by minimizing H[f ] =

n

(vi − f (xi , yi ))2 + λIf .

i=1

The regularization parameter λ, a positive scalar, controls the amount of smoothing; the limiting case of λ = 0 reduces to exact interpolation. As demonstrated in [9,6], we can solve for the TPS coeﬃcients in the regularized case by replacing the matrix K by K + λI, where I is the p × p identity matrix.

3

Approximation Techniques

Since inverting L is an O(p3 ) operation, solving for the TPS coeﬃcients can be very expensive when p is large. We will now discuss three diﬀerent approximation methods that reduce this computational burden to O(m3 ), where m can be as small as 0.1p. The corresponding savings factors in memory (5x) and processing time (1000x) thus make TPS methods tractable when p is very large. In the discussion below we use the following partition of the K matrix: A B K= (2) BT C with A ∈ Rm×m , B ∈ Rm×n , and C ∈ Rn×n . Without loss of generality, we will assume the p points are labeled in random order, so that the ﬁrst m points represent a randomly selected subset. 3.1

Method 1: Simple Subsampling

The simplest approximation technique is to solve for the TPS mapping between a randomly selected subset of the correspondences. This amounts to using A in place of K in Equation (1). We can then use the recovered coeﬃcients to extrapolate the TPS mapping to the remaining points. The result of applying this approximation to some sample shapes is shown in Figure 1. In this case, certain parts were not sampled at all, and as a result the mapping in those areas is poor. 3.2

Method 2: Basis Function Subset

An improved approximation can be obtained by using a subset of the basis functions with all of the target values. Such an approach appears in [10,6] and Section 3.1 of [8] for the case of Gaussian RBFs. In the TPS case, we need to account for the aﬃne terms, which leads to a modiﬁed set of linear equations. Starting from the cost function R[w, ˜ a] =

1 λ T ˜w v − K ˜ − P a2 + w ˜ Aw ˜ , 2 2

24

G. Donato and S. Belongie (a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

Fig. 1. Thin plate spline (TPS) mapping example. (a,b) Template and target synthetic ﬁsh shapes, each consisting of 98 points. (Correspondences between the two shapes are known.) (c) TPS mapping of (a) onto (b) using the subset of points indicated by circles (Method 1). Corresponding points are indicated by connecting line segments. Notice the quality of the mapping is poor where the samples are sparse. An improved approximation can be obtained by making use of the full set of target values; this is illustrated in (d), where we have used Method 2 (discussed in Section 3.2). A similar mapping is found for the same set of samples using Method 3 (see Section 3.3). In (e-h) we observe the same behavior for a pair of handwritten digits, where the correspondences (89 in all) have been found using the method of [2].

we minimize it by setting ∂R/∂ w ˜ and ∂R/∂a to zero, which leads to the following (m + 3) × (m + 3) linear system, T T ˜ + λA K ˜TP ˜ K ˜ v K w ˜ K (3) = T T ˜ T a P v P K P P ˜ T = [A B T ], w ˜ is an m × 1 vector of TPS coeﬃcients, and the rest where K of the entries are as before. Thus we seek weights for the reduced set of basis functions that take into account the full set of p target values contained in v. If we call P˜ the ﬁrst m rows of P and I˜ the ﬁrst m columns of the p × p identity matrix, then under the assumption P˜ T w ˜ = 0, Equation (3) is equivalent to ˜ + λI˜ P w ˜ v K = a o P˜ T O which corresponds to the regularized version of Equation (1) when using the ˜ and P˜ T in place of K and P T . subsampled K The application of this technique to the ﬁsh and digit shapes is shown in Figure 1(d,h).

Approximate Thin Plate Spline Mappings

3.3

25

Method 3: Matrix Approximation

The essence of Method 2 was to use a subset of exact basis functions to approximate a full set of target values. We now consider an approach that uses a full set of approximate basis functions to approximate the full set of target values. The approach is based on a technique known as the Nystr¨om method. The Nystr¨om method provides a means of approximating the eigenvectors of K without using C. It was originally developed in the late 1920s for the numerical solution of eigenfunction problems [1] and was recently used in [11] for fast approximate Gaussian process regression and in [8] (implicitly) to speed up several machine learning techniques using Gaussian kernels. Implicit to the Nystr¨ om method is the assumption that C can be approximated by B T A−1 B, i.e. A B ˆ K= (4) B T B T A−1 B If rank(K) = m and the m rows of the submatrix [A B] are linearly indepenˆ = K. In general, the quality of the approximation can be expressed dent, then K as the norm of the diﬀerence C − B T A−1 B, the Schur complement of K. Given the m × m diagonalization A = U ΛU T , we can proceed to ﬁnd the approximate eigenvectors of K: U T ˜ ˆ ˜ ˜ (5) K = U ΛU , with U = B T U Λ−1 ˜ are not orthogonal. To address this, ﬁrst Note that in general the columns of U ˜ Λ1/2 so that K ˆ = ZZ T . Let QΣQT denote the diagonalization deﬁne Z = U of Z T Z. Then the matrix V = ZQΣ −1/2 contains the leading orthonormalized ˆ i.e. K ˆ = V ΣV T , with V T V = I. eigenvectors of K, From the standard formula for the partitioned inverse of L, we have −1 K + K −1 P S −1 P T K −1 −K −1 P S −1 L−1 = , S = −P T K −1 P −S −1 P T K −1 S −1 and thus

(I + K −1 P S −1 P T )K −1 v w −1 v = =L o a −S −1 P T K −1 v

ˆ −1 = V Σ −1 V T and Using the Nystr¨om approximation to K, we have K w ˆ = (I + V Σ −1 V T P Sˆ−1 P T )V Σ −1 V T v , a ˆ = −Sˆ−1 P T V Σ −1 V T v with Sˆ = −P T V Σ −1 V T P , which is 3 × 3. Therefore, by computing matrixvector products in the appropriate order, we can obtain estimates to the TPS

26

G. Donato and S. Belongie (a)

(b)

(c)

Fig. 2. Grids used for experimental testing. (a) Reference point set S1 : 12 × 12 points on the interval [0, 128]×[0, 128]. (b,c) Warped point sets S2 and S3 with bending energy 0.3 and 0.8, respectively. To test the quality of the diﬀerent approximation methods, we used varying percentages of points to estimate the TPS mapping from S1 to S2 and from S1 to S3 .

coeﬃcients without ever having to invert or store a large p × p matrix. For the regularized case, one can proceed in the same manner, using (V ΣV T + λI)−1 = V (Σ + λI)−1 V T . Finally, the approximate bending energy is given by ˆ = (V T w)T Σ(V T w) wT Kw Note that this bending energy is the average of the energies associated to the x and y components as in [3]. Let us brieﬂy consider what w ˆ represents. The ﬁrst m components roughly correspond to the entries in w ˜ for Method 2; these in turn correspond to the ˆ (i.e. K) ˜ for which exact information is available. The remaining columns of K ˆ with (implicitly) ﬁlled-in values for all but the entries weight columns of K ﬁrst m entries. In our experiments, we have observed that the latter values of w ˆ are nonzero, which indicates that these approximate basis functions are not being disregarded. Qualitatively, the approximation quality of methods 2 and 3 are very similar, which is not surprising since they make use of the same basic information. The pros and cons of these two methods are investigated in the following section.

4 4.1

Experiments Synthetic Grid Test

In order to compare the above three approximation methods, we ran a set of experiments based on warped versions of the cartesian grid shown in Figure 2(a). The grid consists of 12 × 12 points in a square of dimensions 128 × 128. Call this set of points S1 . Using the technique described in Appendix A, we generated

Approximate Thin Plate Spline Mappings If=0.3

8

6

6

5

5

4

4

3

3

2

2

1

1

0

5

10

15 20 Percentage of samples used

25

Method 1 Method 2 Method 3

7

MSE

MSE

7

If=0.8

8

Method 1 Method 2 Method 3

30

27

0

5

10

15 20 Percentage of samples used

25

30

Fig. 3. Comparison of approximation error. Mean squared error in position between points in the target grid and corresponding points in the approximately warped reference grid is plotted vs. percentage of randomly selected samples used. Performance curves for each of the three methods are shown in (a) for If = 0.3 and (b) for If = 0.8.

point sets S2 and S3 by applying random TPS warps with bending energy 0.3 and 0.8, respectively; see Figure 2(b,c). We then studied the quality of each approximation method by varying the percentage of random samples used to estimate the (unregularized) mapping of S1 onto S2 and S3 , and measuring the mean squared error (MSE) in the estimated coordinates. The results are plotted in Figure 3. The error bars indicate one standard deviation over 20 repeated trials. 4.2

Approximate Principal Warps

In [3] Bookstein develops a multivariate shape analysis framework based on −1 −1 eigenvectors of the bending energy matrix L−1 p KLp = Lp , which he refers to as principal warps. Interestingly, the ﬁrst 3 principal warps always have eigenvalue zero, since any warping of three points in general position (a triangle) can be represented by an aﬃne transform, for which the bending energy is zero. The shape and associated eigenvalue of the remaining principal warps lend insight into the bending energy “cost” of a given mapping in terms of that mapping’s projection onto the principal warps. Through the Nystr¨ om approximation in ˆ −1 Method 3, one can produce approximate principal warps using L p as follows: ˆ −1 ˆ −1 + K ˆ −1 P S −1 P T K ˆ −1 L p =K = V Σ −1 V T + V Σ −1 V T P S −1 P T V Σ −1 V T = V (Σ −1 + Σ −1 V T P S −1 P T V Σ −1 )V T ∆ ˆ T = V ΛV

28

G. Donato and S. Belongie

Fig. 4. Approximate principal warps for the ﬁsh shape. From left to right and top to bottom, the surfaces are ordered by eigenvalue in increasing order. The ﬁrst three principal warps, which represent the aﬃne component of the transformation and have eigenvalue zero, are not shown.

where ∆ Λˆ = Σ −1 + Σ −1 V T P S −1 P T V Σ −1 = W DW T

to obtain orthogonal eigenvectors we proceed as in section 3.3 to get ˆΣ ˆW ˆT Λˆ = W ∆ ˆ = ˆ 1/2 and QΣQ ˆ T is the diagonalization of D1/2 W T W D1/2 . where W W D1/2 QΣ Thus we can write

ˆ ˆˆT T ˆ −1 L p = V W ΣW V An illustration of approximate principal warps for the ﬁsh shape is shown in Figure 4, wherein we have used m = 15 samples. As in [3], the principal warps are visualized as continuous surfaces, where the surface is obtained by ˆ −1 applying a warp to the coordinates in the plane using a given eigenvector of L p as the nonlinear spline coeﬃcients; the aﬃne coordinates are set to zero. The

Approximate Thin Plate Spline Mappings

29

Fig. 5. Exact principal warps for the ﬁsh shape.

corresponding exact principal warps are shown in Figure 5. In both cases, warps 4 through 12 are shown, sorted in ascending order by eigenvalue. Given a rank m Nystr¨ om approximation, at most m − 3 principal warps with nonzero eigenvalue are available. These correspond to the principal warps at the “low frequency” end, meaning that very localized warps, e.g. pronounced stretching between adjacent points in the target shape, will not be captured by the approximation. 4.3

Discussion

We now discuss the relative merits of the above three methods. From the synthetic grid tests we see that Method 1, as expected, has the highest MSE. Considering that the spacing between neighboring points in the grid is about 10, it is noteworthy, however, that all three methods achieve an MSE of less than 2 at 30% subsampling. Thus while Method 1 is not optimal in the sense of MSE, its performance is likely to be reasonable for some applications, and it has the advantage of being the least expensive of the three methods. In terms of MSE, Methods 2 and 3 perform roughly the same, with Method 2 holding a slight edge, more so at 5% for the second warped grid. Method 3 has a disadvantage built in relative to Method 2, due to the orthogonalization

30

G. Donato and S. Belongie (a)

(b)

Fig. 6. Comparison of Method 2 (a) and 3 (b) for poorly chosen sample locations. (The performance of Method 1 was terrible and is not shown.) Both methods perform well considering the location of the samples. Note that the error is slightly lower for Method 3, particularly at points far away from the samples.

step; this leads to an additional loss in signiﬁcant ﬁgures and a slight increase in MSE. In this regard Method 2 is the preferred choice. While Method 3 is comparatively expensive and has slightly higher MSE than Method 2, it has the beneﬁt of providing approximate eigenvectors of the bending energy matrix. Thus with Method 3 one has the option of studying shape transformations using principal warp analysis. As a ﬁnal note, we have observed that when the samples are chosen badly, e.g. crowded into a small area, Method 3 performs better than Method 2. This is illustrated in Figure 6, where all of the samples have been chosen at the back of the tail ﬁn. Larger displacements between corresponding points are evident near the front of the ﬁsh for Method 2. We have also observed that the bending energy ˜ exhibits higher variance than that of Method 3; estimate of Method 2 (w ˜ T Aw) e.g. at a 20% sampling rate on the ﬁsh shapes warped using If = 0.3 over 100 trials, Method 2 estimates If to be 0.29 with σ = 0.13 whereas Method 3 gives 0.25 and σ = 0.06. We conjecture that this advantage arises from the presence of the approximate basis functions in the Nystr¨ om approximation, though a rigorous explanation is not known to us.

5

Conclusion

We have discussed three approximate methods for recovering TPS mappings between 2D pointsets that greatly reduce the computational burden. An experimental comparison of the approximation error suggests that the two methods that use only a subset of the available correspondences but take into account the full set of target values perform very well. Finally, we observed that the method based on the Nystr¨om approximation allows for principal warp analysis and per-

Approximate Thin Plate Spline Mappings

31

forms better than the basis-subset method when the subset of correspondences is chosen poorly. Acknowledgments. The authors wish to thanks Charless Fowlkes, Jitendra Malik, Andrew Ng, Lorenzo Torresani, Yair Weiss, and Alice Zheng for helpful discussions. We would also like to thank Haili Chui and Anand Rangarajan for useful insights and for providing the ﬁsh datasets.

Appendix: Generating Random TPS Transformations To produce a random TPS transformation with bending energy ν, ﬁrst choose a set of p reference points (e.g. on a grid) and form L−1 p . Now generate a random vector u, set its last three components to zero, and normalize it. Compute the T diagonalization L−1 p = U ΛU , with the eigenvalues sorted in descending order. √ Finally, compute w = νU Λ1/2 u. Since If is unaﬀected by the aﬃne terms, their values are arbitrary; we set translation to (0, 0) and scaling to (1, 0) and (0, 1).

References 1. C. T. H. Baker. The numerical treatment of integral equations. Oxford: Clarendon Press, 1977. 2. S. Belongie, J. Malik, and J. Puzicha. Matching shapes. In Proc. 8th Int’l. Conf. Computer Vision, volume 1, pages 454–461, July 2001. 3. F. L. Bookstein. Principal warps: thin-plate splines and decomposition of deformations. IEEE Trans. Pattern Analysis and Machine Intelligence, 11(6):567–585, June 1989. 4. H. Chui and A. Rangarajan. A new algorithm for non-rigid point matching. In Proc. IEEE Conf. Comput. Vision and Pattern Recognition, pages 44–51, June 2000. 5. M.H. Davis, A. Khotanzad, D. Flamig, and S. Harms. A physics-based coordinate transformation for 3-d image matching. IEEE Trans. Medical Imaging, 16(3):317– 328, June 1997. 6. F. Girosi, M. Jones, and T. Poggio. Regularization theory and neural networks architectures. Neural Computation, 7(2):219–269, 1995. 7. M. J. D. Powell. A thin plate spline method for mapping curves into curves in two dimensions. In Computational Techniques and Applications (CTAC95), Melbourne, Australia, 1995. 8. A.J. Smola and B. Sch¨ olkopf. Sparse greedy matrix approximation for machine learning. In ICML, 2000. 9. G. Wahba. Spline Models for Observational Data. SIAM, 1990. 10. Y. Weiss. Smoothness in layers: Motion segmentation using nonparametric mixture estimation. In Proc. IEEE Conf. Comput. Vision and Pattern Recognition, pages 520–526, 1997. 11. C. Williams and M. Seeger. Using the Nystr¨ om method to speed up kernel machines. In T. K. Leen, T. G. Dietterich, and V. Tresp, editors, Advances in Neural Information Processing Systems 13: Proceedings of the 2000 Conference, pages 682–688, 2001.

DEFORMOTION Deforming Motion, Shape Average and the Joint Registration and Segmentation of Images Stefano Soatto1 and Anthony J. Yezzi2 1

Department of Computer Science, University of California, Los Angeles, Los Angeles – CA 90095 [email protected] 2 Department of Electrical and Computer Engineering, Georgia Institute of Technology, Atlanta – GA 30332 [email protected]

Abstract. What does it mean for a deforming object to be “moving” (see Fig. 1)? How can we separate the overall motion (a ﬁnite-dimensional group action) from the more general deformation (a diﬀeomorphism)? In this paper we propose a deﬁnition of motion for a deforming object and introduce a notion of “shape average” as the entity that separates the motion from the deformation. Our deﬁnition allows us to derive novel and eﬃcient algorithms to register non-equivalent shapes using region-based methods, and to simultaneously approximate and register structures in grey-scale images. We also extend the notion of shape average to that of a “moving average” in order to track moving and deforming objects through time.

Fig. 1. A jellyﬁsh is “moving while deforming.” What exactly does this mean? How can we separate its “global” motion from its “local” deformation?

1

Introduction

Consider a sheet of paper falling. If it were a rigid object, one could describe its motion by providing the coordinates of one particle and the orientation of an orthonormal reference frame attached to that particle. That is, 6 numbers

This research is supported in part by NSF grant IIS-9876145, ARO grant DAAD1999-1-0139 and Intel grant 8029.

A. Heyden et al. (Eds.): ECCV 2002, LNCS 2352, pp. 32–47, 2002. c Springer-Verlag Berlin Heidelberg 2002

DEFORMOTION

33

would be suﬃcient to describe the object at any instant of time. However, being a non-rigid object, in order to describe it at any instant of time one should really specify the trajectory of each individual particle on the sheet. That is, if γ0 represents the initial collection of particles, one could provide a function f that describes how the entire set of particles evolves in time: γt = f (γ0 , t). Indeed, if each particle can move independently, there may be no notion of “overall motion,” and a more appropriate description of f is that of a “deformation” of the sheet. That includes as a special case a rigid motion, described collectively by a rotation matrix1 R(t) ∈ SO(3) and a translation vector T (t) ∈ R3 , so that γt = f (γ0 , t) = R(t)γ0 + T (t) with R(t) and T (t) independent of the particle in γ0 . In practice, however, that is not how one usually describes a sheet of paper falling. Instead, one may say that the sheet is “moving” downwards along the vertical direction while “locally deforming.” The jellyﬁsh in Fig. 1 is just another example to illustrate the same issue. But what does it even mean for a deforming object to be “moving”? From a mathematical standpoint, rigorously deﬁning a notion of motion for deforming objects presents a challenge. In fact, if we describe the deformation f as the composition of a rigid motion (R(t), T (t)) and a “local deformation” function h(·, t), so that γt = h(R(t)γ0 + T (t), t), we can always ﬁnd inﬁnitely many ˜ t), R(t), ˜ diﬀerent choices h(·, T˜(t) that give rise to the same overall deformation ˜ Rγ ˜ 0 + T˜(t), t) by simply choosing f : γt = f (γ0 , t) = h(R(t)γ0 + T (t), t) = h( . T ˜ ˜ ˜ ˜ T˜). Therefore, we could h(γ, t) = h(RR (γ − T ) + T, t) for any rigid motion (R, ˜ T˜), which is describe the motion of our sheet with (R, T ) as well as with (R, arbitrary, and in the end we would have failed in deﬁning a notion of “motion” that is unique to the event observed. So, how can we deﬁne a notion of motion for a deforming object in a mathematically sound way that reﬂects our intuition? For instance, in Fig. 6, how do we describe the “motion” of a jellyﬁsh? Or in Fig. 5 the “motion” of a storm? In neuroanatomy, how can we “register” a database of images of a given structure, say the corpus callosum (Fig. 9), by “moving” them to a common reference frame? All these questions ultimately boil down to an attempt to separate the overall motion from the more general deformation. Before proceeding, note that this is not always possible or even meaningful. In order to talk about the “motion” of an object, one must assume that “something” of the object is preserved as it deforms. For instance, it may not make sense to try to capture the “motion” of a swarm of bees, or of a collection of particles that indeed all move independently. What we want to capture mathematically is the notion of overall motion when indeed there is one that corresponds to our intuition! The key to this paper is the observation that the notion of motion and the notion of shape are very tightly coupled. Indeed, we will see that the shape average is exactly what allows separating the motion from the deformation.

1

SO(3) denotes the set of 3 × 3 orthogonal matrices with unit determinant.

34

S. Soatto and A.J. Yezzi

1.1

Prior Related Work

The study of shape spans at least a hundred years of research in diﬀerent communities from mathematical morphology to statistics, geology, neuroanatomy, paleontology, astronomy etc. In statistics, the study of “Shape Spaces” was championed by Kendall, Mardia and Carne among others [15,20,8,24,9]. Shapes are deﬁned as the equivalence classes of N points in RM under the similarity group2 , RM N /{SE(M ) × R}. Although the framework clearly distinguishes the notion of “motion” (along the ﬁbers) from the “deformation” (across ﬁbers), the analytical tools are essentially tied to the point-wise representation. One of our goals in this paper is to extend the theory to smooth curves, surfaces and other geometric objects that do not have distinct “landmarks.” In computer vision, a wide literature exists for the problem of “matching” or “aligning” objects based on their images, and space limitations do not allow us to do justice to the many valuable contributions. We refer the reader to [32] for a recent survey. A common approach consists of matching collections of points organized in graphs or trees (e.g. [19,11]). Belongie et al. [4] propose comparing planar contours based on their “shape context.” The resulting match is based on “features” rather than on image intensity directly, similar to [10]. “Deformable Templates,” pioneered by Grenander [12], do not rely on “features” or “landmarks;” rather, images are directly deformed by a (possibly inﬁnite-dimensional) group action and compared for the best match in an “image-based” approach [34]. There, the notion of “motion” (or “alignment” or “registration”) coincides with that of deformation, and there is no clear distinction between the two [5]. Another line of work uses variational methods and the solution of partial diﬀerential equations (PDEs) to model shape and to compute distances and similarity. In this framework, not only can the notion of alignment or distance be made precise [3,33,25,28], but quite sophisticated theories that encompass perceptually relevant aspects can be formalized in terms of the properties of the evolution of PDEs (e.g. [18,16]). Kimia et al. [16] describes a scale-space that corresponds to various stages of evolution of a diﬀusing PDE, and a “reacting” PDE that splits “salient parts” of planar contours by generating singularities. The variational framework has also proven very eﬀective in the analysis of medical images [23,30,22]. Although most of these ideas are developed in a deterministic setting, many can be transposed to a probabilistic context [35,7,31]. Scalespace is a very active research area, and some of the key contributions as they relate to the material of this paper can be found in [14,29,17,1,2] and references therein. The “alignment,” or “registration,” of curves has also been used to deﬁne a notion of “shape average” by several authors (see [21] and references therein). The shape average, or “prototype,” can then be used for recognition in a nearestneighbor classiﬁcation framework, or to initialize image-based segmentation by providing a “prior.” Leventon et al. [21] perform principal component analysis 2

SE(M ) indicates the Euclidean group of dimension M .

DEFORMOTION

35

in the aligned frames to regularize the segmentation of regions with low contrast in brain images. Also related to this paper is the recent work of Paragios and Deriche, where active regions are tracked as they “move.” In [27] the notion of motion is not made distinct from the general deformation, and therefore what is being tracked is a general (inﬁnite-dimensional) deformation. Our aim is to deﬁne tracking as a trajectory on a ﬁnite-dimensional group, despite inﬁnite-dimensional deformations. Substantially diﬀerent in methods, but related in the intent, is the work on stochastic ﬁlters for contour tracking and snakes (see [6] and references therein). Our framework is designed for objects that undergo a distinct overall “global” motion while “locally” deforming. Under these assumptions, our contribution consists of a novel deﬁnition of motion for a deforming object and a corresponding deﬁnition of shape average (Sect. 2). Our deﬁnition allows us to derive novel and eﬃcient algorithms to register non-identical (or non-equivalent) shapes using region-based methods (Sect. 5). We use our algorithms to simultaneously approximate and register structures in images, or to simultaneously segment and calibrate images (Sect. 6). In the context of tracking, we extend our deﬁnition to a novel notion of “moving average” of shape, and use it to perform tracking for deforming objects (Sect. 4). Our deﬁnitions do not rely on a particular representation of objects (e.g. explicit vs. implicit, parametric vs. non-parametric), nor on the particular choice of group (e.g. aﬃne, Euclidean), nor are they restricted to a particular modeling framework (e.g. deterministic, energy-based vs. probabilistic). For the implementation of our algorithms on deforming contours, we have chosen an implicit non-parametric representation in terms of level sets, and we have implemented numerical algorithms for integrating partial diﬀerential equations to converge to the steady-state of an energy-based functional. However, these choices can be easily changed without altering the nature of the contribution of this paper. Naturally, since shape and motion are computed as the solution of a nonlinear optimization problem, the algorithms we propose are only guaranteed to converge to local minima and, in general, no conclusions can be drawn on uniqueness. Indeed, it is quite simple to generate pathological examples where the setup we have proposed fails. In the experimental section we will highlight the limitations of the approach when used beyond the assumptions for which it is designed.

2

Deﬁning Motion and Shape Average

The key idea underlying our framework is that the notion of motion throughout a deformation is very tightly coupled with the notion of shape average. In particular, if a deforming object is recognized as moving, there must be an underlying object (which will turn out to be the shape average) moving with the same motion, from which the original object can be obtained with minimal deformations. Therefore, we will model a general deformation as the composition of a group action g on a particular object, on top of which a local deformation is applied. The shape average is deﬁned as the one that minimizes such deformations.

36

S. Soatto and A.J. Yezzi

ht

γt

gt (µ)

gt

. . .

. . .

h2

g2 µ

T12

γ2

g2 (µ)

h1

γ1

g1

g1 (µ)

Fig. 2. A model (commutative diagram) of a deforming contour.

Let γ1 , γ2 , . . . , γn be n “shapes” (we will soon make the notion precise in Def. 1). Let the map between each pair of shapes be Tij γi = Tij γj , i, j = 1 . . . n.

(1)

It comprises the action of a group g ∈ G (e.g. the Euclidean group on the plane G = SE(2)) and a more general transformation h that belongs to a pre-deﬁned class H (for instance diﬀeomorphisms). The deformation h is not arbitrary, but depends upon another “shape” µ, deﬁned in such a way that γi = hi ◦ gi (µ), i = 1 . . . n.

(2)

Therefore, in general, following the commutative diagram of Fig. 2, we have that . Tij = hi ◦ gi ◦ gj−1 (µ) ◦ h−1 (3) j so that g = gi gj−1 and h is a transformation that depends on hi , hj and µ. Given two or more “shapes” and a cost functional E : H → R+ deﬁned on the set of diﬀeomorphisms, the motion gt and the shape average are deﬁned as the n minimizers of t=1 E(ht ) subject to γt = ht ◦ gt (µ). Note that the only factors which determine the cost of ht are the “shapes” before and after the transfor. mation, µi = gi (µ) and γi , so that we can write, with an abuse of notation, . E(h(µi , γi )) = E(µi , γi ). We are therefore ready to deﬁne our notion of motion during a deformation. Deﬁnition 1 Let γ1 , . . . , γn be smooth boundaries of closed subsets of a diﬀerentiable manifold embedded in RN , which we call pre-shapes. Let H be a class of diﬀeomorphisms acting on γi , and let E : H → R+ be a positive, real-valued functional. Consider now a group G acting on γi via g(γi ). We say that gˆ1 , . . . , gˆn is a motion undergone by γi , i = 1 . . . n if there exists a pre-shape µ ˆ such that gˆ1 , . . . , gˆn , µ ˆ = arg min gt ,µ

n i=1

E(hi ) subject to γi = hi ◦ gi (µ)

i = 1...n

(4)

DEFORMOTION

37

The pre-shape µ ˆ is called the shape average relative to the group G, or Gaverage, and the quantity gˆi−1 (γi ) is called the shape of γi . Remark 1 (Invariance) In the deﬁnition above, one will notice that the shape average is actually a pre-shape, and that there is an arbitrary choice of group action g0 that, if applied to γi and µ, leaves the deﬁnition unchanged (the functional E is invariant with respect to g0 because T (g ◦ g0 , h ◦ g0 ) = T (g, h) ∀ g0 ). For the case of the Euclidean group SE(N ), a way to see this is to notice that the reference frame where µ is described is arbitrary. Therefore, one may choose, for instance, µ = h−1 1 (γ1 ). Remark 2 (Symmetries) In Def. 1 we have purposefully avoided to use the article “the” for the minimizing value of the group action gˆt . It is in fact possible that the minimum of (4) not be unique. A particular case when this occurs is when the pre-shape γ is (symmetric, or) invariant with respect to a particular element of the group G, or to an entire subgroup. Notice, however, that the notion of shape average is still well-deﬁned even when the notion of motion is not unique. This is because any element in the symmetry group suﬃces to register the pre-shapes, and therefore compute the shape average (Fig. 3).

3

Shape and Deformation of a Planar Contour

In this section we consider the implementation of the program above for a simple case: two closed planar contours, γ1 and γ2 , where we choose as cost functional for the deformations h1 , h2 either the set-symmetric diﬀerence ∆ of their interior (the union minus the intersection), or what we call the signed distance score3 ψ . ψ(µ, γ) = ζ(γ)dx (5) µ ¯

where µ ¯ denotes the interior of the contour µ and ζ is the signed distance function of the contour γ; dx is the area form on the plane. In either case, since we have an arbitrary choice of the global reference frame, we can choose g1 = e, the . group identity. We also call g = g2 , so that µ2 = g(µ). The problem of deﬁning the motion and shape average can then be written as gˆ, µ ˆ = arg min g,µ

3

2

E(hi ) subject to γ1 = h1 (µ); γ2 = h2 ◦ g(µ).

(6)

i=1

The rationale behind this score is that one wants to make the signed distance function as positive as possible outside the contour to be matched, and as negative as possible inside. This score can be interpreted as a weighted Monge-Kantorovic functional where the mass of a curve is weighted by its distance from the boundary.

38

S. Soatto and A.J. Yezzi

. As we have anticipated, we choose either E(hi ) = ∆(gi (µ), γi ) or E(hi ) = ψ(gi (µ), γi ). Therefore, abusing the notation as anticipated before Def. 1, we can write the problem above as an unconstrained minimization gˆ, µ ˆ = arg min φ(γ1 , γ2 ) g,µ

where

. φ(γ1 , γ2 ) = E(µ, γ1 ) + E(g(µ), γ2 )

(7)

and E is either ∆ or ψ. The estimate gˆ deﬁnes the motion between γ1 and γ2 , and the estimate µ ˆ deﬁnes the average of the two contours. If one thinks of contours and their interior, represented by a characteristic function χ, as a binary image, then the cost functional above is just a particular case of a more general cost functional where each term is obtained by integrating a function inside and a function outside the contours φ=

2 i=1

µ ¯ in

fin (x, γi )dx +

µ ¯ out

fout (x, γi )dx

(8)

where the bar in µ ¯ indicates that the integral is computed on a region inside or outside µ and we have emphasized the fact that the function f depends upon the contour γi . For instance, for the case of the set symmetric diﬀerence we have fin = (χγ − 1) and fout = χγ . To solve the problem, therefore, we need to minimize the following functional fin (x, γ1 ) + fin (g(x), γ2 )|Jg |dx + fout (x, γ1 ) + fout (g(x), γ2 )|Jg |dx µ ¯ in

µ ¯ out

(9)

where |Jg | is the determinant of the Jacobian of the group action g. This makes it easy to compute the component of the ﬁrst variation of φ along the normal direction to the contour µ, so that we can impose ∇µ φ · N = 0

(10)

to derive the ﬁrst-order necessary condition. If we choose G = SE(2), an isometry, it can be easily shown that ∇µ φ = fin (x, γ1 ) − fout (x, γ1 ) + fin (g(x), γ2 ) − fout (g(x), γ2 ) 3.1

(11)

Representation of Motions and Their Variation

For the speciﬁc case of matrix Lie groups (e.g. G = SE(2)), there exist twist coordinates ξ that can be represented as a skew-symmetric matrix ξ so that4

g = eξ 4

and

∂g ∂ ξ = g ∂ξi ∂ξi

(12)

The “widehat” notation, which indicates a lifting to the Lie algebra, should not be confused with the “hat”ˆ, which indicates an estimated quantity.

DEFORMOTION

39

∂ξ where the matrix ∂ξ is composed of zeros and ones and the matrix exponential i can be computed in closed form using Rodrigues’ formula. To compute the variation of the functional φ with respect to the group action g, we notice that the ﬁrst two terms in φ do not contribute since they are inde. pendent of g. The second two terms are of the generic form A(g) = g(¯µ) f (x)dx. Therefore, we consider the variation of A with respect to the components of ∂A , which we will eventually use to compute the gradient with the twist ξi , ∂ξ i respect to the natural connection ∇G φ = ∂φ g. We ﬁrst rewrite A(g) us∂ξ ing the change of measure g(¯µ) f (x)dx = µ¯ f ◦ g(x)|Jg |dx which leads to ∂A(g) = µ¯ ∂ξ∂ i (f ◦ g(x))|Jg |dx + µ¯ (f ◦ g(x)) ∂ξ∂ i |Jg |dx and note that the Eu∂ξi clidean group is an isometry and therefore the determinant of the Jacobian is one and the second integral can be re-written, using is zero. The last equation ∂g ∂g −1 , g N ds Green’s theorem, as g(µ) f (x) ∂ξi ◦ g (x), N ds = µ f ◦ g(x) ∂ξ ∗ i where g∗ indicates the push-forward. Notice that g is an isometry and therefore it does not aﬀect the arc length; we then have

∂ ξ ∂A(g) = f (g(x)) g, g∗ N ds (13) ∂ξi ∂ξi µ

After collecting all the partial derivatives into an operator evolution of the group action. 3.2

∂φ ∂ξ ,

we can write the

Evolution

The algorithm for evolving the contour and the group action consists of a twostep process where an initial estimate of the contour µ ˆ = γ1 is provided, along with an initial estimate of the motion gˆ = e. The contour and motion are then updated in an alternating minimization where motion is updated according to ∂φ dˆ g = gˆ dt ∂ξ

(14)

Notice that this is valid not just for SE(2), but for any (ﬁnite-dimensional) matrix Lie group, although there may not be a closed-form solution for the exponential map like in the case of SE(3) and its subgroups. In practice, the group evolution (14) can be implemented in local (exponential) coordinates by evolv ∂φ ing ξ deﬁned by g = eξ via dξ dt = ∂ξ . In the level set framework, the derivative of the cost function φ with respect to the coordinates of the group action ξi can be computed as the collection of two terms, one for fin , one for fout where ∂g(x) ∂φ ∂ξi = g(γ1,2 ) ∂ξi , f{in,out} (g(x), γ1,2 )J(g∗ T ) ds. As we have anticipated in Eq. (11), the contour µ ˆ evolves according to dˆ µ = (fin (x, γ1 ) − fout (x, γ1 ) + fin (g(x), γ2 ) − fout (g(x), γ2 ))N dt

(15)

40

S. Soatto and A.J. Yezzi

As we have already pointed out, the derivation can be readily extended to surfaces in space and to multiple objects, as we show in Sect. 6. 3.3

Distance between Shapes

The deﬁnition of motion gˆ and shape average µ ˆ as a minimizer of (6) suggests deﬁning the distance5 between two shapes as the “energy” necessary to deform one into the other via the average shape: . ˆ j ). d(γi , γj ) = E(γi , T (ˆ g , h)γ

(16)

For instance, for the case of the set-symmetric diﬀerence of two contours, we have . (17) d∆ (γ1 , γ2 ) = χµˆ χγ1 + χgˆ(ˆµ) χγ2 dx and for the signed distance transform we have . dψ (γ1 , γ2 ) = ζ(γ1 )dx + ˆ µ ¯

ˆ ¯) g ˆ(µ

ζ(γ2 )dx.

(18)

In either case, a gradient ﬂow algorithm based on Eq. (14) and (15), when it converges to a global minimum, returns an average shape and a set of group elements gi which minimize the sums of the distances between the contours γi and any other common contour modulo the chosen group.

4

Moving Average and Tracking

The discussion above assumes that an unsorted collection of shapes is available, where the deformation between any two shapes is “small” (modulo G), so that the whole collection can be described by a single average shape. Consider however the situation where an object is evolving in time, for instance Fig. 5. While the deformation between adjacent time instants could be captured by a group action and a small deformation, as time goes by the object may change so drastically that talking about a global time average may not make sense. One way to approach this issue is by deﬁning a notion of “moving average”, similarly to what is done in time series analysis. In classical linear time series, however, the uncertainty is modeled via additive noise. In our case, the uncertainty is an inﬁnite-dimensional deformation h that acts on the measured contour. So the model becomes µ(t + 1) = g(t)µ(t) (19) γ(t) = h(µ(t)) 5

Here we use the term distance informally, since we do not require that it satisﬁes the triangular inequality. The term pseudo-distance would be more appropriate.

DEFORMOTION

41

where µ(t) represents the moving average of order k = 1. A similar model can be used to deﬁne moving averages of higher-order k > 1. The procedure described in Sect. 3, initialized with µ(0) = γ1 , provides an estimate of the moving average of order 1, as well as the tracking of the trajectory g(t) in the group G, which in (19) is represented as the model parameter. Note that the procedure in Sect. 3 simultaneously estimates the state µ(t) and identiﬁes the parameters g(t) of the model (19). It does so, however, without imposing restrictions on the evolution of g(t). If one wants to impose additional constraints on the motion parameters, one can augment the state of the model to include the parameters g, for instance g(t + 1) = eξ(t) g(t). This, however, is beyond the scope of this paper. In Fig. 5 we show the results of tracking a storm with a moving average of order one.

5

Simultaneous Approximation and Registration of Non-equivalent Shapes

So far we have assumed that the given shapes are obtained by moving and deforming a common underlying “template” (the average shape). Even though the given shapes are not equivalent (i.e. there is no group action g that maps one exactly onto the other), g is found as the one that minimizes the cost of the deviation from such an equivalence. In the algorithm proposed in Eq. (14)-(15), however, there is no explicit requirement that the deformation between the given shapes be small. Therefore, the procedure outlined can be seen as an algorithm to register shapes that are not equivalent under the group action. A registration is a group element gˆ that minimizes the cost functional (4). To illustrate this fact, consider the two considerably diﬀerent shapes shown in Fig. 7, γ1 , γ2 . The simultaneous estimation of their average µ, for instance relative to the aﬃne group, and of the aﬃne motions that best matches the shape average onto the original ones, g1 , g2 , provides a registration that maps γ1 onto γ2 and vice-versa: g = g2 g1−1 . If instead of considering the images in Fig. 7 as binary images that represent the contours, we consider them as gray-scale images, then the procedure outlined, for the case where the score is computed using the set-symmetric diﬀerence, provides a way to simultaneously jointly segment the two images and register them. This idea is illustrated in Fig. 9 for true gray-scale (magnetic resonance) images of brain sections.

6

Experiments

Fig. 3 illustrates the diﬀerence between the motion and shape average computed under the Euclidean group, and the aﬃne group. The three examples show the two given shapes γi , the mean shape registered to the original shapes, gi (µ) and the mean shape µ. Notice that aﬃne registration allows to simultaneously capture the square and the rectangle, whereas the Euclidean average cannot be registered to either one, and is therefore only an approximation.

42

S. Soatto and A.J. Yezzi

Fig. 3. Euclidean (top) vs. aﬃne (bottom) registration and average. For each pair of objects γ1 , γ2 , the registration g1 (µ), g2 (µ) relative to the Euclidean motion and aﬃne motion is shown, together with the Euclidean average and aﬃne average µ. Note that the aﬃne average can simultaneously “explain” a square and a rectangle, whereas the Euclidean average cannot.

Fig. 4 compares the eﬀect of choosing the signed distance score (left) and the set-symmetric diﬀerence (right) in the computation of the motion and average shape. The ﬁrst choice results in an average that captures the common features of the original shapes, whereas the second captures more of the features in each one. Depending on the application, one may prefer one or the other.

Fig. 4. Signed distance score (left) vs. set-symmetric diﬀerence (right). Original contours (γ1 on the top, γ2 on the bottom), registered shape gi (µ) and shape average µ. Note that the original objects are not connected, but are composed by a circle and a square. The choice of pseudo-distance between contours inﬂuences the resulting average. The signed distance captures more of the features that are common to the two shapes, whereas the symmetric diﬀerence captures the features of both.

DEFORMOTION

43

c Fig. 5. Storm (ﬁrst row) a collection of images from EUMETSAT 2001, (second row) aﬃne motion of the storm based on two adjacent time instances, superimposed to the original images, (bottom) moving average of order 1.

Fig. 6. Jellyﬁsh. Aﬃne registration (top), moving average and aﬃne motion (bottom) for the jellyﬁsh in Fig. 1. (Bottom right) trajectory of the jellyﬁsh (aﬃne component of the group).

Fig. 5 shows the results of tracking a storm. The aﬃne moving average is computed, and the resulting aﬃne motion is displayed. The same is done for the jellyﬁsh in Fig. 6. Fig. 7 and 8 are meant to challenge the assumptions underlying our method. The pairs of objects chosen, in fact, are not simply local deformations of one another. Therefore, the notion of shape average is not meaningful per se in this context, but serves to compute the change of (aﬃne) pose between the two shapes (Fig. 7). Nevertheless, it is interesting to observe how the shape average allows registering even apparently disparate shapes. Fig. 8 shows a representative example from an extensive set of experiments. In some cases, the shape average contains disconnected components, in some other it includes small parts that are shared by the original dataset, whereas in others it removes parts that are not consistent among the initial shapes (e.g. the tails). Notice that our framework is not meant to capture such a wide range of variations. In particular, it does

44

S. Soatto and A.J. Yezzi

Fig. 7. Registering non-equivalent shapes. Left to right: two binary images representing two diﬀerent shapes; aﬃne registration; corresponding aﬃne shape; approximation of the original shapes using the registration of the shape average based on the set-symmetric diﬀerence. Results for the signed distance score are shown in Fig. 8. This example is shown to highlight the limitations of our method.

Fig. 8. Biological shapes. For the signed distance score, we show the original shape with the aﬃne shape average registered and superimposed. It is interesting to notice that diﬀerent “parts” are captured in the average only if they are consistent in the two shapes being matched and, in some cases, the average shape is disconnected.

not possess a notion of “parts” and it is neither hierarchical nor compositional. In the context of non-equivalent shapes (shapes for which there is no group action mapping one exactly onto the other), the average shape serves purely as a support to deﬁne and compute motion in a collection of images of a given deforming shape. Fig. 9 shows the results of simultaneously segmenting and computing the average motion and registration for 4 images from a database of magnetic resonance images of the corpus callosum.

DEFORMOTION

45

Fig. 9. Corpus Callosum (top row) a collection of (MR) images from diﬀerent patients (courtesy of J. Dutta), further translated, rotated and distorted to emphasize their misalignment (second row). Aligned contour (second row, dark gray, superimposed to original regions) and shape average (bottom) corresponding to the aﬃne group.

Finally, Fig. 10 shows an application of the same technique to simultaneously register and average two 3D surfaces. In particular, two 3D models in diﬀerent

Fig. 10. 3D Averaging and registration (left) two images of 3D models in diﬀerent poses (center) registered average (right) aﬃne average. Note that the original 3D surfaces are not equivalent. The technique presented allows “stitching” and registering diﬀerent 3D models in a natural way.

46

S. Soatto and A.J. Yezzi

poses are shown. Our algorithm can be used to register the surfaces and average them, thus providing a natural framework to integrate surface and volume data. We wish to thank S. Belongie and B. Kimia for test data and suggestions.

References 1. L. Alvarez and J. M. Morel. Morphological approach to multiscale analysis: From principles to equations. In In Geometric–Driven Diﬀusion in Computer Vision, 1994. 2. L. Alvarez, J. Weickert, and J. Sanchez. A scale-space approach to nonlocal optical ﬂow calculations. In In ScaleSpace ’99, pages 235–246, 1999. 3. R. Azencott, F. Coldefy, and L. Younes. A distance for elastic matching in object recognition. Proc. 13th Intl. Conf. on Patt. Recog, 1:687–691, 1996. 4. S. Belongie, J. Malik, and J. Puzicha. Matching shapes. In Proc. of the IEEE Intl. Conf. on Computer Vision, 2001. 5. D. Bereziat, I. Herlin, and L. Younes. Motion detection in meteorological images sequences: Two methods and their comparison. In Proc. of the SPIE, 1997. 6. A. Blake and M. Isard. Active contours. Springer Verlag, 1998. 7. C. Bregler and S. Omohundro. Surface Learning with Applications to Lip Reading. In J. D. Cowan, G. Tesauro and J. Alspector (eds), Advances in Neural Information Processing Systems (6), San Francisco, CA: Morgan Kaufmann Publishers, 1994. 8. T. K. Carne. The geometry of shape spaces. Proc. of the London Math. Soc. (3) 61, 3(61):407–432, 1990. 9. T. F. Cootes, C. J. Taylor, D. H. Cooper, and J. Graham. Active Shape Models - Their Training and Application. Computer Vision and Image Understanding, 61(1), 1995, 38-59. 10. H. Chui and A. Rangarajan. A new algorithm for non-rigid point matching. In Proc. of the IEEE Intl. Conf. on Comp. Vis. and Patt. Recog., pages 44–51, 2000. 11. M. Fischler and R. Elschlager. The representation and matching of pictorial structures. IEEE Transactions on Computers, 22(1):67–92, 1973. 12. U. Grenander. General Pattern Theory. Oxford University Press, 1993. 13. U. Grenander and M. I. Miller. Representation of knowledge in complex systems. J. Roy. Statist. Soc. Ser. B, 56:549–603, 1994. 14. P.T. Jackway and R. Deriche. Scale-space properties of the multiscale morphological dilationerosion. IEEE Trans. PAMI, 18(1):38–51, 1996. 15. D. G. Kendall. Shape manifolds, procrustean metrics and complex projective spaces. Bull. London Math. Soc., 16, 1984. 16. B. Kimia, A. Tannenbaum, and S. Zucker. Shapes, shocks, and deformations i: the components of two-dimensional shape and the reaction-diﬀusion space. IJCV, 15:189–224, 1995. 17. R. Kimmel. Intrinsic scale space for images on surfaces: The geodesic curvature ﬂow. In Lecture Notes In Computer Science: First International Conference on Scale-Space Theory in Computer Vision, 1997. 18. R. Kimmel, N. Kiryati, and A. M. Bruckstein. Multivalued distance maps for motion planning on surfaces with moving obstacles. IEEE TAC, 14(3):427–435, 1998.

DEFORMOTION

47

19. M. Lades, C. Borbruggen, J. Buhmann, J. Lange, C. von der Malsburg, R. Wurtz, and W. Konen. Distortion invariant object recognition in the dynamic link architecture. IEEE Trans. on Computers, 42(3):300–311, 1993. 20. H. Le and D. G. Kendall. The Riemannian structure of Euclidean shape spaces: a novel environment for statistics. The Annals of Statistics, 21(3):1225–1271, 1993. 21. M. Leventon, E. Grimson, and O. Faugeras. Statistical shape inﬂuence in geodesic active contours, 2000. 22. R. Malladi, R. Kimmel, D. Adalsteinsson, V. Caselles G. Sapiro, and J. A. Sethian. A geometric approach to segmentation and analysis of 3d medical images. In Proc. Mathematical Methods in Biomedical Image Analysis Workshop, pages 21–22, 1996. 23. R. Malladi, J. A. Sethian, and B. C. Vemuri. Shape modeling with front propagation: A level set approach. IEEE PAMI, 17(2):158–175, 1995. 24. K. V. Mardia and I. L. Dryden. Shape distributions for landmark data. Adv. appl. prob., 21(4):742–755, 1989. 25. M. I. Miller and L. Younes. Group actions, diﬀeomorphisms and matching: a general framework. In Proc. of SCTV, 1999. 26. S. Osher and J. Sethian. Fronts propagating with curvature-dependent speed: algorithms based on Hamilton-Jacobi equations. J. of Comp. Physics, 79:12–49, 1988. 27. N. Paragios and R. Deriche. Geodesic active contours and level sets for the detection and tracking of moving objects. IEEE PAMI, 22(3):266–280, 2000. 28. C. Samson, L. Blanc-Feraud, G. Aubert, and J. Zerubia. A level set model for image classiﬁcation. In in International Conference on Scale-Space Theories in Computer Vision, pages 306–317, 1999. 29. B. ter Haar Romeny, L. Florack, J. Koenderink, and M. Viergever (Eds.). Scalespace theory in computer vision. In LLNCS, Vol 1252. Springer Verlag, 1997. 30. P. Thompson and A. W. Toga. A surface-based technique for warping threedimensional images of the brain. IEEE Trans. Med. Imaging, 15(4):402–417, 1996. 31. K. Toyama and A. Blake. Probabilistic tracking in a Metric Space. In Proc. of the Eur. Conf. on Comp. Vision, 2001. 32. R. C. Veltkamp and M. Hagedoorn. State of the art in shape matching. Technical Report UU-CS-1999-27, University of Utrecht, 1999. 33. L. Younes. Computable elastic distances between shapes. SIAM J. of Appl. Math., 1998. 34. A. Yuille. Deformable templates for face recognition. J. of Cognitive Neurosci., 3(1):59–70, 1991. 35. S. Zhu, T. Lee, and A. Yuille. Region competition: Unifying snakes, region growing, energy /bayes/mdl for multi-band image segmentation. In ICCV, pages 416–423, 1995.

Region Matching with Missing Parts Alessandro Duci1,4 , Anthony J. Yezzi2 , Sanjoy Mitter3 , and Stefano Soatto4 1

Scuola Normale Superiore, Pisa – Italy 56100 [email protected] 2 Georgia Institute of Technology, Atlanta – GA 30332 [email protected] 3 Massachusetts Institute of Technology, Cambridge – MA 02139 [email protected] 4 University of California at Los Angeles, Los Angeles – CA 90095 [email protected]

Abstract. We present a variational approach to the problem of registering planar shapes despite missing parts. Registration is achieved through the evolution of a partial diﬀerential equation that simultaneously estimates the shape of the missing region, the underlying “complete shape” and the collection of group elements (Euclidean or aﬃne) corresponding to the registration. Our technique applies both to shapes, for instance represented as characteristic functions (binary images), and to grayscale images, where all intensity levels evolve simultaneously in a partial diﬀerential equation. It can therefore be used to perform “region inpainting” and to register collections of images despite occlusions. The novelty of the approach lies on the fact that, rather than estimating the missing region in each image independently, we pose the problem as a joint registration with respect to an underlying “complete shape” from which the complete version of the original data is obtained via a group action. Keywords: shape, variational, registration, missing part, inpainting

1

Introduction

Consider diﬀerent images of the same scene, taken for instance from a moving camera, where one or more of the images have been corrupted, so that an entire part is missing. This problem arises, for instance, in image registration with missing data, in the presence of occlusions, in shape recognition when one or more parts of an object may be absent in each view, and in “movie inpainting” where one or more frames are damaged and one wants to “transfer” adjacent frames to ﬁll in the damaged part. We consider a simpliﬁed version of the problem, where we have a compact region in each image i, bounded by a closed planar contour, γi , and a region of the

This research is sponsored in part by ARO grant DAAD19-99-1-0139 and Intel grant 8029.

A. Heyden et al. (Eds.): ECCV 2002, LNCS 2352, pp. 48–62, 2002. c Springer-Verlag Berlin Heidelberg 2002

Region Matching with Missing Parts

49

image, with support described by a characteristic function χi , is damaged. We do not know a-priori what the region χi is, and we do not know the transformation mapping one image onto the other. However, we make the assumption that such a transformation can be well approximated by a ﬁnite-dimensional group gi , for instance the aﬃne or the projective group. In addition, we do not know the value of the image in the missing region. Therefore, given a sequence of images, one has to simultaneously infer the missing regions χi as well as the transformations gi and the occluded portions of each contour γi . We propose an algorithm that stems from a simple generative model, where an unknown contour µ0 , the “complete shape,” is ﬁrst transformed by a group action gi , and then occluded by a region χi (Figure 1). Therefore, one only estimates the complete shape and the group actions: gˆ1 , . . . , gˆk , µ ˆ0 , χ ˆ1 , . . . , χ ˆk = arg min

gi ,µ0

k

φ(χi (γi ), χi ◦ gi (µ0 ))

(1)

i=1

for a given discrepancy measure φ. A simpler case is when the occlusion occurs at the same location in all shapes; in this case, there is only one indicator function χ0 that acts on the complete shape µ0 .

Fig. 1. A contour undergoes a global motion and local occlusions.

50

2

A. Duci et al.

Background and Prior Work

The analysis of “Shape Spaces” was pioneered in Statistics by Kendall, Mardia and Carne among others [12,17,6,20]. Shapes are deﬁned as the equivalence classes of points modulo the similarity group, RM N /SE(M ) × R. These tools have proven useful in contexts where N distinct “landmarks” are available, for instance in comparing biological shapes with N distinct “parts.” Various extensions to missing points have been proposed, mostly using expectationmaximization (EM), alternating between computing the suﬃcient statistics of the missing data and performing shape analysis in the standard framework of shape spaces. However, since this framework is essentially tied to representing shapes as collections of points, they do not extend to the case of continuous curves and surfaces in a straightforward way, and we will therefore not pursue them further here. In computational vision, a wide literature exists for the problem of “matching” or “aligning” discrete representations of collections of points, for instance organized in graphs or trees [16,8]. A survey of shape matching algorithms is presented in [29]. Just to highlight some representative algorithms, Belongie et al. [2] propose comparing planar contours based on the “shape context” of each point along the contour. This work is positioned somewhere in between landmark or feature-based approaches and image-based ones, similarly to [7]. Kang et al. [11] have recently approached the multiple image inpainting problem using a shape context representation. Deformable templates, pioneered by Grenander [9], do not rely on a pointwise representation; rather, images are deformed under the action of a group (possibly inﬁnite-dimensional) and compared for the best match in an imagebased approach [32,3]. Grenander’s work sparked a current that has been particularly successful in the analysis of medical images, for instance [10]. In this work we would like to retain some of the features of deformable templates, but extend them to modeling missing parts. A somewhat diﬀerent line of work is based on variational methods and the solution of partial diﬀerential equations (PDEs) to deform planar contours and quantify their “distance.” Not only can the notion of alignment or distance be made precise [1,31,21,14,26], but quite sophisticated theories of shape, that encompass perceptually relevant aspects, can be formalized in terms of the properties of the evolution of PDEs (e.g. [15,13]). The variational framework has also been proven very eﬀective in the analysis of medical images [19,28,18]. Zhu et al. [33] have also extended some of these ideas to a probabilistic context. Other techniques rely on matching diﬀerent representations, for instance skeletons [13], that are somewhat robust to missing parts. In [27] a similar approach is derived using a generic representation of 2-D shape in the form of structural descriptions from the shocks of a curve evolution process, acting on bounding contours. The possibility of making multiple registration by ﬁnding a mean shape and a rigid transformation was studied by Pennec [24] in the case of 3D landmarks. Leung, Burl and Perona [5] described an algorithm for locating quasi-frontal

Region Matching with Missing Parts

51

views of human faces in cluttered scenes that can handle partial occlusions. The algorithm is based on coupling a set of local feature detectors with a statistical model of the mutual distances between facial features. In this work, we intend to extend these techniques to situations where parts of the image cannot be used for matching (see [30]) and at the same time landmark approaches fail. In the paper of Berger and Gerig [4], a deformable area-based template matching is applied to low contrast medical images. In particular, they use a least squares template matching (LSM) with an automatic quality control of the resulting match. Nastar, Moghaddam and Pentland [22] proposed to use a statistical learning method for image matching and interpolation of missing data. Their approach is based on the idea of modeling the image like a deformable intensity surface represented by a 3D mesh and use principal component analysis to provide a priori knowledge about object-speciﬁc deformations. Rangarajan, Chui and Mjolsness [25] deﬁned a novel distance measure for non-rigid image matching where probabilistic generative models are constructed for the nonrigid matching of point-sets. 2.1

Contributions of This Paper

This work presents a framework and an algorithm to match regions despite missing parts. To the best of our knowledge, work in this area, using region-based variational methods, is novel. Our framework relies on the notion of “complete shape” which is inferred simultaneously with the group actions that map the complete shape onto the incomplete ones. The complete shape and the registration parameters are deﬁned as the ones that minimize a cost functional, and are computed using an alternating minimization approach where a partial diﬀerential equation is integrated using level set methods [23].

3

Matching with Missing Parts

The formulation of the problem and the derivation of the evolution equations are introduced in this section for the case of a planar shape under isometric transformations (rigid motions). In this case, the dimension of the space is 2 and the determinant of the Jacobian of the group is J(g) = 1 for all the elements g of the group G. The main assumption about the shape is that it must be a regular domain, that is an open and bounded subset of R2 with a ﬁnite number of connected components and a piece-wise C ∞ boundary. This regularity is required to avoid singular pathologies and to make the computation possible. The main notation that will be used in this section is listed below. Notation – – – – –

¯ are regular domains in R2 γ¯i , µ γi , µ are the boundaries of γ¯i , µ ¯ χ(γ) is the characteristic function of the set γ¯ A(γ) is the area (volume) of the region γ¯ ·, · is the usual inner product

52

3.1

A. Duci et al.

Formulation of the Problem

Let γ¯1 , . . . , γ¯k be regular domains of R2 , all obtained from the same regular domain µ ¯ ⊂ R2 by composition with characteristic functions χ1 , . . . , χk : R2 → R and actions of Lie group elements g1 , . . . , gk ∈ G. We want to ﬁnd the best solution in the sense expressed by the functional φ=

k

A(¯ γi \ gi (¯ µ)) + α A(¯ µ)

(2)

i=1

where A denotes the area, α is a design constant, µ ¯, χi and gi are the unknowns and the sets γ¯i and the structure of G are given. The rationale behind the choice of the cost function φ is that one wants to maximize the overlap between the incomplete shapes and the registered complete shape (ﬁrst term) while keeping the complete shape as small as possible (second term). This is equivalent to minimizing the area of the γ¯i that is not covered by the image of the complete shape after the application of the group action gi . At the same time, one needs to minimize a quantity related to the complete shape (e.g. the area) to constrain the solution to be non-singular. Without the second term, it is always possible to choose a compact complete shape that covers all the incomplete ones (e.g. a big square) and minimizes the ﬁrst term. 3.2

Minimization with Respect to Shape

The functional φ can be written in integral form φ=

k i=1

γ ¯i

(1 − gi µ ¯)dx + α

dx

(3)

χ(µ)dx

(4)

µ ¯

and using the characteristic function notation φ=

k

χ(γi )(1 − χ(gi µ))dx + α

i=1

=

k

χ(γi )dx −

i=1

k

χ(γi )χ(gi µ))dx + α

χ(µ)dx.

(5)

i=1

Since the ﬁrst term of φ is independent of µ, g and remembering that gi are isometries, the problem of minimizing φ is equivalent to that of ﬁnding the minimum of the energy E(gi , µ) =

µ ¯

α−

k i=1

χ(gi−1 γi ) dx.

(6)

Region Matching with Missing Parts

53

One can show that the ﬁrst variation of this integral along the normal direction of the contour is simply its integrand, by using the divergence theorem, and therefore conclude that a gradient ﬂow that minimizes the energy E with respect to the shape of the contour µ is given by k ∂µ −1 χ(gi γi ) N (7) = α− ∂t i=1 where N is the normal vector ﬁeld of the contour µ.

3.3

Minimization with Respect to the Group Action

In order to compute the variation of the functional φ with respect to the group actions gi , we ﬁrst notice that there is only one term that depends on gi in equation (6). Therefore, we are left with having to compute the variation of (8) − χ(gi−1 γi )dx. µ ¯

In order to simplify the notation, we note that the term above is of the generic form . W (g) = f (g(x))dx (9) µ ¯

with f = χ(γi ). Therefore, we consider the variation of W with respect to the components of the exponential coordinates1 ξi of the group gi = eξi ∂W ∂ = f (g(x))dx (10) ∂ξi ∂ξi µ¯ ∂ ∇f (g(x)) g(x)dx. (11) = ∂ξi µ ¯ Using Green’s theorem it is possible to write the variation as an integral along the contour of µ and one over g(¯ µ) with a divergence integrand

∂W ∂ ∂ ∗ −1 (g) = f (g(x)) g(x), g N d s − f (y)∇y · g(g (y)) d y. ∂ξi ∂ξi ∂ξi µ g(¯ µ) (12) Therefore, the derivative with respect of the group action is ∂ ∂φ = gi (x), gi∗ Ni d s. ∂ξi ∂ξi µ ¯ ∩gi−1 (γi ) 1

(13)

Every ﬁnite-dimensional Lie group admits exponential coordinates. For the simple case of the isometries of the plane, the exponential coordinates can be computed in closed-form using Rodrigues’ formula.

54

A. Duci et al.

Where N is the normal vector ﬁeld to the boundary of gi−1 (γi ) and gi∗ is the push forward induced by the map gi . Detailed calculations are reported in Appendix A. 3.4

Evolution Equations

Within the level set framework, a function ψ is evolved instead of the contour µ. The function ψ is negative inside µ, positive outside and zero on the contour. The evolution of ψ depends on the velocity of µ via the Hamilton-Jacobi equation

k ut + α − i=1 χ(gi−1 γi ) |∇u| = 0, (14) u(0, x) = ψ (t) (x). The evolution equations follow ψ (t+1) (x) = u(1, x |ψ (t) ) µ ¯(t) = {x : ψ(x) < 0} (t+1) (t) = ξi − βξ ξi

(15)

µ ¯ ∩gi−1 (γi )

∂ ∗ gi (x), gi Ni d s ∂ξi

(16) (17)

where βξ is a step parameter and u(·, · |ψ (t) ) is the solution of (14) with initial condition u(0, x) = ψ (t) (x). 3.5

Generalization to Graylevel Images

There are many possible generalizations of the above formulas. Here we present the case of gray level images. The case of color images is very similar so the equations will not be stated. The main idea is to work with a generic level set between the minimum and the maximum intensity levels and write the functional to match all the level sets simultaneously using the partial diﬀerential equation in Eq. (25). To derive this equation, let k images of the same object Ii , i = 1, . . . , k be given and γ¯iδ = {x : Ii < δ}

(18)

be the generic δ-underlevel of Ii , where δ is a positive constant. Then let µ : R2 → R

(19)

be a function that represents the complete image intensity (the choice of the name µ is not casual, since this will turn out to be the complete shape for the case of greylevel images) and µ ¯δ = {x : µ < δ} µδ = ∂ µ ¯δ

(20)

Region Matching with Missing Parts

55

respectively its underlevel and the boundary of the underlevel. From the same considerations developed for the case of a contour, we write the functional φ as φ=

n k

A(¯ γiδ \ gi (¯ µj )) + α

i=1 j=1

n

A(µj )

(21)

j=1

where j is a discretization of the level set δ φ=

k n

χ(γiδ )(1

j

− χ(gi µ ))dx + α

i=1 j=1

n

χ(µj )dx

(22)

j=1

and in the same way the derivatives of the functional can be obtained as k ∂ µj −1 δ = α− χ(gi γi ) N j , ∂t i=1 n ∂ ∂φ ∗ δ =− gi (x), gi Ni d s. ∂ξi ∂ξi ¯ j ∩gi−1 (γiδ ) j=1 µ

(23) (24)

After letting j go to the limit, Eq. (23) gives the Hamilton-Jacobi equation for the function µ: k H (Ii (gi (x)) − µ(t, x)) − α |∇µ(t, x)| = 0 (25) µt (t, x) + i=1

where

1 if s > 0, H(s) = 0 if s ≤ 0.

(26)

Therefore, the evolution equation for both the function µ and the parameters gi is given by  (t) k (t) (t)  µ(t+1) (x) = µ(t) (x) − βµ i=1 H Ii (gi (x)) − µ (x) − α |∇µ (x)| (t) ∂ gi (t+1) (t) (t) (t)  = ξi − βξ H Ii (gi ) − µ(t) dx ξi (t) , (∇γi )(gi (x)) ∂ξi

(27) where βµ and βξ are step parameters.

4

Experiments

A numerical implementation of the evolution equations (15),(16),(17) has been written within the level set framework proposed by Osher and Sethian [23] using

56

A. Duci et al.

Fig. 2. Hands. (Top) a collection of images of the same hand in diﬀerent poses with different missing parts. The support of the missing parts is unknown. (Middle) similarity group, visualized as a “registered” image. (Bottom) estimated template corresponding to the similarity group (“complete shape”).

Fig. 3. Hands Evolution. (Top) evolution of the complete shape for t = 0, . . . , 20. (Bottom) evolution of g2 (µ) for t = 0, . . . , 20.

Region Matching with Missing Parts

57

an ultra narrow band algorithm and an alternating minimization scheme. A set of common shapes (hands, leaves, mice, letters) has been chosen and converted into binary images of 256 × 256 pixels. For each image of this set, a group of binary images with missing parts and with diﬀerent poses has been generated (Figures 2, 4 curves γ1 , . . . , γ5 ). The following level set evolution equation has been used µt +

α−

k i=1

χ(gi−1 γi )

|∇µ| = 0

(28)

with a ﬁrst-order central scheme approximation. The evolution of the pose parameters has been carried out using the integral (17) with the following approximation of the arclength ds ds ≈ |∇(µ)|dx.

(29)

Fig. 4. Letter “A.” (Top) a collection of images of the letter “A” in diﬀerent poses with diﬀerent missing parts. The support of the missing parts is unknown. (Middle) similarity group (“registration”). (Bottom) estimated template corresponding to the similarity group (“complete shape”).

58

A. Duci et al.

Fig. 5. Letter “A” Evolution. (Top) evolution of the complete shape for t = 0, . . . , 20. (Bottom) evolution of g3 (µ) for t = 0, . . . , 20.

The evolution has been initialized with the following settings µt=0 = γ1 Ti = Bγi − Bγ1

cos θi − sin θi Ri = , sin θi cos θi

(30) with θi = Eγ i OEγ1

where Bγi is the baricenter of γi and Eγi is the principal axis of inertia of the region γ¯i . The value of α has been set between 0 and 1. Ri , Ti are the rotational and translational components of gi = (Ri , Ti ) ∈ SE(2). In Figures 2, 4 some results have been illustrated. γj are the starting curves and µ is the complete shape in an absolute system after the computation. The ﬁgures show the computed gj (µ), the estimated rigid motions. Figure 6 shows the method applied to grayscale images of a face, where diﬀerent portions have been purposefully erased. Figure 8 shows the results of matching a collection of images of the corpus callosum of a patient. One way to further improve this technique is to use richer ﬁnite-dimensional groups G that can account for more than simple rotations and translations. Simple examples include the aﬃne and projective groups, for which the derivation is essentially the same, except for a few changes of measure due to the fact that they are not isometric. The experiments show that this method works very well even when the missing part in each image is pretty signiﬁcant, up to about 20% of the area.

Region Matching with Missing Parts

59

Fig. 6. Faces (Top) a collection of images of the same face in diﬀerent poses with different missing parts. The support of the missing parts is unknown. (Middle) similarity group, visualized as a “registered” image. (Bottom) estimated template corresponding to the similarity group (“complete image”).

Fig. 7. Face evolution. (Top) evolution of the complete image for t = 0, . . . , 189. (Bottom) evolution of g5 (µ) for t = 0, . . . , 189.

60

A. Duci et al.

Fig. 8. Corpus Callosum. (Top) a collection of images of the same corpus callosum in diﬀerent poses with diﬀerent missing parts. The support of the missing parts is unknown. (Middle) similarity group, visualized as a “registered” image. (Bottom) estimated template corresponding to the similarity group (“complete image”).

Fig. 9. Corpus Callosum evolution. (Top) evolution of the complete image for t = 0, . . . , 199. (Bottom) evolution of g2 (µ) for t = 0, . . . , 199.

Region Matching with Missing Parts

61

References 1. R. Azencott, F. Coldefy, and L. Younes. A distance for elastic matching in object recognition. Proc. 13th Intl. Conf. on Patt. Recog, 1:687–691, 1996. 2. S. Belongie, J. Malik, and J. Puzicha. Matching shapes. In Proc. of the IEEE Intl. Conf. on Computer Vision, 2001. 3. D. Bereziat, I. Herlin, and L. Younes. Motion detection in meteorological images sequences: Two methods and their comparison. In Proc. of the SPIE, 1997. 4. M. Berger and G. Gerig. Deformable area-based template matching with application to low contrast imagery, 1998. 5. M. Burl, T. Leung, and P. Perona. Face localization via shape statistics. In Proc. Intl. Workshop on automatic face and gesture recognition, pages 154–159, Zurich, June 1995. IEEE Computer Soc. 6. T. K. Carne. The geometry of shape spaces. Proc. of the London Math. Soc. (3) 61, 3(61):407–432, 1990. 7. H. Chui and A. Rangarajan. A new algorithm for non-rigid point matching. In Proc. of the IEEE Intl. Conf. on Comp. Vis. and Patt. Recog., pages 44–51, 2000. 8. M. Fischler and R. Elschlager. The representation and matching of pictorial structures. IEEE Transactions on Computers, 22(1):67–92, 1973. 9. U. Grenander. General Pattern Theory. Oxford University Press, 1993. 10. U. Grenander and M. I. Miller. Representation of knowledge in complex systems. J. Roy. Statist. Soc. Ser. B, 56:549–603, 1994. 11. S. H. Kang, T. F. Chan, and S. Soatto. Multiple image inpainting. In Proc. of the 3DPVT, Padova, IT, June 2002. 12. D. G. Kendall. Shape manifolds, procrustean metrics and complex projective spaces. Bull. London Math. Soc., 16, 1984. 13. B. Kimia, A. Tannebaum, and S. Zucker. Shapes, shocks, and deformations i: the components of two-dimensional shape and the reaction-diﬀusion space. Int’l J. Computer Vision, 15:189–224, 1995. 14. R. Kimmel and A. Bruckstein. Tracking level sets by level sets: a method for solving the shape from shading problem. Computer Vision, Graphics and Image Understanding, (62)1:47–58, 1995. 15. R. Kimmel, N. Kiryati, and A. M. Bruckstein. Multivalued distance maps for motion planning on surfaces with moving obstacles. IEEE Trans. Robot. & Autom., 14(3):427–435, 1998. 16. M. Lades, C. Borbruggen, J. Buhmann, J. Lange, C. von der Malsburg, R. Wurtz, and W. Konen. Distortion invariatn object rcognition in the dynamic link architecture. IEEE Trans. on Computers, 42(3):300–311, 1993. 17. H. Le and D. G. Kendall. The riemannian structure of euclidean shape spaces: a novel environment for statistics. The Annals of Statistics, 21(3):1225–1271, 1993. 18. R. Malladi, R. Kimmel, D. Adalsteinsson, V. Caselles G. Sapiro, and J. A. Sethian. A geometric approach to segmentation and analysis of 3d medical images. In Proc. Mathematical Methods in Biomedical Image Analysis Workshop, pages 21–22, 1996. 19. R. Malladi, J. A. Sethian, and B. C. Vemuri. Shape modeling with front propagation: A level set approach. IEEE Trans. on Pattern Analysis and Machine Intelligence, 17(2):158–175, 1995. 20. K. V. Mardia and I. L. Dryden. Shape distributions for landmark data. Adv. appl. prob., 21(4):742–755, 1989. 21. M. I. Miller and L. Younes. Group action, diﬀeomorphism and matching: a general framework. In Proc. of SCTV, 1999.

62

A. Duci et al.

22. C. Nastar, B. Moghaddam, and A. Pentland. Generalized image matching: Statistical learning of physically-based deformations. In Proceedings of the Fourth European Conference on Computer Vision (ECCV’96), Cambridge, UK, April 1996. 23. S. Osher and J. Sethian. Fronts propagating with curvature-dependent speed: algorithms based on hamilton-jacobi equations. J. of Comp. Physics, 79:12–49, 1988. 24. X. Pennec. Multiple Registration and Mean Rigid Shapes - Application to the 3D case. In I.L. Dryden K.V. Mardia, C.A. Gill, editor, Image Fusion and Shape Variability Techniques (16th Leeds Annual Statistical (LASR) Workshop), pages 178–185, july 1996. 25. Anand Rangarajan, Haili Chui, and Eric Mjolsness. A new distance measure for non-rigid image matching. In Energy Minimization Methods in Computer Vision and Pattern Recognition, pages 237–252, 1999. 26. C. Samson, L. Blanc-Feraud, G. Aubert, and J. Zerubia. A level set model for image classiﬁcation. In in International Conference on Scale-Space Theories in Computer Vision, pages 306–317, 1999. 27. K. Siddiqi, A. Shokoufandeh, S. Dickinson, and S. Zucker. Shock graphs and shape matching, 1998. 28. P. Thompson and A. W. Toga. A surface-based technique for warping threedimensional images of the brain. IEEE Trans. Med. Imaging, 15(4):402–417, 1996. 29. R. C. Veltkamp and M. Hagedoorn. State of the art in shape matching. Technical Report UU-CS-1999-27, University of Utrecht, 1999. 30. A. Yezzi and S. Soatto. Stereoscopic segmentation. In Proc. of the Intl. Conf. on Computer Vision, pages 59–66, 2001. 31. L. Younes. Computable elastic distances between shapes. SIAM J. of Appl. Math., 1998. 32. A. Yuille. Deformable templates for face recognition. J. of Cognitive Neurosci., 3(1):59–70, 1991. 33. S. Zhu, T. Lee, and A. Yuille. Region competition: Unifying snakes, region growing, energy /bayes/mdl for multi-band image segmentation. In Int. Conf. on Computer Vision, pages 416–423, 1995.

A

Group Action Derivation Details ∂ ∂ g(x)d x = ∇f (y) g(g −1 (y))d y ∂ξi ∂ξi µ ¯ g(¯ µ)

∂ ∂ f (y) g(g −1 (y)), N d s − f (y)∇y · g(g −1 (y)) d y = ∂ξi ∂ξi ∂g(¯ µ) g(¯ µ)

∂ ∂ ∗ −1 f (g(x)) g(x), g N d s + ∇y · g(g (y)) d y = ∂ξi ∂ξi µ g(¯ µ)∩¯ γi ∂ ∂ ∗ ∗ g(x), g N d s + g(x), g N d s =− ∂ξi ∂ξi µ∩g −1 (¯ γi ) ∂(¯ µ∩g −1 (¯ γi )) ∂ = g(x), g ∗ N d s ∂ξi µ ¯ ∩g −1 (γi )

∂W = ∂ξi

∇f (g(x))

What Energy Functions Can Be Minimized via Graph Cuts? Vladimir Kolmogorov and Ramin Zabih Computer Science Department, Cornell University, Ithaca, NY 14853 [email protected], [email protected]

Abstract. In the last few years, several new algorithms based on graph cuts have been developed to solve energy minimization problems in computer vision. Each of these techniques constructs a graph such that the minimum cut on the graph also minimizes the energy. Yet because these graph constructions are complex and highly speciﬁc to a particular energy function, graph cuts have seen limited application to date. In this paper we characterize the energy functions that can be minimized by graph cuts. Our results are restricted to energy functions with binary variables. However, our work generalizes many previous constructions, and is easily applicable to vision problems that involve large numbers of labels, such as stereo, motion, image restoration and scene reconstruction. We present three main results: a necessary condition for any energy function that can be minimized by graph cuts; a suﬃcient condition for energy functions that can be written as a sum of functions of up to three variables at a time; and a general-purpose construction to minimize such an energy function. Researchers who are considering the use of graph cuts to optimize a particular energy function can use our results to determine if this is possible, and then follow our construction to create the appropriate graph.

1

Introduction and Summary of Results

Many of the problems that arise in early vision can be naturally expressed in terms of energy minimization. The computational task of minimizing the energy is usually quite diﬃcult, as it generally requires minimizing a non-convex function in a space with thousands of dimensions. If the functions have a restricted form they can be solved eﬃciently using dynamic programming [2]. However, researchers typically have needed to rely on general purpose optimization techniques such as simulated annealing [3,10], which is extremely slow in practice. In the last few years, however, a new approach has been developed based on graph cuts. The basic technique is to construct a specialized graph for the energy function to be minimized, such that the minimum cut on the graph also minimizes the energy (either globally or locally). The minimum cut in turn can be computed very eﬃciently by max ﬂow algorithms. These methods have been successfully used for a wide variety of vision problems including image restoration [7,8,12,14], stereo and motion [4,7,8,13,16,20,21], voxel occupancy [23], multicamera scene reconstruction [18] and medical imaging [5,6,15]. The output of A. Heyden et al. (Eds.): ECCV 2002, LNCS 2352, pp. 65–81, 2002. c Springer-Verlag Berlin Heidelberg 2002

66

V. Kolmogorov and R. Zabih

these algorithms is generally a solution with some interesting theoretical quality guarantee. In some cases [7,12,13,14,20] it is the global minimum, in other cases a local minimum in a strong sense [8] that is within a known factor of the global minimum. The experimental results produced by these algorithms are also quite good, as documented in two recent evaluations of stereo algorithms using real imagery with dense ground truth [22,24]. Minimizing an energy function via graph cuts, however, remains a technically diﬃcult problem. Each paper constructs its own graph speciﬁcally for its individual energy function, and in some of these cases (especially [8,16,18]) the construction is fairly complex. The goal of this paper is to precisely characterize the class of energy functions that can be minimized via graph cuts, and to give a general-purpose graph construction that minimizes any energy function in this class. Our results play a key role in [18], provide a signiﬁcant generalization of the energy minimization methods used in [4,5,6,8,12,15,23], and show how to minimize an interesting new class of energy functions. In this paper we only consider energy functions involving binary-valued variables. At ﬁrst glance this restriction seems severe, since most work with graph cuts considers energy functions that involve variables with more than two possible values. For example, the algorithms presented in [8] for stereo, motion and image restoration use graph cuts to address the standard pixel labeling problem that arises in early vision. In a pixel labeling problem the variables represent individual pixels, and the possible values for an individual variable represent, e.g., its possible displacements or intensities. However, many of the graph cut methods that handle multiple possible values actually consider a pair of labels at a time. Even though we only address binary-valued variables, our results therefore generalize the algorithms given in [4,5,6,8,12,15,23]. As an example, we will show in section 4.1 how to use our results to solve the pixel-labeling problem, even though the pixels have many possible labels. An additional argument in favor of binary-valued variables is that any cut eﬀectively assigns one of two possible values to each node of the graph. So in a certain sense any energy minimization construction based on graph cuts relies on intermediate binary variables. 1.1

Summary of Our Results

In this paper we consider two classes of energy functions. Let {x1 , . . . , xn }, xi ∈ {0, 1} be a set of binary-valued variables. We deﬁne the class F 2 to be functions that can be written as a sum of functions of up to 2 variables at a time, E i (xi ) + E i,j (xi , xj ). (1) E(x1 , . . . , xn ) = i

i<j

We deﬁne the class F 3 to be functions that can be written as a sum of functions of up to 3 variables at a time, E(x1 , . . . , xn ) = E i (xi ) + E i,j (xi , xj ) + E i,j,k (xi , xj , xk ). (2) i

i<j

i<j
What Energy Functions Can Be Minimized via Graph Cuts?

67

Obviously, the class F 2 is a strict subset of the class F 3 . The main result in this paper is a precise characterization of the functions in F 3 that can be minimized using graph cuts, together with a graph construction for minimizing such functions. Moreover, we give a necessary condition for all other classes which must be met for a function to be minimized via graph cuts. Our results also identify an interesting class of class of energy functions that have not yet been minimized using graph cuts. All of the previous work with graph cuts involves a neighborhood system that is deﬁned on pairs of pixels. In the language of Markov Random Fields [10,19], these methods consider ﬁrstorder MRF’s. The associated energy functions lie in F 2 . Our results allow for the minimization of energy functions in the larger class F 3 , and thus for neighborhood systems involve triples of pixels. 1.2

Organization

The rest of the paper is organized as follows. In section 2 we give an overview of graph cuts. In section 3 we formalize the problem that we want to solve. Section 4 contains our main theorem for the class of functions F 2 and shows how it can be used. Section 5 contains our main theorems for other classes. Proofs of our theorems, together with the graph constructions, are deferred to section 6, with some additional details deferred to a technical report [17]. A summary of the actual graph constructions os given in the appendix.

2

Overview of Graph Cuts

Suppose G = (V, E) is a directed graph with two special vertices (terminals), namely the source s and the sink t. An s-t-cut (or just a cut as we will refer to it later) C = S, T is a partition of vertices in V into two disjoint sets S and T , such that s ∈ S and t ∈ T . The cost of the cut is the cut is the sum of costs of all edges that go from S to T : c(u, v). c(S, T ) = u∈S,v∈T,(u,v)∈E

The minimum s-t-cut problem is to ﬁnd a cut C with the smallest cost. Due to the theorem of Ford and Fulkerson [9] this is equivalent to computing the maximum ﬂow from the source to sink. There are many algorithms which solve this problem in polynomial time with small constants [1,11]. It is convenient to denote a cut C = S, T by a labeling f mapping from the set of the nodes V − {s, t} to {0, 1} where f (v) = 0 means that v ∈ S, and f (v) = 1 means that v ∈ T . We will use this notation later.

3

Deﬁning Graph Representability

Let us consider a graph G = (V, E) with terminals s and t, thus V = {v1 , . . . , vn , s, t}. Each cut on G has some cost; therefore, G represents the energy

68

V. Kolmogorov and R. Zabih

function mapping from all cuts on G to the set of nonnegative real numbers. Any cut can be described by n binary variables x1 , . . . , xn corresponding to nodes in G (excluding the source and the sink): xi = 0 when vi ∈ S, and xi = 1 when vi ∈ T . Therefore, the energy E that G represents can be viewed as a function of n binary variables: E(x1 , . . . , xn ) is equal to the cost of the cut deﬁned by the conﬁguration x1 , . . . , xn (xi ∈ {0, 1}). Note that the conﬁguration that minimizes E will not change if we add a constant to E. We can eﬃciently minimize E by computing the minimum s-t-cut on G. This naturally leads to the question: what is the class of energy functions E for which we can construct a graph that represents E? We can also generalize our construction. Above we used each node (except the source and the sink) for encoding one binary variable. Instead we can specify a subset V0 = {v1 , . . . , vk } ⊂ V − {s, t} and introduce variables only for the nodes in this set. Then there may be several cuts corresponding to a conﬁguration x1 , . . . , xk . If we deﬁne the energy E(x1 , . . . , xk ) as the minimum among the costs of all such cuts, then the minimum s-t-cut on G will again yield the conﬁguration which minimizes E. We will summarize graph constructions that we allow in the following deﬁnition. Deﬁnition 1. A function E of n binary variables is called graph-representable if there exists a graph G = (V, E) with terminals s and t and a subset of nodes V0 = {v1 , . . . , vn } ⊂ V −{s, t} such that for any conﬁguration x1 , . . . , xn the value of the energy E(x1 , . . . , xn ) is equal to a constant plus the cost of the minimum s-t-cut among all cuts C = S, T in which vi ∈ S, if xi = 0, and vi ∈ T , if xi = 1 (1 ≤ i ≤ n). We say that E is exactly represented by G, V0 if this constant is zero. The following lemma is an obvious consequence of this deﬁnition. Lemma 2. Suppose the energy function E is graph-representable by a graph G and a subset V0 . Then it is possible to ﬁnd the exact minimum of E in polynomial time by computing the minimum s-t-cut on G. In this paper we will give a complete characterization of the classes F 2 and F in terms of graph representability, and show how to construct graphs for minimizing graph-representable energies within these classes. Moreover, we will give a necessary condition for all other classes which must be met for a function to be graph-representable. Note that it would be suﬃce to consider only the class F 3 since F 2 ⊂ F 3 . However, the condition for F 2 is simpler so we will consider it separately. 3

4

The Class F 2

Our main result for the class F 2 is the following theorem.

What Energy Functions Can Be Minimized via Graph Cuts?

69

Theorem 3. Let E be a function of n binary variables from the class F 2 , i.e. it can be written as the sum E(x1 , . . . , xn ) = E i (xi ) + E i,j (xi , xj ). i

i<j

Then E is graph-representable if and only if each term E i,j satisﬁes the inequality E i,j (0, 0) + E i,j (1, 1) ≤ E i,j (0, 1) + E i,j (1, 0).

4.1

Example: Pixel-Labeling via Expansion Moves

In this section we show how to apply this theorem to solve the pixel-labeling problem. In this problem, are given the set of pixels P and the set of labels L. The goal is to ﬁnd a labeling l (i.e. a mapping from the set of pixels to the set of labels) which minimizes the energy E(l) = Dp (lp ) + Vp,q (lp , lq ) p∈P

p,q∈N

where N ⊂ P × P is a neighborhood system on pixels. Without loss of generality we can assume that N contains only ordered pairs p, q for which p < q (since we can combine two terms Vp,q and Vq,p into one term). We will show how our method can be used to derive the expansion move algorithm developed in [8]. This problem is shown in [8] to be NP-hard if |L| > 2. [8] gives an approximation algorithm for minimizing this energy. A single step of this algorithm is an operation called an α-expansion. Suppose that we have some current conﬁguration l0 , and we are considering a label α ∈ L. During the α-expansion operation a pixel p is allowed either to keep its old label lp0 or to switch to a new label α: lp = lp0 or lp = α. The key step in the approximation algorithm presented in [8] is to ﬁnd the optimal expansion operation, i.e. the one that leads to the largest reduction in the energy E. This step is repeated until there is no choice of α where the optimal expansion operation reduces the energy. [8] constructs a graph which contains nodes corresponding to pixels in P. The following encoding is used: if f (p) = 0 (i.e., the node p is in the source set) then lp = lp0 ; if f (p) = 1 (i.e., the node p is in the sink set) then lp = α. Note that the key technical step in this algorithm can be naturally expressed as minimizing an energy function involving binary variables. The binary variables correspond to pixels, and the energy we wish to minimize can be written formally as Dp (lp (xp )) + Vp,q (lp (xp ), lq (xq )), (3) E(xp1 , . . . , xpn ) = p∈P

p,q∈N

where

∀p ∈ P

lp (xp ) =

lp0 , α,

xp = 0 xp = 1.

70

V. Kolmogorov and R. Zabih

We can demonstrate the power of our results by deriving an important restriction on this algorithm. In order for the graph cut construction of [8] to work, the function Vp,q is required to be a metric. In their paper, it is not clear whether this is an accidental property of the construction (i.e., they leave open the possibility that a more clever graph cut construction may overcome this restriction). Using our results, we can easily show this is not the case. Speciﬁcally, by theorem 1, the energy function given in equation 3 is graph-representable if and only if each term Vp,q satisﬁes the inequality Vp,q (lp (0), lq (0)) + Vp,q (lp (1), lq (1)) ≤ Vp,q (lp (0), lq (1)) + Vp,q (lp (1), lq (0)) or

Vp,q (β, γ) + Vp,q (α, α) ≤ Vp,q (β, α) + Vp,q (α, γ)

where β = lp0 , γ = lq0 . If Vp,q (α, α) = 0, then this is the triangle inequality: Vp,q (β, γ) ≤ Vp,q (β, α) + Vp,q (α, γ) This is exactly the constraint on Vp,q that was given in [8].

5

More General Classes of Energy Functions

We begin with several deﬁnitions. Suppose we have a function E of n binary variables. If we ﬁx m of these variables then we get a new function E of n − m binary variables; we will call this function a projection of E. The notation for projections is as follows. Deﬁnition 4. Let E(x1 , . . . , xn ) be a function of n binary variables, and let I, J be a disjoint partition of the set of indices {1, . . . , n}: I = {i(1), . . . , i(m)}, J = {j(1), . . . , j(n − m)}. Let αi(1) , . . . , αi(m) be some binary constants. A projection E = E[xi(1) = αi(1) , . . . , xi(m) = αi(m) ] is a function of n − m variables deﬁned by E (xj(1) , . . . , xj(n−m) ) = E(x1 , . . . , xn ), where xi = αi for i ∈ I. We say that we ﬁx variables xi(1) , . . ., xi(m) . Now we give a deﬁnition of regular functions. Deﬁnition 5. • All functions of one variable are regular. • A function E of two variables is called regular if E(0, 0)+E(1, 1) ≤ E(0, 1)+ E(1, 0). • A function E of more than two variables is called regular if all projections of E of two variables are regular. Now we are ready to formulate our main theorem for F 3 .

What Energy Functions Can Be Minimized via Graph Cuts?

71

Theorem 6. Let E be a function of n binary variables from F 3 , i.e. it can be written as the sum E i (xi ) + E i,j (xi , xj ) + E i,j,k (xi , xj , xk ). E(x1 , . . . , xn ) = i

i<j

i<j
Then E is graph-representable if and only if E is regular. Finally, we give a necessary condition for all other classes. Theorem 7. Let E be a function of binary variables. If E is not regular then E is not graph-representable.

6

Proofs

6.1

Basic Lemmas

Deﬁnition 8. The functional π will be a mapping from the set of all functions (of binary variables) to the set of real numbers which is deﬁned as follows. For a function E(x1 , . . . , xn ) n (Πi=1 (−1)xi ) E(x1 , . . . , xn ). π(E) = x1 ∈{0,1},...,xn ∈{0,1}

For example, for a function E of two variables π(E) = E(0, 0) − E(0, 1) − E(1, 0) + E(1, 1). Note that a function E of two variables is regular if and only if π(E) ≤ 0. It is trivial to check the following properties of π. Lemma 9. • π is linear, i.e. for a scalar c and two functions E , E of n variables π(E + E ) = π(E ) + π(E ) and π(c · E ) = c · π(E ). • If E is a function of n variables that does not depend on at least one of the variables then π(E) = 0. The next two lemmas provide “building blocks” for constructing graphs for complex functions. Lemma 10. Let I = {1, . . . , n}, I = {i (1), . . . , i (n )} ⊂ I, I = {i (1), . . . , i (n )} ⊂ I be sets of indices. If the functions E (xi (1) , . . . , xi (n ) ) and E (xi (1) , . . . , xi (n ) ) are graph-representable, then so is the function E(x1 , . . . , xn ) = E (xi (1) , . . . , xi (n ) ) + E (xi (1) , . . . , xi (n ) ).

72

V. Kolmogorov and R. Zabih

Proof. Let us assume for simplicity of notation that E and E are functions of all n variables: E = E (x1 , . . . , xn ), E = E (α1 , . . . , αn ). By the deﬁnition of graph respresentability, there exist constants K , K , graphs G = (V , E ), G = (V , E ) and the set V0 = {v1 , . . . , vn }, V0 ⊂ V − {s, t}, V0 ⊂ V − {s, t} such that E + K is exactly represented by G , V0 and E + K is exactly represented by G , V0 . We can assume that the only common nodes of G and G are V0 ∪ {s, t}. Let us construct the graph G = (V, E) as the combined graph of G and G : V = V ∪ V , E = E ∪ E . It is rather straightforward to show that G, V0 exactly represent E+(K +K ) (and, therefore, E is graph-representable). Due to space limitations, the proof is omitted but is contained in a technical report [17]. Lemma 11. Suppose E and E are two functions of n variables such that xk = 0 E(x1 , . . . , xn ), E (x1 , . . . , xn ) = ∀x1 , . . . , xn E(x1 , . . . , xn ) + C, xk = 1, for some constants k and C (1 ≤ k ≤ n). Then • E if graph-representable if and only if E is graph-representable; • E is regular if and only if E is regular. Proof. Let us introduce the following function E C : 0, C E (x1 , . . . , xn ) = ∀x1 , . . . , xn C,

xk = 0 xk = 1.

We need to show that E C is graph-representable for any C then the ﬁrst part of the lemma will follow from the lemma 3 since E ≡ E +E C and E ≡ E +E −C . It is easy to construct a graph which represents E C . The set of nodes in this graph will be {v1 , . . . , vn , s, t} and the set of edges will include the only edge (s, vk ) with the capacity C (if C ≥ 0) or the edge (vk , t) with the capacity −C (if C < 0). It is trivial to check that this graph exactly represents E C (in the former case) or E C + C (in the latter case). Now let us assume that one of the functions E and E is regular, for example, E. Consider a projection of E of two variables: E [xi(1) = αi(1) , . . . , xi(m) = αi(m) ], where m = n − 2 and {i(1), . . . , i(m)} ⊂ {1, . . . , n}. We need to show that this function is regular, i.e. that the functional π of this function is nonpositive. Due to the linearity of π we can write π(E [xi(1) = αi(1) , . . . , xi(m) = αi(m) ]) = = π(E[xi(1) = αi(1) , . . . , xi(m) = αi(m) ])

+

+ π(E C [xi(1) = αi(1) , . . . , xi(m) = αi(m) ]). The ﬁrst term is nonpositive by assumption, and the second term is 0 by lemma 2.

What Energy Functions Can Be Minimized via Graph Cuts?

6.2

73

Proof of Theorems 1 and 2: The Constructive Part

In this section we will give the constructive part of the proof: given a regular energy function from class F 3 we will show how to construct a graph which represents it. We will do it in three steps. First we will consider regular functions of two variables, then regular functions of three variables and ﬁnally regular functions of the form as in the theorem 2. This will also prove the constructive part of the theorem 1. Indeed, suppose a function is from the class F 2 and each term in the sum satisﬁes the condition given in the theorem 1 (i.e. regular). Then each term is graph-representable (as we will show in this section) and, hence, the function is graph-representable as well according to the lemma 3. The other direction of theorems 1 and 2 as well as the theorem 3 will be proven in the section 6.3. Functions of two variables. Let E(x1 , x2 ) be a function of two variables represented by a table E=

E(0, 0) E(0, 1) E(1, 0) E(1, 1)

Lemma 4 tells us that we can add a constant to any column or row without aﬀecting theorem 2. Thus, without loss of generality we can consider only functions E of the form E=

0A 0 0

(we subtracted a constant from the ﬁrst row to make the upper left element zero, then we subtracted a constant from the second row to make the bottom left element zero, and ﬁnally we subtracted a constant from the second column to make the bottom right element zero). π(E) = −A ≤ 0 since we assumed that E is regular; hence, A is non-negative. Now we can easily construct a graph G which represents this function. It will have four vertices V = {v1 , v2 , s, t} and one edge E = {(v1 , v2 )} with the cost c(v1 , v2 ) = A. It is easy to see that G, V0 = {v1 , v2 } represent E since the only case when the edge (v1 , v2 ) is cut (yielding a cost A) is when v1 ∈ S, v2 ∈ T , i.e. when x1 = 0, x2 = 1. Note that we did not introduce any additional nodes for representing binary interactions of binary variables. This is in contrast to the construction in [8] which added auxiliary nodes for representing energies that we just considered. Our construction yields a smaller graph and, thus, the minimum cut can potentially be computed faster. Functions of three variables. Now let us consider a regular function E of three variables. Let us represent it as a table

74

V. Kolmogorov and R. Zabih

E(0, 0, 0) E(0, 0, 1) E(0, 1, 0) E(0, 1, 1) E= E(1, 0, 0) E(1, 0, 1) E(1, 1, 0) E(1, 1, 1) Two cases are possible: Case 1. π(E) ≥ 0. We can apply transformations described in lemma 4 and get the following function: 0 0 E= 0 A3

0 A1 A2 A0

It is easy to check that these transformations preserve the functional π. Hence, A = A0 − (A1 + A2 + A3 ) = −π(E) ≤ 0. By applying the regularity constraint to the projections E[x1 = 0], E[x2 = 0], E[x3 = 0] we also get A1 ≤ 0, A2 ≤ 0, A3 ≤ 0. We can represent E as the sum of functions 0 0 0 0 0 0 0 0 0 A1 0 0 0 0 0 0 + + + E= 0 0 0 A2 0 0 0 0 0 A1 0 A2 A3 A3 0A We need to show that all terms here are graph-representable, then lemma 3 will imply that E is graph-representable as well. The ﬁrst three terms are regular functions depending only on two variables and thus are graph-representable as was shown in the previous section. Let us consider the last term. The graph G that represents this term can be constructed as follows. The set of nodes will contain one auxilary node u: V = {v1 , v2 , v3 , u, s, t}. The set of edges will consist of directed edges E = {(v1 , u), (v2 , u), (v3 , u), (u, t)} with capacities A = −A. It is easy to check that G, V0 = {v1 , v2 , v3 } exactly represent the function E (x1 , x2 , x3 ) = E(x1 , x2 , x3 ) + A ; the proof is contained in [17]. Case 2. π(E) < 0. This case is similar to the case 1. We can transform the energy to A0 A E= 2 A1 0

A3 A1 0 A2 0 A3 A3 A0 0 0 0 A2 0 0 0 0 0 = + + + 0 A1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

where A = A0 − (A1 + A2 + A3 ) = π(E) < 0 and A1 ≤ 0, A2 ≤ 0, A3 ≤ 0 since E is regular. The ﬁrst three terms are regular functions of two variables and the last term can be represented by the graph G = (V, E) where V = {v1 , v2 , v3 , u, s, t} and E = {(u, v1 ), (u, v2 ), (u, v3 ), (s, u)}; capacities of all edges are −A.

What Energy Functions Can Be Minimized via Graph Cuts?

75

Functions of many variables. Finally let us consider a regular function E which can be written as E i,j,k (xi , xj , xk ), E(x1 , . . . , xn ) = i<j
where i, j, k are indices from the set {1, . . . , n} (we omitted terms involving functions of one and two variables since they can be viewed as functions of three variables). Although E is regular, each term in the sum need not necessarily be regular. However we can “regroup” terms in the sum so that each term will be regular (and, thus, graph-representable). This can be done using the following lemma and a trivial induction argument. Deﬁnition 12. Let E i,j,k be a function of three variables. The functional N (E i,j,k ) is deﬁned as the number of projections of two variables of E i,j,k with the positive value of the functional π. Note that N (E i,j,k ) = 0 exactly when E i,j,k is regular. Lemma 13. Suppose the function E of n variables can be written as E i,j,k (xi , xj , xk ), E(x1 , . . . , xn ) = i<j
where some of the terms are not regular. Then it can be written as ˜ i,j,k (xi , xj , xk ), E E(x1 , . . . , xn ) = i<j
where

˜ i,j,k ) < N (E

i<j
N (E i,j,k ).

i<j
Proof. For the simplicity of notation let us assume that the term E 1,2,3 is not regular and π(E 1,2,3 [x3 = 0]) > 0 or π(E 1,2,3 [x3 = 1]) > 0 (we can ensure this by renaming indices). Let Ck =

max π(E 1,2,k [xk = αk ])

αk ∈{0,1}

C3 = −

n

k ∈ {4, . . . , n}

Ck

k=4

Now we will modify the terms E 1,2,3 , . . ., E 1,2,n as follows: ˜ 1,2,k ≡ E 1,2,k − R[Ck ] E

k ∈ {3, . . . , n}

where R[C] is the function of two variables x1 and x2 deﬁned by the table R[C] =

0 0 0C

76

V. Kolmogorov and R. Zabih

˜ i,j,k ≡ E i,j,k , (i, j) = (1, 2)). We have (other terms are unchanged: E ˜ i,j,k (xi , xj , xk ) E E(x1 , . . . , xn ) = n

i<j
n

since k=3 Ck = 0 and k=3 R[Ck ] ≡ 0. If we consider R[C] as a function of n variables and take a projection of two variables where the two variables that are not ﬁxed are xi and xj (i < j), then the functional π will be C, if (i, j) = (1, 2), and 0 otherwise since in the latter case a projection actually depends on at most one variable. Hence, the only projections of two variables that could have changed their value of the ˜ 1,2,k ˜ 1,2,k [x3 = α3 , . . . , xn = αn ], k ∈ {3, . . . , n}, if we treat E functional π are E 1,2,k 1,2,k ˜ ˜ as functions of n variables, or E [xk = αk ], if we treat E as functions of three variables. First let us consider terms with k ∈ {4, . . . , n}. We have π(E 1,2,k [xk = αk ]) ≤ Ck , thus ˜ 1,2,k [xk = αk ]) = π(E 1,2,k [xk = αk ]) − π(R[Ck ][xk = αk ]) ≤ Ck − Ck = 0 π(E Therefore we did not introduce any nonregular projections for these terms. ˜ 1,2,3 [x3 = α3 ]). We can write Now let us consider the term π(E ˜ 1,2,3 [x3 = α3 ]) = π(E 1,2,3 [x3 = α3 ]) − π(R[C3 ][x3 = α3 ]) = π(E n n 1,2,3 [x3 = α3 ]) − (− Ck ) = π(E 1,2,k [xk = αk ]) = π(E k=4

k=3

where αk = arg maxα∈{0,1} π(E 1,2,k [xk = α]), k ∈ {4, . . . , n}. The last expression is just π(E[x3 = α3 , . . . , xn = αn ]) and is nonpositive since E is regular by ˜ 1,2,3 [x3 = 1]) are both ˜ 1,2,3 [x3 = 0]) and π(E assumption. Hence, values π(E nonpositive and, therefore, the number of nonregular projections has decreased. 6.3

Proof of Theorem 3

In this section we will prove a necessary condition for graph representability: if a function of binary variables is graph-representable then it is regular. It will also imply the corresponding directions of the theorems 1 and 2. Note that theorem 1 needs a little bit of reasoning, as follows. Let us consider a graph-representable function E from the class F 2 : E(x1 , . . . , xn ) = E i (xi ) + E i,j (xi , xj ) i

i<j

E is regular as we will prove in this section. It means that the functional π of any projection of E of two variables is nonpositive. Let us consider a projection where the two variables that are not ﬁxed are xi and xj . By lemma 2 the value of the functional π of this projection is equal to π(E i,j ) (all other terms yield zero). Hence, all terms E i,j are regular, i.e. they satisfy the condition in the theorem 1.

What Energy Functions Can Be Minimized via Graph Cuts?

77

Deﬁnition 14. Let G = (V, E) be a graph, v1 , . . . , vk be a subset of nodes V and α1 , . . . , αk be binary constants whose values are from {0, 1}. We will deﬁne the graph G[x1 = α1 , . . . , xk = αk ] as follows. Its nodes will be the same as in G and its edges will be all edges of G plus additional edges corresponding to nodes v1 , . . . , vk : for a node vi , we add the edge (s, vi ), if αi = 0, or (vi , t), if αi = 1, with an inﬁnite capacity. It should be obvious that these edges enforce constraints f (v1 ) = α1 , . . ., f (vk ) = αk in the minimum cut on G[x1 = α1 , . . . , xk = αk ], i.e. if αi = 0 then vi ∈ S, and if αi = 1 then vi ∈ T . (If, for example, αi = 0 and vi ∈ T then the edge (s, vi ) must be cut yielding an inﬁnite cost, so it would not the minimum cut.) Now we can give a deﬁnition of graph representability which is equivalent to the deﬁnition 1. This new deﬁnition will be more convenient for the proof. Deﬁnition 15. We say that the function E of n binary variables is exactly represented by the graph G = (V, E) and the set V0 ⊂ V if for any conﬁguration α1 , . . . , αn the cost of the minimum cut on G[x1 = α1 , . . . , xk = αk ] is E(α1 , . . . , αn ). Lemma 16. Any projection of a graph-representable function is graphrepresentable. Proof. Let E be a graph-representable function of n variables, and the graph G = (V, E) and the set V0 represents E. Suppose that we ﬁx variables xi(1) , . . . , xi(m) . It is straightforward to check that the graph G[xi(1) = αi(1) , . . . , xi(m) = αi(m) ] and the set V0 = V0 − {vi(1) , . . . , vi(m) } represent the function E = E[xi(1) = αi(1) , . . . , xi(m) = αi(m) ]. This lemma implies that it suﬃces to prove theorem 3 only for energies of two variables. Let E(x1 , x2 ) be a graph-representable function of two variables. Let us represent this function as a table: E=

E(0, 0) E(0, 1) E(1, 0) E(1, 1)

Lemma 4 tells us that we can add a constant to any column or row without aﬀecting theorem 3. Thus, without loss of generality we can consider only functions E of the form E=

0 0 0A

(we subtracted a constant from the ﬁrst row to make the upper left element zero, then we subtracted a constant from the second row to make the bottom left element zero, and ﬁnally we subtracted a constant from the second column to make the upper right element zero).

78

V. Kolmogorov and R. Zabih

We need to show that E is regular, i.e. that π(E) = A ≤ 0. Suppose this is not true: A > 0. Suppose the graph G and the set V0 = {v1 , v2 } represent E. It means that there is a constant K such that G, V0 exactly represent E (x1 , x2 ) = E(x1 , x2 ) + K: E =

K K K K +A

The cost of the minimum s-t-cut on G is K (since this cost is just the minimum entry in the table for E ); hence, K ≥ 0. Thus the value of the maximum ﬂow from s to t in G is K. Let G 0 be the residual graph obtained from G after pushing the ﬂow K. Let E 0 (x1 , x2 ) be the function exactly represented by G 0 , V0 . By the deﬁnition of graph representability, E (α1 , α2 ) is equal to the value of the minimum cut (or maximum ﬂow) on the graph G[x1 = α1 , x2 = α2 ]. The following sequence of operations shows one possible way to push the maximum ﬂow through this graph. • First we take the original graph G and push the ﬂow K; then we get the residual graph G 0 . (It is equivalent to pushing ﬂow through G[x1 = α1 , x2 = α2 ] where we do not use edges corresponding to constraints x1 = α1 and x2 = α2 ). • Then we add edges corresponding to these constraints; then we get the graph G 0 [x1 = α1 , x2 = α2 ]. • Finally we push the maximum ﬂow possible through the graph G 0 [x1 = α1 , x2 = α2 ]; the amount of this ﬂow is E 0 (α1 , α2 ) according to the deﬁnition of graph representability. The total amount of ﬂow pushed during all steps is K + E 0 (α1 , α2 ); thus, E (α1 , α2 ) = K + E 0 (α1 , α2 ) or

E(α1 , α2 ) = E 0 (α1 , α2 )

We proved that E is exactly represented by G 0 , V0 . The value of the minimum cut/maximum ﬂow on G 0 is 0 (it is the minimum entry in the table for E); thus, there is no augmenting path from s to t in G 0 . However, if we add edges (v1 , t) and (v2 , t) then there will be an augmenting path from s to t in G 0 [x1 = α1 , x2 = α2 ] since E(1, 1) = A > 0. Hence, this augmenting path will contain at least one of these edges and, therefore, either v1 or v2 will be in the path. Let P be the part of this path going from the source until v1 or v2 is ﬁrst encountered. Without loss of generality we can assume that it will be v1 . Thus, P is an augmenting path from s to v1 which does not contain edges that we added, namely (v1 , t) and (v2 , t). Finally let us consider the graph G 0 [x1 = 1, x2 = 0] which is obtained from 0 G by adding edges (v1 , t) and (s, v2 ) with inﬁnite capacities. There is an augmenting path {P, (v1 , t)} from the source to the sink in this graph; hence, the minimum cut/maximum ﬂow on it greater than zero, or E(1, 0) > 0. We get a contradiction.

What Energy Functions Can Be Minimized via Graph Cuts?

79

Acknowledgements. We thank Olga Veksler and Yuri Boykov for their careful reading of this paper, and for valuable comments which greatly improved its readibility. We also thank Ian Jermyn for helping us clarify the paper’s motivation. This research was supported by NSF grants IIS-9900115 and CCR-0113371, and by a grant from Microsoft Research.

Appendix: Summary of Graph Constructions We now summarize the graph constructions used for regular functions. The notation D(v, c) means that we add an edge (s, v) with the weight c if c > 0, or an edge (v, t) with the weight −c if c < 0. Regular Functions of One Binary Variable Recall that all functions of one variable are regular. For a function E(x1 ), we construct a graph G with three vertices V = {v1 , s, t}. There is a single edge D(v1 , E(1) − E(0)). Regular Functions of Two Binary Variables We now show how to construct a graph G for a regular function E(x1 , x2 ) of two variables. It will contain four vertices: V = {v1 , v2 , s, t}. The edges E are given below. • D(v1 , E(1, 0) − E(0, 0)); • D(v2 , E(1, 1) − E(1, 0)); • (v1 , v2 ) with the weight −π(E). Regular Functions of Three Binary Variables We next show how to construct a graph G for a regular function E(x1 , x2 , x3 ) of three variables. It will contain ﬁve vertices: V = {v1 , v2 , v3 , u, s, t}. If π(E) ≥ 0 then the edges are • • • • • • •

D(v1 , E(1, 0, 1) − E(0, 0, 1)); D(v2 , E(1, 1, 0) − E(1, 0, 0)); D(v3 , E(0, 1, 1) − E(0, 1, 0)); (v2 , v3 ) with the weight −π(E[x1 = 0]); (v3 , v1 ) with the weight −π(E[x2 = 0]); (v1 , v2 ) with the weight −π(E[x3 = 0]); (v1 , u), (v2 , u), (v3 , u), (u, t) with the weight π(E).

If π(E) < 0 then the edges are • • • • • • •

D(v1 , E(1, 1, 0) − E(0, 1, 0)); D(v2 , E(0, 1, 1) − E(0, 0, 1)); D(v3 , E(1, 0, 1) − E(1, 0, 0)); (v3 , v2 ) with the weight −π(E[x1 = 1]); (v1 , v3 ) with the weight −π(E[x2 = 1]); (v2 , v1 ) with the weight −π(E[x3 = 1]); (u, v1 ), (u, v2 ), (u, v3 ), (s, u) with the weight −π(E).

80

V. Kolmogorov and R. Zabih

References 1. Ravindra K. Ahuja, Thomas L. Magnanti, and James B. Orlin. Network Flows: Theory, Algorithms, and Applications. Prentice Hall, 1993. 2. Amir Amini, Terry Weymouth, and Ramesh Jain. Using dynamic programming for solving variational problems in vision. IEEE Transactions on Pattern Analysis and Machine Intelligence, 12(9):855–867, September 1990. 3. Stephen Barnard. Stochastic stereo matching over scale. International Journal of Computer Vision, 3(1):17–32, 1989. 4. S. Birchﬁeld and C. Tomasi. Multiway cut for stereo and motion with slanted surfaces. In International Conference on Computer Vision, pages 489–495, 1999. 5. Yuri Boykov and Marie-Pierre Jolly. Interactive organ segmentation using graph cuts. In Medical Image Computing and Computer-Assisted Intervention, pages 276–286, 2000. 6. Yuri Boykov and Marie-Pierre Jolly. Interactive graph cuts for optimal boundary and region segmentation of objects in N-D images. In International Conference on Computer Vision, pages I: 105–112, 2001. 7. Yuri Boykov, Olga Veksler, and Ramin Zabih. Markov Random Fields with efﬁcient approximations. In IEEE Conference on Computer Vision and Pattern Recognition, pages 648–655, 1998. 8. Yuri Boykov, Olga Veksler, and Ramin Zabih. Fast approximate energy minimization via graph cuts. IEEE Transactions on Pattern Analysis and Machine Intelligence, 23(11):1222–1239, November 2001. 9. L. Ford and D. Fulkerson. Flows in Networks. Princeton University Press, 1962. 10. S. Geman and D. Geman. Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 6:721–741, 1984. 11. A. Goldberg and R. Tarjan. A new approach to the maximum ﬂow problem. Journal of the Association for Computing Machinery, 35(4):921–940, October 1988. 12. D. Greig, B. Porteous, and A. Seheult. Exact maximum a posteriori estimation for binary images. Journal of the Royal Statistical Society, Series B, 51(2):271–279, 1989. 13. H. Ishikawa and D. Geiger. Occlusions, discontinuities, and epipolar lines in stereo. In European Conference on Computer Vision, pages 232–248, 1998. 14. H. Ishikawa and D. Geiger. Segmentation by grouping junctions. In IEEE Conference on Computer Vision and Pattern Recognition, pages 125–131, 1998. 15. Junmo Kim, John Fish, Andy Tsai, Cindy Wible, Ala Willsky, and William Wells. Incorporating spatial priors into an information theoretic approach for fMRI data analysis. In Medical Image Computing and Computer-Assisted Intervention, pages 62–71, 2000. 16. Vladimir Kolmogorov and Ramin Zabih. Visual correspondence with occlusions using graph cuts. In International Conference on Computer Vision, pages 508–515, 2001. 17. Vladimir Kolmogorov and Ramin Zabih. What energy functions can be minimized via graph cuts? Technical report CUCS-TR2001-1857, Cornell Computer Science Department, November 2001. 18. Vladimir Kolmogorov and Ramin Zabih. Multi-camera scene reconstruction via graph cuts. In European Conference on Computer Vision, 2002. 19. S. Li. Markov Random Field Modeling in Computer Vision. Springer-Verlag, 1995.

What Energy Functions Can Be Minimized via Graph Cuts?

81

20. S. Roy. Stereo without epipolar lines: A maximum ﬂow formulation. International Journal of Computer Vision, 1(2):1–15, 1999. 21. S. Roy and I. Cox. A maximum-ﬂow formulation of the n-camera stereo correspondence problem. In International Conference on Computer Vision, 1998. 22. Daniel Scharstein and Richard Szeliski. A taxonomy and evaluation of dense twoframe stereo correspondence algorithms. Technical Report 81, Microsoft Research, 2001. To appear in IJCV. An earlier version appears in CVPR 2001 Workshop on Stereo Vision. 23. Dan Snow, Paul Viola, and Ramin Zabih. Exact voxel occupancy with graph cuts. In IEEE Conference on Computer Vision and Pattern Recognition, pages 345–352, 2000. 24. Richard Szeliski and Ramin Zabih. An experimental comparison of stereo algorithms. In B. Triggs, A. Zisserman, and R. Szeliski, editors, Vision Algorithms: Theory and Practice, number 1883 in LNCS, pages 1–19, Corfu, Greece, September 1999. Springer-Verlag.

Multi-camera Scene Reconstruction via Graph Cuts Vladimir Kolmogorov and Ramin Zabih Computer Science Department, Cornell University, Ithaca, NY 14853 [email protected], [email protected]

Abstract. We address the problem of computing the 3-dimensional shape of an arbitrary scene from a set of images taken at known viewpoints. Multi-camera scene reconstruction is a natural generalization of the stereo matching problem. However, it is much more diﬃcult than stereo, primarily due to the diﬃculty of reasoning about visibility. In this paper, we take an approach that has yielded excellent results for stereo, namely energy minimization via graph cuts. We ﬁrst give an energy minimization formulation of the multi-camera scene reconstruction problem. The energy that we minimize treats the input images symmetrically, handles visibility properly, and imposes spatial smoothness while preserving discontinuities. As the energy function is NP-hard to minimize exactly, we give a graph cut algorithm that computes a local minimum in a strong sense. We handle all camera conﬁgurations where voxel coloring can be used, which is a large and natural class. Experimental data demonstrates the eﬀectiveness of our approach.

1

Introduction

Reconstructing an object’s 3-dimensional shape from a set of cameras is a classic vision problem. In the last few years, it has attracted a great deal of interest, partly due to a number of new applications both in vision and in graphics that require good reconstructions. While the problem can be viewed as a natural generalization of stereo, it is considerably harder. The major reason for this is the diﬃculty of reasoning about visibility. In stereo matching, most scene elements are visible from both cameras, and it is possible to obtain good results without addressing visibility constraints. In the more general scene reconstruction problem, however, very few scene elements are visible from every camera, so the issue of visibility cannot be ignored. In this paper, we approach the scene reconstruction problem from the point of view of energy minimization. Energy minimization has several theoretical advantages, but has generally been viewed as too slow for early vision to be practical. Our approach is motivated by some recent work in early vision, where fast energy minimization algorithms have been developed based on graph cuts [6,7,12,14,20,21]. These methods give strong experimental results in practice, as documented in two recent evaluations of stereo algorithms using real imagery with dense ground truth [22,27]. A. Heyden et al. (Eds.): ECCV 2002, LNCS 2352, pp. 82–96, 2002. c Springer-Verlag Berlin Heidelberg 2002

Multi-camera Scene Reconstruction via Graph Cuts

83

The energy that we minimize has three important properties: – it treats the input images symmetrically, – it handles visibility properly, and – it imposes spatial smoothness while also preserving discontinuities. We begin with a review of related work. In section 3 we give a precise deﬁnition of the problem that we wish to solve, and deﬁne the energy that we will minimize. Section 4 describes how we use graph cuts to compute a strong local minimum of this energy. Experimental data is presented in section 5. Some details of the graph cut construction are deferred to the appendix, along with a proof that computing the global minimum of the energy is NP-hard.

2

Related Work

The problem of reconstructing a scene from multiple cameras has received a great deal of attention in the last few years. One extensively-explored approach to this problem is voxel occupancy. In voxel occupancy [18,25] the scene is represented as a set of 3-dimensional voxels, and the task is to label the individual voxels as ﬁlled or empty. Voxel occupancy is typically solved using silhouette intersection, usually from multiple cameras but sometimes from a single camera with the object placed on a turntable [8]. It is known that the output of silhouette intersection even without noise is not the actual 3-dimensional shape, but rather an approximation called the visual hull [17]. 2.1

Voxel Coloring and Space Carving

Voxel occupancy, however, fails to exploit the consistent appearance of a scene element between diﬀerent cameras. This constraint, called photo-consistency, is obviously quite powerful. Two well-known recent algorithms that have used photo-consistency are voxel coloring [23] and space carving [16]. Voxel coloring makes a single pass through voxel space, ﬁrst computing the visibility of each voxel and then its color. There is a constraint on the camera geometry, namely that no scene point is allowed to be within the convex hull of the camera centers. As we will see in section 3, our approach handles all the camera conﬁgurations where voxel coloring can be used. Space carving is another voxel-oriented approach that uses the photo-consistency constraint to prune away empty voxels from the volume. Space carving has the advantage of allowing arbitrary camera geometry. One major limitation of voxel coloring and space carving is that they lack a way of imposing spatial coherence. This is particularly problematic because the image data is almost always ambiguous. Another (related) limitation comes from the fact that these methods traverse the volume making “hard” decisions concerning the occupancy of each voxel they analyze. Because the data is ambiguous, such a decision can easily be incorrect, and there is no easy way to undo such a decision later on.

84

2.2

V. Kolmogorov and R. Zabih

Energy Minimization Approaches

Our approach to the scene reconstruction problem is to generalize some recently developed techniques that give strong results for stereo matching. It is well known that stereo, like many problems in early vision, can be elegantly stated in terms of energy minimization [19]. The energy minimization problem has traditionally been solved via simulated annealing [2,11], which is extremely slow in practice. In the last few years powerful energy minimization algorithms have been developed based on graph cuts [6,7,12,14,21]. These methods are fast enough to be practical, and yield quite promising experimental results for stereo [22, 27]. Unlike simulated annealing, graph cut methods cannot be applied to an arbitrary energy function; instead, for each energy function to be minimized, a careful graph construction must be developed. In this paper, instead of building a special purpose graph we will use some recent results [15] that give graph constructions for a quite general class of energy functions. While energy minimization has been widely used for stereo, only a few papers [13,20,24] have used it for scene reconstruction. The energy minimization formalism has several advantages. It allows a clean speciﬁcation of the problem to be solved, as opposed to the algorithm used to solve it. In addition, energy minimization naturally allows the use of soft constraints, such as spatial coherence. In an energy minimization framework, it is possible to cause ambiguities to be resolved in a manner that leads to a spatially smooth answer. Finally, energy minimization avoids being trapped by early hard decisions. Although [20] and [24] use energy minimization via graph cuts, their focus is quite diﬀerent from ours. [20] uses an energy function whose global minimum can be computed eﬃciently via graph cuts; however, the spatial smoothness term is not discontinuity preserving, and so the results tend to be oversmoothed. Visibility constraints are not used. [24] computes the global minimum of a diﬀerent energy function as an alternative to silhouette intersection (i.e., to determine voxel occupancy). Their approach does not deal with photoconsistency at all, nor do they reason about visibility. Our method is perhaps closest to the work of [13], which also relies on graph cuts. They extend the work of [7], which focused on traditional stereo matching, to allow an explicit label for occluded pixels. While the energy function that they use is of a similar general form to ours, they do not treat the input images symmetrically. While we eﬀectively compute a disparity map with respect to each camera, they compute a disparity map only with respect to a single camera. [26] also proposed the use of disparity maps for each camera, in an energy minimization framework. Their optimization scheme did not rely on graph cuts, and their emphasis was on the two-camera stereo problem. They also focused on handling transparency, which is an issue that we do not address. Another approach is to formulate the problem in terms of level sets, and to solve a system of partial diﬀerential equations using numerical techniques [9]. The major distinction between this approach and our work is that their problem formulation is continuous while ours is discrete. The discrete energy

Multi-camera Scene Reconstruction via Graph Cuts

85

minimization approach is known to lead to strong results for two-camera stereo [22,27], and is therefore worth investigating for multiple cameras. In [14] we proposed a (two camera) stereo algorithm that shares a number of properties with the present work. That method treats the input images symmetrically, and is based on graph cuts for energy minimization. It properly handles the limited form of visibility constraint that arises with two cameras (namely, occlusions). The major diﬀerence is that [14] is limited to the case of two cameras, while the multi-camera problem is much more diﬃcult.

3

Problem Formulation

Suppose we are given n calibrated images of the same scene taken from diﬀerent viewpoints (or at diﬀerent moments of time). Let Pi be the set of pixels in the camera i, and let P = P1 ∪ . . . ∪ Pn be the set of all pixels. A pixel p ∈ P corresponds to a ray in 3D-space. Consider the point of the ﬁrst intersection of this ray with an object in the scene. Our goal is to ﬁnd the depth of this point for all pixels in all images. Thus, we want to ﬁnd a labeling f : P → L where L is a discrete set of labels corresponding to diﬀerent depths. In the current implementation of our method, labels correspond to increasing depth from a ﬁxed camera. A pair p, l where p ∈ P, l ∈ L corresponds to some point in 3D-space. We will refer to such pairs as 3D-points. The limitation of our method is that there must exist a function D : R3 → R such that for all scene points P and Q, P occludes Q in a camera i only if D(P ) < D(Q). This is exactly the constraint used in voxel coloring [23]. If such a function exists then labels correspond to level sets of this function. In our current implementation, we make a slightly more speciﬁc assumption, which is that the cameras must lie in one semiplane looking at the other semiplane. The interpretation of labels will be as follows: each label corresponds to a plane in 3D-space, and a 3D-point p, l is the intersection of the ray corresponding to the pixel p and the plane l. Let us introduce the set of interactions I consisting of (unordered) pairs of 3D-points p1 , l1 , p2 , l2 “close” to each other in 3D-space. We will discuss several criteria for “closeness” later. In general, I can be an arbitrary set of pairs of 3D-points satisfying the following constraint: – Only 3D-points at the same depth can interact, i.e.if {p1 , l1 , p2 , l2 } ∈ I then l1 = l2 . Now we will deﬁne the energy function that we minimize. It will consist of three terms: E(f ) = Edata (f ) + Esmoothness (f ) + Evisibility (f ) The data term will impose photo-consistency. It is D(p, q) Edata (f ) = p,f (p),q,f (q)∈I

(1)

86

V. Kolmogorov and R. Zabih

where D(p, q) is a non-positive value depending on intensities of pixels p and q. It can be, for example, D(p, q) = min{0, (Intensity(p) − Intensity(q))2 − K} for some constant K > 0. Since we minimize the energy, terms D(p, q) that we sum will be small. These terms are required to be non-negative for technical reasons that will be described in section 4.2. Thus, pairs of pixels p, q which come from the same scene point according to the conﬁguration f will have similar intensities, which causes photo-consistency. The smoothness term involves a notion of neighborhood; we assume that there is a neighborhood system on pixels N ⊂ {{p, q} | p, q ∈ P} This can be the usual 4-neighborhood system: pixels p = (px , py ) and q = (qx , qy ) are neighbors if they are in the same image and |px − qx | + |py − qy | = 1. We will write the smoothness term as V{p,q} (f (p), f (q)) Esmoothess (f ) = {p,q}∈N

We will require the term V{p,q} to be a metric. This imposes smoothness while preserving discontinuities, as long as we pick an appropriate robust metric. For example, we can use the robustiﬁed L1 distance V (l1 , l2 ) = min(|l1 − l2 |, K) for constant K. The last term will encode the visibility constraint: it will be zero if this constraint is satisﬁed, and inﬁnity otherwise. We can write this using another set of interactions Ivis which contains pairs of 3D-points violating the visibility constraint: Evisibility (f ) = ∞ p,f (p),q,f (q)∈Ivis

We require the set Ivis to meet following condition: – Only 3D-points at diﬀerent depths can interact, i.e. if {p1 , l1 , p2 , l2 } ∈ Ivis then l1 = l2 . The visibility constraint says that if a 3D-point p, l is present in a conﬁguration f (i.e. l = f (p)) then it “blocks” views from other cameras: if a ray corresponding to a pixel q goes through (or close to) p, l then its depth is at most l. Again, we need a deﬁnition of “closeness”. We will use the set I this part of the construction of Ivis . The set Ivis can then be deﬁned as follows: it will contain all pairs of 3D-points p, l, q, l such that p, l and q, l interact (i.e. they are in I) and l > l. An important issue is the choice of the sets of interactions I. This basically deﬁnes a discretization. It can consist, for example, of pairs of 3D-points

Multi-camera Scene Reconstruction via Graph Cuts l=2

l=3

C1

87

q

p

C2 Depth labels

Fig. 1. Example of pixel interactions. There is a photo-consistency constraint between the red round point and the blue round point, both of which are at the same depth (l = 2). The red round point blocks camera C2’s view of the green square point at depth l = 3. Color ﬁgures are available in the electronic version of this paper.

{p, l, q, l} such that the distance in 3D-space between these points is less than some threshold. However, with this deﬁnition of closeness the number of interactions per pixel may vary greatly from pixel to pixel. Instead, we chose the following criterion: if p is a pixel in the image i, q is a pixel in the image j and i < j then the 3D-points p, l, q, l interact if the closest pixel to the projection of p, l onto the image j is q. Of course, we can include only interactions between certain images, for example, taken by neighboring cameras. Note that it makes sense to include interactions only between different cameras. An example of our problem formulation in action is shown in ﬁgure 1. There are two cameras C1 and C2. There are 5 labels shown as black vertical lines. As in our current implementation, labels are distributed by increasing distance from a ﬁxed camera. Two pixels, p from C1 and q from C2 are shown, along with the red round 3D-point q, 2 and the blue round 3D-point p, 2. These points share the same label, and interact (i.e., {p, 2, q, 2} ∈ I). So there is a photoconsistency term between them. The green square point q, 3 is at a diﬀerent label (greater depth), but is behind the red round point. The pair of 3D-points {p, 2, q, 3} is in Ivis . So if the ray p from camera C1 sees the red round point p, 2, the ray q from C2 cannot see the green square point q, 3. We show in the appendix that minimizing our energy is an NP-hard problem. Our approach, therefore, is to constuct an approximation algorithm based on graph cuts that ﬁnds a strong local minimum.

88

4

V. Kolmogorov and R. Zabih

Graph Construction

We now show how to eﬃciently minimize E among all conﬁgurations using graph cuts. The output of our method will be a local minimum in a strong sense. In particular, consider an input conﬁguration f and a disparity α. Another conﬁguration f is deﬁned to be within a single α-expansion of f when for all pixels p ∈ P either f (p) = f (p) or f (p) = α. This notion of an expansion was proposed by [7], and forms the basis for several very eﬀective stereo algorithms [3,7,13,14]. Our algorithm is very straightforward; we simply select (in a ﬁxed order or at random) a disparity α, and we ﬁnd the unique conﬁguration within a single α-expansion move (our local improvement step). If this decreases the energy, then we go there; if there is no α that decreases the energy, we are done. Except for the problem formulation and the choice of energy function, this algorithm is identical to the methods of [7,14]. One restriction on the algorithm is that the initial conﬁguration must satisfy the visibility constraint. This will guarantee that all subsequent conﬁgurations will satisfy this constraint as well, since we minimize the energy, and conﬁgurations that do not satisfy the visibility constraint have inﬁnite energy. The critical step in our method is to eﬃciently compute the α-expansion with the smallest energy. In this section, we show how to use graph cuts to solve this problem. 4.1

Graph Cuts

Let G = V, E be a weighted graph with two distinguished terminal vertices {s, t} called the source and sink. A cut C = V s , V t is a partition of the vertices into two sets such that s ∈ V s and t ∈ V t . (Note that a cut can also be equivalently deﬁned as the set of edges between the two sets.) The cost of the cut, denoted |C|, equals the sum of the weights of the edges between a vertex in V s and a vertex in V t . The minimum cut problem is to ﬁnd the cut with the smallest cost. This problem can be solved very eﬃciently by computing the maximum ﬂow between the terminals, according to a theorem due to Ford and Fulkerson [10]. There are a large number of fast algorithms for this problem (see [1], for example). The worst case complexity is low-order polynomial; however, in practice the running time is nearly linear for graphs with many short paths between the source and the sink, such as the one we will construct. We will use a result from [15] which says that for energy functions of binary variables of the form E i,j (xi , xj ) (2) E(x1 , . . . , xn ) = i<j

it is possible to construct a graph for minimizing it if and only if each term E i,j satisﬁes the following condition: E i,j (0, 0) + E i,j (1, 1) ≤ E i,j (0, 1) + E i,j (1, 0)

(3)

Multi-camera Scene Reconstruction via Graph Cuts

89

If these conditions are satisﬁed then the graph G is constructed as follows. We add a node vi for each variable xi . For each term E i,j (xi , xj ) we add edges which are given in the appendix. Every cut on such a graph corresponds to some conﬁguration x = (x1 , . . . , xn ), and vice versa: if vi ∈ V s then xi = 0, otherwise xi = 1. Edges on a graph were added in such a way that the cost of any cut is equal to the energy of the corresponding conﬁguration plus a constant. Thus, the minimum cut on G yields the conﬁguration that minimizes the energy. 4.2

α-Expansion

In this section we will show how to convert our energy function into the form of equation 2. Note that it is not necessary to use only terms E i,j for which i < j since we can swap the variables if necessary without aﬀecting condition 3. Although pixels can have multiple labels and in general cannot be represented by binary variables, we can do it for the α-expansion operation. Indeed, any conﬁguration f within a single α-expansion of the initial conﬁguration f can be encoded by a binary vector x = {xp |p ∈ P} where f (p) = f (p) if xp = 0, and f (p) = α if xp = 1. Let us denote a conﬁguration deﬁned by a vector x as f x . Thus, we have the energy of binary variables: ˜ ˜data (x) + E ˜smoothness (x) + E ˜visibility (x) E(x) =E where

˜data (x) = Edata (f x ), E ˜smoothness (x) = Esmoothness (f x ), E ˜visibility (x) = Evisibility (f x ). E

Let’s consider each term separately, and show that each satisﬁes condition (3). 1. Data term. ˜data (x) = D(p, q) = T (f x (p) = f x (q) = l) · D(p, q) E p,f x (p),q,f x (q)∈I

p,l,q,l∈I

where T (·) is 1 if its argument is true and 0 otherwise. Let’s consider a single term E p,q (xp , xq ) = T (f x (p) = f x (q) = l) · D(p, q). Two cases are possible: 1A. l =α. If at least one of the labels f (p), f (q) is not l then E p,q ≡ 0. Suppose that f (p) = f (q) = l. Then E(xp , xq ) is equal to D(p, q), if xp = xq = 0, and is zero otherwise. We assumed that D(p, q) is non-positive, thus, condition (3) holds: E p,q (0, 0) + E p,q (1, 1) = D(p, q) ≤ 0 = E p,q (0, 1) + E p,q (1, 0) 1B. l = α. If at least one of the labels f (p), f (q) is α then E p,q depends only on at most one variable so condition (3) holds. (Suppose, for example, that f (q) = α, then f x (q) = α for any value of xq and, thus, E p,q (xp , 0) = E p,q (xp , 1) and E p,q (0, 0) + E p,q (1, 1) = E p,q (0, 1) + E p,q (1, 0).)

90

V. Kolmogorov and R. Zabih

Now suppose that both labels f (p), f (q) are not α. Then E(xp , xq ) is equal to D(p, q), if xp = xq = 1, and is zero otherwise. We assumed that D(p, q) is non-positive, thus, condition (3) holds: E p,q (0, 0) + E p,q (1, 1) = D(p, q) ≤ 0 = E p,q (0, 1) + E p,q (1, 0) 2. Smoothness term. ˜smoothness (x) = E

V{p,q} (f x (p), f x (q)).

{p,q}∈N

Let’s consider a single term E p,q (xp , xq ) = V{p,q} (f x (p), f x (q)). We assumed that V{p,q} is a metric; thus, V{p,q} (α, α) = 0 and V{p,q} (f (p), f (q)) ≤ V{p,q} (f (p), α)+ V{p,q} (α, f (q)), or E p,q (1, 1) = 0 and E p,q (0, 0) ≤ E p,q (0, 1) + E p,q (1, 0). Therefore, condition (3) holds. 3. Visibility term. ˜visibility (x) = ∞ E p,f x (p),q,f x (q)∈Ivis

=

T (f x (p) = lp ∧ f x (q) = lq ) · ∞.

p,lp ,q,lq ∈Ivis

Let’s consider a single term E p,q (xp , xq ) = T (f x (p) = lp ∧ f x (q) = lq ) · ∞. E p,q (0, 0) must be zero since it corresponds to the visibility cost of the initial conﬁguration and we assumed that the initial conﬁguration satisﬁes the visibility constraint. Also E p,q (1, 1) is zero (if xp = xq = 1, then f x (p) = f x (q) = α and, thus, the conditions f x (p) = lp and f x (q) = lq cannot both be true since Ivis includes only pairs of 3D-points at diﬀerent depths). Therefore, condition (3) holds since E p,q (0, 1) and E p,q (1, 0) are non-negative.

5

Experimental Results

Some details of our implementation are below. The planes for labels were chosen to be orthogonal to the main axis of the middle camera, and to increase with depth. The labels are distributed so that disparities depend on labels approximately linearly. The depth diﬀerence between planes was chosen in such a way that a label change by one results in disparity change by at most one both in x and y directions. We selected the labels α in random order, and kept this order for all iterations. We performed three iterations. (The number of iterations until convergence was at most ﬁve but the result was practically the same). We started with an initial solution in which all pixels are assigned to the label having the largest depth. For our data term D we made use of the method of Birchﬁeld and Tomasi [4] to handle sampling artifacts, with a slight variation: we compute intensity intervals for each band (R,G,B) using four neighbors, and then take the average

Multi-camera Scene Reconstruction via Graph Cuts

91

data penalty. (We used color images; results for grayscale images are slightly worse). The constant K for the data term was chosen to be 30. We used a simple Potts model for the smoothness term: lq U{p,q} if lp = (4) V{p,q} (lp , lq ) = 0 otherwise The choice of U{p,q} was designed to make it more likely that a pair of adjacent pixels in one image with similar intensities would end up with similar disparities. For pixels {p, q} ∈ N the term U{p,q} was implemented as an empirically selected decreasing function of ∆I(p, q) as follows: (5) U{p,q} = 3λ if ∆I(p, q) < 5 λ otherwise where ∆I(p, q) is the average of values |Intensity(p) − Intensity(q)| for all three bands (R,G,B). Thus, the energy depends only on one parameter λ. For diﬀerent images we picked λ empirically. We performed experiments on three datasets: the ’head and lamp’ image from Tsukuba University, the ﬂower garden sequence and the Dayton sequence. We used 5 images for the Tsukuba dataset (center, left, right, top, bottom), 8 images for the ﬂower garden sequence and 5 images for the Dayton sequence. For the Tsukuba dataset we performed two experiments: in the ﬁrst one we used four pairs of interacting cameras (center-left, center-right, center-top, center-bottom), and in the second one we used all 10 possible pairs. For the ﬂower garden and Dayton sequences we used 7 and 4 interacting pairs, respectively (only adjacent cameras interact). The table below show image dimensions and running times obtained on 450MHz UltraSPARC II processor. We used the max ﬂow algorithm of [5], which is speciﬁcally designed for the kinds of graphs that arise in vision. Source code for this algorithm is available on the web from http://www.cs.cornell.edu/People/vnk. dataset Tsukuba Tsukuba Flower garden Dayton

number of number of image number of images interactions size labels 5 4 384 x 288 16 5 10 384 x 288 16 8 7 352 x 240 16 5 4 384 x 256 16

running time 369 secs 837 secs 693 secs 702 secs

We have computed the error statistics for the Tsukuba dataset, which are shown in the table below. We determined the percentage of the pixels where the algorithm did not compute the correct disparity (the “Errors” column), or a disparity within ±1 of the correct disparity (“Gross errors”). For comparison, we have included the results from the best known algorithm for stereo reported in [27], which is the method of [7]. The images are shown in 2, along with the areas where we diﬀer from ground truth (black is no diﬀerence, gray is a diﬀerence of ±1, and white is a larger diﬀerence). Inspecting the image shows that we in general achieve greater accuracy at discontinuities; for example, the camera in the background and the lamp

92

V. Kolmogorov and R. Zabih

are more accurate. The major weakness of our output is in the top right corner, which is an area of low texture. The behavior of our method in the presence of low texture needs further investigation. Errors Gross errors 4 interactions 6.13% 2.75% 10 interactions 4.53% 2.30% 3.99% Boykov-Veksler-Zabih [7] 9.76% We have also experimented with the parameter sensitivity of our method for the Tsukuba dataset using 10 interactions. Since there is only one parameter, namely λ in equation 5, it is easy to experimentally determine the algorithm’s sensitivity. The table below shows that our method is relatively insensitive to the exact choice of λ. λ 2 5 10 20 40 Error 4.67% 4.53% 5.28% 5.78% 8.27% Gross errors 2.43% 2.30% 2.30% 2.23% 2.86%

6

Extensions

It would be interesting to extend our approach to more general camera geometries, for example the case when cameras look at the object in the center from all directions. Labels could then correspond to spheres. More precisely, there would be two labels per sphere corresponding to the closest and farthest intersection points. Acknowledgements. This research was supported by NSF grants IIS-9900115 and CCR-0113371, and by a grant from Microsoft Research. We also thank Rick Szeliski for providing us with some of the imagery used in section 5. Appendix: Edges for Terms E i,j In this section we show how to add edges for a term E i,j in the equation 2 assuming that condition (3) holds. This is a special case of a more general construction given in [15]. – if E(1, 0) > E(0, 0) then we add an edge (s, vi ) with the weight E(1, 0) − E(0, 0), otherwise we add an edge (vi , t) with the weight E(0, 0) − E(1, 0); – if E(1, 0) > E(1, 1) then we add an edge (vj , t) with the weight E(1, 0) − E(1, 1), otherwise we add an edge (s, vj ) with the weight E(1, 1) − E(1, 0); – the last edge that we add is (vi , vj ) with the weight E(0, 1) + E(1, 0) − E(0, 0) − E(1, 1). We have omitted indices i, j in E i,j for simplicity of notation. Of course it is not necessary to add edges with zero weights. Also, when several edges are added from one node to another, it is possible to replace them with one edge with the weight equal to the sum of weights of individiual edges.

Multi-camera Scene Reconstruction via Graph Cuts

93

Center image

Ground truth

Our results - 4 interactions

Our results - 10 interactions

Boykov-Veksler-Zabih results [7]

Comparison of our results with ground truth

Fig. 2. Results on Tsukuba dataset.

Appendix: Minimizing Our Energy Function Is NP-Hard It is shown in [7] that the following problem, referred to as Potts energy minimization, is NP-hard. We are given as input a set of pixels S with a neighborhood system N ⊂ S × S, and a set of label values L and a non-negative function C : S × L → + . We seek the labeling f : S → Lthat minimizes EP (f ) = C(p, f (p)) + T (f (p) = f (q)). (6) p∈P

{p,q}∈N

94

V. Kolmogorov and R. Zabih

Middle image

Our results

Fig. 3. Results on the ﬂower garden sequence.

Middle image

Our results

Fig. 4. Results on the Dayton sequence.

We now sketch a proof that an arbitrary instance of the Potts energy minimization problem can be encoded as a problem minimizing the energy E deﬁned in equation 1. This shows that the problem of minimizing E is also NP-hard. We start with a Potts energy minimization problem consisting of S, V, N and C. We will create a new instance of our energy minimization problem as follows. There will be |L| + 1 images. The ﬁrst one will be the original image S, and for each each label l ∈ L there will be an image Pl . For each pixel p ∈ S there will be a pixel in Pl which we will denote as pl ; thus, |Pl | = |S| for all labels l ∈ L. The set of labels L, the neighborhood system N and the smoothness function V{p,q} will be the same as in the Potts energy minimization problem: V{p,q} (lp , lq ) = T (lp = lq ),

{p, q} ∈ N

The set of interactions I will include all pairs {p, l, pl , l}, where p ∈ S, l ∈ L. The corresponding cost will be D(p, pl ) = C(p, l) − Kp where Kp is a constant which ensures that all costs D(p, pl ) are non-positive (for example, Kp = maxl∈L C(p, l)). There will be no visibility term (i.e. the set of

Multi-camera Scene Reconstruction via Graph Cuts

95

interactions Ivis will be empty). The instance of our minimization problem is now completely speciﬁed. Any labeling f : S → Lcan be extended to the labeling f¯ : P → Lif we set f¯(pl ) = l, pl ∈ Pl . It is easy to check that E(f¯) = EP (f ) − K where K = p∈S Kp is a constant. Moreover, there is no labeling f¯ : P → L with the smaller value of the energy E such that f¯ (p) = f¯(p) for all pixels p ∈ S. Thus, the global minimum f¯ of the energy E will yield the global minimum of the Potts energy EP (we need to take the restriction of f¯ on S).

References 1. Ravindra K. Ahuja, Thomas L. Magnanti, and James B. Orlin. Network Flows: Theory, Algorithms, and Applications. Prentice Hall, 1993. 2. Stephen Barnard. Stochastic stereo matching over scale. International Journal of Computer Vision, 3(1):17–32, 1989. 3. S. Birchﬁeld and C. Tomasi. Multiway cut for stereo and motion with slanted surfaces. In International Conference on Computer Vision, pages 489–495, 1999. 4. Stan Birchﬁeld and Carlo Tomasi. A pixel dissimilarity measure that is insensitive to image sampling. IEEE Transactions on Pattern Analysis and Machine Intelligence, 20(4):401–406, April 1998. 5. Yuri Boykov and Vladimir Kolmogorov. An experimental comparison of mincut/max-ﬂow algorithms for energy minimization in computer vision. In International Workshop on Energy Minimization Methods in Computer Vision and Pattern Recognition, volume 2134 of LNCS, pages 359–374. Springer-Verlag, September 2001. 6. Yuri Boykov, Olga Veksler, and Ramin Zabih. Markov Random Fields with efﬁcient approximations. In IEEE Conference on Computer Vision and Pattern Recognition, pages 648–655, 1998. 7. Yuri Boykov, Olga Veksler, and Ramin Zabih. Fast approximate energy minimization via graph cuts. IEEE Transactions on Pattern Analysis and Machine Intelligence, 23(11):1222–1239, November 2001. 8. R. Cipolla and A. Blake. Surface shape from the deformation of apparent contours. International Journal of Computer Vision, 9(2):83–112, November 1992. 9. O.D. Faugeras and R. Keriven. Complete dense stereovision using level set methods. In European Conference on Computer Vision, 1998. 10. L. Ford and D. Fulkerson. Flows in Networks. Princeton University Press, 1962. 11. S. Geman and D. Geman. Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 6:721–741, 1984. 12. H. Ishikawa and D. Geiger. Occlusions, discontinuities, and epipolar lines in stereo. In European Conference on Computer Vision, pages 232–248, 1998. 13. S.B. Kang, R. Szeliski, and J. Chai. Handling occlusions in dense multi-view stereo. In IEEE Conference on Computer Vision and Pattern Recognition, 2001. Expanded version available as MSR-TR-2001-80.

96

V. Kolmogorov and R. Zabih

14. Vladimir Kolmogorov and Ramin Zabih. Visual correspondence with occlusions using graph cuts. In International Conference on Computer Vision, pages 508–515, 2001. 15. Vladimir Kolmogorov and Ramin Zabih. What energy functions can be minimized via graph cuts? In European Conference on Computer Vision, 2002. Also available as Cornell CS technical report CUCS-TR2001-1857. 16. K.N. Kutulakos and S.M. Seitz. A theory of shape by space carving. International Journal of Computer Vision, 38(3):197–216, July 2000. 17. A. Laurentini. The visual hull concept for silhouette-based image understanding. IEEE Transactions on Pattern Analysis and Machine Intelligence, 16(2):150–162, February 1994. 18. W.N. Martin and J.K. Aggarwal. Volumetric descriptions of objects from multiple views. IEEE Transactions on Pattern Analysis and Machine Intelligence, 5(2):150– 158, March 1983. 19. Tomaso Poggio, Vincent Torre, and Christof Koch. Computational vision and regularization theory. Nature, 317:314–319, 1985. 20. S. Roy. Stereo without epipolar lines: A maximum ﬂow formulation. International Journal of Computer Vision, 1(2):1–15, 1999. 21. S. Roy and I. Cox. A maximum-ﬂow formulation of the n-camera stereo correspondence problem. In International Conference on Computer Vision, 1998. 22. Daniel Scharstein and Richard Szeliski. A taxonomy and evaluation of dense twoframe stereo correspondence algorithms. Technical Report 81, Microsoft Research, 2001. To appear in IJCV. An earlier version appears in CVPR 2001 Workshop on Stereo Vision. 23. S.M. Seitz and C.R. Dyer. Photorealistic scene reconstruction by voxel coloring. International Journal of Computer Vision, 35(2):1–23, November 1999. 24. Dan Snow, Paul Viola, and Ramin Zabih. Exact voxel occupancy with graph cuts. In IEEE Conference on Computer Vision and Pattern Recognition, pages 345–352, 2000. 25. R. Szeliski. Rapid octree construction from image sequences. Computer Vision, Graphics and Image Processing, 58(1):23–32, July 1993. 26. R. Szeliski and P. Golland. Stereo matching with transparency and matting. In International Conference on Computer Vision, pages 517–523, 1998. 27. Richard Szeliski and Ramin Zabih. An experimental comparison of stereo algorithms. In B. Triggs, A. Zisserman, and R. Szeliski, editors, Vision Algorithms: Theory and Practice, number 1883 in LNCS, pages 1–19, Corfu, Greece, September 1999. Springer-Verlag.

A Markov Chain Monte Carlo Approach to Stereovision Julien S´en´egas12 1 2

Centre de G´eostatistique, Ecole des Mines de Paris, 35 rue Saint-Honor´e, 77305 Fontainebleau Cedex, France Istar, 2600 route des Crˆetes, BP282 06905 Sophia-Antipolis Cedex, France [email protected]

Abstract. We propose Markov chain Monte Carlo sampling methods to address uncertainty estimation in disparity computation. We consider this problem at a postprocessing stage, i.e. once the disparity map has been computed, and suppose that the only information available is the stereoscopic pair. The method, which consists of sampling from the posterior distribution given the stereoscopic pair, allows the prediction of large errors which occur with low probability, and accounts for spatial correlations. The model we use is oriented towards an application to mid-resolution stereo systems, but we give insights on how it can be extended. Moreover, we propose a new sampling algorithm relying on Markov chain theory and the use of importance sampling to speed up the computation. The eﬃciency of the algorithm is demonstrated, and we illustrate our method with the computation of conﬁdence intervals and probability maps of large errors, which may be applied to optimize a trajectory in a three dimensional environment. Keywords: stereoscopic vision, digital terrain models, disparity, uncertainty estimation, sampling algorithms, Bayesian computation, inverse problems

1

Introduction

In this paper, we address the problem of assessing the uncertainty of depth estimates computed from a stereo system. We exclude the possibility of using any external source of information on the observed scene, i.e. only the radiometric information provided by the stereoscopic pair will be used. The application we have in mind is the certiﬁcation of digital terrain models for the aircraft industry, but the methods we propose here are quite general and can be applied to any stereo system with some adaptations of the model used. 1.1

Errors in Stereovision

A stereo system consists of two cameras observing the same scene from two different points. With this set-up, a left and right image are obtained. A physical A. Heyden et al. (Eds.): ECCV 2002, LNCS 2352, pp. 97–111, 2002. c Springer-Verlag Berlin Heidelberg 2002

98

J. S´en´egas

point M projects onto the image plane of each camera. These projections form a pair of homologous points, and their coordinates in each image plane can be expressed as a function of the stereo system parameters (mainly the baseline b and the focal length f ) and the coordinates of M . The principle of depth estimation from stereovision relies on the inversion of this relation. An extensive introduction to three dimensional computer vision and depth from stereo can be found in [10]. Many diﬀerent techniques are used for depth computation, depending on the type of sensors and the nature of the scene. However, they generally consist of the following steps. First, the stereo system parameters are computed, using calibration methods (see [8] for an example). It is then possible to reproject the images in the epipolar geometry, in which two homologous points have the same vertical coordinate. The diﬀerence between their horizontal coordinates is called the disparity. Hence, the rectiﬁcation step reduces the search of homologous points to a mono-dimensional one (along the rows of the images). The stereo problem itself amounts then to the computation of the disparity map, which is done by matching pixels (or features) in the left image with the corresponding pixels in the right image [9]. Lastly, the computation of the real world coordinates, the triangulation, follows from the disparity map and the geometry of the stereo system. Uncertainty in depth estimates arises from errors or approximations in each step described above, however it is important to distinguish between these errors. Calibration, rectiﬁcation and triangulation rely on geometric relations veriﬁed by the stereo system. Even if these are only approximations of the true geometric constraints, the errors associated with them can be controlled and their range computed. For instance, quantization errors, which come from the knowledge of the image points coordinates at ﬁxed positions only, with a precision equal to half the sampling interval δ [20], depend only on the stereo system parameters. Possibly, the precision can be improved by modifying the stereo system: for example, an increase of the product bf enables to obtain more accurate depth estimates [20]. In opposition, matching errors are strongly dependent on the nature of the radiometric information, and cannot be entirely controlled by the design of the stereo system. The reason for this lies in the implicit matching assumption that pairs of homologous points can be identiﬁed. In many situations, violations of this assumption are encountered: the geometric deformations induced by the projections may be strong, the images are corrupted by sensor noise or the radiometric information itself may be ambiguous (existence of repetitive features, lack of texture in the images,...). As a result, matching errors are likely to be dependent on the speciﬁc features in the images and to vary strongly over the disparity map. 1.2

A Stochastic Approach

A convenient framework to handle uncertainty in disparity computation is to consider the disparity D and the stereoscopic pair Y = (I1 , I2 ), as random

A Markov Chain Monte Carlo Approach to Stereovision

99

functions. This approach has been successfully applied in a Bayesian context to compute disparity estimates [1]. However, to our knowledge, the problem of uncertainty assessment has been only partly handled, mostly through the computation of standard deviation [23,16]. Standard deviation estimates give only a rough idea of the range of possible errors, and are not appropriate to describe large errors, which appear with very low probability and are spatially correlated. Therefore, speciﬁc methods have to be proposed. Typically, if we consider an object in motion in a 3D environment, one must ﬁnd the path to a given target which minimizes the risk of collisions. Such problems require the computation of the whole conditional probability distribution πD (d|Y = y) of the disparity D given the stereoscopic pair Y . Note that once πD (d|Y = y) is known, we can easily compute the corresponding probability distributions for the real world coordinates, using the geometric transformation of the stereo system. This motivates the choice of the disparity as variable of interest. The main contribution of this paper is to propose a theoretically founded approach to the problem of uncertainty assessment in disparity computation. Especially, we show in Sect. 2 that this problem amounts to sample from the posterior distribution πD and to use Monte Carlo integrations. We also explicit the model πD used for computation, but point out that this one is very dependent on the application. We give insight on how this model can be adapted to speciﬁc applications. On the opposite, the sampling algorithms we propose in Sect. 3 are quite general and can be used for diﬀerent forms of probability models πD . The only restriction is a gaussian assumption for the prior model. Lastly, in Sect. 4, we illustrate our methods with an application to the study of a stereoscopic pair of SPOT images. In the sequel, we will assume that an estimate dˆ of the disparity has been already computed, for example with a correlation based algorithm. This means that the algorithms we propose in the sequel are not used for disparity computation, but only for uncertainty assessment. One reason is that Monte Carlo algorithms are computationally cumbersome. Another reason is that we are interested in the uncertainty which is intrinsic to the radiometric information. Indeed, we consider the uncertainty estimation mostly at a postprocessing stage, and would like to propose a method that is widely independent of the disparity computation itself.

2

Stochastic Framework: Stereovision as an Inverse Problem

Let Y = (I1 , I2 ) be a stereoscopic pair in the epipolar geometry. We consider a physical point M , and its projections m1 with coordinates (u1 , v1 ) in I1 and m2 , with coordinates (u2 , v2 ) in I2 . The epipolar constraint imposes v1 = v2 and we can deﬁne the disparity in the reference system of I1 as the diﬀerence: d(u1 , v1 ) = u2 − u1

(1)

100

J. S´en´egas

Therefore, a pair of homologous points is fully deﬁned by the coordinates (u, v) of m1 in I1 and the disparity d(u, v). Note that a symmetric deﬁnition of disparity in the cyclopean image is possible, see [1]. Although we proceed with deﬁnition 1, the approach we propose is readily applicable to this alternative deﬁnition. This purely geometric presentation however corresponds to the direct problem, where the coordinates of the homologous points are known. In practice, these are to be found, and generally a loss function Cu,v (d), which expresses the similarity of the images in the neighborhood of I1 (u, v) and I2 (u + d, v), is used [9]. The estimate dˆ of the “true” disparity is then taken to be the value which maximizes Cu,v (d), possibly under some constraints, i.e.: ˆ v) = arg max Cu,v (d) d(u, d

(2)

Therefore, when computing the disparity, and whatever method is used, one must move from a purely geometric deﬁnition 1 to an optimization criterion 2. In other words, disparity computation in stereovision is an inverse problem, and therefore, the result is only an estimate of the underlying disparity. This estimate is sensible to the amount of noise in the stereoscopic pair, but, even if perfect signal is assumed, uncertainty remains, because of the ambiguity of the information, due to occlusions, repetitive features, lack of texture in the images,... The Bayesian framework [3] is well adapted to inverse problems. Bayesian inference has been widely used in image analysis [2], and eﬃcient algorithms have been proposed to solve the restoration problem, as in [13]. More recently, this formalism has been applied to the stereo-reconstitution problem [23,1]. Let πD (d|Y = y) be the posterior (or conditional) probability of the disparity D given the stereoscopic pair Y = y. Using Bayes relation, this one can be expressed by means of the conditional distribution of Y given D = d, πY (y|D = d), and the marginal distributions πD (d) and πY (y). Since here Y is known and remains equal to y, πY (y) is a constant, and we can introduce the likelihood L(y|D = d) = πY π(y|D=d) , which can be computed up to a multiplicative conY (y) stant, and for clarity, we denote g(d) = πD (d) the prior distribution. With these notations, the posterior distribution writes: πD (d|Y = y) = L(y|D = d)g(d)

(3)

In stereovision, this formalism has been mostly applied to compute an estimate of the disparity, through maximum a posteriori (MAP) estimation [1], where the maximization of πD (d|Y = y) is used as criterion. However, here, we are interested in the computation of the whole posterior distribution πD (d|Y = y). Especially, for any πD −measurable function h, we would like to compute the integral: EπD (h) =

h(z)πD (z|Y = y)dz

(4)

With an appropriate choice of h, every probability concerning D can be computed. For example, with h(d) = 1|d−d|≥s where dˆ is an estimate of the disparity, ˆ

A Markov Chain Monte Carlo Approach to Stereovision

101

we obtain the probability that the estimation error D− dˆ is greater than a threshold s: ˆ ≥ s) = πD (z|Y = y)dz (5) P (|D − d| ˆ |z−d|≥s

In the following, we explicit the computation of 4 and derive an expression for πD . 2.1

MCMC Methods

In practice, the numerical computation of 4 is not possible since πD is a function of n variables, where n is the size of the image. Another reason is that πD is known only up to a multiplicative factor. A solution is provided by Monte Carlo integration which evaluates the integral in 4 by drawing samples {zi , i = 1, ..., m} from πD and then approximating m

EπD (h) ≈

1 h(zi ) m i=1

(6)

The laws of large numbers ensure that the approximation can be made as accurate as desired by increasing the sample size m. Monte Carlo methods were used in similar problems [11,7], but mostly to compute map estimates. We show in this paper that more information can be extracted from the posterior distribution. A method to draw samples from πD consists of building a Markov chain having πD as its stationary distribution. In practice, the problem amounts to determine a transition probability kernel which ensures the convergence to πD . This is the principle of the well known Metropolis-Hastings algorithm [15] or the popular Gibbs sampler [13]. For a detailed introduction to Markov chain theory, the interested reader is referred to [18] and [24]. Practical applications of Markov chain Monte Carlo can be found in [14]. As a conclusion, assessment of disparity uncertainty and disparity estimation are two very diﬀerent problems, and the former requires the development of speciﬁc methods. Uncertainty can be assessed under a Bayesian framework by drawing samples from the posterior distribution of the disparity given the stereoscopic pair. Two points remain to be solved: the speciﬁcation of the model, which is treated in Sect. 2.2, and the choice of a sampling algorithm, which is the subject of Sect. 3. 2.2

Model Derivation

We need to specify the likelihood term L(y|D = d), which describes the relation between the images of the pair at ﬁxed disparity, and the prior model g, which reﬂects the spatial structure of the disparity. Let us point out that the choice of the probability model πD is speciﬁc to the application, i.e. to the type of images recorded and the nature of the scene observed. In our case, we use

102

J. S´en´egas

stereoscopic pairs of SPOT images for the computation of mid-resolution digital terrain models (with a ground resolution approximately equal to 10 m). We ﬁrst derive an expression for the image model (likelihood) and then for the disparity model (prior). A common modeling for the likelihood [1] consists of assuming that the recorded intensity I for a speciﬁc feature is the same for both images I1 and I2 and sensor noise is additive. Therefore, for ﬁxed disparity d, the relation between I1 and I2 writes: I1 (u, v) − I2 (u + d(u, v), v) = η(u, v)

(7)

where η(u, v) can be interpreted as the residual between the intensities after a correction on the coordinates with the disparity. The choice of the image model amounts hence to the modeling of η, and especially its spatial correlation. If we assume the residual η to be uncorrelated gaussian noise with mean µ and standard deviation κ, as in [4,6,1], we end up with the following expression of the likelihood:   (ηij − µ)2  L(y|D = d) ∝ exp − (8) 2κ2 i,j The assumption of uncorrelated noise η may be questioned. Especially, one may argue that if the deformations induced by the stereo system are relatively important, the residual η will show strong patterns. Therefore, this point is very dependent on the setting of the stereo system and the nature of the scene. For satellite images at mid-resolution, these deformations may be neglected in a ﬁrst approach, and the assumption of uncorrelated gaussian noise is reasonable. However, this is deﬁnitely a challenging aspect of the problem. Note that, as we have an estimate dˆ of the disparity, we can compute from I1 ˆ v), v). Hence, and I2 an estimate of the residual ηˆ(u, v) = I1 (u, v) − I2 (u + d(u, the model parameters µ and κ can be estimated by maximum likelihood. For the disparity model, we adopt a gaussian random function framework, which is relatively common [23,4,1]. Under the gaussian assumption, the prior distribution is entirely speciﬁed by the mean m and the spatial covariance C. ˆ for example under These parameters can be estimated from the disparity map d, an assumption of spatial stationarity [5]. Finally, the prior distribution g takes the following form: 1 −1 g(d) ∝ exp − (d − m) C (d − m) (9) 2 Here again, let us point out that the choice of the prior model is dependent on the application. Especially, the behavior of the covariance at the origin will reﬂect the smoothness of the disparity: for example, a model with a linear behavior (such as Brownian motion [1]) is liable to reproduce discontinuities. Our experience is that covariances which are diﬀerentiable at the origin (such as the

A Markov Chain Monte Carlo Approach to Stereovision

103

cubic covariance) are well adapted in the case of mid-resolution stereo systems. However, for high-resolution applications, such as urban or indoor scenes, other models may be chosen. For example, in [1], Belhumeur proposes to model directly the discontinuities at the edges of objects through Poisson processes. More generally, one may argue that for such applications, stochastic geometric models, which directly take into account the form of objects, may be better adapted. Such models were applied to building detection [12] and road extraction [22]. Let us point out that, although we made the gaussian assumption for the prior distribution g, the posterior distribution is not necessarily gaussian. The nature of the posterior distribution depends largely on the likelihood term L, which is not a quadratic form of d. For example, the presence of repetitive features in the stereoscopic pair would result in a multimodal posterior distribution.

3

Sampling Algorithm

In this section we expose the details of the sampling algorithm. We recall that the probability distribution to be sampled from is of the form: π(z|Y = y) = L(y|Z = z)g(z)

(10)

where L(y|Z = z) is the likelihood function, which may be known up to multiplicative factor, and g(z) is the prior distribution. 3.1

Markov Chain Sampling Algorithms

As stated previously, iterative Markov chain algorithms have to be used. In our case, the diﬃculty lies in the form of the likelihood term L(Y |D = d), which cannot be approximated eﬃciently and is expected to vary strongly with the coordinates (u, v). The Gibbs sampler, which in combination with Markov random ﬁelds models is very popular in image analysis, relies on the use of conditional densities of the form π(zS |z−S ), where S is a subspace of the grid to be simulated and −S its complementary part [3]. Unfortunately, because of the expression of the distribution πD , it is not possible to sample directly from the conditional distributions. Other alternatives are Metropolis-Hastings algorithms [14] and variants such as discretized Langevin diﬀusions [19]. The principle of the Metropolis-Hastings algorithm is to generate a transition kernel q(z, from the current state z to the state z from a proposal transition

z ), )q(z ,z) and to accept this transition with probability a(z, z ) = min 1, π(z π(z)q(z,z ) . The eﬃciency of this algorithm relies merely on the proposal transition kernel q, but when the target distribution π cannot be easily approximated, optimal choices of q are not possible, and very often an ad hoc transition kernel is used, like in the independence sampler or the random walk [14].

104

3.2

J. S´en´egas

A New Sampling Algorithm

These diﬃculties have led us to propose a new sampling algorithm which makes use of the target distribution π to generate the transitions. Let us assume that the prior distribution g(z) denotes now a gaussian distribution. The algorithm is based on the following remark. Let W1 and W2 be two independent gaussian processes with zero mean and covariance C (which we denote W1 , W2 ∼ g). Then, for any θ ∈ [0, 2π[, the linear combination W (θ) = W1 cos θ + W2 sin θ is also a gaussian process with zero mean and covariance C. We propose to use this relation, together with an appropriate choice of θ, to build the transitions of a Markov chain Z = (Zk , k ≥ 0). The algorithm is the following: 1. sample w ∼ g, 2. compute the proposal transitions: 2π 2π + w sin i , zi = zk cos i m m

∀i = 0, m − 1

(11)

and the corresponding transition probabilities: L(y|Z = zi ) a(z, zi ) = m−1 , j=0 L(y|Z = zj )

∀i = 0, m − 1

(12)

3. choose zi ∼ a(z, zi ) and set zk+1 = zi . We prove in [21] that the Markov chain built according to this algorithm has π as stationary distribution. A comparison study of this algorithm with the independence sampler and the Langevin diﬀusion can be found in [21]. The interesting feature of this algorithm is that at each iteration, a whole path of possible transitions is proposed, in opposition to a single one in the Metropolis-Hastings algorithm, and this is done at a low computational cost since it only requires the computation of linear combinations of the form 11. Therefore, transitions are likely to occur more often, which improves mixing within the chain and speeds up the convergence. An illustration of this feature is displayed on Fig. 1. 3.3

Size Reduction and Importance Sampling

For large grids, the computational burden of Markov chain algorithms can become very cumbersome. Gaussian samples can be generated very quickly (with algorithms such as those relying on the Fast Fourier Transform [5]), but the transition rate of the Markov chain tends to decrease drastically with the grid size, so that the sampling algorithm must be run longer to achieve convergence. To cope with this diﬃculty, we propose to split the sampling in two consecutive phases. Consider a subgrid S of the total sampling grid G and its complementary −S in

1.0

A Markov Chain Monte Carlo Approach to Stereovision

105

0.05 0.06

0.5

0.07

0.12

0.11 0.1 0.09 0.08

y 0.0

0.08 0.07 0.06

-0.5

0.05 0.04 0.03

-1.0

0.02 0.01

-1.0

-0.5

0.0 x

0.5

1.0

Fig. 1. Path of possible transitions generated from the current point and an independent simulation (black dots). The target distribution, displayed in the background with level lines, is a bigaussian distribution. The circles along the path, which correspond each to a diﬀerent value of θ, are proportional to the transition probability a(z, zi ).

G. zS denotes the restriction of the vector z to S. The posterior distribution π can be decomposed as follows: π(z|Y = y) =

L(y|Z = z) g(z−S |ZS = zS )L(y|ZS = zS )g(zS ) L(y|ZS = zS )

(13)

The term L(y|ZS = zS )g(zS ) is the posterior distribution π(zS |Y = y) on S, whereas g(z−S |ZS = zS ) is the conditional distribution of the gaussian vector z−S given zS (this is the conditional distribution which is used in the Gibbs sampler). Sampling from this conditional distribution can be done directly and very eﬃciently using an algorithm such as conditioning by kriging [5]. The ﬁrst L(y|Z=z) term w = L(y|Z is a correction factor which shall be used to weight the S =zS ) samples in the Monte Carlo integration. We end up with the following algorithm: 1. run the Markov chain algorithm on the subgrid S, i 2. for each generated sample zSi , generate z−S using the conditional gaussian i ) the sample on distribution g(z−S |ZS = zS ), and construct zi = (zSi , z−S the total grid G, L(y|Z=zi ) 3. compute the weight wi = L(y|Z i , S =zS ) 4. compute the integral 1 Eπ (h) ≈ m

i=1

m

wi

wi h(zi ) .

(14)

i=1

The last step is an application of importance sampling theory [14]. One may run the risk that all weights wi but a few become equal to 0, and only a few samples take the total weight. This could happen if the distribution used for sampling π (z|Y = y) = π(zS |Y = y)g(z−S |ZS = zS ) and the target distribution

106

J. S´en´egas

π(z|Y = y) are very diﬀerent. In our case, this should be avoided due to the strong spatial correlations exhibited by the disparity D. The idea is that if we know the disparity on a subgrid S, then we are able to compute accurate estimates of D on the total grid G, and correct ﬂuctuations can be reproduced by sampling from the conditional distribution g(z−S |ZS = zS ). An overview of the sampling algorithm is shown on Fig. 2. Markov chain

Importance

Monte Carlo

sampling

sampling

integration

Left

Probability

Image

Σ

map

Stereoscopic pair

Right Image

Fig. 2. Overview of the sampling algorithm. First, the Markov chain sampling algorithm is run on a sub-grid, conditional to the stereoscopic pair. Then, for each generated sample, the whole grid is generated. Lastly, the Monte Carlo integration is performed.

4

Experimental Results

We have applied the model of Sect. 2.2 and the sampling algorithms introduced in Sect. 3 to the study of a stereoscopic pair of rectiﬁed SPOT images of the Marseille area, of size 512 × 512 pixels (Fig. 3). A disparity map (Fig. 3) has been computed using a correlation based algorithm. 2000 samples have been generated. Using the importance sampling algorithm, the size of the sampling grid has been reduced to 64 × 64, yielding thus a size reduction factor equal to 64. To illustrate the eﬃciency of the sampling algorithms, we run a Metropolis-Hastings algorithm (denoted as MH) on the total grid, then the sampling algorithm of Sect. 3 (denoted TCP for transitions along continuous paths), ﬁrst on the total grid, and then on the reduced grid with the importance sampling method (denoted further TCPIS). We compared the computation time required to generate 100 samples with at least 10 state transitions between two samples, assuming that these conditions ensure similar convergence properties. Computations were performed on a SUN Ultra 60 computer with a 400 Mhz ULTRA SPARC II processor. The results are shown on Tab. 1. The TCP algorithm allows to reduce the computation time by a factor 10 in comparison to the independence Metropolis-Hastings sampler. The size reduction coupled with

A Markov Chain Monte Carlo Approach to Stereovision

(a) One image of the stereoscopic pair

107

(b) Disparity

Fig. 3. One rectiﬁed image of the stereoscopic pair (Marseille area, France) and the corresponding disparity (right) Table 1. Comparison of computation time for MH, TCP and TCPIS. MH TCP TCPIS CPU time (sec.) 642571 55454 1326

the importance sampling algorithm (TCPIS) allows to reduce again the computation time, by a factor 40. The overall computation time is still relatively large (a few hours for our case study), but the TCPIS algorithm allows a tremendous gain in comparison with crude sampling algorithms. The application we aim at is the detection of possible large errors; that is, given an error threshold s, which are the pixels for which disparity errors larger than s are likely to occur? This is typically the type of questions one may face in applications in the aircraft industry. This problem can be formulated as the computation of the probability P (D− dˆ ≥ s), and, for a given risk α, the location ˆ v) ≥ s) ≤ α, which correspond to of the pixels (u, v) for which P (D(u, v) − d(u, the safe area. Figure 4 displays the probability maps obtained for s = 2 and s = 3. The probability of errors varies strongly with the location (u, v), that is why usual methods estimating error ranges over a large domain are not appropriate for such applications. It is then possible to threshold these maps to a given risk value, say 0.1%, to identify the safe areas. In real applications however, it is not realistic to consider that such information is required for every pixel. For example, if one thinks of an object in motion over a domain, then one must guarantee that no collision occurs along

108

J. S´en´egas

Fig. 4. Probability map of positive errors larger than 2 (left) and 3 (right) pixels. One scale unit on the vertical axis equals 10%.

its trajectory. Hence, we need to consider a whole domain, and compute the probability that no error greater than a given threshold s occurs in this domain, with a probability 1 − α. Since errors are spatially correlated, this is not equal to the product of the corresponding probabilities over the domain. We point out that this type of computation deﬁnitely requires the knowledge of the entire multivariate distribution πD and not only the marginal distribution of a given pixel. For this purpose, we divided the disparity map in 64 sub-domains of equal surface, and computed the probability that errors larger than 3 (respectively 4) pixels occur anywhere in the sub-domain (Fig. 5). Thus, as previously, it is possible to determine the safe areas as the sub-domains for which this probability is lower than α = 0.1%. The results obtained are rather pessimistic: the 3 pixel threshold map has only 2 safe domains, this contrasts with the situation we would have expected from the right map displayed on Fig. 4. This surprising result emphasizes the importance of spatial correlations and justiﬁes the use of Monte Carlo methods, although they are computationally demanding. We consider now the opposite situation, which is often encountered in practice: a risk value α is given, and one needs to compute a conﬁdence interval [Zinf , Zsup ] for the variable Z such that P (Z ∈ [Zinf , Zsup ]) ≥ 1 − α. An example of the use of conﬁdence intervals in a diﬀerent context of computer vision can be found in [17]. To derive a conﬁdence interval, we propose to make use of the min and max of the disparity samples. Indeed, for n + 1 independent identically distributed random variables Zi , we have the following result: P (Zn+1 ∈ [min Zi , max Zi ]) = i≤n

i≤n

n−1 n+1

(15)

A Markov Chain Monte Carlo Approach to Stereovision

109

Fig. 5. Probability map of positive errors larger than 3 (left) and 4 (right) pixels in a subdomain. One scale unit on the vertical axis equals 10%.

Therefore, for any risk value α, we can choose the smallest integer n such that n−1 n+1 ≥ 1 − α and compute for each pixel the min and the max of the n samples. The conﬁdence interval has then the form of two embedding surfaces. In order to validate our results, we have used a reference disparity map obtained from high resolution images, which has been degraded to the SPOT resolution. Using this reference map, it is possible to compute the percentage of points lying in the conﬁdence interval computed previously and to compare this value to the theoretical value given by 15. The risk value was set equal to 0.1%, and therefore we used n = 2000 samples to construct the conﬁdence interval. From the reference map, we found that 0.11% of the total points fell outside the conﬁdence interval, which is in very good agreement with the theoretical value 0.1%. Moreover, the mean size over the grid of the computed disparity interval is 4.4 pixels: although the risk value is very small, the precision given by the conﬁdence interval is reasonable. Therefore, the method proposed in this paper allows to compute accurate statistics.

5

Conclusion

In this paper, we have considered the problem of assessing the uncertainty of a disparity map with only the use of the stereoscopic pair as information. Since the usual standard deviation approach does not permit to handle the case of large errors which occur with very low probability and are spatially correlated, speciﬁc methods have to be proposed. A solution consists of computing the posterior distribution of the disparity given the stereoscopic pair. Hence, uncertainty assessment amounts to sample from this posterior distribution and to compute the desired probabilities using a Monte Carlo approach. From a

110

J. S´en´egas

practical point of view, a model for the posterior distribution must be chosen, and an eﬃcient sampling algorithm must be used to keep the computation time reasonable. This method is very general and can be applied to diﬀerent types of stereovision systems, but the choice of the stochastic model is speciﬁc to the application. The model explicited in this paper considers an application to the computation of mid-resolution digital terrain models, for which the disparity map can be assumed to vary relatively smoothly and the gaussian assumption is reasonable. For more complex applications, models which take explicitly into account the geometry of the scene may be more appropriate. An important problem is the choice of the sampling algorithm. The algorithm we propose considers the case of gaussian priors, which is quite general. It is based on the construction of a continuous path of possible transitions, and compared to standard sampling algorithms such as the independence sampler proves very competitive. Moreover, we have shown that the computation can be tremendously speeded up by ﬁrst running the Markov chain on a subgrid and then generating conditionally the samples on the whole grid. This Markov chain Monte Carlo approach has been applied to the study of a stereoscopic pair of SPOT images. The results show that the disparity statistics are highly non-stationary. This is due to the nature of the radiometric information itself. These statistics can be used to determine the safe areas, for which large errors occur with probability inferior to a given risk value, or to compute precise conﬁdence intervals. The range of applications is however brighter: for example, they may be used to optimize a functional over the domain under study. We think especially of the motion of an object in a three-dimensional environment, for which the path to a given target has to be optimized and collisions avoided.

References [1] P.N. Belhumeur. A Bayesian approach to binocular stereopsis. International Journal of Computer Vision, 19(3):237–262, 1996. [2] J. Besag. On the statistical analysis of dirty pictures (with discussion). J. Roy. Statist. Soc. Ser. B, 48:259–302, 1986. [3] J. Besag, P. Green, D. Higdon, and K. Mengersen. Bayesian computation and stochastic systems. Statistical Science, 10(1):3–66, 1995. [4] C. Chang and S. Chatterjee. Multiresolution stereo - A Bayesian approach. IEEE International Conference on Pattern Recognition, 1:908–912, 1990. [5] J.P. Chil`es and P. Delﬁner. Geostatistics: Modeling Spatial Uncertainty. Wiley and Sons, 1999. [6] I.J. Cox. A maximum likelihood n-camera stereo algorithm. International Conference on Computer Vision and Pattern Recognition, pages 733–739, 1994. [7] F. Dellaert, S. Seitz, S. Thrun, and C. Thorpe. Feature correspondance: A Markov chain Monte Carlo approach. In Advances in Neural Information Processing Systems 13, pages 852–858, 2000.

A Markov Chain Monte Carlo Approach to Stereovision

111

[8] F. Devernay and O. Faugeras. Automatic calibration and removal of distorsions from scenes of structured environments. In Leonid I. Rudin and Simon K. Bramble, editors, Investigate and Trial Image Processing, Proc. SPIE, volume 2567. SPIE, San Diego, CA, 1995. [9] U.R. Dhond and J.K. Aggarwal. Structure from stereo - A review. IEEE Transactions on systems, man and cybernetics, 19(6):1489–1510, 1989. [10] O. Faugeras. Three-Dimensional Computer Vision. Massachussets Institute of Technology, 1993. [11] D. Forsyth, S. Ioﬀe, and J. Haddon. Bayesian structure from motion. In International Conference on Computer Vision, volume 1, pages 660–665, 1999. [12] L. Garcin, X. Descombes, H. Le Men, and J. Zerubia. Building detection by markov object processes. In Proceedings of ICIP’01, Thessalonik, Greece, 2001. [13] S. Geman and D. Geman. Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 6(6):721–741, 1984. [14] W.R. Gilks, S. Richardson, and D.J. Spiegelhalter. Markov Chain Monte Carlo in Practice. Chapman and Hall, 1996. [15] W.K. Hastings. Monte carlo sampling methods using Markov chains and their applications. Biometrika, 57:97–109, 1970. [16] T. Kanade and M. Okutomi. A stereo matching algorithm with an adaptive window: Theory and experiment. IEEE Transactions on Pattern Analysis and Machine Intelligence, 16(9), september 1994. [17] R. Mandelbaum, G. Kamberova, and M. Mintz. Stereo depth estimation: a conﬁdence interval approach. In International Conference on Computer Vision, pages 503–509, 1998. [18] S.P. Meyn and R.L. Tweedie. Markov Chains and Stochastic Stability. SpringerVerlag London, 1993. [19] G.O. Roberts and J.S. Rosenthal. Optimal scaling of discrete approximations to Langevin diﬀusions. J. Roy. Statist. Soc. Ser. B, 60:255–268, 1998. [20] J.J. Rodriguez and J.K. Aggarwal. Stochastic analysis of stereo quantization error. IEEE Transactions on Pattern Analysis and Machine Intelligence, 12(5):467–470, 1990. [21] J. S´en´egas, M. Schmitt, and P. Nonin. Conditional simulations applied to uncertainty assessment in DTMs. In G. Foody and P. Atkinson, editors, Uncertainty in Remote Sensing and GIS. Wiley and Sons, 2002. to appear. [22] R. Stoica, X. Descombes, and J. Zerubia. Road extraction in remote sensed images using a stochastic geometry framework. In Proceedings of MaxEnt’00, Gif Sur Yvette, July 8-13, 2000. [23] R. Szeliski. Bayesian Modeling of Uncertainty in Low-level Vision. Kluwer Academic Publishers, 1989. [24] R.L. Tweedie. Markov chains: Structure and applications. In B. N. Shanbhag and C.R. Rao, editors, Handbook of Statistics 19, pages 817–851. Elsevier Amsterdam, 1998.

A Probabilistic Theory of Occupancy and Emptiness Rahul Bhotika1 , David J. Fleet2 , and Kiriakos N. Kutulakos3 1

Computer Science Department, University of Rochester Rochester, NY 14627, USA [email protected] 2 Palo Alto Research Center, Palo Alto CA 94304, USA [email protected] 3 Computer Science Department, University of Toronto Toronto, ON M5S3G4, Canada [email protected]

Abstract. This paper studies the inference of 3D shape from a set of n noisy photos. We derive a probabilistic framework to specify what one can infer about 3D shape for arbitrarily-shaped, Lambertian scenes and arbitrary viewpoint configurations. Based on formal definitions of visibility, occupancy, emptiness, and photo-consistency, the theoretical development yields a formulation of the Photo Hull Distribution, the tightest probabilistic bound on the scene’s true shape that can be inferred from the photos. We show how to (1) express this distribution in terms of image measurements, (2) represent it compactly by assigning an occupancy probability to each point in space, and (3) design a stochastic reconstruction algorithm that draws fair samples (i.e., 3D photo hulls) from it. We also present experimental results for complex 3D scenes.

1

Introduction

Many applications, from robotics [1] to computer graphics [2] and virtual reality, rely on probabilistic representations of occupancy to express uncertainty about the physical layout of the 3D world. While occupancy has an intuitive and well-defined physical interpretation for every 3D point (i.e., a point is occupied when the scene’s volume contains it), little is known about what probabilistic information regarding occupancy is theoretically computable from a set of n noisy images. In particular, what information about the occupancy and emptiness of space is intrinsic to a set of n stochasticallygenerated photos taken at arbitrary viewpoints? Importantly, how can we answer this question for general scenes whose shape is arbitrary and unknown? This paper outlines a general probabilistic theory of occupancy and emptiness for Lambertian scenes that is intended to handle arbitrary 3D shapes, arbitrary viewpoint configurations and noisy images. Based on formal definitions of several key concepts, namely, visibility, occupancy, emptiness and photo-consistency, our theoretical development yields a formulation of the Photo Hull Distribution. The Photo Hull Distribution is the tightest probabilistic bound on the scene’s true shape that can be inferred from the n input photos. Importantly, we show that the theory leads directly to a stochastic algorithm that (1) draws fair samples from the Photo Hull Distribution and (2) converges A. Heyden et al. (Eds.): ECCV 2002, LNCS 2352, pp. 112–130, 2002. c Springer-Verlag Berlin Heidelberg 2002

A Probabilistic Theory of Occupancy and Emptiness L

R

L

R

1 3

2 4

7

1 6

2

3

5

4

(a)

113

(b)

Fig. 1. (a) Noiseless images of four Lambertian points with identical albedos afford seven consistent scene interpretations—{2,3},{1,4} {1,2,3}, {1,2,4}, {1,3,4},{2,3,4}, and {1,2,3,4}. Maximizing image consistency in the presence of even small amounts of noise will cause one of these interpretations to be arbitrarily preferred. The resulting shape (e.g., {2, 3}), however, cannot conclusively rule out the possibility that other points may also be occupied (e.g., {1}). (b) The probabilistic event “5 is visible from L” implies that points 6 and 7 are in empty space, thereby inducing a cycle of probabilistic dependencies (arrows) that make the events “4 is visible from L” and “5 is visible from L” dependent on each other. Such cycles characterize the visibility and occupancy dependencies between every pair of points in IR3 within the cameras’ view.

to an optimal conservative estimate of the occupancy probability, i.e., to the lowest upper bound on that probability that is theoretically computable from the photos. Experimental results illustrating the applicability of our framework are given for real scenes. Our work is motivated by four objectives that we believe are key to designing a general and useful probabilistic theory of occupancy and emptiness: – Understanding stereo ambiguities: Since stereo reconstruction can be ambiguous even in the noiseless case, a clear understanding of the mathematical relationship between shape ambiguity and shape uncertainty, and their effect on occupancy computation, becomes crucial in the design of stable algorithms (Figure 1a). – Probabilistic treatment of visibility: Unconstrained scenes induce arbitrarily complex visibility relationships between cameras and surface points, and arbitrarily long probabilistic dependency cycles between any two 3D points (Figure 1b). One must provide a physically-valid and self-consistent probabilistic interpretation of occupancy (which refers to individual points) and visibility (which depends on a scene’s global shape). – Algorithm-independent analysis of occupancy: Without a clear understanding of what is theoretically computable from a set of input images, it is difficult to assess the performance of individual algorithms and the effect of heuristics or approximations they employ. – Handling sensor and model errors: Since computational models are inherently limited in their ability to describe the physical world and the true image formation process, it is important to account for such errors. A key aspect of our analysis is the explicit distinction we make between the notions of ambiguity and uncertainty. Intuitively, we interpret shape ambiguity as the multiplicity of solutions to the n-view reconstruction problem given noiseless images. Shape

114

R. Bhotika, D.J. Fleet, and K.N. Kutulakos

uncertainty, on the other hand, exists in the presence of noise and modeling errors and is modeled as a probability distribution that assigns a probability to every 3D shape. While ambiguities have long been recognized as an important problem in stereo (e.g., [3]), the question of how these ambiguities affect the estimation of occupancy and emptiness is not well understood. Existing stereo algorithms make one or more assumptions that allow one to estimate a unique shape. Most methods compute the shape that maximizes some form of image consistency likelihood (i.e., that a reconstructed shape is consistent with the image measurements), assuming constraints or prior beliefs about shape and image disparities. Examples include the use of SSD-based likelihoods [4], window-based correlation [5], ordering constraints on correspondence [6,7], shape priors [8], as well as MRF [9,10] and parametric models [11,12]. But these approaches do not estimate the probability that a region is empty or occupied. This is because image consistency is not equivalent to occupancy, nor is it the opposite of emptiness (Figure 1a). In general, ascertaining the emptiness of a region requires that one consider the likelihood that no consistent scene interpretation intersects that region, a problem for which no formal analysis currently exists. Even though recent formulations of the n-view stereo problem have demonstrated an ability to resolve global visibility relationships among views [13,14,15,16,17,18,19], most of these ignore the existence of noise and the resulting model uncertainty. Two notable exceptions are the Roxel Algorithm [20] and the Probabilistic Space Carving Algorithm [21], which attempt to capture shape uncertainty by assigning a “soft” occupancy value between zero and one to every voxel in space. Working under the assumption that an accurate treatment of long-range probabilistic dependencies is intractable, these approaches rely on heuristic approximations to assess a voxel’s visibility. For instance, the Roxel Algorithm assesses a voxel’s visibility by assuming that voxels in front of it are “semi-transparent,” an assumption that is not consistent with image formation models for opaque scenes. Analogously, the Probabilistic Space Carving Algorithm assumes that visibilities of distant voxels are independent from each other and that visibility information can be propagated locally from voxel to voxel without regard to any specific global 3D shape. Neither algorithm takes into account the global nature of visibility, leading to occupancy calculations that do not afford a probabilistic interpretation in a strict mathematical sense. Similar problems would arise in any method that relies on a local model for propagating probabilities since long-distance dependency cycles, like those depicted in Figure 1b, must be taken into account. To our knowledge, no attempts have been made to define the notions of “occupancy probability” and “emptiness probability” in a mathematically rigorous manner when noisy images are given as input. Instead, existing approaches define their goal procedurally, by giving an algorithm that assigns soft occupancies to 3D points. The question of what probabilistic information is theoretically computable from a set of n noisy images, without regard to specific algorithms or implementations, is therefore still open. In practice, this makes it difficult to characterize the behavior of individual algorithms, to define precise notions of optimality and to compare results from different algorithms. In this paper, we argue that a probabilistic treatment of occupancy and emptiness that accounts for stereo ambiguities, shape uncertainty and all long-range probabilistic dependencies caused by visibility can be formulated. Moreover, it leads to a fairly

A Probabilistic Theory of Occupancy and Emptiness

115

straightforward reconstruction algorithm. At the heart of our approach lies the realization that we can overcome technical difficulties via a probabilistic modeling of shape space, i.e., the space of all 3D reconstructions. Using the photo-consistency theory in [19] as a starting point, we show that shape-space modeling allows one to derive a complete theory of occupancy and emptiness. This theory arises from a small set of “axioms” (a generative image formation model and a radiance independence assumption) and does not rely on heuristic assumptions about scene shape or the input photos.

2

Shape-Space Modeling: Photo Hulls, Occupancy, and Emptiness

In the formulation of shape ambiguity and uncertainty that follows, we represent shapes using a discrete volumetric approximation and we further define a bounding volume V ⊆ 3 that is assumed to contain the scene. The shape space of V is defined simply as its powerset, 2V . With these definitions, we use the photo-consistency theory of Kutulakos and Seitz [19] to model shape ambiguities and to introduce the concept of the Photo Hull Distribution to model shape uncertainty. 2.1 The Photo Hull Photo-consistency is based on the idea that a valid solution to the reconstruction problem should reproduce the scene’s input photographs exactly when no image noise is present. When one or more such photographs are available, each taken from a known viewpoint, they define an equivalence class of solutions called photo-consistent shapes. For Lambertian shapes, a voxel v ∈ S is photo-consistent if (1) v is not visible to any camera, or (2) we can assign a radiance to v that is equal to the color at v’s projection in all photos where v is visible. Accordingly, a shape S ∈ 2V is photo-consistent if all its voxels are photo-consistent. The equivalence class of photo-consistent shapes captures all the ambiguities present in a set of n noiseless images and is closed under set union (Figure 2a): Theorem 1 (Photo Hull Theorem [19]) For every volume V that contains the true scene and

every noiseless image set I, there exists a unique shape called the Photo Hull, H(V, I), that is the union of all shapes in 2V that are photo-consistent with I. This shape is a superset of the true scene and is itself photo-consistent.

The photo hull is the tightest possible bound on the true scene that can be recovered from n photographs in the absence of a priori scene information. The photo hull is especially important in our analysis for two reasons. First, it is algorithm-independent and is applicable to scenes of arbitrary shape. Second, it suggests that even though photoconsistency is an ambiguous property (i.e., many distinct 3D shapes can possess it), emptiness of space is not; the volume V − H(V, I) cannot intersect any photo-consistent 3D shape. Hence, by studying how noise affects the information we can recover about the photo hull and about the emptiness of space (rather than about individual photoconsistent shapes) we can establish a clear mathematical distinction between the analyses of shape ambiguity and shape uncertainty. For simplicity and without loss of generality we assume in this paper that the true scene S ∗ is equal to the photo hull.

116

R. Bhotika, D.J. Fleet, and K.N. Kutulakos

(a)

(b)

(c)

Fig. 2. (a) The photo hull H(V, I) contains all photo-consistent shapes (e.g., S ∗ and a shape S). (b) When noise is present, the photo hull cannot be established with certainty. The Photo Hull Distribution assigns a probability to every possible hull (e.g., H1 and H2 ). (c) Foreground and background modeling: the foreground and background models are satisfied for voxels v and v , respectively.

2.2 The Photo Hull Distribution When the set I of input images is generated stochastically, the emptiness or occupancy of voxels cannot be determined with certainty any more (Figure 2b). To model this uncertainty, we introduce the Photo Hull Distribution: De nition 1 (Photo Hull Probability) The Photo Hull Probability of S ∈ 2V is the conditional probability

p [Hull(S, V, I) | V, I] : 2V → [0, 1], where Hull(S, V, I) is the event that S is the photo hull H(V, I).

The Photo Hull Distribution captures all the information that we can possibly extract from noisy photographs of an arbitrarily-shaped scene in the absence of prior information. This is because (1) scene radiances are not observable and hence shape information cannot be extracted with certainty, and (2) the inherent ambiguities of the (noiseless) n-view stereo problem preclude bounding the scene with anything but its photo hull. While useful as an analytical tool, the Photo Hull Distribution is impractical as a shape representation because it is a function over 2V , an impossibly large set. To overcome this limitation, we also study how to reconstruct 3D shapes that have high probability of being the photo hull and how to compute scene reconstructions that represent shape uncertainty in a compact way. To achieve the latter objective, we introduce a probabilistic definition of occupancy: De nition 2 (Occupancy Probability) The occupancy probability of a voxel v is given by the marginal probability p [Occ(v, V, I) | V, I] = p [Hull(S, V, I) | V, I] . {S|S∈2V and v∈S}

A Probabilistic Theory of Occupancy and Emptiness

117

Intuitively, the occupancy probability defines a scalar field over V whose value at a voxel v ∈ V is equal to the probability that v is contained in the photo hull. Our goal in the remainder of this paper is to characterize the information that n noisy photos provide about the shape of an arbitrary 3D scene. We achieve this by deriving an image-based expression for the photo hull distribution and a method for drawing fair samples from it. Toward this end, we first factor the photo hull probability of a shape S into two components, namely, the probability that S is photo-consistent conditioned on the emptiness of its exterior V − S (so that the shape is visible), and the probability that its exterior is empty: p [Hull(S, V, I ) | V, I] = p [PhotoC(S, I ) | Empty(V − S, I ), V, I] p [Empty(V − S, I ) | V, I] . (1) Here, PhotoC(S, I) denotes the event that S is photo-consistent with image set I. Similarly, Empty(V − S, I) denotes the event that the exterior volume V − S does not intersect any shape in 2V that is photo-consistent with I. The following sections derive expressions for these two fundamental distributions in terms of image measurements.

3

Probabilistic Modeling of Photo-Consistency

When confronted with the problem of formulating the photo-consistency probability p [PhotoC(S, I) | Empty(V − S, I), V, I] in terms of image measurements two questions arise: (1) how do we express the photo-consistency probability of a 3D shape S in terms of individual pixels (which are projections of individual voxels on the shape), and (2) how do we ensure that all global dependencies between 3D points are correctly accounted for? To answer these questions, we use two key ideas. First, we assume radiance independence, that the radiances of any two surface points on the true scene are independent. This simplifies the probabilistic modeling of the photo-consistency of individual voxels. The second idea, which we call the visibility-conditioning principle, states that, with the radiance independence assumption, the photo-consistency of individual voxels are conditionally independent, when conditioned on the voxels’ visibility in the true scene, S ∗ . This conditional independence is derived from our image formation model that is outlined below, and leads to the following factorization of the photo-consistency probability of shape S:1 Lemma 1 (Photo-Consistency Probability Lemma) p [PhotoC(S, I) | Empty(V − S, I), V, I] =

p [PhotoC(v, I, βS ) | I, βS (v)] ,

(2)

v∈S

where PhotoC(v, I, βS ) denotes the event that voxel v is photo-consistent, conditioned on the set of cameras from which it is visible, βS (v). The following sections give our simple model of image formation and derive the form of the voxel photo-consistency probability that it entails. 1

See the Appendix for proof sketches of some of the paper’s main results. Complete proofs can be found in [22].

118

3.1

R. Bhotika, D.J. Fleet, and K.N. Kutulakos

Image Formation Model

Let S ∗ be an opaque, Lambertian 3D scene that occupies an arbitrary volume in 3 . Also assume that the scene is viewed under perspective projection from n distinct and known viewpoints, c1 , . . . , cn , in 3 − S ∗ . Then, the measured RGB value of a pixel depends on three factors: (1) visibility; (2) surface radiance; and (3) pixel noise. We use the term visibility state to denote the set of input viewpoints from which x is visible. It can be represented as an n-bit vector βS ∗ (x) whose i-th bit, [βS ∗ (x)]i , is one if and only if x is visible from ci . Each 3D scene produces a unique visibility function, βS ∗ : 3 → {0, 1}n , mapping points to their visibility states. The radiance, r(.), is a function that assigns RGB values to surface points on S ∗ . Let r(.) be a random variable with a known distribution, R, over [0, 1]; i.e., ∀x : r(x) ∼ R3 [0, 1]. Ignoring noise, if a surface point x is visible from viewpoint ci , then the RGB values observed from x in the camera at ci are equal to x’s radiance [23]; more formally, (x ∈ S ∗ ) ∧ ([βS ∗ (x)]i = 1)

=⇒

proji (x) = r(x),

where proji (x) denotes the image RGB value at x’s projection along ci . In practice, image measurements are corrupted by noise. Here, we model pixel noise, n, as meanzero Gaussian additive noise with variance σn2 . The RGB value proji (x) is therefore a random variable given by proji (x) = r(x) + n. Assuming that radiances and noise are independent, it follows that (1) the mean of proji (x) is E [r(x)], (2) its variance is V ar [r(x)] + σn2 , and (3) conditioned on radiance, the measured pixel color is distributed according to the noise distribution: proji (x)|r(x) ∼ N (r(x), σn2 ). In what follows we use proj(x) to denote the vector of random variables {proji (x) : [βS ∗ (x)]i } of a point x with visibility state βS ∗ (x). We use I to denote the set of images obtained from the n cameras, and I(x, βS ∗ ) to denote a sample of the random vector proj(x), i.e., the vector of image RGB values observed at x’s projections. 3.2

Photo-Consistency

Given only βS ∗ (v) and I(v, βS ∗ ), what can we say about whether or not v belongs to S ∗ ? To answer this question we formulate two generative models that characterize the distribution of the set of observations, I(v, βS ∗ ), that corresponds to v. The foreground model captures the fact that if v lies on the surface and βS ∗ (v) is v’s correct visibility state, then all radiances in proj(v) are identical since they arise from the same location on the surface: Observation 1 (Foreground Model) If v ∈ S ∗ and k = |I(v, βS ∗ )| is the number of cameras

for which v is visible, then (1) the distribution of proj(v) conditioned on the actual radiance, r(v), is a multi-variate Gaussian proj(v) | r(v) ∼ N k (r(v), σn2 ) , and

(3)

(2) the sample variance of the RGB measurements, V ar [I(v, βS ∗ )], will converge to the noise variance σn2 as k tends to infinity.

A Probabilistic Theory of Occupancy and Emptiness

119

In practice, k is finite. It follows from Eq. (3) that the observation likelihood, p [V ar [I(v, βS ∗ )] | v ∈ S ∗ ], is a χ2 density with 3(k − 1) degrees of freedom. If v is not a voxel on S ∗ , the color at each of its k projections is due to a distinct scene voxel vj (Figure 2c), with projj (v) = r(vj ) + n for j = 1, . . . , k. This, along with the radiance independence assumption, that radiances at different voxels are independent, leads to our simple background model: Observation 2 (Background Model) If v ∈ S ∗ , the distribution of proj(v) conditioned on

the actual radiances is given by

proj(v) | {r(vj )}kj=1 ∼

k

N (r(vj ), σn2 ),

(4)

j=1

the sample variance of which, V ar [I(v, βS ∗ )], converges to V ar [r] + σn2 as k tends to infinity.

In practice, with finite cameras and an unknown background distribution, we model the likelihood p [V ar [I(v, βS ∗ )] | v ∈S ∗ ], using the input photos themselves: we generate 106 sets of pixels, each of size |I(v, βS ∗ )|, whose pixels are drawn randomly from distinct photos. This collection of variances is used to model the distribution of sample variances for voxels that do not lie on the surface. We can now use Bayes’ rule to derive the probability that a voxel is on the scene’s surface given a set of image measurements associated with its visibility state β: p [v ∈ S ∗ | V ar [I(v, β)]] =

p [V ar [I(v, β)] | v ∈ S ∗ ] . p [V ar [I(v, β)] | v ∈ S ∗ ] + p [V ar [I(v, β)] | v ∈S ∗ ] (5)

Note that this particular formulation of the foreground and background likelihoods was chosen for simplicity; other models (e.g., see [21]) could be substituted without altering the theoretical development below. Finally, by defining the voxel photo-consistency probability to be the probability that v is consistent with the scene surface conditioned on visibility, i.e., p [PhotoC(v, I, βS ) | I, βS (v)] = p [v ∈ S ∗ | V ar [I(v, βS (v))]] , def

we obtain the factored form for the shape photo-consistency probability in Lemma 1.

4

Probabilistic Modeling of Emptiness

Unlike photo-consistency, the definition of emptiness of region V − S does not depend only on the shape S; it requires that no photo-consistent shape in 2V intersects that region. In order to express this probability in terms of individual pixel measurements, we follow an approach analogous to the previous section: we apply the visibility-conditioning principle to evaluate p [Empty(V − S, I) | V, I], the emptiness probability of a volume V − S, as a product of individual voxel-specific probabilities. But what voxel visibilities do we condition upon? The difficulty here is that we cannot assign a unique visibility state to a voxel in V − S because its visibility depends on which shape in 2V we are

120

R. Bhotika, D.J. Fleet, and K.N. Kutulakos

Fig. 3. Visualizing voxel visibility strings and carving sequences. The carving sequence Σ = v1 · · · vm defines a sequence of m shapes, where the (i − 1)-th shape determines the visibility of the i-th voxel and S(Σ, i − 1) = V − {v1 , v2 , . . . , vi−1 }. Voxels in V − S and S are shown as light and dark gray, respectively. Also shown is the shape-space path defined by Σ.

considering. To overcome this difficulty, we introduce the concept of voxel visibility strings. These strings provide a convenient way to express the space of all possible visibility assignments for voxels in V − S. Importantly, they allow us to reduce the estimation of emptiness and photo hull probabilities to a string manipulation problem: De nition 3 (Voxel Visibility String) (1) A voxel visibility string, Σ = q1 · · · q|Σ| , is a string

of symbols from the set {v, v | v ∈ V }. (2) The shape S(Σ, i) induced by the first i elements of the visibility string Σ is obtained by removing from V all voxels in the sub-string q1 · · · qi that occur in their v-form. (3) The visibility state of the voxel v corresponding to string element qi is determined by the shape S(Σ, i − 1), i.e., it is the bit vector βS(Σ,i−1) (v). (4) The value function, F (Σ), of string Σ is the product |Σ| p PhotoC(v, I, βS(Σ,i−1) ) | I, βS(Σ,i−1) (v) (qi = v) F (Σ) = f (qi ), with f (qi ) = β I, PhotoC(v, p − 1 β (q I, | ) (v) i = v) S(Σ,i−1) S(Σ,i−1) i=1 (5) A voxel visibility string that is a permutation of the voxels in V − S, each of which appears in its v-form, is called a carving sequence.

Note that the set of voxel visibility strings is infinite since voxels can appear more than once in a string. Carving sequences, on the other hand, are always of length |V − S| and have an intuitive interpretation (Figure 3): a carving sequence Σ = v1 · · · vm can be thought of as a path in 2V that “joins” V to S. That is, beginning with the initial volume, V , every time we encounter a string element with a bar we remove the corresponding voxel and thereby generate a new shape. When we reach the end of the string we will have generated the desired shape S. Thus, the set of all carving sequences for a shape S gives us all the possible ways of obtaining S by carving voxels from V . Carving sequences provide an avenue for relating the emptiness of space to the nonphoto-consistency of individual voxels in V − S. This is because the order of voxels in a carving sequence assigns a unique visibility state to every voxel it contains, allowing us to apply the visibility-conditioning principle through the following definition:

A Probabilistic Theory of Occupancy and Emptiness

121

Definition 4 (Σ-Non-Photo-Consistency Event) We use NPhotoC(V −S, I, Σ) to denote the event that V − S is non-photo-consistent with respect to carving sequence Σ = v1 · · · vm . This is the event that no voxel v in Σ is photo-consistent when its visibility state is given by Σ: NPhotoC(V − S, I, Σ) ⇐⇒

m

¬PhotoC(vi , I, βS(Σ,i−1) ) .

i=1

Using this definition, it is easy to show that the non-photo-consistency of the volume V − S, when conditioned on the visibilities induced by a carving sequence, is simply the product of the non-photo-consistency probabilities of its constituent voxels: p [NPhotoC(V − S, I, Σ ) | V, I, Σ] = F (Σ). To fully specify the emptiness probability of the volume V − S it therefore suffices to establish a relation between the events NPhotoC(V − S, I, Σ) and Empty(V − S, I). The following proposition gives us a partial answer to this problem: Lemma 2 (Empty-Space Event) If a non-photo-consistent carving sequence exists, V − S is empty:

NPhotoC(V − S, I, Σ)

=⇒

Empty(V − S, I).

Σ∈C(V,S)

where C (V, S) is the set of all carving sequences that map V to S. Lemma 2 suggests a rule to establish a volume’s emptiness without exhaustive enumeration of all photo-consistent shapes. It tells us that it suffices to find just one carving sequence whose voxels are all non-photo-consistent. While this rule is sufficient, it is not a necessary one: it is possible for V − S to be empty without any of the events NPhotoC(V − S, I, Σ) being true. This seemingly counter-intuitive fact can be proven by contradiction because it leads to an expression for the Photo Hull Probability that is not a distribution,2 i.e., S∈2V p [Hull(S, V, I) | V, I] < 1. From an informal standpoint, this occurs because the set of voxel visibility strings that are carving sequences does not adequately cover the space of all possible visibility assignments for voxels in V − S, resulting in an under-estimation of the “probability mass” for the event Empty(V − S, I). Fortunately, the following proposition tells us that this mass can be correctly accounted for by a slightly expanded set of voxel visibility strings, all of which are mutually disjoint, which consider all sets of visibility assignments for voxels in V − S and contain every voxel in V − S at most |V − S| times: Proposition 1 (Emptiness Probability) p [Empty(V − S, I) | V, I] =

1 |V − S|!

F (Σ) +

Σ∈C(V,S)

R(P ) G(P )

Σ∈C(V,S) P ∈P(Σ)−C(V,S)

(6) where P(Σ) is the set of strings P = p1 · · · pl with the following properties: (1) P is created by adding voxels in their v-form to a carving sequence Σ, (2) each voxel vi occurs at most once 2

A simple way to verify this fact is to expand the sum in Eq. (7) for a 2-voxel initial volume V . Details are provided in Appendix A.3.

122

R. Bhotika, D.J. Fleet, and K.N. Kutulakos

between any two successive voxels appearing in their v-form, (3) no voxel in P can occur in its v-form after it has occurred in its v-form; and G(P ) and R(P ) are given by   f (pi )      f (pi )/f (pj )

(1st occurrence) (voxel occurs in v-form, |P | g(pi ) , where g(pi ) = G(P ) = previous occurrence at pj )   i=1  ) + f (p ) − 1] /f (p ) (voxel occurs in v-form, [f (p  i j j    previous occurrence at pj )

R(P ) =

(2 − lm−1 )! (m − l1 )! (m − 1 − l2 )! ... m! (m − 1)! 2!

where m = |V − S|; and lj is the number of voxels, vj inclusive, that exist in P between two voxels vj−1 and vj that are consecutive in Σ.

Proposition 1 tells us that the “extra” mass needed to establish the emptiness of a region consists of the G(P )-terms in Eq. (6). This result is somewhat surprising, since it suggests that we can capture all relevant visibility configurations of voxels in V − S by only considering a very constrained set of voxel visibility strings, whose length is at most |V −S|2 +|V −S| . Moreover, it not only leads to a probability distribution for the photo hull 2 but, as shown in Section 6, it leads to a simple stochastic algorithm that produces fair samples from the photo hull distribution. We consider the implications of this proposition below.

5 The Photo Hull Distribution Theorem One of the key consequences of our probabilistic analysis is that despite the many intricate dependencies between global shape, visibility, emptiness, and photo-consistency, the expression of a shape’s photo hull probability in terms of image measurements has a surprisingly simple form. This expression is given to us by the following theorem that re-states Eq. (1) in terms of image measurements by bringing together the expressions in Lemma 1 and Proposition 1. It tells us that for a shape S, this probability is a sum of products of individual voxel photo-consistency or non-photo-consistency probabilities; each voxel in S contributes exactly one voxel photo-consistency factor to the product, while each voxel in V − S contributes multiple non-photo-consistency factors. Theorem 2 (Photo Hull Distribution Theorem) (1) If every pixel observation in I takes RGB values over a continuous domain, the probability that shape S ⊆ V is the photo hull H(V, I) is G(P || Π) R(P ), (7) p [Hull(S, V, I) | V, I] = Σ∈C(V,S) P ∈P(Σ)

where Π is a string corresponding to an arbitrary permutation of the voxels in S, all of which appear in their v-form; C (V, S) , P(Σ) and G(·), R(·) are defined in Proposition 1; and “||” denotes string concatenation. (2) Eq. (7) defines a probability distribution function, i.e., S∈2V p [Hull(S, V, I) | V, I] = 1.

A Probabilistic Theory of Occupancy and Emptiness

6

123

Probabilistic Occupancy by Fair Stochastic Carving

Even though the photo hull probability of a shape S has a relatively simple form, it is impractical to compute because of the combinatorial number of terms in Eq. (7). Fortunately, while the Photo Hull Distribution cannot be computed directly, we can draw fair samples from it (i.e., complete 3D photo hulls) using a stochastic variant of the space carving algorithm [19]. Moreover, by computing a set of 3D photo hulls through repeated applications of this algorithm, we can approximate the occupancy probability of every voxel in V according to Definition 2. Occupancy Field Reconstruction Algorithm – Set trial = 1 and run the following algorithm until trial = K: Stochastic Space Carving Algorithm Initialization Set Strial = V and repeatedly perform the following two steps. Voxel Selection Randomly choose a voxel v ∈ Strial that either has not been selected or whose visibility state changed from β to β since its last selection. Stochastic Carving If no such voxel exists, set trial = trial + 1 and terminate. Otherwise, carve v from Strial with probability p [PhotoC(v, I, β) | I, β] − p [PhotoC(v, I, β ) | I, β ] p [PhotoC(v, I, β) | I, β] – For every v ∈ V , set p [Occ(v, V, I) | V, I] = L/K where L is the number of carved shapes in {S1 , . . . , SK } containing v.

From a superficial point of view, the Stochastic Space Carving Algorithm in the above inner loop is almost a trivial extension of standard space carving—the only essential difference is the stochastic nature of the carving operation, which does not depend on a pure threshold any more. This carving operation and the precise definition of the carving probability are extremely important, however. This is because if carving is performed according to the above rule, the algorithm is guaranteed to draw fair samples from the Photo Hull Distribution of Theorem 2. In the limit, this guarantees that the occupancy probabilities computed by the algorithm will converge to the theoretically-established values for this probability: Proposition 2 (Algorithm Fairness) When the number of trials goes to infinity, the frequency by which the Stochastic Carving Algorithm generates shape S will tend to p [Hull(S, V, I) | V, I].

A second feature of the Stochastic Space Carving Algorithm, which becomes evident from the proof of Proposition 2 in the Appendix, is that it provides us with an approximate means for comparing the likelihoods of two photo hulls it generates: since each run of the algorithm corresponds to a single term in the sum of Eq. (7), i.e., a particular combination of the Σ, P, Π strings, and since we can compute this term from the algorithm’s trace as it examines, re-examines and removes voxels, we can compare two photo hulls by simply comparing the value of these terms. This effectively allows us to develop an approximate method for selecting a maximum-likelihood photo hull from a set of generated candidate shapes.3 3

Note that this method is approximate because we can only compute a single term in the sum in Eq. (7) that defines p [Hull(S, V, I) | V, I], not the probability itself.

124

R. Bhotika, D.J. Fleet, and K.N. Kutulakos

Fig. 4. Cactus: Raw views of the occupancy field were created by computing at each pixel, the average over all voxels that project to it and have non-zero occupancy probability. Each image in the last row compares the recovered occupancy field to volumes obtained from standard space carving, shown directly above it. A light (dark) gray pixel indicates that at least one voxel in the occupancy field with non-zero probability projects to it, and that no (at least one) voxel in the carved volume projects to it. Intuitively, light gray pixels mark regions that the standard algorithm “over-carves” , i.e., regions that were carved even though they have a non-negligible probability of being in the photo hull. Here, P F and P B correspond to the foreground and background likelihoods in Section 3.2.

7

Experimental Results

To demonstrate the practical utility of our formulation, we applied the algorithm of Section 6 to different scenes of varying complexity and structure.4 The cactus dataset (from [18]) is especially challenging because of the drastic changes in visibility between photos, the complex scene geometry, and because the images contain 4

The complete input sequences are available at http://www.cs.toronto.edu/˜kyros/research/occupancy/.

A Probabilistic Theory of Occupancy and Emptiness

125

many “mixed” pixels, quantization errors, and 1-4 pixel calibration errors that make pixel matching difficult. The carving probabilities in the Stochastic Carving Step of the algorithm require foreground and background likelihoods (denoted P F and P B in Figure 4). The foreground likelihood was given by a χ2 density with σ = 20. The background likelihood was modeled from the input photos as described in Section 3.2. The resulting plots in Figure 4 (bottom left) show these likelihoods. When the number of input views is small, the carving probability changes slowly as a function of the variance of a voxel’s projections. For many views, where a coincidental pixel match across views is unlikely, it approaches a step function and carving becomes almost deterministic. We ran the algorithm for K = 400 iterations on a cubical, 1283 -voxel initial volume V , generating 400 distinct photo hulls that were used to compute the occupancy field shown in Figure 4 (top right). Each iteration took about 17 minutes on an R12000 SGI O2. Views of two of the 400 photo hulls are shown in Figure 4 (top middle). Despite the large number of stochastic carving decisions made by the algorithm, the generated photo hulls are similar in both shape and appearance, with volumes differing by less than 3% from hull to hull. The number of voxels with non-zero probability in the final occupancy field was only 7% more than the voxels in a single sample, suggesting that despite the high dimensionality of shape-space, the Photo Hull Distribution is “highly peaked” for this dataset. In Figure 4 (bottom right) we compare the occupancy field to photo hulls obtained from the standard space carving algorithm [19] for different variance thresholds. Note that the standard space carving algorithm fails catastrophically for lower values of σ. Even at σ = 35, standard space carving results in a volume that appears over-carved and has artifacts. This behavior of standard space carving over a wide range of variance thresholds shows that it fails to model the uncertainty in the data set. The occupancy field represents this uncertainty quite well: the volume corresponding to the pot base has voxel occupancies close to 1, while near the top of the tallest cactus, where detail is highest, voxels have lower occupancy probabilities. Results with K = 400 for a second dataset (the gargoyle from [19]) are summarized in Figure 5. Each iteration took about 6 minutes. The gargoyle statuette has more structure than the earlier example and the resulting occupancy field is less diffuse. However, as each voxel is visible in at most 7 of the 16 input images, the voxel probability distribution has a gentler slope, and the occupancy field shows a smoother gradation of voxel occupancy probabilities moving inwards of the object, most noticeably in regions with discontinuities and irregularities on the surface, and in the cavity between the gargoyle’s hands and body (Figure 5, bottom row).

8

Concluding Remarks

This paper introduced a novel, probabilistic theory for analyzing the 3D occupancy computation problem from multiple images. We have shown that the theory enables a complete analysis of the complex probabilistic dependencies inherent in occupancy calculations, provides an expression for the tightest occupancy bound recoverable from noisy images, and leads to an algorithm with provable properties and good performance on complex scenes. Despite these features, however, our framework relies on a shape representation that is discrete and on a radiance independence assumption that does not

126

R. Bhotika, D.J. Fleet, and K.N. Kutulakos

Fig. 5. Gargoyle dataset (input images not shown): (Top) 3 of 400 sample hulls generated and the occupancy field obtained by combining them. (Bottom) Vertical slices of the occupancy field. The bar at the bottom shows the mapping from voxel occupancy probabilities to gray levels.

hold for scenes with smoothly-varying textures. Overcoming these limitations will be a major focus of future work. Additional topics will include (1) providing a mechanism for introducing shape priors, (2) developing efficient strategies for sampling the Photo Hull Distribution and for computing likely Photo Hulls, and (3) developing fast, sampling-free approximation algorithms for occupancy field computation. Acknowledgements. We would like to thank Allan Jepson for many helpful discussions. The support of the National Science Foundation under Grant No. IIS-9875628 and of the Alfred P. Sloan Foundation for Research Fellowships to Fleet and Kutulakos is gratefully acknowledged.

References 1. A. Elfes, “Sonar-based real-world mapping and navigation,” IEEE J. Robotics Automat., vol. 3, no. 3, pp. 249–266, 1987. 2. B. Curless and M. Levoy, “A volumetric method for building complex models from range images,” in Proc. SIGGRAPH’96, pp. 303–312, 1996. 3. D. Marr and T. Poggio, “A computational theory of human stereo vision,” Proceedings of the Royal Society of London B, vol. 207, pp. 187–217, 1979. 4. L. Matthies, “Stereo vision for planetary rovers: Stochastic modeling to near real-time implementation,” Int. J. Computer Vision, vol. 8, no. 1, pp. 71–91, 1992. 5. T. Kanade and M. Okutomi, “A stereo matching algorithm with an adaptive window: Theory and experiment,” IEEE Trans. PAMI, vol. 16, no. 9, pp. 920–932, 1994. 6. I. Cox, S. Hingorani, S. Rao, and B. Maggs, “A maximum likelihood stereo algorithm,” CVIU: Image Understanding, vol. 63, no. 3, pp. 542–567, 1996.

A Probabilistic Theory of Occupancy and Emptiness

127

7. H. Ishikawa and D. Geiger, “Occlusions, discontinuities and epipolar line in stereo,” in Proc. 5th European Conf. on Computer Vision, pp. 232–248, 1998. 8. P. N. Belhumeur, “A Bayesian approach to binocular stereopsis,” Int. J. on Computer Vision, vol. 19, no. 3, pp. 237–260, 1996. 9. R. Szeliski, “Bayesian modeling of uncertainty in low-level vision,” Int. J. Computer Vision, vol. 5, no. 3, pp. 271–301, 1990. 10. D. Geiger and F. Girosi, “Parallel and deterministic algorithms from MRF’s: surface reconstruction,” IEEE Trans. PAMI, vol. 13, no. 5, pp. 401–412, 1991. 11. K. Kanatani and N. Ohta, “Accuracy bounds and optimal computation of homography for image mosaicing applications,” in Proc. 7th Int. Conf. on Computer Vision, vol. 1, pp. 73–78, 1999. 12. P. Torr, R. Szeliski, and P. Anandan, “An integrated bayesian approach to layer extraction from image sequences,” IEEE Trans. PAMI, vol. 23, no. 3, pp. 297–303, 2001. 13. P. Fua and Y. G. Leclerc, “Object-centered surface reconstruction: Combining multi-image stereo and shading,” Int. J. Computer Vision, vol. 16, pp. 35–56, 1995. 14. R. T. Collins, “A space-sweep approach to true multi-image matching,” in Proc. Computer Vision and Pattern Recognition Conf., pp. 358–363, 1996. 15. S. M. Seitz and C. R. Dyer, “Photorealistic scene reconstruction by voxel coloring,” in Proc. Computer Vision and Pattern Recognition Conf., pp. 1067–1073, 1997. 16. S. Roy and I. J. Cox, “A maximum-flow formulation of the N-camera stereo correspondence problem,” in Proc. 6th Int. Conf. on Computer Vision, pp. 492–499, 1998. 17. O. Faugeras and R. Keriven, “Complete dense stereovision using level set methods,” in Proc. 5th European Conf. on Computer Vision, pp. 379–393, 1998. 18. K. N. Kutulakos, “Approximate N-View stereo,” in Proc. European Conf. on Computer Vision, pp. 67–83, 2000. 19. K. N. Kutulakos and S. M. Seitz, “A theory of shape by space carving,” Int. J. Computer Vision, vol. 38, no. 3, pp. 199–218, 2000. Marr Prize Special Issue. 20. J. D. Bonet and P. Viola, “Roxels: Responsibility-weighted 3D volume reconstruction,” in Prof. 7th. Int. Conf. on Computer Vision, pp. 418–425, 1999. 21. A. Broadhurst, T. Drummond, and R. Cipolla, “A probabilistic framework for space carving,” in Proc. 8th Int. Conf. on Computer Vision, pp. 388–393, 2001. 22. R. Bhotika, D. J. Fleet, and K. N. Kutulakos, “A probabilistic theory of occupancy and emptiness,” Tech. Rep. 753, University of Rochester Computer Science Department, Dec. 2001. 23. B. K. P. Horn, Robot Vision. MIT Press, 1986.

A A.1

Proof Sketches Lemma 1: Photo-Consistency Probability

We rely on the following result which is stated here without proof (see [22] for details): Proposition 3 (Conditional Independence of Photo-consistency and Emptiness) If every pixel observation in I takes RGB values over a continuous domain, the photo-consistency probability of S, conditioned on the visibility function βS , is independent of the emptiness of V − S: p [PhotoC(S, I) | Empty(V − S, I), V, I] = p [PhotoC(S, I) | I, βS ] .

(8)

128

R. Bhotika, D.J. Fleet, and K.N. Kutulakos

To prove the lemma, it therefore suffices to derive an expression for the right-hand side of Eq. (8). A shape is photo-consistent if and only if every voxel v ∈ S is photo-consistent. Considering an arbitrary voxel v1 ∈ S and applying the chain rule gives

p [PhotoC(S, I) | I, βS ] = p

PhotoC(v, I, βS ) | I, βS =

v∈S

 p PhotoC(v1 , I, βS ) |

 

PhotoC(v, I, βS ), I, βS  p 

v∈S−{v1 }

 PhotoC(v, I, βS ) | I, βS  .

v∈S−{v1 }

The visibility function βS defines the visibility state for all voxels in S, including v1 . Now, the photo-consistency probability of v1 , when conditioned on v1 ’s visibility state in S, is independent of the photo-consistency probability of all other voxels in S:   p PhotoC(v1 , I, βS ) | PhotoC(v, I, βS ), I, βS  = p [PhotoC(v1 , I, βS ) | I, βS (v1 )] . v∈S−{v1 }

This equation is a precise statement of the visibility conditioning principle. When applied to all voxels in S and combined with Eq. (8), it yields the desired factorization: p [PhotoC(v, I, βS ) | I, βS (v)] . (9) p [PhotoC(S, I) | Empty(V − S, I), V, I] = v∈S

A.2

Lemma 2: Empty Space Events

We prove the lemma by contradiction. In particular, we show that if V − S intersects a photoconsistent shape, i.e., if Empty(V − S) is false, no carving sequence Σ ∈ C (V, S) exists for which NPhotoC(V − S, I, Σ) is true. Suppose V − S intersects a photo-consistent shape and let S1 , S2 be the shape’s intersections with V − S and S, respectively. We partition the set of carving sequences, C (V, S), into two subsets C1 , C2 as follows. Let C1 be the set of those carving sequences in C (V, S) where every voxel in S1 appears after the entire set of voxels in V − (S ∪ S1 ), and let C2 = C (V, S) − C1 . Now consider any carving sequence Σ ∗ ∈ C1 . If voxel vk is the first voxel in S1 to appear in Σ ∗ , this voxel will be photo-consistent with respect to the visibility state βSk , where Sk = S1 ∪S. This is because the cameras that view vk according to βSk are a subset of the cameras that view it when vk lies on the shape S1 ∪S2 , which is both a subset of Sk and is photo-consistent. Therefore, NPhotoC(V − S, I, Σ ∗ ) is not true. Moreover, voxel vk will be photo-consistent in every carving sequence in C2 in which it appears before every other voxel in S1 . This is because in that case vk will be visible from even fewer cameras than those that view vk in Σ ∗ . It follows that the above argument is true for every carving sequence in C1 and can be extended to cover every sequence in C2 as well. It follows that the event NPhotoC(V − S, I, Σ) is not satisfied for any carving sequence in C (V, S).

A.3

Emptiness and Inadequacy of Carving Sequences

If it were possible to define the emptiness of a volume V − S simply by the event that at least one carving sequence is not photo-consistent, i.e., Empty(V − S, I) ⇐⇒ Σ∈C(V,S) NPhotoC(V − S, I, Σ), the expression for the photo-hull probability of S would reduce to

A Probabilistic Theory of Occupancy and Emptiness

L

v1

129

R

v2

Fig. 6. A two voxel initial volume, V = {v1 , v2 }, viewed by cameras L and R.

p [Hull(S, V, I) | V, I] =

G(Σ || Π)R(Σ),

(10)

Σ∈C(V,S)

where Π is a voxel visibility string corresponding to an arbitrary permutation of voxels in S, all of which appear in their v-form. Eq. (10) is obtained by setting P(Σ) = {Σ} in Eq. (7). Despite the fact that Eq. (10) may seem intuitively correct, this equation does not hold in general. This is because the emptiness of V − S does not imply the existence of a non-photo-consistent carving sequence. We prove this fact by contradiction, by showing that the photo-hull probability defined by Eq. (10) is not a distribution in general, i.e., S∈2V p [Hull(S, V, I) | V, I] < 1. Consider the volume and viewing geometry illustrated in Figure 6. Let V = {v1 , v2 }, viewed by two cameras L and R. Voxel v1 is always visible from L and is occluded from R by v2 , while v2 is always visible from R and is occluded from L by v1 . The shape space consists of S1 = V , S2 = {v1 }, S3 = {v2 }, and S4 = ∅. From Eq. (10), we get the following expressions for the photo-hull probability of these shapes: p [Hull(S1 , V, I) | V, I] = G(∅ || v1 v2 )R(∅), p [Hull(S2 , V, I) | V, I] = G(v2 || v1 )R(v2 ), p [Hull(S3 , V, I) | V, I] = G(v1 || v2 )R(v1 ), and p [Hull(S4 , V, I) | V, I] = G(v1 v2 || ∅)R(v1 v2 ) + G(v2 v1 || ∅)R(v2 v1 ) Using Definition 3, the equations above, and the following notational shorthand pβvS = p [PhotoC(v, I, βS ) | I, βS (v)] , the sum of photo-hull probabilities of all shapes in the shape space is given by βS βS βS βS βS βS p [Hull(S, V, I) | V, I] = pv1 1 pv2 1 + (1 − pv2 1 )pv1 2 + (1 − pv1 1 )pv2 3 S∈2V βS βS βS βS 1 1 (1 − pv1 1 )(1 − pv2 3 ) + (1 − pv2 1 )(1 − pv1 2 ) 2 2 βS βS βS βS βS βS 1 1 = 1 − (1 − pv1 1 )(pv2 1 − pv2 3 ) − (1 − pv2 1 )(pv1 1 − pv1 2 ). 2 2

+

The second and third “residual” terms in the final expression of the above sum will be nonzero in general. This means that S∈2V p [Hull(S, V, I) | V, I] < 1, implying that the photo-hull probability of shapes S ⊆ V in Eq. (10) is underestimated. Hence, the emptiness probability of V − S is not completely explained by the non-photo-consistency of the carving sequences in C (V, S). The two residual terms correspond to the voxel visibility strings v2 v1 v2 and v1 v2 v1 , respectively. These strings belong to ∪Σ∈C(V,S4 ) P(Σ), but were omitted when Eq. (7) was reduced to Eq. (10).

A.4

Proposition 2: Algorithm Fairness

Proof (Sketch). The proof involves four steps: (1) showing that each run of the algorithm is described by a voxel visibility string T of bounded length; (2) defining an onto mapping µ between

130

R. Bhotika, D.J. Fleet, and K.N. Kutulakos

voxel visibility strings that characterize a run of the algorithm and voxel visibility strings P || Π in Theorem 2; (3) showing that G(T ) = G(µ(T )); and (4) using the above steps to prove fairness. Sketch of Step 1: Define string T to be the exact trace of voxels selected by the algorithm’s Voxel Selection Step, where a voxel appears in its v-form if it was selected and not carved in the Stochastic Carving Step immediately following it, or in its v-form if it was carved in that step. The algorithm terminates when all surface voxels have been examined and no visibility transitions have occurred for these voxels. Since carving can only increase the number of cameras viewing a voxel, the total number of visibility transitions is n|V |. This gives an upper bound on |T |. Sketch of Step 2: Let S be the shape computed by the algorithm and let T be its trace. To construct the string µ(T ) we change T as follows: (i) delete from T all but the last occurrence of every voxel in S, creating string T1 ; (ii) move the voxels in S to the end of the string, creating T2 ; and (iii) define µ(T ) = T2 . String µ(T ) will be of the form P ||Π , with P ∈ P(Σ) for some Σ ∈ C (V, S). This mapping is onto since every string in P(Σ) is a valid trace of the algorithm. Sketch of Step 3: (i) The visibility state of points in Π that is induced by the string µ(T ) = P ||Π is identical to the one assigned by the visibility function βS . It follows that G(Π) = F (Π) = p [PhotoC(S, I) | Empty(V − S, I), V, I]. (ii) Since the voxels in T that are contained in Π always occur in their v-form, their position in T does not affect the visibilities induced by T to those voxels in T that belong to P . Hence, G(T2 ) = G(T1 ). (iii) If a voxel in V − S appears multiple times in T , this means that it was examined and left uncarved for all but its last occurrence in T . It now follows from the definition of G(.) that G(T ) = G(T2 ). Sketch of Step 4: The frequency that the algorithm arrives at shape S is equal to the probability that the algorithm produces a trace in ∪Σ∈C(V,S) P(Σ). It can be shown that Steps 1-3 now imply that the probability the algorithm produces a trace T is equal to G(µ(T ))R(µ(T )), where R(.) is defined in Proposition 1. Intuitively, the R(µ(T )) factor accounts for the portion of all traces T that map to the same µ(T ). Since each trial of the algorithm is independent of all previous ones, it follows that the probability that the algorithm generates shape S is equal to the shape’s photo hull probability, as defined in Theorem 2.

Texture Similarity Measure Using Kullback-Leibler Divergence between Gamma Distributions John Reidar Mathiassen1 , Amund Skavhaug2 , and Ketil Bø3 1

2

SINTEF Fisheries and Aquaculture, N-7465 Trondheim, Norway [email protected] Department of Engineering Cybernetics (ITK), NTNU, N-7491 Trondheim, Norway [email protected] 3 Department of Computer and Information Science (IDI), NTNU, N-7491 Trondheim, Norway [email protected]

Abstract. We propose a texture similarity measure based on the Kullback-Leibler divergence between gamma distributions (KLGamma). We conjecture that the spatially smoothed Gabor ﬁlter magnitude responses of some classes of visually homogeneous stochastic textures are gamma distributed. Classiﬁcation experiments with disjoint test and training images, show that the KLGamma measure performs better than other parametric measures. It approaches, and under some conditions exceeds, the classiﬁcation performance of the best non-parametric measures based on binned marginal histograms, although it has a computational cost at least an order of magnitude less. Thus, the KLGamma measure is well suited for use in real-time image segmentation algorithms and time-critical texture classiﬁcation and retrieval from large databases.

1

Introduction

Measuring the similarity between textures is an important task in low-level computer vision. A similarity measure for textures is essential for texture classiﬁcation, supervised- and unsupervised image segmentation. The motivation of our research has been to ﬁnd a similarity measure that is both low in computational cost and is statistically justiﬁed for classiﬁcation purposes, and is therefore well suited for use in both texture classiﬁcation/retrieval and real-time texture segmentation. Many similarity measures have been developed, based on various feature extraction schemes ([20],[21],[22],[23], [24]). In this paper we will focus on texture similarity between features extracted using Gabor ﬁlters. Observations of many textures, lead us to conjecture that some classes of visually homogeneous

The authors would like to thank Henrik Schumann-Olsen for his useful comments and discussions. We also thank the anonymous reviewers for their feedback. This research was supported by The Research Council of Norway.

A. Heyden et al. (Eds.): ECCV 2002, LNCS 2352, pp. 133–147, 2002. c Springer-Verlag Berlin Heidelberg 2002

134

J.R. Mathiassen, A. Skavhaug, and K. Bø

stochastic textures have gamma distributed spatially smoothed Gabor magnitude responses. We introduce the KLGamma texture similarity measure, which is based on the Kullback-Leibler divergence between gamma distributions. We perform classiﬁcation experiments on 35 texture images. These experiments are done to evaluate classiﬁcation performance on image blocks of various sizes. Testing is done with several parametric and non-parametric similarity measures, with both disjoint and non-disjoint test and training sets. All the tested measures assume independence between features to avoid the computational cost of estimating and using the feature covariance matrix in the similarity measure. In addition to the proposed method, the following parametric methods are tested: weighted mean variance (WMV) [14], weighted nearest neighbour (WNN) and the Kullback-Leibler divergence between Gaussians (KLGauss). We also test several non-parametric measures applied to binned marginal probability distributions: Kullback-Leibler divergence (KL), χ2 statistic (Chi) and Jeﬀrey divergence (JD) [15],[16]. In classiﬁcation experiments with disjoint test and training sets, KLGamma performs better than the other tested parametric measures and KL. For block sizes smaller than 32x32 pixels KLGamma outperforms all other measures, and for larger blocksizes it is slightly worse than JD and Chi. We analyze the computional cost of all the tested similarity measures and ﬁnd that KLGamma is somewhat slower than WNN, and faster than the other parametric measures. Compared to non-parametric measures, which have similar classiﬁcation performance, KLGamma is at least an order of magnitude less computationally intensive. In tests with disjoint test and training sets, KLGamma achieves slightly better results than the second best parametric measure, KLGauss, while being computationally simpler. Research by Puzicha et.al. ([15] and [16]) has shown that the non-parametric Chi and JD measures outperform other non-parametric measures and the parametric WMV measure in texture retrieval tests. In their experiments, the extracted texture features were Gabor ﬁlter magnitude responses. It has been shown [2] that spatially smoothing the Gabor ﬁlter magnitude responses can improve classiﬁcation performance, and we will therefore do this kind of smoothing in our tests. Our main contributions are as follows: – We introduce the KLGamma texture similarity measure based on the Kullback-Leibler divergence between gamma distributions. – We conjecture that the spatially smoothed Gabor ﬁlter magnitude responses of some classes of visually homogeneous stochastic textures are gamma distributed. – We compare the classiﬁcation performance of several non-parametric and parametric similarity measures on spatially smoothed Gabor features. In addition to being justiﬁed from a statistical point of view, the combination of speed and classiﬁcation performance implies that the KLGamma measure is well suited to real-time texture segmentation and texture classiﬁcation/retrieval. In the next section we present texture feature extraction using spatially smoothed Gabor ﬁlter magnitude responses. We then introduce all the tested

Texture Similarity Measure Using Kullback-Leibler Divergence

135

similarity measures in the context of maximum-likelihood classiﬁcation. Then we present results that show how well the gamma distribution ﬁts the probability density function of the extracted features, and then present results from classiﬁcation tests that compare the KLGamma similarity measure with other parametric and non-parametric measures. Moreover, we present a comparison of the computational cost of the various similarity measures. Finally we discuss the results of this paper and oﬀer some conclusions and suggestions for future work.

2 2.1

Texture Feature Extraction Gabor Filtering

When extracting image features for use in texture analysis, it seems wise to look to the way texture is analyzed in the human visual cortex. Marcelja [4] and Daugman [5], [6] showed that the responses of simple cells in the visual cortex can be modeled by Gabor ﬁlter response. The 2D Gabor functions proposed by Daugman also achieve the theoretical resolution limit in the conjoint spatial-spatial frequency domain. Thus it is understandable that Gabor ﬁlters are commonly used in the feature extraction step of texture analysis. Based on the Gabor functions proposed by Daugman, Lee[1] derived a family of Gabor wavelets that satisfy the neurophysiological constraints for simple cells. 2 κ2 ω0 − ω02 (x2 +y2 ) i(ω0 x ) − e− 2 · e e 8κ ψ (x, y, ω0 , θ) = √ (1) 2πκ Here x = x cos θ +y sin θ and y = −x sin θ +y cos θ. ω0 is the radial frequency in radians per unit length and θ is the wavelet orientation in radians. The wavelet is centered at (x = 0, y = 0) and κ ≈ π for a frequency bandwidth of one octave. κ2

The term e− 2 in (1) is the d.c. response of the wavelet, and is subtracted to achieve invariance with respect to image brightness. For simplicity, let ψω0 ,θ denote the Gabor wavelet with parameters ω0 and θ. Let ψωr 0 ,θ be the real part of ψω0 ,θ and ψωi 0 ,θ be the imaginary part, and let Ixy be the image centered at (x, y) to be ﬁltered by ψω0 ,θ . Then the magnitude response of Ixy ﬁltered by ψω0 ,θ is |ψω0 ,θ ∗ Ixy | =

ψωr 0 ,θ ∗ Ixy

2

+ ψωi 0 ,θ ∗ Ixy

2

(2)

where ∗ is the convolution operator. Experiments have shown [2] that using this magnitude response achieves better texture separability than using full-wave rectiﬁcation or the real component only. This is most likely due to the fact that optimal conjoint resolution in the spatial-spatial frequency domain is achieved with the complex form (quadrature-phase pair) of the Gabor ﬁlter [1]. Pollen and Ronner’s [7] discovery, that the simple cells in the visual cortex exist in quadrature-phase pairs, suggests that the design of these cells might indeed be optimal with respect to the conjoint resolution in the spatial-spatial frequency domains.

136

2.2

J.R. Mathiassen, A. Skavhaug, and K. Bø

Spatial Smoothing

The performance of Gabor ﬁlters for texture analysis is known to improve by performing Gaussian smoothing of the Gabor ﬁlter magnitude response images ω2 2 2 0 [2]. If g(x, y, ω , θ) = √ω0 e− 8κ2 (x +y ) is the Gaussian term in (1), the magni0

2πκ

tude response images are smoothed with oriented Gaussian ﬁlter g(γx, γy, ω0 , θ). We denote the value at (x, y) of a smoothed magnitude image by cxy,ω0 ,θ,γ = g(γx, γy, ω0 , θ) ∗ |ψω0 ,θ ∗ Ixy | .

3

(3)

Similarity Measures

In this section we will present the similarity measures. To understand the reason for choosing a given similarity measure, one must look at the underlying assumptions that are made in deducing the measure. Therefore we will describe the maximum-likelihood classiﬁcation principle, and the similarity measures that result from making assumptions on the type of feature probability distributions. We then deﬁne the asymptotically (for large sample sizes) optimal similarity measures for Gaussian distributions, gamma distributions and binned marginal histogram distributions. For binned histograms we will also present some further measures used in [15] and [16]. Finally we will describe the parametric WMV similarity measure proposed by Manjunath and Ma [14] which was compared with several non-parametric methods in [15] and [16]. 3.1

Maximum-Likelihood Texture Classiﬁcation

Assume that a texture is described by a featurespace F. Given this featurespace we deﬁne a mapping g : F → C = {1, . . . , K} from the featurespace F to the set C of possible texture class labels. The goal we set for a texture classiﬁcation system is to minimize the probability of classiﬁcation error. This can be formalized as the minimization of p (g (x) =y), with x being a set of feature observations from a texture sample of class y. The optimal mapping g for this problem formulation is [13] the Bayes classifer g ∗ (x) = arg max p (x|y = i) P (y = i) i

(4)

where p (x|y = i) is the likelihood function for the ith class, and P (y = i) is it’s prior probability. When all texture classes have the same prior probability P (y = i) = 1/K (4) reduces to the maximum-likelihood (ML) classiﬁer [12] M 1 g (x) = arg max log p (xj |y = i) . i M j=1

(5)

It is in general computionally expensive to use (5) for large sample sizes M .

Texture Similarity Measure Using Kullback-Leibler Divergence

137

Kullback-Leibler Divergence. In [12] it is shown that for large values of M we can apply the law of large numbers to (5), resulting in M →∞ (6) g (x) −→ arg max Eq [log p (x|y = i)] = arg min KL q; p(i) i

i

where KL (q; pi ) is the Kullback-Leibler divergence (KL) between the query density q and the density p(i) associated with texture class i. This shows that KL is the asymptotic limit of the ML classiﬁcation criteria. If F has N dimensions, and these dimensions are independent other, the Kullback-Leibler di of each (i) N (i) = j=1 KL qj ; pj . This follows from (6) assuming vergence is KL q; p

N p(x) = j=1 pj (xj ). Here j denotes the feature dimension. 3.2

Gaussian Distribution

The univariate Gaussian distribution is given by pGauss (x; µ, σ) = 2 √ 1 e−0.5[(x−µ)/σ] . If the N dimensions of F are independent and Gaussian 2πσ distributed we have from [3] the following Kullback-Leibler divergence  

(q)

(q) 2 (p) 2 (p) 2 N σj µj − µj σj 1  log + + − 1 KLGauss (q; p) = (q) (p) (p) 2 j=1 σj σj σj (7) A simpliﬁcation of (7) gives us the weighted nearest neighbour classiﬁer (WNN)

(q) (p) 2 N µj − µj (8) DW N N (q; p) = (p) σj j=1 (q)

(p)

The condition under which this simpliﬁcation is justiﬁed is σj ≈ σj . We observe that the WNN measure is equivalent to the Mahalanobis measure [12] if the feature covariance matrix is diagonal. 3.3

Gamma Distribution

The gamma distribution is given by pGamma (x; α, β) = ∞ Γ (z) = 0 e−t tz−1 dt, z > 0.

1 α−1 −x/β e β α Γ (α) x

where

Parameter Estimation. There are two main methods of calculating estimates of the parameters α and β of the gamma distribution: the moment method and maximum-likelihood (ML) method. Using the moment method [9] we obtain the µ σ2 parameters by β = and α = , where µ and σ denote the estimated mean βˆ µ and standard deviation of the distribution. Using the ML estimation method [10] we obtain parameter estimates using n

1 Xi α β = X = n i=1

(9)

138

J.R. Mathiassen, A. Skavhaug, and K. Bø n

log α − Ψ ( α) = log X − log X =

1 log Xk − log X n

(10)

k=1

where X and log X are the mean and mean logarithm of the distribution given by a set of independent observations X = {X1 , . . . , Xn }, and Ψ (·) is the digamma function, i.e. Ψ (z) = Γ (z) /Γ (z). Kullback-Leibler Divergence between Gamma Distributions. A closedform expression for the KL-divergence between two Gamma distributions is given by [3] and results in the following expression KLGamma (q; p) =

N j=1 (p) +αj

(q) (q) (q) (q) (q) (p) αj − 1 Ψ αj − log βj −αj − log Γ αj + log Γ αj (p) log βj −

(p) αj

α(q) β (q) j j (q) (q) − 1 Ψ αj + log βj + (p) βj

(11)

Without any loss of classiﬁcation performance we can simplify (11) for use in (6), leading to the following similarity measure DKLGamma (q; p) =

N j=1

log Γ

(p) αj

(p) +αj

(p) (p) log βj −αj

α(q) β (q) j j (q) (q) Ψ αj + log βj + (p) βj

(12)

which we introduce in this paper for use as a texture similarity measure. 3.4

Binned Histograms

Marginal probability density functions can be modeled as binned histograms. For a given feature j we then have the following empirical distribution dimension xj pj (xj ) = , where · denotes the ﬂoor operator, and pj (z) = 0 if ∆lj z∈ / {0, . . . , L − 1}. L is the number of bins and ∆lj is the bin size. Kullback-Leibler Divergence. The Kullback-Leibler divergence between two empirical distributions is DKL (q, p) =

N L−1 j=1 l=0

qj (l) log

qj (l) . pj (l)

(13)

Appendix A shows that when the feature probabilities are modelled with binned histograms, g (x) = arg mini DKL q, p(i) is exactly the maximum-likelihood classiﬁer.

Texture Similarity Measure Using Kullback-Leibler Divergence

139

Fig. 1. 128x128 pixel patches of the textures used in the experiments. The textures are displayed in order of goodness-of-ﬁt with the gamma distribution model, with the ﬁt decreasing from left to right, and top to bottom

Jeﬀrey Divergence. A symmetrized version of the Kullback-Leibler divergence is the Jeﬀrey divergence [15] which is more numerically stable for empirical distributions. For binned histograms it is N L−1 DJD (q, p) = qj (l) log j=1 l=0

qj (l) pj (l) + pj (l) log . qj (l) + pj (l) qj (l) + pj (l)

(14)

χ2 Statistic. An approximation to the Kullback-Leibler divergence is the χ2 statistic [12]. A symmetrized version of this similarity measure is used in [15],[16] and is deﬁned for binned histograms as follows Dχ2 (q, p) =

N L−1 [qj (l) − pj (l)]2 j=1 l=0

3.5

qj (l) + pj (l)

.

(15)

Weighted Mean Variance (WMV)

A parametric similarity measure used for comparing textures is the weighted (i) mean variance (WMV) measure proposed by Manjunath and Ma [14]. Let µj (i)

and σj be the mean and standard deviation of feature dimension j. The superscript (i) denotes the texture class. The WMV measure between two textures q and p is then deﬁned as

140

J.R. Mathiassen, A. Skavhaug, and K. Bø Goodness-of-fit with gamma distribution 10

8

log(C/n)

6

4

2

0 A1 B1 C1 D1 E1 F1 G1 A2 B2 C2 D2 E2 F2 G2 A3 B3 C3 D3 E3 F3 G3 A4 B4 C4 D4 E4 F4 G4 A5 B5 C5 D5 E5 F5 G5

-2 Texture

Fig. 2. Goodness-of-ﬁt with gamma distribution

DW M V

(q) N µ − µ(p) σ (q) − σ (p) j j j j (q; p) = + α (µj ) α (σj )

(16)

j=1

where α (µj ) and α (σj ) are the standard deviations of the respective features over the entire database, and are used to normalize the individual feature components.

4 4.1

Results Validity of Gamma Distribution Model

In this section we illustrate how well the gamma distribution model ﬁts the distribution of the extracted texture features. As a goodness-of-ﬁt measure we use the N L (p (l)−π (l))2 , χ2 statistic [18]. With N feature dimensions it is C = n j=1 l=1 j πj (l)j where n is the number of samples, L is the number of bins, pj (l) the observed probability of bin l and πj (l) the expected probability of bin i assuming the feature is gamma distributed. πj (l) = Fj (l∆lj ) − Fj ((l − 1)∆lj ), where ∆lj is the bin size and Fj (z) is the cumulative gamma distribution. Given a gamma distrix Γ (αj ,z/βj ) bution with parameters αj and βj , Fj (z) = x Γ (α , Γx (α, t) = 0 e−t tα−1 dt j) [19]. The goodness-of-ﬁt with the gamma distribution for a given texture is calculated for feature distributions extracted from the entire 512x512 texture image. From ﬁgures 1 and 2 we see that the textures that best ﬁt the gamma distribution model are visually homogeneous and appear to be stochastic in nature. We also see that the textures which are either structered, such as G5, or nonhomogeneous such as C5 and F5, do not ﬁt the said model very well. We would like to point out that the texture patches in ﬁgure 1 are 128x128 blocks taken from larger 512x512 images. A consequence of this, is that some 128x128 patches

Texture Similarity Measure Using Kullback-Leibler Divergence Best (A1)

25% (B2)

Median (D3)

75% (F4)

141

Worst (G5)

ω0 = π ω0 = π / 2 ω0 = π / 4

Fig. 3. An illustration of how well the gamma distribution model ﬁts the extracted features. We show the histogram (jagged line) and ﬁtted gamma model (smooth line) for a representative set of textures. In these ﬁgures we have θ = π/4

may look homogeneous, while the entire 512x512 is not. This is for example the case for textures D4 and E4, and explains why they look very homogeneous in ﬁgure 1, but are not among the textures with the best goodness-of-ﬁt wrp the gamma distribution. The degree to which the features histograms are gamma distributed can be observed in ﬁgure 3. It is also worth noting that in preliminary experiments, we ﬁnd that adding a small amount of Gaussian noise to the texture images increased the goodness-of-ﬁt with the gamma model. From these observations we make the following conjecture: Conjecture 1. Spatially smoothed Gabor ﬁlter magnitude responses of some classes of visually homogeneous stochastic textures are gamma distributed. 4.2

Classiﬁcation Performance

We perform two diﬀerent sets of experiments to validate the classiﬁcation performance when using the diﬀerent similarity measures: – Experiment A: Training and testing on the same data – Experiment B: Training and testing disjoint data sets The ﬁrst set of experiments will indicate how well the feature set and similarity measure represent the textures. The second set of experiments will shed light on such issues as overtraining and generalization. Gabor Filter Parameters. For our classiﬁcation experiments the π π we3π ﬁlter images with Gabor ﬁlters of 4 diﬀerent orientations, θ ∈ 0, , , , and 5 4 2 4 π π π π diﬀerent scales, ω0 = π, 2 , 4 , 8 , 16 for a total of 20 ﬁlters. In the experiments with spatially smoothed magnitude images, a smoothing factor of γ = 2/3 was used for smoothing all Gabor ﬁlter magnitude response images. This is just a small portion of the possible parameter range of {θ, ω0 , γ}, and we see it as being similar to what is commonly used in the texture analysis community [11],[2]. A vector x in the featurespace F is deﬁned as x = [x1 , . . . , xN ] where N is the number of dimensions in the featurespace. The parameters used to extract the feature xj using (3) are {θj , ω0j } .

142

J.R. Mathiassen, A. Skavhaug, and K. Bø Table 1. Misclassiﬁed blocks in experiment A (in percent) Block size (in pixels) Measure 128x128 64x64 32x32 DKLGamma 0.9 4.8 10.6 DKL 0.4 4.3 7.9 DJD 0.9 4.2 10.3 Dχ2 0.9 3.9 10.0 0.7 4.4 10.8 KLGauss DW N N 1.6 6.0 14.0 DW M V 6.1 16.5 34.6

16x16 15.9 13.5 16.8 17.3 16.8 20.5 53.4

8x8 19.2 17.4 22.3 23.5 20.5 24.8 62.5

4x4 21.1 19.4 26.5 28.6 22.8 27.3 65.5

Experimental Setup. In our experiments we used a 35 image subset of the 40 images used in [16]. The image size is 512x512 pixels. For experiment A, training was performed on the entire images, and testing was performed on all non-overlapping blocks of size 128x128, 64x64, 32x32,...,4x4. One might question whether blocks as small as 4x4 pixels can represent the variance of the texture they are extracted from. Recall that the features we are analyzing are spatially smoothed Gabor ﬁlter responses. Thus the Gabor features in a 4x4 block contain information extracted from a ’receptive ﬁeld’ slightly larger than the (largest) Gabor ﬁlter. In experiment B we look at the ability to generalize. Training is performed on one quadrant of the image and testing on all non-overlapping blocks of size 128x128, 64x64, 32x32,...,4x4 in the remaining three quadrants. This procedure is repeated for all four quadrants, and the results presented are the average performance over all combinations of training and test sets. In both experiment A and B, the classiﬁcation performance was measured by the classiﬁcation precision, i.e. the percentage of correctly classiﬁed image blocks. (i) The histograms were binned by setting ∆lj = L1 maxx,y,i cxy,ω0j ,θj ,γ , where (i) denotes the texture and j the feature dimension. To avoid numerical instability when using the non-parametric methods we initialized the histogram bins to a small value δ = 10−5 before estimating the histograms. We experimented with values of L between 32 and 256, but found that classiﬁcation errors did not decrease noticably when increasing L from 64 to 256. Decreasing L from 64 to 32 gave marginally better classiﬁcation results for small block sizes, and worse for large block sizes. Tables 1 and 2 therefore display the results of the nonparametric measures with 64 bins. Our experiments showed that there was no noticable diﬀerence in classiﬁcation error between KLGamma with gamma distribution parameters estimated using the moment-method and KLGamma with parameters estimated using the ML method. We therefore display results from the former, due to its computational simplicity. Interpretation. In experiment A we see that the best measure is KL. This is expected since it is equivalent to using the maximum-likelihood classiﬁer, and the

Texture Similarity Measure Using Kullback-Leibler Divergence

143

Table 2. Misclassiﬁed blocks in experiment B (in percent) Block size (in pixels) Measure 128x128 64x64 32x32 DKLGamma 8.0 12.7 18.7 DKL 13.9 17.7 22.8 DJD 7.6 11.4 18.2 Dχ2 7.4 11.3 17.8 9.7 13.8 20.2 KLGauss DW N N 9.9 15.1 22.9 DW M V 11.1 22.7 39.2

16x16 23.7 27.1 24.5 24.7 25.4 28.9 57.3

8x8 26.9 30.2 29.6 30.7 28.8 32.9 68.3

4x4 28.8 31.2 33.6 35.5 30.8 35.0 73.3

training data are representative of the test data. We also observe that KLGamma is the best parametric method for blocksizes smaller than 64x64 pixels, which is also understandable since the assumption that the data are gamma distributed is a valid one for most of the textures. For blocksizes of 128x128 and 64x64 pixels, the KLGauss measure is slightly better than KLGamma. The diﬀerence between KL and KLGamma can be attributed to the fact the conjecture 1 does not hold for all of the textures in the dataset. JD and Chi are marginally better than KLGamma for block sizes larger than 16x16, and worse for smaller block sizes. Experiment B reveals that KL does not do to well when the test and training sets are disjoint. This is perhaps due to problems in estimating accurate histograms from the training data, either because the sample size is too small (256x256 pixels) or the training data are not representaive of the test data. Chi is the best method for block sizes larger than 16x16 with KLGamma being only slightly worse. KLGamma is the best parametric measure, and for smaller block sizes it is the overall best similarity measure. We attribute the diﬀerence in classiﬁcation error between experiment A and B to the fact that quite a few of the test images had internal textural variations, and were not entirely homeogeneous across the whole image. 4.3

Computational Cost

In this section we will look at how computationally intensive the similarity measures tested in the previous section are. We will analyze them in terms of how many operations are required to classify a query texture as being one of M textures. Parametric Measures. To give a fair speed comparison we will for each similarity measure deﬁne a set of parameters Q that must be calculated for each feature dimension of the query texture, and parameters P that must be calculated for each feature dimension of the database textures, in order for the similarity measure calculation to be as simple as possible. The equations in the second column of table 3 show the simpliﬁed calculations of each similarity measure for one feature dimension.

144

J.R. Mathiassen, A. Skavhaug, and K. Bø Table 3. Computational cost of parametric similarity measures

Measure

Expression

Q

DW N N

(Q1 −P1 )2 P2

DW M V

|Q1 − P1 | + |Q2 − P2 |

KLGauss P3 − Q3 + (Q1 −PP12) 2 DKLGamma P3 − P1 Q1 + Q P2

2

+Q2

P No. of ops 2 4N M µ, σ √ √ µ µ σ2 σ2 6N M + 4N α(µ) , α(σ) α(µ) , α(σ) 2 2 2 2 µ, σ , log σ µ, σ , log σ 7N M + 2N [Ψ (α) + log β, αβ] [α, β, log Γ (α) + α log β] 5N M + 7N [µ]

Table 4. Computational cost of non-parametric similarity measures Measure Expression DKL Q1 P1 DJD (−Q1 − P1 ) log (Q1 + P1 ) + P2 (Q1 −P1 )2 Dχ2 Q1 +P1

Q [−q] [q] [q]

P [log p] [p, p log p] [p]

No. of ops 2LN M 8LN M 5LN M

We assume that multiplications/divisions are as fast as additions/subtractions, one table lookup takes two operations and the |·| operator takes one operation. A further assumption is that the quantities µ and σ 2 are already calculated and the moment method is used to calculate the parameters of the gamma distribution. Also assume that the texture featurespace has N dimensions and the database texture parameters P are calculated in advance. In the rightmost column of table 3 we display the number of operations required to classify a query texture as being one of M possible database textures using the diﬀerent similarity measures. From these results we see that WNN is the fastest method. If M > 3 the proposed KLGamma measure is the second fastest. Non-parametric Measures. We will do a similar analysis for the nonparametric measures. Simpliﬁcations will be done where possible. Here Q and P will be calculated for each of the L histogram bins in all the N feature dimensions. q and p denote the bin values for the query texture and database texture respectively. These results show that the KL measure is the easiest to compute, and the Chi measure is signiﬁcantly faster than the JD measure. If we compare the computational cost of the parametric measures with the non-parametric measures we see that they are approximatly L times faster.

5

Discussion and Future Work

A question that naturally arises is whether the classiﬁcation results we have obtained represent what would typically be found in texture classiﬁcation applications. The ﬁrst issue that should be addressed in order to answer this question is whether the test data are are representative of ”typical” textures. To say they are would be jumping to conclusions without much evidence. However, one can not avoid noticing how well the gamma distribution model ﬁts the features

Texture Similarity Measure Using Kullback-Leibler Divergence

145

extracted from the visually homogeneous and stochastic textures. We suggest that further research should be done to determine for which classes of homogeneous stochastic textures our conjecture holds, and if it holds for homogeneous deterministic (e.g. polkadot) textures when there is noise present. Preliminary experiments we have done, suggest that this might be the case. In a later paper we will investigate more thoroughly the validity of our conjecture, give a rationale for why the extracted Gabor features appear to be gamma distributed, and determine how to extend the use of the KLGamma similarity measure to scale and rotation invariant texture classiﬁcation/retrieval. The second issue we wish to touch upon, is whether our computational cost comparison gives results that are valid in a realistic implementation scenarios. If the implementation scenario follows the assumptions we have made, the results should be valid. The pletora of diﬀerent specialized hardware and software platforms in existence makes it almost impossible to give a computional cost comparison that is valid in general. However, we can determine that the relative diﬀerence in computional cost between our proposed KLGamma measure and non-parametric measures is approximately an order of magnitude.

6

Conclusion

Based on observations of 35 texture images, we have conjectured that the spatially smoothed Gabor ﬁlter magnitude responses of some classes of visually homogeneous stochastic textures are gamma distributed. For the class of textures under which this conjecture holds, we have derived a computationally simple parametric similarity measure, the KLGamma measure, which is asymptotically optimal when used for texture classiﬁcation/retrieval. This texture similarity measure is based on the Kullback-Leibler divergence between gamma distributions. We compared the texture classiﬁcation performance of KLGamma with several parametric and non-parametric measures. Classiﬁcation experiments conﬁrmed that KLGamma is close to optimal. When training and testing was performed on the same dataset, the only method that noticably outperformed our method was the Kullback-Leibler divergence between binned histograms. We attribute this to the fact that some of the textures in our test suite were nonhomogeneous. Our conjecture does not hold for these textures. In tests with disjoint test and training sets, KLGamma was only slightly outperformed by the non-parametric Jeﬀrey divergence and symmetric χ2 statistic for block sizes larger than 16x16 pixels. For smaller blocks, KLGamma achieved the best classiﬁcation results of all the similarity measures we have tested. An analysis of the computational cost of all the tested measures, indicate that KLGamma is at least an order of magnitude faster than non-parametric measures, while having comparable, and under some conditions better, classiﬁcation performance. Only the simplest parametic method, WNN, is faster. Judging from the results we have obtained, we can conclude that, in scenarios under which our conjecture holds, our proposed KLGamma measure is the

146

J.R. Mathiassen, A. Skavhaug, and K. Bø

method of choice in e.g. real-time block-based segmentation algorithms such as that proposed by Hofman et.al. in [17]. KLGamma is also well suited for texture classiﬁcation applications where speed is of the essence and the number of texture classes is large.

Appendix A Let X be a sample of S feature vectors from a texture q. Xjk denotes sample k from feature dimension j. Let N be the number of dimensions in the feature (i) vectors. If a texture i is described by a set of binned histograms pj , the likelihood S

N

(i) Xjk . The pj of the samples X being from texture i is λi = j=1 ∆lj k=1 X maximum-likelihood classiﬁer is g(X) = arg maxi λi . If we let bjk = ∆ljkj N S (i) and take the log of λi we have log λi = j=1 k=1 log pj (bjk ). Now let njl denote the number of samples Xjk for which bjk = l. With this notation we N L−1 (i) obtain log λi = j=1 l=0 njl log pj (l). Furthermore, dividing by S on each side implies that g(X) = arg max λi = arg max i

(q)

Where pj

i

N L−1 (q) 1 (i) log λl = arg max λi pj (l) log pj (l) i S j=1 l=0

(17)

are the binned histograms of the query texture.

g(X) = arg max λi = arg min DKL q, p i

i

(i)

= arg min i

N L−1

(q) pj (l) log

(q)

pj (l) (i)

pj (l) (18) and thus minimizing the KL divergence is identical to the maximum-likelihood classiﬁer. j=1 l=0

References 1. Tai Sing Lee, Image Representation Using 2D Gabor Wavelets, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol 18, No. 10, October 1996 2. David A. Clausi, M. Ed Jernigan, Designing Gabor ﬁlters for optimal texture separability, Pattern Recognition 33 (2000) 1835-1849 3. W.D. Penny, KL-Divergences of Normal, Gamma, Direchlet and Wishart densities, Wellcome Department of Cognitive Neurology, University College London, March 30, 2001 4. S. Marcelja, Mathematical Description of the Responses of Simple Cortical Cells, J. Optical Soc. Am., vol. 70, pp. 1297-1300, 1980 5. J.G. Daugman, Two-dimensional Analysis of Cortical Receptive Field Proﬁle, Vision Research, vol. 20, pp. 847-856, 1980

Texture Similarity Measure Using Kullback-Leibler Divergence

147

6. J.G. Daugman, Uncertainty Relation for Resolution in Space, Spatial Frequency, and Orientation Optimized by Two-Dimensional Visual Cortical Filters, J. Optical Soc. Am., vol. 2, no. 7, pp. 1160-1169, 1985 7. D.A. Pollen and S.F. Ronner, Phase Relationship Between Adjacent Simple Cells in the Visual Cortex, Science, vol. 212, pp. 1409-1411, 1981 8. J.H. van Hateren and A. van der Schaf, Independent Component Filters of Natural Images Compared with Simple Cells in Primary Visual Cortex, Proc.R.Soc.Lond.B 265:359-366, 1998 9. Ronald E. Walpole and Raymond H. Myers, Probability and Statistics for Scientists and Engineers 5th Edition, Prentice Hall, 1993 10. C.T.J. Dodson and J. Scharcanski, Information Geometry For Stochastic Reﬁnement of Database Search and Retrieval Algorithms, 2001 11. A.C. Bovik, M. Clark, W.S. Geisler, Multichannel Texture Analysis Using Localized Spatial Filters, IEEE Trans. Pattern Anal. Machine Intell. 12 (1) (1990) 55-73 12. Nuno Vasconcelos and Andrew Lippman, A Unifying View of Image Similarity, MIT Media Lab 13. K. Fukunaga, Introduction to Statistical Pattern Recognition, Academic Press, 1990 14. B.S. Manjunath and W.Y. Ma, Texture Features for Browsing and Retrieval of Image Data, IEEE Trans. Pattern Anal. Machine Intell. 18 (8) (1996) 837-842 15. Jan Puzicha, Joachim M. Buhmann, Yossi Rubner and Carlo Tomasi, Empirical Evaluation of Dissimilarity Measures for Color and Texture, Proceedings of the Internation Conference on Computer Vision (ICCV’99), 1165-1173, 1999 16. Jan Puzicha, Thomas Hofmann and Joachim M. Buhmann, Non-parametric Similarity Measures for Unsupervised Texture Segmentation and Image Retrieval, Proc. of the IEEE Int. Conf. on Computer Vision and Pattern Recognition, San Juan, 1997 17. Thomas Hofmann, Jan Puzicha and Joachim M. Buhmann, Deterministic Annealing for Unsupervised Texture Segmentation, Proc. of the International Workshop on Energy Minimization Methods in Computer Vision and Pattern Recognition (EMMCVPR’97) 18. Richard J. Larsen and Morris L. Marx, An Introduction to Mathematical Statistics and Its Applications 3’rd Edition, Prentice Hall, 2001 19. W.H. Press, S.A. Teukolsky, W.T. Vetterling, and B.V.P. Flannery, Numerical Recipes in C, Cambridge, 1992 20. G. R. Cross and A. K. Jain. Markov Random Field Texture Models. IEEE Transactions on Pattern Analysis and Machine Intelligence 5/1, 1983 21. Chun-Shien Lu and Pau-Choo Chung. Wold Features for Unsupervised Texture Segmentation. Department of Electrical Engineering, National Cheng Kung University, Tainan, Taiwan, ROC, 1998 22. K. I. Laws. Rapid texture identiﬁcation. In Proc. of the SPIE Conference on Image Processing for Missile Guidance, pages 376-380, 1980 23. Timo Ojala and Matti Pietik¨ ainen, Unsupervised texture segmentation using feature distributions. Pattern Recognition, Vol. 32, pp. 477-486, 1999 24. Trygve Randen and John H˚ akon Husøy. Filtering for Texture Classiﬁcation: A Comparative Study. IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 21, no. 4, Apr 1999. Web: http://www.ux.his.no/˜tranden/

All the Images of an Outdoor Scene Srinivasa G. Narasimhan, Chi Wang, and Shree K. Nayar Columbia University, New York NY 10027, USA 500 West 120th street, Rm 450 Computer Science {srinivas, chi, nayar}@cs.columbia.edu

Abstract. The appearance of an outdoor scene depends on a variety of factors such as viewing geometry, scene structure and reﬂectance (BRDF or BTF), illumination (sun, moon, stars, street lamps), atmospheric condition (clear air, fog, rain) and weathering (or aging) of materials. Over time, these factors change, altering the way a scene appears. A large set of images is required to study the entire variability in scene appearance. In this paper, we present a database of high quality registered and calibrated images of a ﬁxed outdoor scene captured every hour for over 5 months. The dataset covers a wide range of daylight and night illumination conditions, weather conditions and seasons. We describe in detail the image acquisition and sensor calibration procedures. The images are tagged with a variety of ground truth data such as weather and illumination conditions and actual scene depths. This database has potential implications for vision, graphics, image processing and atmospheric sciences and can be a testbed for many algorithms. We describe an example application - image analysis in bad weather - and show how this method can be evaluated using the images in the database. The database is available online at http://www.cs.columbia.edu/CAVE/. The data collection is ongoing and we plan to acquire images for one year.

1

Variability in Scene Appearance

The appearance of a ﬁxed scene depends on several factors - the viewing geometry, illumination geometry and spectrum, scene structure and reﬂectance (BRDF or BTF) and the medium (say, atmosphere) in which the scene is immersed. The estimation of one or more of these appearance parameters from one or more images of the scene has been an important part of research in computer vision. Several researchers have focused on solving this inverse problem under speciﬁc conditions of illumination (constant or smoothly varying), scene structure (no discontinuities), BRDF (lambertian) and transparent media (pure air). Images captured to evaluate their methods adhere to the speciﬁc conditions. While understanding each of these speciﬁc cases is important, modeling scene appearance in the most general setting is ultimately the goal of a vision system. To model, develop and evaluate such a general vision system, it is critical to collect a comprehensive set of images that describes the complete variability in the appearance of a scene. Several research groups have collected images of a scene (for example, faces, textures, objects) under varying lighting conditions A. Heyden et al. (Eds.): ECCV 2002, LNCS 2352, pp. 148–162, 2002. c Springer-Verlag Berlin Heidelberg 2002

All the Images of an Outdoor Scene

149

and/or viewpoints in controlled lab environments. The CMU-PIE database [1] has 40000 facial images under diﬀerent poses, illumination directions and facial expressions. The FERET [2] database consists of 1196 images of faces with varying facial expressions. Similarly, the Yale Face database [3] has around 165 images taken under diﬀerent lighting, pose and occlusion conﬁgurations. The SLAM database [4] provides a set of 1500 images of toy objects under diﬀerent poses. The color constancy dataset collected by Funt et al. [5] provides a large set of images of objects (boxes, books and so on) acquired under diﬀerent poses and with diﬀerent illuminants (ﬂuorescent, halogen, etc). The CURET database [6] provides a set of 12000 images of real world textures under 200 illumination and viewing conﬁgurations. It also provides an additional set of 14000 Bi-directional Texture Function (BTF) measurements of 61 real world surfaces. Several databases of images of outdoor scenes have also been collected. The “natural stimuli collection” [7] has around 4000 images of natural scenes taken on clear, foggy and hazy days. Parraga et al. [8] provide a hyperspectral dataset of 29 natural scenes. The MIT city scanning project [9] provides a set of 10000 geo-referenced calibrated images acquired over a wide area of the MIT campus. These databases, however, do not cover the complete appearance variability (due to all outdoor illumination and weather conditions) in any one particular scene. Finally, web-cams [10] used for surveillance capture images regularly over long periods of time. However, they are usually low quality, non-calibrated, not tagged with ground truth data and focus only on activity in the scene. Note that the references we have provided for various databases are by no means complete. We refer the reader to [11] for a more comprehensive listing. In this paper, we present a set of very high quality registered images of an outdoor scene, captured regularly for a period of 5 months. The viewpoint (or sensor) and the scene are ﬁxed over time. Such a dataset is a comprehensive collection of images under a wide variety of seasons, weather and illumination conditions. This database serves a dual purpose; it provides an extensive testbed for the evaluation of existing appearance models, and at the same time can provide insight needed to develop new appearance models. To our knowledge, this is the ﬁrst eﬀort to collect such data in a principled manner, for an extended time period. The data collection is ongoing and we plan to acquire images for one year. We begin by describing the image acquisition method, the sensor calibration procedures, and the ground truth data collected with each image. Next, we illustrate the various factors that eﬀect scene appearance using images from our database captured over 5 months. We demonstrate thorough evaluation of an existing model for outdoor weather analysis, using the image database.

2 2.1

Data Acquisition Scene and Sensor

The scene we image is an urban scene with buildings, trees and sky. The distances of these buildings range from about 20 meters to about 5 kilometers. The large

150

S.G. Narasimhan, C. Wang, and S.K. Nayar Kodak DCS315 10-bit Camera

Weather Proof Enclosure Pan/Tilt Stage

24 -70 mm Zoom Lens

FireWire (IEEE 1394)

Anti-reflection Glass

Fig. 1. Acquisition setup

distance range facilitates the observation of weather eﬀects on scene appearance. See ﬁgure 5 for the entire ﬁeld of view. The digital camera we use for image capture is a single CCD KODAK Professional DCS 315 (see ﬁgure 1). As usual, irradiance is measured using 3 broadband R, G, and B color ﬁlters. An AF Nikkor 24 mm - 70 mm zoom lens is attached to the camera. 2.2

Acquisition Setup

The setup for acquiring images is shown in ﬁgure 1. The camera is rigidly mounted over a pan-tilt head which is ﬁxed rigidly to a weather-proof box (see black box in ﬁgure 1). The weather-proof box is coated on the inside with two coats of black paint to prevent inter-reﬂections within the box. An anti-reﬂection glass plate is attached to the front of this box through which the camera views the scene. Between the camera and the anti-reﬂection plate, is a ﬁlter holder (for, say, narrow band spectral ﬁlters). The entire box with the camera and the anti-reﬂection glass plate is mounted on a panel rigidly attached to a window. 2.3

Image Quality and Quantity

Images are captured automatically every hour for 20 hours each day (on an average). The spatial resolution of each image is 1520 × 1008 pixels and the intensity resolution is 10 bits per pixel per color channel. Currently, we have acquired images for over 150 days. In total, the database has around 3000 images. Due to maintainance issues that arise from prolonged camera usage (camera power failures and mechanical problems in controlling camera shutter), we have had to remove the camera twice from the enclosure. We believe the resulting loss of few days in the database can be tolerated since the dataset has enormous redundancy. The new image sets are registered with existing ones using the matlab image registration utility. To capture both subtle and large changes in illumination and weather, high dynamic range images are required. So, we acquire images with multiple exposures (by changing the camera shutter speed while keeping the aperture constant) and apply prior techniques to compute a high dynamic range image (≈ 12 bits per pixel) of the scene. Since the illumination intensity is expected to vary with time, the set of exposures are chosen adaptively. First, an auto-exposure image is taken and its shutter speed is noted. Then 4 more images are captured with exposures around this auto-exposure value. This type of adaptive exposure selection is commonly used by photographers and is called exposure bracketing.

All the Images of an Outdoor Scene

Focal Length u

0

,

v

0

151

2845 mm 7.2 , 9.5 pixels

Radial Dist. (C1 )

0.07

Reprojection Err

0.18 pixels

Fig. 2. Geometric calibration using planar checkerboard patterns. Table shows estimated intrinsic parameters. The distortion parameters not shown are set to zero.

3 3.1

Sensor Calibration Geometric Calibration

Geometric calibration constitutes the estimation of the geometric mapping between 3D scene points and their image projections. Since the calibration was done in a location diﬀerent from that used for image acquisition, we estimate only the intrinsic parameters of the camera. Intrinsic parameters include the eﬀective focal length, f , skew, s, center of projection (u0 , v0 ) and distortion parameters, C1 ...Cn (radial) and P1 , P2 (tangential). Then, the relation between observed image coordinates and the 3D scene coordinates of a scene point is: ui = Du s(xi vi = Dv (yi (r)

f (r) (t) + δui + δui ) + u0 zi

f (r) (t) + δvi + δvi ) + v0 , zi

(r)

(1) (t)

where (δui , δvi ) are functions of the radial distortion parameters, and (δui , (t) δvi ) are functions of the tangential distortion parameters. Du , Dv are the conversion factors from pixels to millimeters. See [12] for more details. We captured the images of a planar checkerboard pattern under various orientations (see ﬁgure 2). The corresponding corners of the checkerboard patterns in these images were marked. These corresponding corner points were input to a calibration routine [13] to obtain the intrinsic parameters. Figure 2 shows the estimated intrinsic parameters. The CCD pixels are square and hence skew is assumed to be 1. The deviation of the principal point from the image center is given by ∆u0 , ∆v0 . Only the ﬁrst radial distortion parameter, C1 , is shown. The remaining distortion parameters are set to zero. 3.2

Radiometric Calibration

Analysis of image irradiance using measured pixel brightness requires the radiometric response of the sensor. The radiometric response of a sensor is the mapping, g, from image irradiance, I, to measured pixel brightness, M : M = g(I) . Then, the process of obtaining I from M : I = g −1 (M ) , upto a global scale factor, is termed as radiometric calibration.

152

S.G. Narasimhan, C. Wang, and S.K. Nayar

(b) Auto Exposure

(a) High Exposure M Measured Intensity

( c) Low Exposure 2

G

R

1

2

1

B

0.5

3 0

0

0.5

1

3

1

I Image

Irradiance

(d) Radiometric Response

(e) Computed High Dynamic Range Image (Histogram Equalized)

Fig. 3. Radiometric Self-Calibration. (a) - (c) Three images (10 bits per pixel per RGB channel) captured with diﬀerent camera exposures. (d) Computed radiometric response functions of the 3 RGB channels. The response functions are linear with slopes 1.5923, 1.005 and 0.982 for R, G, and B respectively. The colors can be balanced by normalizing the slope of each response function. (e) Histogram equalized high dynamic range image irradiance map obtained by combining images taken with multiple exposures. Insets indicate that the dynamic range in this image is much higher than the dynamic range in any image captured using a single exposure.

The response functions of CCD cameras (without considering the gamma or color corrections applied to the CCD readouts) are close to linear. We computed the response functions of the 3 RGB channels separately using Mitsunaga and Nayar’s [14] radiometric self-calibration method. In this method, images captured with multiple exposures and the their relative exposure values are used to estimate the inverse response function in polynomial form. The results of the calibration are shown in the plots of ﬁgure 3(d). Notice that the response functions of R, G, and B are linear and they have diﬀerent slopes - 1.5923, 1.005 and 0.982 respectively. To balance the colors, we normalize the response functions by the respective slopes. The images taken with diﬀerent exposures are linearized using the computed response function. A high dynamic range image irradiance map (see ﬁgure 3) is obtained by using a weighted combination of the linearized images. This image has signiﬁcantly more dynamic range then any of the original images taken with single exposures [15]. The high dynamic range can prove very useful when analyzing both subtle and large changes in weather and illumination.

4

Ground Truth Data

Any database is incomplete without the accompanying ground truth. We have tagged our images with a variety of ground truth information. Most important categories of the ground truth we collected are scene depth and weather information. The depths of scene points are mainly obtained using satellite digital

All the Images of an Outdoor Scene

153

Conditions at 2001.03.06 11:51am

FOV

O

Wind

NNW (340 ) 10 MPH

Visibility Sky conditions

1 1/4 mile(s) Overcast Light snow, Mist A trace 32.0 F (0.0 C) 32.0 F (0.0 C) 100%

Weather Precipitation last hour Temperature Dew Point Relative Humidity

Fig. 4. Sample ground truth data. [Left] A satellite digital orthophoto of a portion of the scene. The red spot indicates the position of the sensor and bright region indicates the ﬁeld of view. Arcview [17] is used to measure the orthographic distance between any two scene points (seen in top view) with an accuracy of 1 meter. [Right] The weather data obtained from National Weather Service websites [18].

orthophotos supplied by the United States Geological Survey [16]. Arcview [17] is a mapping software that is used to measure the orthographic distance between two scene points (visible in the orthophotos) up to an accuracy of 1 meter. See ﬁgure 4 for an example of a satellite orthophoto. Note that accurate depth is not available at all pixels. However, since the ﬁeld of view consists of mainly vertical buildings, rough planar models can be used. The position (longitude, latitude and altitude) of the sensor is included in the database. This information along with the date and time of day, can be used to accurately compute sun and moon orientation relative to the sensor. For exact equations, see [19,20]. Every hour we automatically collect standard weather information from the National Weather Service web sites [18]. This includes information about sky condition (sunny, cloudy), weather condition (clear, fog, haze, rain), visibility, temperature, pressure, humidity and wind (see ﬁgure 4). Such information can be used to estimate the scattering coeﬃcient of the atmosphere [21].

5

WILD: Weather and ILlumination Database

We illustrate the variability in scene appearance due to weather, illumination, season changes, and surface weathering using images from our dataset captured over ﬁve months. 5.1

Variation in Illumination

The distribution of environmental illumination on a scene produces a wide variety of scene appearances. Commonly noticed eﬀects include shadows, colors of sunrise and sunset, and illumination from stars and moon at night. The human visual system relies on illumination in the scene to perceive scene reﬂectance (retinex and color constancy [22]) and shape [23] correctly. As a result, rendering a scene with consistent illumination is critical for realism in graphics. Considerable amount of eﬀort has been put into modeling outdoor illumination.

154

S.G. Narasimhan, C. Wang, and S.K. Nayar

The book “Daylight and its Spectrum” [24] provides a compendium of color and intensity distributions of skylight for many years of the 20th century. Daylight spectrum has also been represented using a set of linear bases [25]. Works that model clouds and their eﬀect on the ambient illumination also exist in literature [26,27]. In graphics, scenes have been rendered under diﬀerent daylight [28] and night illuminations [29]. Shadows are a powerful cue for shape and illumination perception. Rendering shadows and extracting shape information from shadows [30,31] are also important problems. Let us consider the various sources of illumination in any outdoor scene. The primary sources (self-luminous) include the sun during the day, the stars and lamps during night. There are numerous other secondary illumination sources such as skylight, ground light, moonlight, airlight [32] (due to scattering of light by the atmosphere), and scene points themselves (inter-reﬂections [33]). Our goal is to include the eﬀects of all these sources in one comprehensive database. Figure 5 shows 6 images from our database illustrating the various shadow conﬁgurations on a sunny day. Figure 6 shows diﬀerent illumination colors and intensities at sunrise, noon, sunset and night. Figure 7 depicts the variations in ambient lighting due to varying cloud covers. When viewed up-close, rough surfaces appear to have 3D textures (due to surface height variations) rather than 2D textures. The appearance of 3D textures has been modeled in [6]. Figure 8 shows the appearance of a rooftop with ridges at diﬀerent times of the day. Notice the change in appearance of cast shadows due to surface height variations. The above variability in a single database facilitates research groups to study the illumination eﬀects individually as well as simultaneously. For instance, one may use just sunny day images at one particular time of day, when the sun position remains constant and shadows do not change. In another instance, one can consider images captured only on cloudy days to model scenes under ambient lighting. 5.2

Variation in Weather Conditions

Most vision algorithms assume that light travels from a scene point to an observer unaltered. This is true only on clear days. In bad weather, however, the radiance from a scene point is severely scattered and thus, the appearance of a scene changes dramatically. The exact nature of scattering depends on a variety of factors such as shapes, orientations and densities of atmospheric particles and the colors, polarizations, and intensities of incident light [34]. Recently, there has been work on computing scene depths from bad weather images [35] as well as to remove weather eﬀects from images. Some works extract structure and clear day scene colors using images of a scene taken under diﬀerent weather conditions [36] or through diﬀerent polarizer orientations [37]. Other works use pre-computed or measured distances to restore scene contrast [38,39]. In graphics, scenes (including skies) have been rendered considering multiple scattering of light in the atmosphere [28,40].

All the Images of an Outdoor Scene

09/07/2001, 10 AM Clear and Sunny

09/07/2001, 1 PM Clear and Sunny

09/07/2001, 11 AM Clear and Sunny

09/07/2001, 12 Noon Clear and Sunny

09/07/2001, 2 PM Clear and Sunny

09/07/2001, 3 PM Clear and Sunny

155

Fig. 5. Images illustrating diﬀerent shadow conﬁgurations on a clear and sunny day. Shadows provide cues for illumination direction and the scene structure. Notice the positions of the sharp shadows on the buildings.

09/05/2001, 6 AM Sun Rise

09/05/2001, 12 noon Noon

09/05/2001, 7 PM Sun Set

09 / 05/2001, 10 PM Night

Fig. 6. Images illustrating the various colors and intensities of illumination at sunrise, noon, sunset and night. Notice the signiﬁcant change in the colors of the sky.

7 AM, Partly Cloudy, Partly Sunny

7 AM, Increased Cloud Cover

7 AM, Overcast Sky

Fig. 7. Images showing various levels of cloud cover. The image on the left shows the appearance of the scene with a few scattered clouds. The two images on the right were taken under mostly cloudy and completely overcast conditions. Notice the soft shadows due to predominant ambient lighting.

156

S.G. Narasimhan, C. Wang, and S.K. Nayar

12 Noon

9 AM

3 PM

Fig. 8. When viewed at close proximity (ﬁne scale), the appearances of surfaces should be modeled using the bi-directional texture function (BTF) instead of the BRDF. Notice the change in cast shadows due to the ridges on the rooftop. All images are histogram equalized to aid visualization.

06/08/2001, 1 PM Clear and Sunny

06/02/2001, 1 PM Light Mist and Rain

06/14/2001, 1 PM Hazy

09/14/2001, 1 PM Foggy

Fig. 9. Images taken at the same time of day but under diﬀerent weather conditions. Notice the degradation in visibility, especially, of far away scene points in bad weather.

05/28/01, 11 PM Clear Night

05/26/01, 11 PM Misty Night

Fig. 10. Night images showing light sources under clear and misty conditions. The sources appear like specks of light on a clear night. Notice the light spreading due to multiple scatterings on a misty night.

How can the above models for weather analysis be evaluated? Under what conditions do these models fail or perform well? To satisfactorily answer such questions, we need to evaluate the performance of such models and algorithms on images of a scene captured under a wide variety of illumination and weather

All the Images of an Outdoor Scene

157

conditions. Our database includes images of the scene under many atmospheric conditions including clear and sunny, fog, haze, mist, rain and snow. Figure 9 shows 4 images of the same scene captured under diﬀerent weather conditions. Notice the signiﬁcant reduction in contrast (and increase in blurring) in far away buildings. Broadening of light beams due to multiple scatterings in the atmosphere is clearly illustrated by the lamps imaged at night (see ﬁgure 10). Consider the event of mild fog setting in before sunrise, becoming dense as time progresses and ﬁnally clearing by noon. We believe that such lengthy, time varying processes can be studied better using our database. Study of such processes have potential implications for image based rendering. 5.3

Example Evaluation: Weather Analysis

Consider images taken under diﬀerent weather conditions. The observed color, E, of a scene point in bad weather is linearly related to its clear day color ˆ and the color of the weather condition (say, fog or haze), A, ˆ by the direction, D, dichromatic model of atmospheric scattering [35]: ˆ + nA ˆ. E = mD (2) ˆ ˆ So, E, A, and D lie on the same “dichromatic plane” (see ﬁgure 11). Here, m = E∞ ρe−βd , n = E∞ (1 − e−βd ), d is the depth of the scene point from the observer, β is called the scattering coeﬃcient of the atmosphere [41], E∞ is the brightness at the horizon, and ρ is a function of the scene point BRDF [36]. Under what weather conditions is the dichromatic model valid? How well does the dichromatic model describe the colors of scene points under a particular weather condition (say, mist)? Figure 11 shows the results of evaluating the model for fog, haze, mist and rain using multiple (5 in this case) images taken under each weather condition. Based on the dichromatic model, Narasimhan and Nayar [36] developed constraints on changes in scene colors under diﬀerent atmospheric conditions. Using these constraints, they developed algorithms to compute 3D structure and clear day colors of a scene from two or more images taken under diﬀerent but unknown weather conditions. How do we evaluate the performance of such an algorithm? Figure 12 shows the comparison of the defogged image with the actual clear day image of the scene under similar illumination spectra (overcast skies). The accuracy of the computed scaled depth, βd, is compared against the ground truth relative depth values obtained from the satellite orthophotos. This demonstrates that models and algorithms pertaining to weather can be evaluated more thoroughly using images from this database. Narasimhan and Nayar’s algorithm described above shows that clear day images can be obtained using two images taken under diﬀerent weather (say, foggy) conditions. Given two foggy images, can we generate novel foggy images? ˆ fog color A, ˆ and optical We show this is possible using the defogged color ρ D, depth βd, at each pixel. Note that these quantities can be computed using the above algorithm. Scattering coeﬃcient β is a measure of the density of fog. Thus, the density of fog can either be increased or decreased by appropriately scaling

158

S.G. Narasimhan, C. Wang, and S.K. Nayar ∧

B

D

∧

D

E

Dichromatic Plane

m ∧

n

E1

A

O

Rain

E2

R

O

A

(b)

Overcast

Dense Haze Overcast ∧

G

(a)

o Weather Sky Avg Err Err < 3 o Fog Overcast 0.58 95 % Mist Overcast 1.25 o 88 %

Mild Haze

Sunny

1.13o 2.27o 3.61o

91 % 76 % 44 %

(c)

Fig. 11. Dichromatic plane geometry and its evaluation. (a) Dichromatic Model [35]. (b) The observed color vectors Ei of a scene point under diﬀerent (two in this case) weather conditions (say, mild fog and dense fog) lie on a plane called the dichromatic plane. (c) Experimental veriﬁcation of the dichromatic model with the scene imaged 5 times under each of the diﬀerent foggy, misty, rainy and hazy conditions. The third column is the mean angular deviation (in degrees) of the observed scene color vectors from the estimated dichromatic planes, over 1.5 megapixels in the images. The fourth column provides the percentage of pixels whose color vectors were within 3 degrees of the estimated dichromatic plane. Note that the dichromatic model works well for fog, mist, rain and dense haze under overcast skies. For mild haze conditions under sunny skies, the model does not perform well. Such evaluation is possible only since our database has several images under each weather condition.

the optical depth βd. Substituting the scaled optical depth k βd, clear day color ˆ and fog color A, ˆ into equation 2, we compute the colors of scene points ρD, under novel fog conditions. The horizon brightness E∞ is kept constant since it is just a global scale factor. Figure 13 shows 4 novel images generated with increasing fog densities. 5.4

Seasonal Variations

The types of outdoor illumination and weather conditions change with seasons. For instance, the intensity distribution of sunlight and skylight diﬀer from summer to winter [24]. Similarly, the atmospheric conditions that manifest in fall are signiﬁcantly diﬀerent from those that occur in winter. Models of the atmosphere in diﬀerent seasons can be found in [21] and other related papers [42]. Since we acquire the images for over a year, changes in scene appearance due to changes in seasons can be studied. For instance, one might easily compare the images taken on a clear day in spring with images taken on a clear day in winter under identical sensor settings. Figure 14 shows 2 images taken at the same time of day in summer and fall. 5.5

Surface Weathering

Over substantial periods of time, we commonly see oxidation of materials (rusting), deposition of dirt on materials and materials becoming wet or dry. These eﬀects are important for realistic scene rendering and have been modeled by Dorsey and Hanrahan [43]. Since our images have high spatial resolution, portions of the image corresponding to small regions in the scene (say, a portion of

All the Images of an Outdoor Scene

5 PM, Fog

d2

d3

159

6 PM, Fog

d4

d1 Computed Depth Map (Brightened)

Computed Defogged Image

d5

Relative Depth

Ground Truth

Computed Value

d3 / d1

9.55

10.457

d2 / d1

6.98

7.72

d4 / d1

6.41

6.49

d5 / d1

8.82

9.30

Relative Depth Verification using Satellite Orthophoto Data

Actual clear day image (under mostly cloudy skies)

Fig. 12. Computing structure and defogging from two foggy images. Table comparing the computed relative depths with ground truth relative depths (obtained using satellite orthophotos) of 5 diﬀerent regions, d1 − d5 , in the scene. The relative depths are averaged over small neighborhoods. The window regions do not remain constant and thus produce erroneous depth values. All the images are contrast stretched for display purposes.

160

S.G. Narasimhan, C. Wang, and S.K. Nayar

Fig. 13. Generation of novel images with increasing fog densities (or scattering coefﬁcients). The relative scattering coeﬃcients used in this case are β, 2β, 3β and 4β respectively.

Summer Season, 06/15/2001, 11 AM

Fall Season, 09/15/2001, 11 AM

Fig. 14. Images taken at the same time of day but on days in summer and fall. Both the images were taken on clear and sunny days. Notice the subtle diﬀerences in colors and the positions of shadows.

Fig. 15. Portions of a rooftop in the scene when it is dry, partly and completely wet.

a wall) can be analyzed. Figure 15 shows a small patch in the scene when it is dry and wet.

All the Images of an Outdoor Scene

6

161

Summary

The general appearance of a scene depends on a variety of factors such as illumination, scene reﬂectance and structure, and the medium in which the scene is immersed. Several research groups have collected and analyzed images of scenes under diﬀerent conﬁgurations of illuminations (both spectrum and direction), and viewpoints, in controlled lab environments. However, the processes that effect outdoor scene appearance such as climate, weather and illumination are very diﬀerent from indoor situations. Ultimately, vision algorithms are expected to work robustly in outdoor environments. This necessitates a principled collection and study of images of an outdoor scene under all illumination and weather conditions. We have collected a large set of high quality registered images of an outdoor urban scene captured periodically for ﬁve months. We described the acquisition process, calibration processes and ground truth data collected. The utility of the database was demonstrated by evaluating an existing model and algorithm for image analysis in bad weather. This dataset has potential used in vision, graphics and atmospheric sciences. Acknowledgements. This work was supported in parts by a DARPA HumanID Contract (N00014-00-1-0916) and an NSF Award (IIS-99-87979). The authors thank E. Rodas for building the weather-proof camera enclosure.

References 1. Sim, T., Baker, S., Bsat, M.: The CMU pose, illumination and expression (PIE) database of faces. Tech Report CMU-RI-TR-01-02 (2001) 2. Phillips, P., Wechsler, H., Huang, J., Rauss, P.: The feret database and evaluation procedure for face-recognition algorithms. Image and Vision Computing 16 (1998) 3. Belhumeur, P.N., Hespanha, J.P., Kriegman, D.J.: Eigenfaces vs. Fisherfaces: Recognition using class speciﬁc linear projection. PAMI 19 (1997) 711–720 4. Murase, H., Nayar, S., Nene, S.: Software library for appearance matching (slam). In: ARPA94. (1994) I:733–737 5. Funt, B., Barnard, K., Martin, L.: Is machine colour constancy good enough? In: Proc ECCV. (1998) 6. Dana, K., Nayar, S., van Ginneken, B., Koenderink, J.: Reﬂectance and texture of real-world surfaces. In: Proc CVPR. (1997) 151–157 7. van Hateren, J.H., van der Schaaf, A.: Independent component ﬁlters of natural images compared with simple cells in primary visual cortex. In Proc. Royal Society of London B 265 (1998) 359 – 366 8. Parraga, C.A., Brelstaﬀ, G.J., Troscianko, T., Moorhead, I.: Color and illumination information in natural scenes. JOSA A 15 (1998) 563–569 9. Teller, S., Antone, M., Bodnar, Z., Bosse, M., Coorg, S., Jethwa, M., Master, N.: Calibrated registered images of an extended urban area. In: Proc. CVPR. (2001) 10. Hazecam: A live webcam. (In: http://www.hazecam.net) 11. CMUPage: The computer vision home page. (In: http://www.cs.cmu.edu/ cil/vision.html) 12. Heikkila, J., Silven, O.: A four-step camera calibration procedure with implicit image correction. In: Proc. CVPR. (1997) 1106–1112

162

S.G. Narasimhan, C. Wang, and S.K. Nayar

13. Bouguet, J.Y.: Camera calibration toolbox. (In: http://www.vision.caltech.edu/ bouguetj/calib doc) 14. Mitsunaga, T., Nayar, S.: Radiometric self calibration. In: CVPR. (1999) I:374–380 15. Nayar, S., Mitsunaga, T.: High dynamic range imaging: Spatially varying pixel exposures. In: Proc. CVPR. (2000) I:472–479 16. USGS: U.S. Geological Survey Mapping home page. (In: http://mapping.usgs.gov) 17. ESRI: The ESRI home page. (In: http://www.esri.com) 18. NWS: The national weather service home page. (In: http://www.nws.noaa.gov) 19. Aurora: Sun angle basics. (In: http://aurora.crest.org/basics/solar/angle/) 20. Naval-Observatory: The Astronomical Alamanac for the Year 2001. US R. G. O Government Printing Oﬃce (2001) 21. Acharya, P.K., Berk, A., Anderson, G.P., Larsen, N.F., Tsay, S.C., Stamnes, K.H.: Modtran4: Multiple scattering and BRDF upgrades to modtran. SPIE Proc. Optical Spectroscopic Techniques and Instrumentation for Atmospheric and Space Research III 3756 (1999) 22. Land, E., McCann, J.: Lightness and retinex theory. JOSA 61 (1971) 23. Ramachandran, V.: Perceiving shape from shading. SciAmer 259 (1988) 76–83 24. Henderson, S.T.: Daylight and its Spectrum. New York : Wiley (1977) 25. Slater, D., Healey, G.: What is the spectral dimensionality of illumination functions in outdoor scenes? In: Proc. CVPR. (1998) 105–110 26. Moon, P., Spencer, D.: Illumination from a non-uniform sky. Illum Engg 37 (1942) 27. Gordon, J., Church, P.: Overcast sky luminances and directional luminous reﬂectances of objects and backgrounds under overcast skies. App. Optics 5 (1966) 28. Tadamura, K., Nakamae, E., Kaneda, K., Baba, M., Yamashita, H., Nishita, T.: Modeling of skylight and rendering of outdoor scenes. In: Eurographics. (1993) 29. Jensen, H.W., Durand, F., Stark, M.M., Premoze, S., Dorsey, J., Shirley, P.: A physically-based night sky model. In Proc. SIGGRAPH (2001) 30. Kender, J., Smith, E.: Shape from darkness. In: Proc. ICCV. (1987) 539–546 31. Kriegman, D., Belhumeur, P.: What shadows reveal about object structure. JOSAA 18 (2001) 1804–1813 32. Koschmieder, H.: Theorie der horizontalen sichtweite. Beitr. Phys. freien Atm., 12 (1924) 33–53,171–181 33. Nayar, S., Ikeuchi, K., Kanade, T.: Shape from interreﬂections. IJCV 6 (1991) 34. Hulst, V.D.: Light Scattering by small Particles. John Wiley and Sons (1957) 35. Nayar, S., Narasimhan, S.: Vision in bad weather. In Proc. ICCV (1999) 36. Narasimhan, S., Nayar, S.: Chromatic framework for vision in bad weather. In Proc. CVPR (2000) 37. Schechner, Y., Narasimhan, S., Nayar, S.: Instant dehazing of images using polarization. In Proc. CVPR (2001) 38. Oakley, J., Satherley, B.: Improving image quality in poor visibility conditions using a physical model for degradation. IEEE Trans. on Image Processing 7 (1998) 39. Yitzhaky, Y., Dror, I., Kopeika, N.: Restoration of altmospherically blurred images according to weather-predicted atmospheric modulation transfer function. Optical Engineering 36 (1998) 40. T. Nishita, Y. Dobashi, E.N.: Display of clouds taking into account multiple anisotropic scattering and sky light. SIGGRAPH (1996) 41. McCartney, E.: Optics of the Atmosphere: Scattering by molecules and particles. John Wiley and Sons (1975) 42. AFRL/VSBM: Modtran. (In: http://www.vsbm.plh.af.mil/soft/modtran.html) 43. Dorsey, J., Hanrahan, P.: Digital materials and virtual weathering. Scientiﬁc American 282 (2000)

Recovery of Reﬂectances and Varying Illuminants from Multiple Views Q.-Tuan Luong1 , Pascal Fua2 , and Yvan Leclerc1 1

2

AI Center,SRI International, CA-94025 Menlo Park, USA, {luong,leclerc}@ai.sri.com Computer Graphics Lab, EPFL, CH-1015 Lausanne, Switzerland [email protected]

Abstract. We introduce a new methodology for radiometric reconstruction from multiple images. It opens new possibilities because it allows simultaneous recovery of varying unknown illuminants (one per image), surface albedoes, and cameras’ radiometric responses. Designed to complement geometric reconstruction techniques, it only requires as input the geometry of the scene and of the cameras. Unlike photometric stereo approaches, it is not restricted to images taken from a single viewpoint. Linear and non-linear implementations in the Lambertian case are proposed; simulation results are discussed and compared to related work to demonstrate the gain in stability; and results on real images are shown.

1

Introduction

Consider cdhaving several magazines with pictures of a mountain villa taken at diﬀerent times of day and from diﬀerent vantage points. Using modern bundle adjustment and geometric reconstructions techniques, it is possible to recover the geometry of the scene from the pictures. However, there is currently no way to automatically recover the unknown radiometric responses of the cameras/digitizers and the unknown illumination for each picture. Without these, it is not possible to create a photo-realistic rendering of the scene. Given the scene geometry and the camera’s geometric parameters, we call the recovery of these unknowns the radiometric recovery problem. Here, we focus on an important special case of the general problem in which the unknown camera/digitizer responses are modelled by an aﬃne transformation (scale and oﬀset), the unknown illuminants in each picture are modelled by a single point light sources at inﬁnity plus ambient illumination, and the

This work was sponsored in part by the Defense Advanced Research Projects Agency under contract F33615-97-C-1023 monitored by Wright Laboratory and in part by the Swiss Federal Oﬃce for Education and Science. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the oﬃcial policies, either expressed or implied, of the Defense Advanced Research Projects Agency, the United States Government, SRI International or EPFL.

A. Heyden et al. (Eds.): ECCV 2002, LNCS 2352, pp. 163–179, 2002. c Springer-Verlag Berlin Heidelberg 2002

164

Q.-T. Luong, P. Fua, and Y. Leclerc

scene BRDF is textured Lambertian. This is reasonably close to the mountain villa scenario above. The main insight of this work is that these unknowns can be recovered from multiple images in much the same way as the geometry of a rigid scene can be recovered from multiple images with unknown but varying viewpoints. Compared to the photometric stereo approaches, there is no need to use a constant viewpoint, thus allowing us to take advantage of the reliable and ﬂexible existing solutions for the geometric reconstruction problem. The decoupling of geometry and radiometry that we advocate makes sense for many applications. In some cases, for example in an industrial setting, 3–D models of target objects may be available. In many other cases, they can be built using well understood vision-based techniques: When performing change detection using images of the same sites taken at diﬀerent times of day, digital terrain maps are available or can be computed with geometric reconstruction techniques, such as stereo based on normalized cross-correlation, which can handle moderate lighting variations. It is then natural to combine maps and actual images to estimate the illumination parameters, recover surface albedoes and detect their changes over time. When creating 3–D models of objects or people from video sequences for rendering purposes, the illumination may vary from image to image because the subject moves with respect to ﬁxed illuminants. After using structure-from-motion techniques that are relatively insensitive to illumination changes to recover geometry, one can attenuate the lighting eﬀects in the original images to create high-quality texture maps. Table 1. Radiometric reconstruction in relation with previous general frameworks

Framework Camera geom. Scene geom. Reﬂectance Illumination Geom. reconstruction recovered recovered some variation some variation Shape from shading single recovered constant recovered Photometric stereo constant recovered recovered known General. phot. stereo constant partly recov’d partly recov’d partly recov’d Inverse lighting single known known or constant recovered Reﬂectance modeling known known recovered known Our approach known known recovered recovered

Previous work is summarized in Table 1 and discussed in detail in Section 2. While the fact that pixel intensities are in a bilinear relation with surface albedo and illumination radiances is well known, it has not been used before for radiometric recovery from multiple views and varying illuminants. Our goal is to introduce a framework for such a task and associated algorithms. As a ﬁrst step towards the general problem of radiometric reconstruction, we investigate the case of Lambertian surfaces. The relevant lighting model consists of a single light source for each input image. In Section 3, we describe the linear radiometry of multiple illuminants, give a characterization of the minimal data required, and then propose in Section 4 a linear algorithm for the recovery of the radiometry of multiple views. A more

Recovery of Reﬂectances and Varying Illuminants from Multiple Views

165

detailed account of this theory, as well as a parallel between the radiometric reconstruction problem and the geometric reconstruction problem can be found in [12]. Simulations shows this linear method to be quite sensitive to measurement noise. To obtain a more robust solution, we turn in Section 5 to a non-linear method. The results of this approach compare very favorably with those of the SVD method [8,3,26]. Section 6 discusses limitations of the current system and areas for future developments.

2

Previous Work

Shape from shading [10] assumes that the reﬂectance is Lambertian and uniform. Surface shape, and the direction of the illuminant are recovered from a single [27] or multiple images [17]. Photometric stereo, by using multiple images, each taken with the same viewpoint but a diﬀerent illuminant explicitly known [23,24,9] or implicitly determined [1], recovers a non-uniform albedo, as well as the surface orientation. Subsequently, it was shown [8] that even with unknown illuminants, albedo and shape can be recovered using SVD, which generalized photometric stereo. However, both of them are subject to a Generalized Bas-Relief ambiguity [3,26]. This is intrinsic to the problem and can be entirely resolved only by suﬃcient prior knowledge about the object’s geometry. For novel view generation, knowing that the object belongs to an instance of a class [7] provides an adequate approximation. These methods are powerful. If given a suﬃcient number of 3D measurement on the object, they can recover both the illuminant and reﬂectance. However their precision has not been documented, nor their applicability to unconstrained or large environments. In Sec. 5.2 we compare experimentally the SVD approach with our radiometric reconstruction approach. The primary purpose of all the previous approaches is to recover the shape based on the Lambertian model. A second family of methods recovers radiometric parameters assuming that the camera and object geometry is available. Inverse lighting [13,14] assumes that the reﬂectance is Lambertian, and known or constant. The illuminant is modeled as a linear combination of a basis of predeﬁned lights, whose coeﬃcients are recovered from a single image using a linear relationship. The dual approach, reﬂectance modeling, consists in assuming that the point light source is known. It recovers sophisticated reﬂectance properties with specular and diﬀuse parameters [11,2,21,13]. This has been extended to multiple known point light sources, as well as indirect illumination eﬀects [25]. These approaches recover rich models of either the illuminant or the reﬂectance, but unlike the method that we propose, they cannot recover both. By contrast, the approach of [20] can. It is, however, not generic because it depends on a speciﬁc situation where the shadow of an object of known shape is projected onto an object also of known shape. Recent work [18,19] done simultaneously to ours [12] demonstrated that a complex lighting model, including multiple lightsources and specular properties, can be recovered given several images of an

166

Q.-T. Luong, P. Fua, and Y. Leclerc

object for which a 3–D model is available but, unlike in our approach, requires assuming that the lighting is the same in all images.

3 3.1

Linear Model for Multiple-Illuminant Radiometry The Lambertian Model

We model the eﬀect of illumination as the sum of a Lambertian term from a distant point light source, and an ambient term. For a surface element of normal n and albedo α that projects into an image i at coordinates (u, v), the radiance at the image plane is given by: Li (u, v) = α(lTi n + µi )

(1)

where li is the light source vector, µi is the ambient light. Each surface element is characterized by 3 parameters, the direction of n, and the albedo α. While n is a unit vector, li is not: its magnitude encodes the light source’s “intensity.” The complete illumination model for each image is therefore given in terms of 4 parameters described by the illumination vector: Li = [li1 , li2 , li3 , µi ]T . Using t the illumination vector, and the notation N = [n, 1] , Eq. (1) can be written T more compactly as: Li (u, v) = αLi N. 3.2

Radiometric Calibration

The measures that we can access are pixel gray-level values. However, the quantity that has a physical meaning is the radiance at the image plane. A camera is said to be radiometrically calibrated when the relationship between these quantities is known. This (non-linear) relationship can be recovered using multiple diﬀerent known [4] or unknown [15] exposures of a ﬁxed scene. In this paper, we illustrate in a simple case how the radiometric (self) calibration can be attacked, rather than investigating it thoroughly. We will approximate the relationship by an aﬃne transformation speciﬁed with a radiometric scale factor ai and radiometric oﬀset bi : Ii (u, v) = ai Li (u, v) + bi

(2)

We will see that within this limitation, there is enough information to recover linearly one of the parameters, the radiometric oﬀset. When using our non-linear approach, a suﬃciently large amount of data could allow to recover non-linear parametric terms. 3.3

Multiple Images

The key idea of our approach is to use multiple images of the same surface, taken with diﬀerent illuminations. We assume that the geometry of the surfaces and cameras has been previously recovered, which makes it possible to express

Recovery of Reﬂectances and Varying Illuminants from Multiple Views

167

all the 3-D spatial relationship in a common coordinate system. In this paper, we restrict ourselves to the case where only one point light source is used at a time. Each image i is therefore characterized by its illumination vector Li . In this context, the only relevant geometric information are the normals nj , of each surface element j , which we assume to have been determined. The only radiometric parameter for each surface element is its albedo αj . We have p surface elements, with n diﬀerent illuminants, which give rise to the p × n radiance values Lij and pixel gray-level values Iij , according to Eq. (1) and Eq. (2) : Iij = ai Lij + bi = ai αj LTi Nj + bi (3) Since there are a total number of p + 4 × n + 2 × n unknowns: – albedoes αj , j = 1 . . . p, – illumination vectors Li , i = 1 . . . n, – radiometric calibration parameters ai , bi , i = 1 . . . n, and a total number of p×n equations, with a suﬃcient number of diﬀerent surface elements and illuminations, we can expect to solve the radiometric reconstruction problem. 3.4

Ambiguities and Minimal Data

Going back to the radiometric reconstruction problem, we notice from Eq. (3) that if: ai , bi , αj , Li is a solution, then another acceptable solution is obtained for any scalars s > 0 and ki , i = 1 . . . n αj ai , bi , , ski Li ki s The ambiguity due to the multiplier s is a uniform scale ambiguity, which means that we cannot distinguish between brighter surfaces lit by a dimmer illuminant or darker surfaces lit by a more intense illuminant. Only the positive values of the global scale s are acceptable because of the physical constraint that the albedoes be positive. In the case of radiometrically calibrated cameras, the scale ambiguity is the only generic ambiguity of the problem. The ambiguity due to the multipliers ki means that for photometrically uncalibrated cameras, the intensity of the illuminants and the radiometric scale factor of the camera cannot be distinguished without further assumptions. In the reminder of this paper, we will therefore take ai = 1. Taking these ambiguities into account, the total number of unknowns in the problem is 5 × n + p − 1 in the uncalibrated case, and 4 × n + p − 1 in the calibrated case. From this count, we can calculate the minimal amount of data needed to have a unique solution. In the calibrated case, we have unique solution when np ≥ 4n + p − 1: 7 surface elements for 2 views, 6 surface elements for 3 views, 5 surface elements for 4 views and more.

168

4

Q.-T. Luong, P. Fua, and Y. Leclerc

A Linear Solution

Given a surface element of normal n that projects into two images, each with a diﬀerent light source, its albedo can be estimated using Eq. (3) as: Ik − bk LTk N

αk =

or as

αl =

Il − bl , LTl N

(4)

where index k (resp. l) denotes gray levels, light source vector parameters, and radiometric calibration parameters in the image k (resp. l). Since the value of the albedo is presumed constant, by expanding αk = αl we obtain:

Ik − bk −Il + bl

T

LTl N=0 , LTk

(5)

linear in the 12 variables: Lki , Lli , Lki bl − Lli bk , 1 ≤ i ≤ 4 . 4.1

The Calibrated Case

Intensities can be transformed so that we can assume without loss of generality that ai = 1 and bi = 0. If nT = [n1 , n2 , n3 ], Eq. (5) can be rewritten as: UT f = 0 U = [Il n1 , Il n2 , Il n3 , Il , −Ik n1 , −Ik n2 , −Ik n3 , −Ik ]T f = [lk1 , lk2 , lk3 , µk , ll1 , ll2 , ll3 , µl ]T

(6)

Combining the rows U for each surface element provides a linear system of ˜ = 0. In practice, this is solved in the least-squares sense, taking the form Uf into account the global scale factor on the illuminant f : ˜ subject to f = 1 min Uf f

(7)

With more than two views, we solve simultaneously for all the illuminants, by concatenating the equalities of Eq. (5). 4.2

Minimal Data and Degeneracies

A surface visible in n views yields n − 1 independent equations. With p surfaces, the number of rows is p(n − 1), and the number of columns is 4n. Because of the scale factor, the number of independent equations is at most max(p(n−1), 4n−1). To prove that this data is suﬃcient when the illuminants and surface normals are fully general, we built a speciﬁc system of linear equations for each value of p = 1..7 and n = 1..5. The MapleT M computer algebra system checked the rank of the system, and conﬁrmed that there is no rank loss.

Recovery of Reﬂectances and Varying Illuminants from Multiple Views

169

There are two types of degeneracies, one which is caused by illuminants, the other by the object’s shape. When all N illumination vectors are identical the ˜ has rank 4×(n−1). The larger the distance between illuminants is, the matrix U more stable the solution, as will be conﬁrmed by Sec. 4.4. Besides this obviously expected degeneracy, we found no case of degeneracy caused by illuminants with generic surface normals. In particular for n = 5, the system does not lose rank in the generic case, despite the fact [16,22] that the space of images (in the common image plane, or equivalently, taken from the same viewpoint) is of dimension four. ˜ to have full rank, it is necessary to have at least seven surFor the matrix U face elements with diﬀerent orientations, and that all these orientations vectors be non-planar. In particular planes and cylinders constitute degenerate surfaces. 4.3

Recovering the Oﬀsets

According to Sec. 3.4, in the uncalibrated case, the only radiometric calibration parameters which can be recovered are the oﬀsets bi . Eq. (5) is rewritten as: VT g = 0

V = [Il n1 , Il n2 , Il n3 , Il , −Ik n1 , −Ik n2 , −Ik n3 , −Ik , n1 , n2 , n3 , 1]T g = [Lk1 , Lk2 , Lk3 , Lk4 , Ll1 , Ll2 , Ll3 , Ll4 , mkl1 , mkl2 , mkl3 , mkl4 ]T

where we have used the notations: Lji = lji , i = 1 . . . 3 Lj4 = µj

and

mjli = bl lji − bj lli , i = 1 . . . 3 mjl4 = bl µj − bj µl

After solving these linear equations, to recover the n oﬀsets, we notice that: (Lpj Lqi − Lpi Lqj )bk = mpqi Lkj − mpqj Lki

(8)

After the Lij and mijk are solved for, this equation can be used in a redundant way to recover the bk . 4.4

Synthetic Experiments

To test our approach, we randomly draw values for the light source vectors, values for the surface normals that are such that the surface elements are frontlit, and values between 0 and 1 for the albedoes. We then compute the expected gray level values and add Gaussian noise. Finally, we use the normals and noisy gray-levels to recover estimated illuminants and compare them to the real values. To this end, we form “concatenated illuminant vectors” using the components − → in all images. The “distance” between two concatenated illuminant vectors V1 − → and V2 is: → − →− V1 .V2 − → − → Dvect (V1 , V2 ) = 1 − − → − → . ||V1 ||||V2 ||

(9)

170

Q.-T. Luong, P. Fua, and Y. Leclerc

Fig. 1. Sensitivity of the linear algorithm to gray level noise. Each curve corresponds to a diﬀerent amount of noise. (a) Using 2 images. (b) Using 3 images.

It is one when the two vectors are proportional to each other and zero when they are perpendicular. Figure 1(a,b) shows the results of such experiments using 2 and 3 illuminants respectively. Each curve corresponds to a diﬀerent value of the gray-level noise. For each noise value, we ran the algorithm 10000 times and computed both the true distance between illuminants1 and the recovery error of Eq.(9). We then divided the range of distances between illuminants, that is the x-axis of the graphs into 20 bins. Within each bin, for all runs whose randomly chosen illuminants have a distance that falls into that bin, we compute and plot the 90 percentile value of all the recovery errors, that is, the smallest value which is larger than 90% of these recovery errors. All these experiments were performed using 200 surface elements. We have veriﬁed experimentally that, as soon as the number of surface elements becomes larger than 50, the results become statistically independent of the actual number being used. As expected, the recovery error increases with the amount of noise and also when the distance between the individual illuminants decreases. Note that increasing the number of illuminants from two to three, and beyond, does not bring any meaningful improvement.

5

Non-linear Solution: Radiometric Bundle Adjustment

The radiometric bundle adjustment method is the counterpart of the geometric bundle adjustment used in photogrammetry. It is promising because the latter one has proved to be one of the most robust methods for geometric reconstruction. In this approach, we seek to ﬁnd the values of the 5 × n + p − 1 unknowns αj , j = 1 . . . p, and Li , bi , 1 . . . n, which minimize the error function: 1

In the two-illuminant case, the distance is Dvect of Eq. (9) between the two individual illuminant vectors. In the three-illuminant case, it is the average distance of the three possible pairs of illuminants.

Recovery of Reﬂectances and Varying Illuminants from Multiple Views

i

(Iij − (αj LTi Nj + bi ))2

171

(10)

j

where Iij are the measured-gray levels in the images, and the sums are taken over all the images and for each projection of a surface element. The main drawback is that non-linear optimization is required to minimize the criterion of Eq. (10). However, as will be shown below, the increase in reliability is well worth the additional computational burden. 5.1

Synthetic Experiments

We ran experiments similar to those of Section 4.4, but using this non-linear approach and much higher noise levels. The results are depicted by Figure 2(a,b). As in Figure 1, the individual curves correspond to diﬀerent levels of gray-level noise and are the 90 percentile of recovery errors shown as functions of the distance between illuminants. In the case of two images, the results are excellent at the 0.5% and 1.0% noise levels, levels for which the linear algorithm’s results are already becoming incorrect. The precision then degrades gracefully as the noise level increases. Also, unlike in the linear case, using three illuminants instead of two yields a marked improvement. The method is still sensitive to noise but, because a 1.0% noise level corresponds to about 3 gray-levels in a typical 8-bit image, it becomes much more applicable.

Fig. 2. Sensitivity of the non-linear algorithm to gray level noise. Each curve corresponds to a diﬀerent amount of noise. (a,b) 90 percentiles of recovery errors of illuminant direction using 2 and 3 images. (c,d) Root-mean square errors between true and recovered albedoes using 2 and 3 images.

172

Q.-T. Luong, P. Fua, and Y. Leclerc

The non-linear algorithm also recovers an estimated albedo value for each facet. In Figure 2(c,d), we divide the x-axis of the graphs in the same 20 bins as before. We compute and plot the 90 percentile of the root mean-square diﬀerence between recovered albedoes and real ones, instead of the 90 percentile of the distance between true and recovered illuminants as in Figure 2(a,b). Note that the graphs of both ﬁgures have exactly the same shapes, indicating that our distance measure between illuminants is also a good indicator of the quality of albedo recovery. In practice, normals will only be known up to some precision. We have therefore veriﬁed in a similar manner that the results are relatively insensitive to fairly large imprecisions in the normals. Another source of errors that must be taken into account are gray-levels that are not merely imprecise but that do not at all conform to the Lambertian model, for example because some surface elements are specular or in shadow. To handle these, we adopt a RANSAC style approach [5] to robustly minimizing the criterion of Eq. (10). We have checked that it tolerates up to 15% of outliers [12]. 5.2

Comparison with the SVD Approach

As discussed earlier, when enough images are available, albedoes and light source directions can be recovered, up to a Generalized Bas-Relief (GBR) ambiguity, using an SVD based-approach that does not require the use of normals [3]. Following this approach, we form the p × n matrix X whose rows are the average gray levels of the p facets in each of the n images. We then use SVD to decompose it as X = BS , (11) where B is a p × 5 matrix in which the ﬁrst three elements of all rows are proportional to the normals multiplied by the facet albedoes and S is a 5 × n matrix whose columns are the parameters of the light sources associated to each image. In an ideal no-noise case, for our model with ambient light source and additive camera oﬀset, the matrix B can be taken to be the matrix with rows αj nxj , αj nyj , αj nzj , αj , 1 ,

(12)

where αj is the albedo of facet j and {nxj , nyj , nzj } is its normal vector. 2 The GBR ambiguity comes from the fact that, for any invertible 5 × 5 matrix T , BT and T −1 S is also a solution of equation 11. In our speciﬁc case, we do not know the albedoes but we do know the normals. We can therefore use them to resolve the ambiguity by computing a matrix T and the αj that jointly minimize: ||BT − A||2 2

(13)

In the original SVD work [3], B is p × 3 or a p × 4 matrix. Here we use a p × 5 to account for the additive oﬀset associated to each illuminant. This approach thus requires at least 5 images.

Recovery of Reﬂectances and Varying Illuminants from Multiple Views

173

Fig. 3. Sensitivity of the SVD algorithm to gray level noise in terms of the 90 percentile of the root-mean square error between true and recovered albedoes. Each curve corresponds to a diﬀerent amount of gray-level noise. (a) Using 6 images. (b) Using 7 images. (c) Using 10 images. (d) Using 20 images.

where A is the matrix whose rows are those deﬁned by Equation 12. Because this system is homogeneous, this involves ﬁxing one of the albedoes and ﬁnding the 25 unknown coeﬃcients of T and the p − 1 unknown albedoes. We then take the αj to be the recovered albedoes and T −1 S to be the matrix whose columns are the recovered illuminants Li . Figure 3 depicts the results of experiments similar to those of Figure 2(c,d), but using the SVD approach instead of our proposed method. Note that, even using 20 illuminants instead of the 2 or 3 of Figure 2(c,d), the recovery errors are higher for identical values of the distance between illuminants. Furthermore, the precision quickly degrades as the number of illuminants decreases. This is not surprising since the SVD method, unlike ours, does not use the normal information early in the computation. 5.3

Using Real Images

In Figure 4, we show one of two aerial images of the same site, but taken at diﬀerent times of day and from slightly diﬀerent viewpoints. We treated them as a stereo pair. We reconstructed the terrain’s shape using a deformable surface technique, resulting in a triangulated mesh. We computed the average gray-level of each mesh facet’s projection in each image. We then fed these gray-levels along with the facet normals to our algorithm. Figure 4(b) shows the mesh shaded using the illuminant recovered for the ﬁrst image. In Figure 4(c), we show the albedo at each pixel location that would produce the observed image gray-levels, assuming that the illuminant we computed is correct. The resulting

174

Q.-T. Luong, P. Fua, and Y. Leclerc

Fig. 4. Cartographic Modeling: (a) One of two images of a site taken at two diﬀerent times of day. (b) The corresponding terrain, shaded using the illuminant recovered by our system. (c) The corresponding de-lighted image, stretched so that it has the same dynamic range as the original one. (d,e) Thresholded versions of the original and delighted image: The pixels whose gray level is inferior to 50 appear in black, those whose gray level is greater than 150 in white, and the rest in gray. In (e), unlike in (d), most of the grassy areas appear in solid gray, which is correct since the physical properties are roughly similar. This would make the task of a region-based segmentation algorithm much easier.

de-lighted image exhibits much less structure than the original one, even though both images have the same dynamic range: Shading eﬀects have been greatly attenuated because they are accounted for by changes in shape. To illustrate this point, in Figure 4(d,e), we show versions of the original and de-lighted images coded using three gray-levels only: The pixels whose gray level is inferior to 50 are painted in black, those whose gray level is greater than 150 in white, and the rest in gray. In (e) unlike in (d), most of the grassy areas appear in gray, the shadows in black and the road pixels in white. In other words, a gray-level based segmentation algorithm would have a much easier time of the de-lighted image than of the original one. In Figure 5, we show results in a case where we can use a CAD/CAM 3–D model of a target object but where there are strong specularities: We acquired four diﬀerent images of an O2 computer by using a static camera and moving the computer before each exposure. We aligned the geometric model with each image by running a bundle-adjustment algorithm. This method yields the motion parameters from one view to the other and, thus, allows us to reduce the number of degrees of freedom: Only one illuminant is free and the others are taken to be rotated versions of it. Again, as shown in Figure 5(e), the recovered illumination estimate allows us to de-light the original image. And, as depicted by the graphs of Figure 5(f,g,h), the de-lighted gray-levels corresponding to object points having the same physical material, such as those on the vertical sides of the computer tower, are much more uniform than the original ones, except for specular pixels. As shown in Figure 5(i,j), it becomes much easier to segment meaningful regions from the de-lighted image than from the original one: In reality, the computer’s base and the rest of the tower have two diﬀerent and roughly constant albedoes. As a result, we can pick out these two regions in the de-lighted image using simple gray-level thresholds, which is impossible in the original images.

Recovery of Reﬂectances and Varying Illuminants from Multiple Views

175

In Figure 6, we present a similar behavior in the more diﬃcult case where we do not have an a-priori shape-model: The three images to the left were acquired using a static camera watching a moving head. We computed the head motion and reconstructed its shape using a stereo-based facial reconstruction technique [6] that is relatively insensitive to illumination changes. Because our geometric reconstruction method also yields motion parameters, as in the case of the images of Figure 5, we can assume that illuminants are rotated versions of each other. To overcome the inﬂuence of specularities, we further reduce the number of degrees of freedom: We constrain the facets that are symmetric with respect to a vertical plane that goes through the middle of the head to have the same albedoes. In Figure 6(d), the resulting surface is shaded using the illuminant we recover while Figure 6(e) shows the de-lighted image corresponding to image (b). For comparison purposes, in Figure 6(f), we show the corresponding portion of the original image, stretched to have the same dynamic range and with two scan-lines overlaid. In Figure 6(g), we plot the gray levels along both scan-lines. Each graph corresponds to one scan-line and contains two curves: One for the original gray levels in (f) and one for the de-lighted gray-levels in (e). Note that the de-lighted gray levels are again much ﬂatter than the original ones: Most of the gray-level variations are accounted for by changes in shape and we are left with the almost constant albedoes one would expect on a face. This is true for most of the skin surface, except on the right side the nose and the cheeks: The recovered illuminant comes from the left side of the face, resulting in exceedingly bright pixels at places where cast shadows should be, but are not, modeled. As a result, the estimated albedoes are too large. Nevertheless, these de-lighted images can then be averaged to produce de-lighted texture-mapped models such as the one of Figure 6(h,i,j). These, in turn, could re-lighted from a diﬀerent direction and inserted in an animatiom.

6

Conclusion

We have introduced a new methodology for recovering the illuminants and surface albedoes from multiple views taken with varying illuminants in the Lambertian case. When coupled with a method for shape reconstruction, it allows us to recover both the geometric and radiometric attributes of a scene. We have also shown that in a linear setting, the only observable camera radiometric parameter, the oﬀset, can be recovered. The new theory indicate the minimal data as well as degenerate conﬁgurations and sets the limits for radiometric reconstruction. We proposed two diﬀerent algorithmic implementations: 1. A linear algorithm It allows fast computation, but is sensitive to noise. 2. A radiometric bundle-adjustment algorithm. It involves non-linear optimization that makes it slower but also much more robust. Using synthetic data, we have quantiﬁed the sensitivity of the algorithms to image noise, outliers, imprecision of surface normals, distance between illuminants,

176

Q.-T. Luong, P. Fua, and Y. Leclerc

Fig. 5. Modeling an object with known geometry and strong specularities: (a,b,c) Three image of an O2 computer acquired using a static camera. The computer was moved before each shot. (d) CAD model shaded using the recovered illuminant for image (b). (e) The corresponding de-lighted image, stretched so that it has the same dynamic range as the original one. (f,g,h) Top: Gray-levels along the three horizontal scan-lines overlaid in (b). Because of illumination eﬀects, the gray levels vary a lot, even though, in reality, the albedo remains almost constant. Bottom: Gray-levels along the same scanlines in the de-lighted image (e). They are almost constant, except for the specular pixels that produce a peak. (i,j) Thresholded versions of the original and de-lighted image: The background appears in black, the foregrounnd pixels whose gray level is inferior to 80 appear in dark gray, those whose gray level is greater than 80 and smaller than 140 in light gray, and the rest in white.

and number of surface elements. Because reliable results can be obtained using relatively few surface elements, we have not found the extra computational expense involved by the non-linear method to be a severe drawback. The usefulness of this algorithm has been illustrated by two experiments that show that, even with a minimal number of real images, we can obtain 3–D models and plausible estimates of the light source direction and albedoes. The implementation represents a ﬁrst step towards a solution of the radiometric reconstruction problem, rather than a complete solution, and could be combined with existing approaches. This work, like many, is based on Lambertian reﬂectance model with ambient light and distant point sources. The surface elements that do not conform to it are treated as outliers and discarded as such by the bundle-adjustment algorithm. Note, however, that the recovery algorithm is implemented using a general least-squares solver in which the constraints derived using the Lam-

Recovery of Reﬂectances and Varying Illuminants from Multiple Views

177

Fig. 6. Facial Reconstruction: (a,b,c) Three image of a moving head acquired using a static camera. (d) Close-up view of the reconstructed head model, shaded using the recovered illuminant. (e) The corresponding de-lighted image. (f) Corresponding part of image (b) stretched so that is has the dynamic range (e). (g) The gray-levels along the two scan-lines overlaid in (f). Each graph, contains two curves: One for the original gray levels in (f) and one for the de-lighted gray-levels in (e). (h,i,j) De-lighted texture-mapped view of the 3-D model.

bertian model could easily be replaced by ones computed using more complex illumination models, such as those that take specularities into account. Similarly, we do not explicitly model shadows but treat them as outliers. There are, however, shadow extraction techniques that do not depend on geometry and could be used. Moreover, given a known surface geometry, the shadow boundaries provide clues as to the location of point light sources. In an iterative scheme, after a ﬁrst estimation of the light sources has been obtained, it would be possible to use the geometry to predict potential shadowed areas and take that information into account in the next iteration. In the absence of shadows and specularities, any combination of light sources could be accounted for by a single point source and an ambient term. Understanding their roles is therefore crucial to extending our approach from the simple case where there is a single point-light source per image to more realistic situations that involve multiple light-sources per image. In future work, focus will be on introducing shadows, specularity models, and multiple light sources into the proposed framework.

178

Q.-T. Luong, P. Fua, and Y. Leclerc

References 1. E. Angelopoulou and J. Williams. Photometric surface analysis in a tri-luminal environment. In ICCV, pages 442–447, 1999. 2. R. Baribeau, M. Rioux, and G. Godin. Color reﬂectance modeling using a polychromatic laser range sensor. PAMI, 14(2):263–269, 1992. 3. P.N. Belhumeur, D.J. Kriegman, and A.L. Yuille. The bas-relief ambiguity. IJCV, 35(1):33–44, November 1999. 4. P. Debevec and J. Malik. Recovering high dynamic range radiance maps from photographs. SIGGRAPH, 31:369–378, 1997. 5. M.A Fischler and R.C Bolles. Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography. Communications ACM, 24(6):381–395, 1981. 6. P. Fua. Regularized Bundle-Adjustment to Model Heads from Image Sequences without Calibration Data. International Journal of Computer Vision, 38(2):153– 171, July 2000. 7. A.S. Georghiades, P.N. Belhumeur, and D.J. Kriegman. Illumination-based image synthesis: Creating novel images of human faces under diﬀering pose and lighting. In IEEE Workshop on multiple-view modeling and analysis of visual scenes, pages 47–54, 1999. 8. H. Hayakawa. Photometric stereo under a light-source with arbitrary motion. JOSA-A, 11(11):3079–3089, November 1994. 9. B.K.P. Horn. Robot Vision. MIT Press, 1986. 10. B.K.P. Horn and M.J. Brooks. Shape from Shading. MIT Press, 1989. 11. K. Ikeuchi and K. Sato. Determining reﬂectance properties of an object using range and brightness images. PAMI, 13(11):1139–1153, 1991. 12. Q.T. Luong, P. Fua, and Y. G. Leclerc. The Radiometry of Multiple Images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(1):19–33, January 2002. 13. S. Marschner. Inverse rendering for computer graphics. PhD thesis, Cornell University, 1998. 14. S. Marschner and D. Greenberg. Inverse lighting for photography. In IST/SID Fifth Color Imaging Conference, pages 262–265, 1997. 15. T. Mitsunaga and S.K. Nayar. Radiometric self calibration. In CVPR, pages I:374–380, 1999. 16. Y. Moses. Face recognition: generalization to novel images. PhD thesis, The Weizmann Institute of Science, Israel, 1993. 17. N. Mukawa. Estimation of shape, reﬂection coeﬃcients and illuminant direction from image sequences. In ICCV, pages 507–512, 1990. 18. K. Nishino, Z. Zhang, and K. Ikeuchi. Determining Reﬂectance Parameters and Illumination Distribution from a Sparse Set of Images for View-Dependent Image Synthesis. In International Conference on Computer Vision, pages 599–606, Vancouver, Canada, July 2001. 19. R Ramamoorthi and P. Hanrahan. A signal processing framework for inverse rendering. In SIGGRAPH, pages 117–128, 2001. 20. I. Sato, Y. Sato, and K. Ikeuchi. Illumination distribution from brightness in shadows: Adaptive estimation of illumination distribution with unknown reﬂectance properties in shadow regions. In ICCV, pages 875–883, 1999. 21. M. Sato, Y. Wheeler and K. Ikeuchi. Object shape and reﬂectance modeling from observation. In SIGGRAPH, pages 379–387, 1997.

Recovery of Reﬂectances and Varying Illuminants from Multiple Views

179

22. A. Shashua. On photometric issues in 3d visual recognition from a single 2d image. IJCV, 21(1-2):99–122, 1997. 23. W. Silver. Determining shape and reﬂectance using multiple images. PhD thesis, MIT, Cambridge, MA, 1990. 24. R.J. Woodham. Photometric method for determining surface orientation from multiple images. OptEng, 19(1):139–144, 1980. 25. Y. Yu, P. Debevec, J. Malik, and T. Hawkins. Inverse global illumination: Recovering reﬂectance models of real scenes from photographs. SIGGRAPH, pages 215–224, August 1999. 26. A.L. Yuille, D. Snow, R. Epstein, and P.N. Belhumeur. Determining generative models of objects under varying illumination: Shape and albedo from multiple images using svd and integrability. IJCV, 35(3):1–20, 1999. 27. Q. Zheng and R. Chellappa. Estimation of illuminant direction, albedo, and shape from shading. PAMI, 13(7):680–702, 1991.

Composite Texture Descriptions Alexey Zalesny1, Vittorio Ferrari1, Geert Caenen2, Dominik Auf der Maur1, and Luc Van Gool1, 2 1

Swiss Federal Institute of Technology Zurich, Switzerland {zalesny, ferrari, aufdermaur, vangool}@vision.ee.ethz.ch http://www.vision.ee.ethz.ch/~zales 2 Catholic University of Leuven, Belgium {Geert.Caenen, Luc.VanGool}@esat.kuleuven.ac.be

Abstract. Textures can often more easily be described as a composition of subtextures than as a single texture. The paper proposes a way to model and synthesize such “composite textures”, where the layout of the different subtextures is itself modeled as a texture, which can be generated automatically. Examples are shown for building materials with an intricate structure and for the automatic creation of landscape textures. First, a model of the composite texture is generated. This procedure comprises manual or unsupervised texture segmentation to learn the spatial layout of the composite texture and the extraction of models for each of the subtextures. Synthesis of a composite texture includes the generation of a layout texture, which is subsequently filled in with the appropriate subtextures. This scheme is refined further by also including interactions between neighboring subtextures.

1

Composite Textures, Divide, and Conquer

Natural textures can be of a very high complexity, which renders them difficult to emulate by texture synthesis methods. Many of such textures consist of patches, which in turn contain patterns of their own. Good examples are the textures of building materials like marbles or limestones, or the textures of different landscape types. A countryside texture could, e.g. be a mixture of meadows and forests. We will refer to such textures as composite textures. Whereas the patterns within the patches may already be homogeneous enough to be modeled and synthesized successfully with existing techniques, modeling and synthesizing the mixture directly tends to be less effective. It therefore stands to reason to divide the synthesis of composite textures into several steps: 1. Segment (by hand or otherwise) example image(s) of the composite texture to be modeled, where each class is given its own label. 2. Consider the resulting label map (with labels assigned to all pixels) as a texture and extract a model for it. 3. Also extract models for the textures corresponding to the different labels (i.e. the different classes/segments). 4. Generate a synthetic label map, based on the model of step 2. 5. Fill in the segments generated under step 4 with textures according to their labels, based on the models of step 3. A. Heyden et al. (Eds.): ECCV 2002, LNCS 2352, pp. 180–194, 2002. © Springer-Verlag Berlin Heidelberg 2002

Composite Texture Descriptions

181

This scheme tallies with ideas about “macrotextures” vs. “microtextures” that have emerged as soon as texture research started in computer vision. The label map can be considered a macrotexture, whereas the textures within the segments of constant label values would be microtextures. In this paper, we will refer to the macrotextures as “label textures”, and to the microtextures as “subtextures”. Although such ideas have been around for quite a while, it seems that so far such hierarchical approach has not really been implemented for textures of a strong stochastic nature. When ideas about hierarchical analysis are used, it is usually in the context of multi-resolution schemes, which have contributed greatly to the state-ofthe-art in texture synthesis (see e.g. [2, 7] as seminal examples). Contributions that come closest to the work presented here have used label textures that were handdrawn by the user and the corresponding textures were then filled in with texture either by “smart copying” from example [8] or by synthesis techniques [13]. The composite texture approach as just described does not include mutual influences between the subtextures. It is not exceptional that the subtextures are not completely stationary within their domains. In particular, regularly a kind of transition zone is found near their boundaries. These changes may well depend on the texture on the other side of the boundary. Later in the paper we will modify the composite texture scheme to include such effects. The remainder of the paper is organized as follows. Section 2 describes the texture synthesis approach used to generate the label textures and the subtextures. Section 3 shows some results for the synthesis of composite textures. Section 4 describes the inclusion of subtexture interactions (nonstationary aspects). Section 5 concludes the paper.

2

Image-Based Texture Synthesis

This section describes the approach that we use to synthesize the textures, i.e. both the label textures and the subtextures. More information about this texture modeling and synthesis approach can be found in [13]. It follows the cooccurrence paradigm in that texture is synthesized as to mimic the pairwise statistics of the example texture. This means that the joint probabilities of the intensities at pixel pairs with a fixed relative position are approximated. Such pairs will be referred to as cliques, and pairs of the same type (same relative position between the pixels) as clique types. This is illustrated in Fig. 1.

Fig. 1. Dots represent pixels. Pixels connected by lines represent cliques. Left: cliques of the same type, right: cliques of different types.

The texture model consists of statistics for a selected set of clique types. Clique type selection follows an iterative approach, where clique types are added one by one to the texture model, a texture synthesized based on the model is each time updated

182

A. Zalesny et al.

accordingly, and the statistical difference between the example texture and the synthesized texture is analyzed to decide which further clique type addition to make. The set of selected clique types is called the neighborhood system. The complete texture model consists of this neighborhood system and the statistical parameter set. The latter contains the histograms of intensity differences between the two pixels of all cliques of the same type. A sketch of the texture model extraction algorithm is as follows: step 1: Collect the complete 2nd-order statistics for the example texture, i.e. the statistics of all clique types. After this step the example texture is no longer needed. As a matter of fact, the current implementation focuses on clique types up to a maximal length. step 2: Generate an image filled with independent noise and with values uniformly distributed in the range of the example texture. This noise image serves as the initial synthesized texture, to be refined in subsequent steps. step 3: Collect the statistics for all clique types from the current synthesized image. step 4: For each clique type, compare the intensity difference histograms of the example texture and the synthesized texture and calculate their Euclidean distance. In fact, the intensity histograms pure (singletons) are considered as well. step 5: Select the clique type with the maximal distance (see step 4). If this distance is less than a threshold, then leave the algorithm with the current model. Otherwise add the clique type to the current (initially empty) neighborhood system and all its statistical characteristics to the current (initially empty) statistical parameter set. step 6: Synthesize a new texture using the updated neighborhood system and texture parameter set. step 7: Go to step 3. After running this algorithm we have the final neighborhood system of the texture and its statistical parameter set. A more detailed description of this texture modeling approach is given elsewhere [13]. In that paper it is also explained how the synthesis step works and how we generalize this texture modeling to colored textures. The proposed algorithm produces texture models that are very small compared to the complete 2nd-order statistics extracted in step 1 and also compared to the example image. Typically only 10 to 40 clique types are included and the model amounts from a few hundred to a few thousand bytes. Another advantage is that the method avoids verbatim copying of parts of the example images. This is an advantage that it shares with similar approaches [5, 6, 9, 12]. Verbatim copying becomes particularly salient when large patches of texture need to be created, e.g. when covering a large wall with a marble texture. Then the repeated appearance of the same structures quickly becomes salient to the human eye. For the texture mapping on such large surfaces this problem is more serious than may transpire from state-of-the-art publications that suffer from this problem (e.g. [3, 11]), as the extent of the textures that can be shown in papers is rather small.

Composite Texture Descriptions

183

The basic texture synthesis approach described in this section can handle quite broad classes of textures. Nevertheless, it has problems with the composite type of textures considered here. Comparisons between single and composite texture synthesis results will be shown later.

3

The Segmentation of Composite Textures

The steps of the composite texture modeling scheme proposed in section 1 are illustrated on the basis of a few examples. Fig. 2 shows a modern “thorn-cushion steppe” type of landscape (left). It consists of several ground cover types, like “rock”, “green bush”, “sand”, etc., for which the corresponding segments are drawn in the figure on the right.

Fig. 2. Left: an example of “thorn-cushion steppe”, Right: manual segmentation into basic ground cover types (also see Fig. 4).

If one were to directly model this composite ground cover as a single texture, the basic texture analysis and synthesis algorithm proposed in the last section would not be able to capture all the complexity in such a scene. Fig. 3 shows the result of such synthesis.

Fig. 3. Attempt to directly model the scene in Fig. 2 as a single texture.

In keeping with the composite texture scheme of section 1, an example image like this is first decomposed into different subtextures, as shown in Fig. 2 (right). The

184

A. Zalesny et al.

segments that correspond with different subtextures have been given the same intensity (i.e. the same label). This segmentation has been done manually. Fig. 4. shows the image patterns corresponding to the different segments. The textures within the different segments are simple enough to be handled by the basic algorithm. Hence, in this case 6 subtexture models are created, one for each of the ground cover types (see caption of Fig. 4.). But also the map with segment labels (Fig. 2 right image) can again be considered to be a texture, describing a typical landscape layout in this case. This label texture is quite simple again, and can be modeled by the basic algorithm of section 2. Hence, similar label textures can be generated automatically as well. The synthesis of this composite texture first generates a landscape layout as a label texture. Subsequently, the different segments are filled in with the corresponding subtextures, based on their models. As an alternative, a graphical designer or artist can draw the layout, after which the computer fills in the subtextures in the segments that s/he has defined, according to their labels. Fig. 5 shows one example for both procedures.

Fig. 4. Manual segmentation of the terrain texture shown in Fig. 2. Left: segments corresponding to 1-green bush, 2-rock, 3-grass, 4-sand, 5-yellow bush. Right: left-over regions are grouped into an additional mixed class.

Fig. 5. Synthetically generated landscape textures. Left: based on a manually drawn label texture. Right: based on an automatically generated label texture.

Composite Texture Descriptions

185

Note that except for the hand segmentation of the label texture in Fig. 2, the right image has been created fully automatically and arbitrary amounts of such texture can be generated, enough to cover a terrain model with never-repeating, yet detailed texture. As mentioned before, the fact that this approach doesn’t use verbatim copying of parts in the example images has the advantage that no disturbing repetitions are created. A complete automation of the composite texture scheme would have to include an initial, unsupervised segmentation of the scene, so that no hand segmented label texture is needed. Such unsupervised segmentation is not an easy matter of course, but in certain cases may nevertheless be possible. As Paget [10] noticed, the optimal complexity of texture models for segmentation is typically lower than that required for synthesis. This can be key to avoiding a chicken-and-egg situation with such fully automated process. In case the features needed for the segmentation were of the same complexity as the models that we extract from segmented data, the whole procedure can no longer run automatically. Fig. 6 left shows two images that could be segmented automatically. As a matter of fact, truly textural features were not needed at all in this case. Segmentation based on Lab color data was sufficient. The data around each pixel were smoothed with a Gaussian filter of size 5x5. Each pixel is described by a three dimensional color feature vector. For more general case, an additional vector is built of simple texture features, obtained as the outputs of a bank of simple filters. A cosine metric between such feature vectors is used as similarity measure between pairs of pixels (Bhattacharyya distance). A sample pixel set is drawn from the image and a complete graph is constructed where vertices correspond to pixels and edges are weighted by the similarity measure. The graph is then partitioned into completely connected, disjoint subsets of vertices so as to maximize the total sum of the similarities over all remaining edges. These subsets are also referred to as “cliques”, but this time in the graph theoretical sense. The pixels in each subset now represents a subtexture. This robust segmentation procedure is efficiently obtained via the Clique Partitioning approximation algorithm of [4]. It is worth noting that the algorithm is completely unsupervised: it doesn’t need to know the number of subtexture classes, their size, or any information other than the image data itself. The segmentation results for the two images are shown in Fig. 6 right. A more detailed description of the unsupervised segmentation scheme is given elsewhere [1]. Fig. 7 shows the result of a fully automatic composite texture synthesis for the top texture of Fig. 6. No user input whatsoever was needed to generate this result. Fig. 7 right is the label texture that was generated. Automatic segmentation of a composite texture will not always be possible or even desirable. Fig. 8 shows one of the more complicated types of limestones that was used as building material at the ancient city of Sagalassos in Turkey, now a center of intensive archaeological excavations. If one wants to recreate the original appearance of the buildings, such textures need to be shown with all their complexities and subtleties, but without the effects of erosion. Fig. 8 (a) shows an example image of this limestone (pink-gray breccia), which has a kind of patchy structure and where several darker cracks and pits are the result of century long erosion. First, the original limestone image was manually segmented, whereupon texture models were generated for the different parts, making sure that those parts left out

186

A. Zalesny et al.

erosion effects. An automatically generated, synthetic result is shown in (c). To simulate the visual effect of erosion, simple thresholding of the image (a) yielded the erosion related cracks and pits. These dark areas were then superimposed on texture (c), yielding (d). As can be seen the visual appearance of this texture comes close to that of (a). It is texture (c), however, that is the desired output for this application. This is a case were manual segmentation and user assisted subtexture modeling is hard to avoid.

Fig. 6. Left: landscape images. Right: unsupervised segmentation.

Composite Texture Descriptions

4

187

Interactions between Subtextures

The subtextures are not always sharply delineated and may not be stationary within their patches. It is quite usual that neighboring subtextures exhibit a mutual influence near their boundaries. The observed nonstationary behavior near the subtexture boundaries may therefore well depend on the specific subtexture on the other side. Such effects have been included in a refined version of the composite texture model. The “upgraded” model includes additional cliques, where head and tail pixels correspond to different subtextures. For all pairs of subtextures that come within a user-specified distance of each other, a separate neighborhood system and statistical parameter set are derived. These are characterized by the fact that the two pixels of a clique (we call them “head” and “tail”) fall within a different subtexture. Note that the interactions impose a strict order on their cliques: the head lying in a first subtexture and the tail in a second has to be distinguished from the reverse situation. Fig. 9 illustrates the three types of cliques that are relevant as soon as subtexture interactions are taken account of.

Fig. 7. Fully automatic composite texture synthesis of the top texture in Fig. 6. Right: the automatically generated label map modeled as a label texture from the unsupervised segmentation in Fig. 6 top-right.

Currently, in the upgraded approach the subtextures are synthesized in order of “complexity”. So far, we have only used simple criteria like entropy of the color histogram as complexity measures. We start the synthesis with the simplest texture. The rationale is that in this way those textures that need the least information are synthesized first. By the time more complex textures are to be synthesized, the simpler ones are already in place and can assist in this synthesis. Suppose we are about to analyze the interactions of texture n in this sequence. We can include the real intensity difference histograms into the model for those textures that come earlier in the sequence, because they will already have been synthesized by the time texture n is. For the interactions with textures further down the line, the intensity differences

188

A. Zalesny et al.

between the real intensities within texture n and 0 outside are taken, as at the time of synthesis the system will find the initialized values 0 there. This in fact corresponds to simply taking the intensity histogram for points of texture n within the relative positions to the neighboring texture that are prescribed by the clique type. Work to synthesize all subtextures simultaneously instead of sequentially as described here is under way. Hence, we are not too concerned with the generality of our texture complexity ranking procedure, as it will become obsolete once simultaneous synthesis is in place. Simultaneous synthesis should be superior as all subtexture interactions are considered.

(a) example image of a limestone

(b) synthesized as a single texture

(c) synthesized as a composite texture, omitting the effects of erosion

(d) synthesis like in (c) but with erosion added as the darker parts of (a)

Fig. 8. Results for the synthesis of a limestone texture.

Composite Texture Descriptions

189

Subtexture 1 Subtexture 2

1

2 2 2

2 1

Fig. 9. The cliques (1,2), (2,2), and (2,1) are of different types despite of their equal relative position of head and tail. The arrows show the direction of the interaction for this specific sequence of subtexture modeling.

Fig. 10 shows an example image of a limestone texture. Figure (b) is the result of unsupervised segmentation. The result consists of only two subtextures in this case, consisting of the dark and bright areas, respectively. The dark areas were considered “simpler” by the upgraded approach and it therefore dealt with them first. The difference between the composite texture approach of section 1 and the upgraded approach doesn’t lie in the label texture as its generation in both cases is identical. In order to best compare the results of the original and the upgraded approach we therefore take the example label texture of (b) as our label texture for subtexture synthesis. Fig. 10 (d) shows the result with the approach of section 1, (c) with the upgraded approach. As can be seen, the upgraded version looks more natural. In the example image the edges between the two subtextures are less sharp than in (d). Near either side of the border the intensities tend to get more similar. A good example is the white spot at the top. The spatial distribution of white and gray pixels in such spot in image (c) is better than that of (d). Fig. 11 (a) gives the result of the upgraded approach for the top texture of Fig. 6. The label texture of Fig. 7 right was reused. As can be seen (cf. independent subtexture synthesis Fig. 11 (b) repeated here for comparison), the quality of this texture is better. Fig. 12 (a) gives the result for the bottom texture in Fig. 6. In this case the quality of the texture synthesized without subtextures’ interactions (not shown) is more comparable. Fig. 13 illustrates the use of the composite textures for virtual reality. Part (a) corresponds to the original model of an ancient building as graphic designers have produced it. Part (b) shows the same building with some of our textures mapped onto the pillars. Our goal is to map the original textures onto complete monuments and the surrounding landscape. Finally, an example is given that shows that the composite texture approach can also be beneficial even for textures that have traditionally been treated as a single texture. Fig. 14 (a) shows one of the Brodatz textures (D20, French canvas). Image (b) is the result of a texture synthesis when this texture is modeled as a single texture. Image (d) is the result of a composite texture synthesis without subtexture interactions (the black and bright pixels formed the two classes). Image (c) is the result of composite texture synthesis with interactions between the two subtextures. This result is clearly the best.

190

A. Zalesny et al.

(a)

(b)

(c)

(d)

Fig. 10. (a) original limestone; (b) unsupervised segmentation; (c) composite synthesis with interactions between subtextures; (d) composite synthesis with independent subtextures.

5

Conclusions and Future Work

Our texture method learns a model of a complicated texture pattern like a landscape or geological structure from example images. This model can be used e.g. to generate more of similarly looking texture without verbatim copying of parts of the example textures. A first “upgrade” of this technique has also been presented. It takes account of nonstationary aspects in the different subtextures, due to their interactions. Several extensions are planned. The first is to do away with the rather clumsy, sequential generation of the subtextures in case of the upgraded model. After an initial generation of the label texture, all subtextures and their interaction effects should better be generated together. In that way, mutual influences between all interacting subtextures are included.

Composite Texture Descriptions

(a)

191

(b)

Fig. 11. (a) composite synthesis with interactions between subtextures for the top texture of Fig. 6; (b) repeats for comparison the result without interdependency between textures.

(a)

(b)

Fig. 12. (a) fully automatic composite texture synthesis of the bottom texture in Fig. 6; (b) the automatically generated label map modeled as a label texture from the unsupervised segmentation in Fig. 6 bottom-right.

192

A. Zalesny et al.

(a)

(b) Fig. 13. (a) original model of an ancient building at Sagalassos as graphic designers have produced it; (b) the same building with some of our textures mapped onto the pillars.

Composite Texture Descriptions

(a)

(b)

(c)

(d)

193

Fig. 14. (a) Brodatz D20, French canvas; (b) synthesis as a single texture; (c) composite texture synthesis with interaction between the subtextures; (d) composite synthesis without interaction between subtextures.

A second extension is the use of the composite texture models to analyze and recognize complete scenes, e.g. the classification of land use and land cover in satellite images. Some of these classes can have intricate structures and could be better dealt with if described as composite textures. As Paget’s work [10] has shown, we can expect that even partial information from the extracted models can suffice for this analysis. A third extension could be the use of deeper hierarchies and to have more levels that just a label texture and its subtextures. A fourth extension could be the enhancement of our unsupervised segmentation scheme.

194

A. Zalesny et al.

Acknowledgments. The authors gratefully acknowledge support by the European IST projects "3D Murale" and "CogViSys".

References 1. G. Caenen, V. Ferrari, L. Van Gool, and A. Zalesny. Unsupervised Texture Segmentation through Clique Partitioning. Tech. Rep. KUL/ESAT/PSI, Catholic University of Leuven, Belgium, 2002. 2. J.S. De Bonet. Multiresolution Sampling Procedure for Analysis and Synthesis of Texture Images. Proc. Computer Graphics, ACM SIGGRAPH’97, 1997, pp. 361-368. 3. A. Efros and T. Leung. Texture Synthesis by Non-Parametric Sampling. Proc. Int. Conf. Computer Vision (ICCV’99), Vol. 2, 1999, pp. 1033-1038. 4. V. Ferrari, T. Tuytelaars, and L. Van Gool. Real-Time Affine Region Tracking and Coplanar Grouping. IEEE Computer Soc. Conf. on Computer Vision and Pattern Recognition (CVPR'01), December 2001, Vol. 2, pp. 226-233. 5. A. Gagalowicz and S.D. Ma. Sequential Synthesis of Natural Textures. Computer Vision, Graphics, and Image Processing, Vol. 30, 1985, pp. 289-315. 6. G. Gimel’farb, Image Textures and Gibbs Random Fields. Kluwer Academic Publishers: Dordrecht, 1999, 250 p. 7. D. Heeger and J. Bergen. Pyramid-Based Texture Analysis/Synthesis, ASM SIGGRAPH ’95, pp.239-238, 1995. 8. A. Hertzmann, C. Jacobs, N. Oliver, B. Curless, and D. Salesin. Image analogies. ACM SIGGRAPH 2001, pp.327-340, 2001. 9. B. Julesz and R.A. Schumer. Early Visual Perception. Ann. Rev. Psychol. Vol. 32, 1981, pp. 575-627 (p. 594). 10. R. Paget. Nonparametric Markov Random Field Models for Natural Texture Images. PhD Thesis, University of Queensland, February 1999. 11. L.-Y. Wei and M. Levoy. Fast Texture Synthesis Using Tree-Structured Vector Quantization. SIGGRAPH 2000, pp.479-488. 12. Y.N. Wu, S.C. Zhu, and X. Liu. Equivalence of Julesz and Gibbs Texture Ensembles. Proc. Int. Conf. Computer Vision (ICCV’99), 1999, pp. 1025-1032. 13. A. Zalesny and L. Van Gool. A Compact Model for Viewpoint Dependent Texture Synthesis. SMILE 2000, Workshop on 3D Structure from Images, Lecture Notes in Computer Science 2018, M. Pollefeys, L. Van Gool, A. Zisserman, and A. Fitzgibbon (Eds.), 2001, pp. 124-143.

Constructing Illumination Image Basis from Object Motion Akiko Nakashima, Atsuto Maki, and Kazuhiro Fukui Corporate Research and Development Center, TOSHIBA Corporation 1, KomukaiToshiba-cho, Saiwai-ku, Kawasaki 212-8582 Japan

Abstract. We propose to construct a 3D linear image basis which spans an image space of arbitrary illumination conditions, from images of a moving object observed under a static lighting condition. The key advance is to utilize the object motion which causes illumination variance on the object surface, rather than varying the lighting, and thereby simpliﬁes the environment for acquiring the input images. Since we then need to re-align the pixels of the images so that the same view of the object can be seen, the correspondence between input images must be solved despite the illumination variance. In order to overcome the problem, we adapt the recently introduced geotensity constraint that accurately governs the relationship between four or more images of a moving object. Through experiments we demonstrate that equivalent 3D image basis is indeed computable and available for recognition or image rendering.

1

Introduction

In appearance-based object recognition, lighting variation is one of the most signiﬁcant issues. As Moses et al.[10] pointed out in the context of face recognition, “the variations between the images of the same face due to illumination and viewing direction are almost always larger than image variations due to change in face identity.” In order to deal with this problem several approaches using image bases have been proposed, and successful results were reported. See, for example, [2,5]. The advantage of these methods is due to the fact that an arbitrary image of an object under a distant light source has low-dimensional representation of the object. The observation concerning the low-dimensional representation originated in Shashua’s works[14,15]. He showed that images of an object under arbitrary lighting conditions are represented by linear combinations of three base images, given that the surface of the object follows the Lambertian model without shadows. Yuille and Snow [21] further considered ambient background illumination, Belhumeur and Kriegman [3] extended the representation to the case with attached shadows. They proved that a set of images with attached shadows under arbitrary lighting conditions forms a convex polyhedral cone which is again based on three base images. As is obvious from these observations, the notion of base images provides a foundation for representing images under arbitrary illumination. In this paper, we call the set of the three base images illumination image basis. A. Heyden et al. (Eds.): ECCV 2002, LNCS 2352, pp. 195–209, 2002. c Springer-Verlag Berlin Heidelberg 2002

196

A. Nakashima, A. Maki, and K. Fukui

In previous approaches the illumination image basis has been obtained from images of an object in a ﬁxed pose under various lighting conditions. To acquire such images, therefore, the object and the camera must be kept stationary, and the direction of illumination be varied. As opposed to this situation, if we could equally construct the illumination image basis from images of an object that is in motion under a stationary lighting condition, the process of capturing input images would be substantially simpliﬁed. This is especially true for an object such as a human head, which we mainly deal with in this article, since the only required action is to move the object in 3D space. However, to our knowledge there has been no attempt to consider the possibility, and the purpose of this paper is to propose a technique for constructing the illumination image basis from object motion. Given images of an object captured in diﬀerent poses, the set of pixels at the same coordinate in the images do not correspond to an identical point in the 3D surface. For the construction of illumination image basis, thus, we basically need to re-align the pixels in the images as if the object were seen from an identical direction, and the problem therefore becomes that of ﬁnding the corresponding image points. For the point correspondence, typically exploited is the constraint that the corresponding parts of the images have equivalent intensity, regarding the variation in illumination as noise. Unfortunately, however, the constraint is nearly always invalid for an object in motion, since relatively non-uniform lighting causes the intensity at a speciﬁc location on the surface of the object to change as the object moves [13]. Among the few eﬀorts to address this issue, recently the geotensity constraint1 [8] has been derived to overcome the problem with respect to camera geometry, and to replace the constant intensity constraint. Based on the notion of linear intensity subspaces [14], the geotensity constraint governs the relationship between four or more images of a moving object. The algorithm for point correspondence using the geotensity constraint proceeds basically in two stages. The ﬁrst stage is to derive the parameters of the Geotensity constraint by analyzing coordinates and image intensity of some sample points on the object in motion. That is, computing structure from motion gives the geometric parameters of the situation, whereas computing the linear image subspace gives the lighting parameters of the situation. By combining both sets of parameters we arrive at the Geotensity constraint. Using the same set of input images, the second stage is to take each pixel in an arbitrary reference image in turn and search for the depth along the ray from the optical center of the camera passing through the pixel. The correspondence is evaluated by measuring the agreement of the entire set of projected intensities of a point on the object surface with the Geotensity constraint. Although the geotensity constraint was proposed originally for the task of 3D surface reconstruction, we will show that the constraint can be applied directly for our porpose, that is, to obtain the illumination image basis from images of an object in motion under stationary illumination. Further, once we obtain the illumination image basis, images of the object under arbitrary lighting con1

Geotensity stands for “geometrically corresponding pixel intensity.”

Constructing Illumination Image Basis from Object Motion

197

ditions can be generated simply by linear combinations of the basis as stated earlier. Since the Geotensity constraint can be applied to any choice of reference frame, diﬀerent choices allow us to construct illumination bases corresponding to diﬀerent viewing directions and thereby we can synthesize variations of object images both in terms of lighting conditions and object poses. While such variations are obviously useful for recognition problems in general, it should be noted that they become available from a minimum of four input images. Although the resulting variations may look similar to what is obtained by the method in [5], we should point out that the entire scheme is diﬀerent in that we explicitly solve for the point correspondence instead of estimating the surface geometry in the framework of photometric stereo [20]. This paper is organized as follows: First, we introduce the low-dimensional representation for images captured under arbitrary illumination in Section 2 and the geotensity constraint in Section 3. Then, in Section 4 we describe an algorithm to construct the illumination image basis by applying the geotensity constraint. Experiments are shown in Section 5. In Section 6, the ﬁnal section, discussions and conclusions are presented.

2

Illumination Image Basis

In this section we brieﬂy describe the low-dimensional representations for variations of illumination which have been successfully utilized in appearance-based object recognition. Consider the case of a convex object with a surface conforming to a Lambertian reﬂectance model with varying albedo, the ratio of outgoing to incoming light intensity, under a single point light source at inﬁnity. Let us assume further that both the camera and object are stationary, but that the light source can be moved. The key advantage of this assumption is that the correspondence problem between frames is solved in advance since the same view of the object is always seen. If we have three pictures of the object I(j)(j = 1, 2, 3) from light source directions s(j)(j = 1, 2, 3), respectively, then the intensities in any fourth frame, taken from a novel setting of the light source, must simply be a linear combination of the intensities in the ﬁrst three frames. Mathematically, I(4) =

3

a(j)I(j)

j=1

for some coeﬃcient a(j) [14]. Epstein et al. [4] showed empirically that this relationship is generally quite good, but it was left to Belhumeur and Kriegman [3] to prove that the correct relationship assuming no self-shadowing is in fact 3 I(4) = max( a(j)I(j), 0) , j=1

(1)

198

A. Nakashima, A. Maki, and K. Fukui

Fig. 1. Images of an object under variable lighting and an illumination basis.

Fig. 2. Images of an object in motion and an illumination basis.

where images I(j)(j = 1, 2, 3) must be taken with the light source in the bright cell, which is the cell of light source directions that illuminate all points on the object, and the maximum operator is eﬀected independently on each pixel and establishes whether or not that pixel was illuminated at all by the light source. These ﬁrst three images form a 3D image basis from which all other images can be derived and hence can be replaced by any equivalent basis of images. Hence Equation 1 can be replaced by, I(j) = max([˘I(1) ˘I(2) ˘I(3)]a(j), 0) ,

(2)

where a set of ˘I(j)(j = 1, 2, 3) is an image basis, which we call illumination image basis, and the 3D vector a(j) contains linear coeﬃcients that map the basis to the j th image. An example is shown in Figure 1. The four images on the left are original images of a statue of Caesar in frontal view under variable lighting. Applying the principle component analysis (PCA) to these images, we obtain an illumination image basis. It is shown in the three images on the right in Figure 1 arranged in the order of the eigenvalues. In theory, the illumination image basis can be obtained directly as three images captured in a ﬁxed pose under three diﬀerent lighting conditions. In this case, however, the illumination image basis remains to be contaminated if the captured images contain any noise. In order to obtain an illumination image basis in which the noise is attenuated, PCA is often applied to a larger number of images which are captured as inputs. Although we illustrate it in Figure 1 using only four images as the minimum number for the inputs, it is to show the relevancy of the method, and also for the consistency with the remaining examples. The concept of illumination image basis is quite general, and has been employed to develop further sophisticated algorithms for object recognition as well as image rendering as seen for example in [16].

Constructing Illumination Image Basis from Object Motion

199

In this paper, we deal with images of an object in motion captured by a ﬁxed camera under stationary illumination. In such a situation, the object is observed diﬀerently according to the motion. Figure 2 shows four images of the statue of Caesar in various poses illuminated from its frontal direction. From these images we wish to construct an illumination image basis as shown in the right-hand part of Figure 2 that is a counterpart to the illumination image basis obtained in the conventional way in Figure 1. Methods for object recognition which give successful result for arbitrary lighting conditions should then become analogously available without collecting images under various illumination. Unlike the conventional situation, however, it is not so straightforward to construct an illumination image basis from images of an object in motion. While this is due to the correspondence problem between frames which is not solved in advance, we do overcome the problem by adapting the geotensity constraint which was originally proposed for 3D surface reconstruction in [8]. We introduce the geotensity constraint in the context of constructing an illumination image basis in the next section, and describe the actual algorithm constructing the illumination image basis in Section 4.

3

Geotensity Constraint

The geotensity constraint is a constraint on geometrically corresponding pixel intensity between four or more images of an object seen from diﬀerent views under static lighting conditions. In this section, we introduce the geotensity constraint for a single light source from the viewpoint of constructing an illumination image basis; a detailed explanation of the geotensity constraint can be found in [8]. In general, the geotensity constraint consists of two parts, the geometry constraint and the intensity constraint, from which the geotensity constraint is derived. 3.1

Intensity Constraint

As shown in Equation 2, images of an object in a ﬁxed pose captured under arbitrary illumination can be represented by three base images, and the representation is also valid for an intensity at each pixel. That is, the key advantage of Equation 2 is that the same view of the object is always seen and the correspondence problem between frames is solved in advance. Now let us assume a static camera and light source and a moving object. Although we no longer have correspondence between images, we consider the intensity Ii (j) of the ith point on the surface of the object projected into the j th image. Ii (j) must then satisfy Ii (j) = max([I˘i (1) I˘i (2) I˘i (3)]a(j), 0) ,

(3)

where a set of ˘I(j)(j = 1, 2, 3) represents the intensity at corresponding points in the three base images. Equation 3 is the intensity constraint between images.

200

3.2

A. Nakashima, A. Maki, and K. Fukui

Geometry Constraint

Regarding the ﬁrst image as the reference frame, for simplicity we discuss the aﬃne and scaled-orthographic camera models [11] for projection. Consider the ith world point Xi = (Xi , Yi , Zi ) on the surface of an object projected to image point xi (j) = (xi (j), yi (j)) in the j th frame. The aﬃne camera model deﬁnes this projection as xi (j) = M(j)Xi + t(j) , where M(j), an arbitrary 2×3 matrix, and t(j), an arbitrary 2-dimensional vector, encode the motion parameters of the object. The scaled orthographic camera model replaces M(j) with the ﬁrst two rows of a scaled 3×3 rotation matrix R(j) so that R1 (j) xi (j) = λ Xi + t(j) . R 2 (j) Note that R(j) represents the rotation of the object from the ﬁrst frame to the j th frame. The Euclidean structure and motion parameters ﬁtting the weak perspective camera model can be recovered by the technique of structure from motion [18]. There is an arbitrary choice of aﬃne or Euclidean frame which can be partially ﬁxed by choosing the ﬁrst frame to be canonical [7] such that 100 0 Xi + . (4) xi (1) = 010 0 A result of choosing the canonical frame is that the structure vectors have the form, Xi = (x i (1), Zi ) and we can derive the relationship, xi (1) xi (j) = M(j) + t(j) . (5) Zi This relationship eﬀectively describes the epipolar constraint between two images. This constraint says that given a point, xi (1), in one image then the corresponding point in another image, xi (j), will lie on a line deﬁned by the object motion parameters, M(j) and t(j), between the two frames and that the exact position along that line depends on the correct depth, Zi , of the point. The corresponding point on an epipolar line is shown in Figure 3. Indeed, given a point in one image and the motion parameters for a number of other frames, then the corresponding points in all the other images are uniquely determined by a particular choice of Zi . We will use Equation 5 as a practical aﬃne form of the geometric constraint later. A similar form of this geometric (epipolar) constraint can be easily derived for the more complex camera models. 3.3

Geotensity Constraint

Consider a set of nj images, I(j)(j = 1, . . . , nj ), and a point (xTi , Zi )T on the surface of an object with depth Zi that projects to point, xi (1), in the ﬁrst image.

Constructing Illumination Image Basis from Object Motion

201

Fig. 3. Given a point in one image, according to the choice of depth Z, corresponding points in the other images are searched on the epipolar lines while investigating the intensity constraint.

Under object motion and aﬃne imaging conditions we can recall Equation 5 for the geometric constraint imposed on a sequence of images and the corresponding pixel in the j th image, xi (j), is given by Equation 5. Hence, the corresponding pixel intensity, Ii (j), is given by xi Ii (j) = I(j)[xi (j)] = I(j)[M(j) + t(j)] . (6) Zi If the motion parameters, M(j) and t(j), and the depth of the point on the object surface, Zi , are given, Ii (j) is measured from the intensity at the corresponding pixels in the j th image. On the other hand, when the corresponding pixel intensity Ii (j) is non-zero, it has another representation by the intensity constraint deﬁned by Equation 3 which we can rewrite as Ii (j) = I˘i (1) I˘i (2) I˘i (3) a(j) . (7) For nj measured intensities Ii (j) which are non-zero, it follows ˘ ˘ ˘ I n = Ii (1) Ii (2) Ii (3) anj (j) , j

(8)

where Inj and anj are nj -dimensional vector consisting of the non-zero measured intensity and a 3 × nj matrix containing the columns a(j), respectively. When the linear coeﬃcients a(j) are known in advance, using the non-zero measured intensity Ink , I˘i (j)(j = 1, 2, 3) can be computed by the solution of Equation 8 in the form −1 . (9) I˘i (1) I˘i (2) I˘i (3) = I nk ank (ank ank ) Then, the other representation of the corresponding pixel intensity, Iˆi (j), is given by Iˆi (j) = max( I˘i (1) I˘i (2) I˘i (3) a(j) , 0). (10)

202

as

A. Nakashima, A. Maki, and K. Fukui

Now let us deﬁne an error between the measured and estimated intensities (Ii (j) − Iˆi (j))2 . (11) Ei = j

If the estimated values Iˆi (j) are correct, they agree with the measured intensity Ii (j). Hence, the constraint on geometrically corresponding intensity between captured images, namely the geotensity constraint, can be stated simply as Ei = 0. Since the measured intensity Ii (j)(j = 1, · · · , nj ) includes noise in practice, the corresponding intensity between captured images, Iˆi (j)(j = 1, · · · , nj ), is estimated so as to minimize the error in Equation 11.

4

Construction of Illumination Image Basis

In this section we propose a method to construct an illumination image basis from an image sequence by applying the geotensity constraint. The basic concept is to search for the depth, Zi , of the surface of the object at each pixel, xi , in a reference image while computing the error Ei in the geotensity constraint. Then, the corresponding pixels are searched along the epipolar lines in the other images as schematically depicted in Figure 3. The algorithm for constructing an illumination image basis is detailed, preceded by descriptions of practical schemes for solving for the geometry and the intensity constraints. 4.1

Solving for Geometry

Computing geometric correspondence requires that M(j) and t(j) in Equation 5 be known. To recover these components we solve the well-known aﬃne structure from motion problem using SVD [17]. An initial measurement matrix containing the coordinates of a few corresponding sample points is thus needed and these must be provided by an independent mechanism. One way to sample proper points through a sequence is to employ a scheme to extract corners and ﬁnd their correspondence automatically, as seen for example in [1,19]. For the obtained corresponding sample points, we solve not only for motion parameters but also compute the correspondences between pixel intensity that are required for the next stage of solving for intensity. 4.2

Solving for Intensity

Using the sample corresponding points obtained in the process of solving for motion parameters, we must now acquire the linear coeﬃcients a(j) acting on the illumination image basis in Equations 9 and 10. We assume Lambertian surface properties, a convex object, and a single point light source. Observing ni sample corresponding pixels in each image through nj frames, we can form the matrix equation I = ˘Ia

(12)

Constructing Illumination Image Basis from Object Motion

203

Fig. 4. Re-assigned images in frontal view.

Fig. 5. Synthesized images under various illumination.

where I is an ni × nj matrix containing the corresponding sample pixel intensity Ii (j), ˘I is an ni × 3 matrix containing a 3D basis for the sparse elements in ˘I (j)(j = 1, 2, 3) such that ˘I = [˘I (1) ˘I (2) ˘I (3)] and a is a 3 × nj matrix whose columns are the linear coeﬃcients a(j) as a = [a(1) a(2) . . . a(nj )]. We record the corresponding sample pixel intensity, Ii (j), in I, which we call illumination matrix. Equation 12 is then in the familiar form for solution by singular value decomposition to obtain a rank 3 approximation to the matrix I. Hence, the linear coeﬀecints are acquired from the intensity of the small number of sample pixels in advance. In order to solve for singular value decomposition of I, in this case, at least four sample points are required through four images in the sequence. The astute reader would have noticed that Equation 12 can only be used if all the pixels used in forming I were illuminated in all j images. This is easy to ensure since under a single light source these pixels will all have non-zero intensity values and thus it is not necessary for the images to have been taken with the light in the bright cell. In practice we employ a robust random sampling and consensus technique (RANSAC) to ensure that artifacts that are caused by an object not fulﬁlling the assumed conditions (eg. specularities and self-shadowing) do not distort the correct solution. As is well known, the solution is unique up to an arbitrary invertible 3×3 transformation A since ˘Ia = (˘IA)(A−1 a). (13) However, we may leave A undetermined, since A is canceled when computing the estimated intensity using Equations 9 and 10. 4.3

Constructing Illumination Image Basis

We describe an algorithm for constructing illumination image basis by way of dense search for the corresponding pixel intensity applying the geotensity constraint. Note that we have acquired M(j), t(j) and a(j) through the techniques

204

A. Nakashima, A. Maki, and K. Fukui

discussed in the previous two sections. At each pixel in the reference image, xi (1), we measure the corresponding pixel intensity, Ii (j), estimate Iˆi (j), and compute the error, Ei , in the geotensity constraint at depth, Zi , at regular small intervals. The corresponding pixel intensities are computed from Equations 6, 9, and 10 armed with M(j), t(j) and a(j). As is apparent in Equation 6, the pixel coordinate where we acquire the intensity depends on the depth map. When the depth is correct we expect the error to approach zero. The eﬀective search range for the depth is determined according to the pointalistic model of the object computed in Section 4.1 so that it is limited to the range where the object may exist in space. By the minimization of the error at each pixel in the reference frame, xi (1), we acquire the intensity at the corresponding pixels in the other frames, Ii (j). With the acquired intensity in the pixels xi (j), we can construct the images of the object seen from the same view as the reference frame. Figure 4 shows the images constructed from four images on the left shown in Figure 2 in various views. The viewing direction in the constructed images is aligned to that in the reference frame and the images appear to be illuminated by variable lighting source. We can either employ a set of three images from the constructed images directly as an illumination image basis, or apply the PCA to the constructed images and form an illumination basis by the three eigenvectors corresponding to the highest eigenvalues. The three images on the right in Figure 2 compose an illumination image basis obtained by applying the PCA to the constructed images shown in Figure 4. Indeed, images under arbitrary illumination can be synthesized by the resulting illumination basis as shown in Figure 5. Given nj images of an object in motion, the framework of the algorithm is summarized as follows: 1◦ We set a reference image as the ﬁrst one, I(1). 2◦ For a pointalistic model, we compute the motion parameters M(j)(j = 1, · · · , nj ) and t(j)(j = 1, · · · , nj ), and linear coeﬃcients, a(j)(j = 1, · · · , nj ). 3◦ For a particular guess of depth, Zi , at point xi , we measure Ii (j)(j = 1, · · · , nj ) using Equation 6 with M(j), t(j), and a(j). ◦ 4 These measurements are fed into Equation 9 to calculate the values I˘i (j)(j = 1, 2, 3). 5◦ The estimated values, Iˆi (j)(j = 1, · · · , nj ), are computed using Equation 10. 6◦ The error Ei can be obtained using Equation 11. 7◦ Do 3◦ – 6◦ while searching for a depth that minimizes the error Ei . 8◦ Do 3◦ – 7◦ for all pixels in the reference frame, xi (1). 9◦ The corresponding pixel intensity Ii (j)(j = 2, · · · , nj ) is aligned to the pixel in the reference image xi (1). 10◦ We apply the PCA to the aligned images and the reference frame. Then, an illumination basis is acquired as three eigenvectors corresponding to the highest eigenvalues. In the process 10◦ , alternatively, we may choose three images directly from the reference and aligned images as an illumination basis.

Constructing Illumination Image Basis from Object Motion

205

Fig. 6. Images of a face under variable lighting and an illumination basis.

Fig. 7. Input images of a face in motion and an illumination basis in frontal view.

Fig. 8. Images of another face under variable lighting and an illumination basis.

Fig. 9. Input images of a face in motion and an illumination basis in frontal view.

5

Experiments

In this section, we show several experiments to conﬁrm the validity of the proposed algorithm. We apply the algorithm to a statue of Caesar and human faces. First, we compare the results obtained by the proposed algorithm with those obtained in the conventional way. Then, we explore the possibility of utilizing the proposed method for face recognition by comparing the diﬀerence between the obtained illumination image bases for faces of diﬀerent subjects. It is also shown that the method makes it possible to construct illumination image bases in various views according to the choice of the reference frame. 5.1

Illumination Image Bases of a Statue of Caesar and Human Faces

Let us begin by conﬁrming the validity of the proposed method. Using a statue of Caesar and human faces as objects, we compare illumination bases constructed from images of the objects under motion with those constructed from images under variable illumination. Although the results for Caesar have already been referred to in the previous sections for explanation, we discuss more of the details here.

206

A. Nakashima, A. Maki, and K. Fukui

Table 1. (a) Similarities between the subspaces of the same subjects spanned by illumination image bases constructed in the conventional and proposed methods. (b) Similarities between person A and B.

subjects Caesar person A person B similarities 0.99 0.99 0.99 (a)

person A \ B illumination object motion illumination 0.94 0.95 object motion 0.95 0.94 (b)

In Figures 1, 6, and 8, four images on the left are captured under variable lighting. We extract the central area including eyes, nose, and mouth from the images. By applying PCA to the extracted areas, each illumination image basis is obtained by three eigenvectors. It is shown in the right-hand parts of Figures 1, 6, and 8 and arranged according to eigenvalues. As observed in [6], the ﬁrst illumination base image is illuminated from the front, the second from the right or left, and the third from the top or bottom. In Figures 2, 7, and 9, on the other hand, four images on the left are input images of the objects in motion. The left image in frontal view is set to be a reference frame. By applying the geotensity constraint to the input images, we search correspondences between the images, and then images in frontal view under variable illumination are obtained by realigning the intensity in the process 9◦ in Section 4.3. The re-alinged images for Caesar are shown in Figure 4. It is observed that searching of correspondences is successful. These images are the analogues to the four left-hand images of the object under variable lighting shown in Figure 1. As in Figures 1, 6, and 8 PCA is applied to the re-aligned images to obtain an illumination image basis. The results are shown in the right-hand parts of Figures 2, 7, and 9. The ﬁrst base image is illuminated from the front, the second from the right or left, and the third from the top or bottom. These aspects are similar to those in Figures 1, 6, and 8 except the reverse of white and black which means that only the signs of elements of the base image are opposite. Hence, each of the base images in Figures 2, 7, and 9 appears to correspond with each of those in Figures 1, 6, and 8. Mathematically, however, since there exist inﬁnite illumination image bases which span the illumination subspace unique to one object in one pose, it is not prerequisite for an illumination image basis constructed from images under object motion to exactly correspond with the counterpart from images under varying illumination. In order to evaluate the illumination image basis acquired by the proposed method, thus, we investigate its closeness to the counterpart due to varying illumination by comparison in the level of the subspaces. For computing the similarities between the subspaces spanned by illumination image bases constructed in the conventional and proposed methods in quantity, we introduce a canonical angle θ. It represents an angle between two spaces, S1 and S2 , and deﬁned by

Constructing Illumination Image Basis from Object Motion

cos2 θ =

sup

s1 ∈S1 ,s2 ∈S2 ||s1 ||=0,||s2 ||=0

(s1 , s2 ) . ||s1 ||2 ||s2 ||2

207

(14)

If S1 exactly corresponds to S2 , then cos2 θ is equal to 1. The results of computing Equation 14 between the illumination subspaces represented by illumination image bases shown in Figures 1 and 2, 6 and 7, and 8 and 9 are 0.99 as listed in Table 1 (a). From the results, it can be inferred that an illumination image basis is constructed from object motion by the proposed algorithm as well as from variable illumination. As for the human faces, the second and third base images in Figures 7 and 9 have some destroyed parts in their appearance, which are seemingly caused by the non-rigidity of the subjects especially around the eyes (the statue of Caesar is rigid on the other hand). Nevertheless, the resulting ﬁgures of similarity in terms of the illumination subspaces are not deteriorated and thereby show the validity of the proposed algorithm. We also explore the possibility of utilizing the illumination image bases obtained by the proposed method for face recognition, by computing the diﬀerences between the illumination subspaces for two diﬀerent human subjects. The resulting angles computed by Equation 14 for all possible combinations of the illumination subspaces derived from the two diﬀerent human subjects are listed in Table 1 (b). The computed similarities for the two subjects, shown in Figures 6 through 9, are 0.94 or 0.95 for diﬀerent cases of utilizing varying illumination and object motion interchangeably. The values are smaller than those for the similarity between the illumination image bases of the same subject, 0.99. In the same way, illumination image bases are constructed for several diﬀerent subjects, and similar results are obtained. That is, the similarities for the same subjects are 0.99 and for diﬀerent subjects are around 0.94. Hence, the diﬀerence between the two subjects can be regarded as reasonably well conserved. Although the diﬀerence between 0.94 and 0.99 may seem trivial, it is in practice a signiﬁcant diﬀerence when considered in algorithms for face recognition. In Fisher face [2], for example, the ratio of dispersion of images classiﬁed in a correct category to dispersion of images classiﬁed in a false category is minimized. Since the minimization has the eﬀect of emphasizing diﬀerences between categories, the observed diﬀerence will be feasible recognized in the algorithm. 5.2

Illumination Image Bases in Various Views

In the proposed method, illumination image bases in various views can be constructed according to the selected reference frames. Here, we use the input images in Figure 7 again. The four input images are set to be a reference frame in turn, and illumination image bases are constructed. Then, we synthesize images under various illumination represented by the linear combination of the obtained illumination base images with various linear coeﬃcients. The synthesized images in views corresponding to the reference frames are shown in Figure 10. It can be seen that the proposed method makes construction of illumination image bases in various views possible only by replacing the reference image with other images

208

A. Nakashima, A. Maki, and K. Fukui

Fig. 10. Illumination bases in various views. The reference images are selected from the input images in Figure 7.

in diﬀerent views. Therefore, constructing illumination image bases from object motion is useful to deal with variations of not only illumination but also views in the appearance-based object recognition. When replacing the reference frame, we do not need to search the small number of sample points and to recompute motion parameters and linear coefﬁcients. Once the motion parameters and linear coeﬃcients are computed, the new motion parameters for a diﬀerent reference image are easily derived from the previous motion parameters which represent the relative motion between the frames and the linear coeﬃcients can be used directly as they are.

6

Discussions and Conclusions

We have proposed a method to construct an illumination image basis from images of an object in motion under ﬁxed illumination. The experiments showed the validity of the proposed method for a statue of Caesar and human faces. The advantages of the method are: 1. An illumination image basis is constructed from images of an object in motion without changing illumination or camera view. 2. Images under arbitrary illumination are synthesized by linear combination of the obtained illumination base images equivalently to the conventional method. 3. Illumination image bases in various poses are constructed from an image sequence only by replacing the reference frame. 4. By utilizing the illumination image bases in various poses, we can enjoy the advantages of an object recognition algorithm that considers the varieties of illumination and poses. Although we dealt with the geotensity constraint in the case of a single lighting source for simplicity in this paper, extension to the case of multiple light sources is possible by applying the geotensity constraint under multiple light sources [9]. In general, illumination image bases in various views constructed by the proposed method can be utilized for image rendering [16], and for object recognition [12], although occlusion problem is left as the future work.

Constructing Illumination Image Basis from Object Motion

209

Acknowledgments. The authors wish to thank Hiroshi Hattori for providing the RANSAC program to ﬁnd corner correspondence.

References 1. P.A. Beardsley, P. Torr, and A.P. Zisserman. 3D model acquisition from extended image sequences. In 4rd ECCV, pages 683–695, Cambridge, UK, 1996. 2. P.N. Belhumeur, J.P. Hespanha, and Kriegman D.J. Eigenfaces vs. ﬁsherfaces: recognition using class speciﬁc linear projection. IEEE-PAMI, 19(7):711–720, 1997. 3. P.N. Belhumeur and D.J. Kriegman. What is the set of images of an object under all possible illumination conditions? IJCV, 28:3:245–260, 1998. 4. R. Epstein, P.W. Hallinan, and A. Yuille. 5 ± 2 Eigenimages suﬃce: an empirical investigation of low-dimensional lighting models. In Proc. IEEE Workshop on Physics-based Modeling in CV, 1995. 5. A.S. Georghiades, D.J. Kriegman, and P.N. Belhumeur. Illumination cones for recognition under variable lighting: Faces. In CVPR, pages 52–58, 1998. 6. P.W. Hallinan. A low-dimensional representation of human faces for arbitrary lighting conditions. In CVPR, pages 995–999, 1994. 7. R.I. Hartley. Euclidean reconstruction from uncalibrated views. In J.L. Mundy, A. Zisserman, and D. Forsyth, editors, Proc. Applications of Invariance in computer vision, pages 238–256. Springer-Verlag, 1994. Lecture Notes in Computer Science 825. 8. A. Maki, M. Watanabe, and C.S. Wiles. Geotensity: Combining motion and lighting for 3d surface reconstruction. In 6th ICCV, pages 1053–1060, 1998. 9. A. Maki and C. Wiles. Geotensity constraint for 3d surface reconstruction under multiple light sources. In 6th ECCV, pages 725–741, 2000. 10. Y. Moses, Y. Adini, and S. Ullman. Face recognition: the problem of compensating for changes in illumination direction. In J.O. Eckland, editor, 3rd ECCV, pages 286–296, Stockholm, Sweden, 1994. Springer-Verlag. 11. J.L. Mundy and A. Zisserman, editors. Geometric invariance in computer vision. The MIT Press, 1992. 12. H. Murase and S.K. Nayer. Visual learning and recognition of 3-d objects from appearance. IJCV, 14(1):5–24, 1995. 13. A.P. Pentland. Photometric motion. IEEE-PAMI, 13:9:879–890, 1991. 14. A. Shashua. Geometry and photometry in 3D visual recognition. PhD thesis, Dept. Brain and Cognitive Science, MIT, 1992. 15. A. Shashua. On photometric issues in 3d visual recognition from a single 2d image. IJCV, 21:99–122, 1997. 16. A. Shashua and T. Riklin-Raviv. The quotient image: Class-based re-rendering and recognition with varying illuminations. IEEE-PAMI, 23(2):129–139, 2001. 17. C. Tomasi and T. Kanade. Shape and motion from image streams under orthography: a factorization method. IJCV, 9:2:137–154, 1992. 18. D. Weinshall and C. Tomasi. Linear and incremental acquisition of invariant shape models from image sequences. In 4th ICCV, pages 675–682, 1993. 19. C.S. Wiles, A. Maki, and N. Matsuda. Hyper-patches for 3d model acquisition and tracking. IEEE-PAMI, 23(12):1391–1403, 2001. 20. R.J. Woodham. Photometric method for determining surface orientation from multiple images. Optical Engineering, 19:139–144, 1980. 21. A. Yuille and D. Snow. Shape and albedo from multiple images using integrability. In CVPR, pages 158–164, 1997.

Diﬀuse-Specular Separation and Depth Recovery from Image Sequences Stephen Lin1 , Yuanzhen Li1,2 , Sing Bing Kang3 , Xin Tong1 , and Heung-Yeung Shum1 2

1 Microsoft Research, Asia Chinese Academy of Science 3 Microsoft Research

†

Abstract. Specular reﬂections present diﬃculties for many areas of computer vision such as stereo and segmentation. To separate specular and diﬀuse reﬂection components, previous approaches generally require accurate segmentation, regionally uniform reﬂectance or structured lighting. To overcome these limiting assumptions, we propose a method based on color analysis and multibaseline stereo that simultaneously estimates the separation and the true depth of specular reﬂections. First, pixels with a specular component are detected by a novel form of color histogram diﬀerencing that utilizes the epipolar constraint. This process uses relevant data from all the stereo images for robustness, and addresses the problem of color occlusions. Based on the Lambertian model of diﬀuse reﬂectance, stereo correspondence is then employed to compute for specular pixels their corresponding diﬀuse components in other views. The results of color-based detection aid the stereo correspondence, which determines both separation and true depth of specular pixels. Our approach integrates color analysis and multibaseline stereo in a synergistic manner to yield accurate separation and depth, as demonstrated by our results on synthetic and real image sequences.

1

Introduction

The diﬀerence in behavior between diﬀuse and specular reﬂections poses a problem in many computer vision algorithms. While purely diﬀuse reﬂections ordinarily exhibit little variation in color from diﬀerent viewing directions, specular reﬂections tend to change signiﬁcantly in both color and position. A vast majority of techniques in areas such as stereo and image segmentation account only for diﬀuse reﬂection and disregard specularities as noise. While this diﬀuse assumption is generally valid for much of an image, processing of regions that contain specular reﬂections can result in signiﬁcant inaccuracies. For instance, traditional stereo correspondence of specular reﬂections does not give the true depth of a scene point, but instead an incorrect virtual depth. To avoid such problems, methods for image preprocessing have been developed to detect and to separate specular reﬂections in images. †

This work was performed while the second author was visiting Microsoft Research, Asia.

A. Heyden et al. (Eds.): ECCV 2002, LNCS 2352, pp. 210–224, 2002. c Springer-Verlag Berlin Heidelberg 2002

Diﬀuse-Specular Separation and Depth Recovery from Image Sequences

1.1

211

Related Work

To distinguish between diﬀuse and specular image intensities, most separation methods utilize color images and the dichromatic reﬂectance model proposed by Shafer [15]. This model suggests that, in the case of dielectrics (non-conductors), diﬀuse and specular components have diﬀerent spectral distributions. The spectral distribution of the specular component is similar to that of the illumination, while the distribution of the diﬀuse component is a product of illumination and surface pigments. The color of a given pixel can be viewed in RGB color space as a linear combination of a vector for object reﬂectance color and a vector for illumination color. All image points on a uniform-colored surface lie on a dichromatic plane which is spanned by these two vectors. Rich literature exists on separation using the dichromatic model. Klinker et al. [7] developed a method based on the observation that the color histogram of a surface with uniform reﬂectance takes the shape of a “skewed T” with two limbs. One limb represents purely diﬀuse points while the other corresponds to highlight points. Their separation algorithm automatically identiﬁes the two limbs, then computes the diﬀuse component of each highlight point as its projection on the diﬀuse limb. To estimate specular color, Tong and Funt [16] suggest computing the dichromatic planes of several uniform-reﬂectance regions and ﬁnding the line most parallel to these planes. Sato and Ikeuchi [14] also employed the dichromatic model for separation by analyzing color signatures produced from many images taken under a moving light source. Besides color, polarization has also been an eﬀective cue for specular separation. Wolﬀ and Boult [17] proposed a polarization-based method for separating reﬂection components in regions of constant Fresnel reﬂectance. Nayar et al. [12] used polarization in conjunction with color information from a single view to separate reﬂection components, where a constraint is provided by neighboring pixels with similar diﬀuse color. These previous approaches have produced good separation results, but they have requirements that limit their applicability. Nearly all of these methods assume some sort of consistency among neighboring pixels. For the color algorithms, color segmentation is needed to avoid problematic overlaps in color histograms among regions with diﬀerent diﬀuse reﬂection color. Segmentation algorithms are especially unreliable in regions containing specularity, and errors in segmentation signiﬁcantly degrade the results of dichromatic plane analysis. Additionally, these color methods generally assume uniform illumination color, so interreﬂections cannot be present. For typical real scenes, which have textured objects and interreﬂections, anatomizing the color histogram is essentially infeasible. The need for segmentation also exists for polarization methods, which additionally require polarizer rotation. The algorithm by Sato and Ikeuchi [14] notably avoids these assumptions, but the need for an image sequence taken under a moving light source restricts its use. 1.2

Stereo in the Presence of Specularities

Stereo in the presence of specular reﬂection has been a challenging problem. Bhat and Nayar [1] consider the likelihood of correct stereo matching by analyzing

212

S. Lin et al.

the relationship between stereo vergence and surface roughness, and in [2] they further propose a trinocular system where only two images are used at a time in the computation of depth at a point. Jin et al. [5] poses this problem within a variational framework and seeks to estimate a suitable shape model of the scene. Of these methods, only [2] has endeavored to recover true depth information for specular points, but it involves extra eﬀorts to determine a suitable trinocular conﬁguration. 1.3

Our Approach: Color-Based Separation with Multibaseline Stereo

We circumvent the limiting assumptions made in previous works by framing the separation problem as one of stereo correspondence. By matching specular pixels to their corresponding diﬀuse points in other views, we can determine the diﬀuse components of the specularities using the Lambertian model of diﬀuse reﬂection. In this way, we employ multibaseline stereo in a manner that computes separation and depth together. Without special consideration of camera positions, we handle highlights in multibaseline stereo by forming correspondence constraints based on the following assumptions: diﬀuse reﬂection satisﬁes the Lambertian property; specular reﬂections vary in color from view to view; scene points having specular reﬂection exhibit purely diﬀuse reﬂection in some other views. The ﬁrst assumption is commonplace in computer vision, and the second assumption is not unreasonable, because specular reﬂections change positions in the scene for diﬀerent viewpoints. Even when a specular reﬂection lies on the same uniformly colored surface between two views, the underlying diﬀuse shading and the specular intensity will likely change, thus yielding diﬀerent color measurements [9]. The third assumption is dependent on surface roughnesses in the scene, but is often satisﬁed by having a suﬃciently long stereo baseline for the entire image sequence. With epipolar geometry and the aforementioned viewing conditions, our proposed method ﬁrst detects specular reﬂections using stereo-enhanced color processing. The color-based detection results are then used to heighten the accuracy of stereo correspondence, which yields simultaneous separation and depth recovery. It is this synergy between color and stereo that leads to better estimation of both separation and depth. With their combination, we develop an algorithm that is novel in the following aspects: – The epipolar constraint is used to promote color histogram diﬀerencing (CHD) by correspondence of image rows. – A technique we call Multi-CHD is introduced to handle color occlusions and to make full use of available images for robust specularity detection. – Diﬀuse components and depth of specular reﬂections are computed by stereo correspondence constrained by epipolar geometry and detection results. In comparison with previous separation approaches, our use of multibaseline stereo provides signiﬁcant practical advantages. First, prior image segmentation and consistency among neighboring pixels are not assumed. Second, our method

Diﬀuse-Specular Separation and Depth Recovery from Image Sequences

213

is robust to moderate amounts of interreﬂection, which are present in most real scenes. Third, capture of image sets can potentially be done instantaneously, since we do not require polarizer rotations or changes in lighting conditions that prohibit possible frame-rate execution. These three properties make our method more feasible than any previous method. The remainder of this paper is organized as follows. Section 2 describes the details of specularity detection by color histogram diﬀerencing under epipolar constraints. These detection results are used in Section 3 to constrain stereo correspondence, which determines specularity separation and depth. The eﬀectiveness of our algorithm is supported by experimental results presented in Section 4, and the paper closes with a discussion and some conclusions in Sections 5 and 6, respectively.

2

Color Histogram Diﬀerencing

For specularity detection, we develop a novel method that takes advantage of multibaseline stereo to improve color histogram diﬀerencing. CHD, introduced in [9], is based upon changes in the color of a specular reﬂection from view to view. Under the Lambertian property, the diﬀuse colors of a scene are viewpoint independent, with the exception of occlusions and disocclusions. Since the scene location of a specular reﬂection changes from view to view, its underlying diffuse reﬂection can diﬀer in shading or color. The underlying diﬀuse color is a component of the overall specular color, so a specular reﬂection will change in color between diﬀerent views. This diﬀerence in behavior between specular and purely diﬀuse reﬂections is exploited by CHD to detect specularities. Suppose that we have two images I1 and I2 of the same scene taken from diﬀerent viewpoints. For each image Ik , its pixel colors can be mapped into a binary RGB histogram Hk , such as those shown in Fig. 1. If a scene point contains purely diﬀuse reﬂection, then its positions in H1 and H2 will be identical. A specular reﬂection in H1 , however, will have a shifted histogram position in H2 . By subtracting points in H2 from H1 , we can detect the specular pixels in I1 : H1,spec = H1 − H2 = {p | p ∈ H1 , p ∈ H2 }. Because of image noise and other unmodelled eﬀects, some leeway is allowed in the diﬀerencing such that slight diﬀerences in histogram point positions between H1 and H2 do not indicate specularity. In the remainder of this section, we present our method of specularity detection based on CHD. First we describe tri-view CHD, which extends standard CHD to a trinocular multibaseline stereo setting in a way that takes advantage of epipolar geometry and addresses color occlusion. Then we generalize tri-view CHD to make full use of the abundant stereo images in a method we call multiCHD.

214

S. Lin et al.

specular

-

diffuse

H1

specular

=

specular

diffuse

-

H2

=

H1,spec

Fig. 1. Standard CHD. Since specular reﬂections generally change in color from one view to another, they can be detected by histogram diﬀerencing.

2.1

Tri-View CHD

Two major obstacles for standard CHD are histogram clutter and color occlusions. When parts of a histogram are crowded with points, CHD can fail because specular histogram points in one view can possibly be masked by other points, diﬀuse or specular, with similar color in another view. When this happens, specular pixels go undetected. The change in viewpoint also presents diﬃculties for CHD because diﬀuse colors present in one view may be occluded in another view by scene geometry or specular reﬂections. As a result, diﬀuse pixels are mistakenly detected as specular. To address both of these problems, we present a method called tri-view CHD. In standard CHD, diﬀerencing is done between histograms formed from entire images. Because of the large number of pixels involved, there often exists much crowding in color histograms. To reduce this clutter, we take advantage of epipolar geometry, which allows us to decrease the number of histogram points by diﬀerencing image rows rather than entire images. In multibaseline stereo, the correspondence between images rows is simply the matching scanlines. For tri-view CHD, matching scanlines for three images are illustrated in Fig. 2 and are denoted as IC for the central (reference) viewpoint, IL for the left viewpoint, and IR for the right viewpoint. The histograms corresponding to these scanlines are respectively labelled HC , HL and HR . We use three images to lessen the eﬀects of occlusion caused by both geometry and specular reﬂections. Diﬀuse reﬂection in IC that is geometrically occluded in IL or IR are present in the histogram HL ∪ HR , because diﬀerent parts of the scene are occluded when changing the viewpoint to the left or to the right. The same is true for diﬀuse reﬂection in IC that is occluded by specularity in IL or IR . Based on this observation, we can formulate the set of histogram points in HC that contain specular reﬂection as HC,spec = HC − (HL ∪ HR ). The detected specular histogram points are backprojected to the image to locate the specular pixels, which we represent as a binary image: 0 if IC (x, y) is non-specular (1) SC (x, y) = 1 if IC (x, y) is specular.

Diﬀuse-Specular Separation and Depth Recovery from Image Sequences

215

scene

L

R

C

IL

IC

IR

HL

HC

HR

Fig. 2. Tri-view CHD. In the multibaseline camera conﬁguration, matching epipolar lines are used for histogram diﬀerencing.

This consideration of occlusion eﬀects and cluttering in CHD diminishes both the number of false positives and the number of missed detections. Although we can compute the detection using tri-view CHD, the results can be sensitive to viewpoint intervals. For intervals that are too small, specularity color may not diﬀer signiﬁcantly, and for very large intervals visibility diﬀerences arise. Since an appropriate interval is dependent on scene geometry and surface roughnesses, it is impractical to compute. But since a multibaseline stereo sequence contains many viewpoints from which to form image triplets, we can make use of these available images for more robust detection. 2.2

Multi-CHD

We refer to use of the entire stereo sequence for tri-view CHD as multi-CHD. To explain this technique, let us consider the camera conﬁguration illustrated in Fig. 3, with (2n + 1) viewpoints from which we capture (2n + 1) images labelled as {ILn . . . IL2 , IL1 , IC , IR1 , IR2 . . . IRn }. For the center reference image IC , we can form n diﬀerent uniform-interval triplets {ILk , IC , IRk } for k = 1, 2 . . . n. Tri-view CHD can be performed on each of these triplets as described in the previous subsection to form k specular point sets SC,k . These sets are used to vote for the ﬁnal detection results. Voting is an eﬀective method for obtaining a more reliable result by combining data from a number of sources. The voted output reﬂects a consensus or compromise of the information. For each image pixel, it is deemed nspecular if the number of votes from SC,k exceeds a given threshold t, i.e., if k=1 SC,k > t.

216

S. Lin et al.

Fig. 3. Camera conﬁguration for Multi-CHD.

Although we employ this basic voting scheme, it may be enhanced by weighting votes according to its distance from the reference viewpoint. Additionally, the reliability of voting can be improved by increasing the number of statistically independent voters [8]. This can be done by considering image triplets where the viewpoints are unevenly spaced, or by generalizing the multi-CHD concept to a 2D grid of cameras.

3

Stereo Correspondence for Specular Reﬂections

Once we detect the specular pixels in an image, we attempt to associate them to their corresponding diﬀuse points in other images to determine separation and depth. Because of the large diﬀerences in intensity and color, a correspondence cannot directly be computed between a specular pixel and its diﬀuse counterpart, so we instead correspond diﬀuse pixels of other images under the constraint of their disparity relationship to the specular pixel in the reference image. In our approach, we ﬁrst use multibaseline stereo to compute initial disparity estimates for diﬀuse reﬂection. With these diﬀuse disparities, we impose a continuity constraint to determine a search range for estimation of specular disparities. This second round of stereo combines the use of spatially ﬂexible windows and a dynamically selected subset of images with which to compute the matches. Upon determining the correspondence, a diﬀuse component is estimated for producing the separation, and the computed disparity gives the true depth. 3.1

Initial Depth Estimation for Diﬀuse Reﬂection

Our specular correspondence method begins with an initial depth estimate that employs the epipolar constraint. This preliminary estimate is reasonably accurate for purely diﬀuse pixels, and it is used to facilitate more precise depth

Diﬀuse-Specular Separation and Depth Recovery from Image Sequences

217

estimation for specular pixels. The uncertainty of this estimate can also be used as an additional independent cue for specularity detection. For a forward-facing multibaseline stereo conﬁguration [13], disparity varies linearly with horizontal pixel displacement. In order to estimate the disparity for a pixel (x, y), we ﬁrst aggregate the matching costs over a window as the sum of sum of squared diﬀerences (SSSD), namely ρ I0 (u, v) − Iˆk (u, v, d) , (2) ESSSD (x, y, d) = k=0 (u,v)∈W (x,y)

where ρ(•) is the per-pixel squared Euclidean distance in RGB between reference image I0 and Iˆk (warped image of Ik at disparity d). W (x, y) is a square window centered at (x, y). For each non-specular pixel (x, y) in the reference image as determined by multi-CHD, the minimum ESSSD for diﬀerent disparity values determines the estimated disparity d and the uncertainty u of the estimation: d(x, y) = arg min ESSSD(x,y,d) and u(x, y) = min ESSSD(x,y,d) . d

d

Similar to conﬁdence measures in other stereo works, u gauges uncertainty because match quality is poor when its value is high. This quantity will later be used within a conﬁdence weight for disparity estimates, and additionally, it can be used as a cue for specular reﬂection. Pixels with uncertainty above a speciﬁed threshold can be added to the multi-CHD detection results. To handle occlusions which pose a problem in dense multi-view stereo, we incorporate shiftable windows and temporal image selection [6] to improve the matching of boundary pixels and semi-occluded pixels. Since specular points cannot be matched in this way, we more carefully correspond them using the continuity constraint. 3.2

Continuity Constraint

With the computed disparity estimates for diﬀuse pixels, we calculate a disparity search interval for specular pixels using the continuity constraint. Based on the cohesiveness of matter, the continuity constraint proposed by Marr and Poggio [10] claims that disparity should vary smoothly throughout an image, given opaque material and piecewise continuous surfaces. The failure of this assumption at depth discontinuities can be remedied by using a large window for SSSD computation. We can safely impose this constraint to get an initial estimate and search range of disparity for specular points, even if they are located near depth discontinuities. We implement the continuity constraint using k-Nearest Neighbors (k-NN). For a particular specular point S(x, y), we ﬁnd k nearest neighbor points that are nonspecular around it. Their disparity estimates dl for l = 1 . . . k are accompanied by corresponding uncertainty values ul that indicate estimation quality. The initial estimate for disparity is expressed in terms of the k nearest neighbor disparities weighted by the reciprocal of their uncertainty values, i.e., di (x, y) = l dl ul −1 / l ul −1 .

218

S. Lin et al.

From this initial estimate, we restrict the search range of possible disparities to be within a preset bound dr so that the disparity candidate interval D(x, y) for S(x, y) is D(x, y) = di (x, y) − dr , di (x, y) + dr . (3) We take dr in our experiments to be equal to the standard deviation of the initial disparity estimates over the image. 3.3

Shiftable and Flexible Windows

Since pixels with specular reﬂection degrade area-based correlation, we employ shiftable and ﬂexible windows, rather than a ﬁxed window, to get a selectively aggregated matching error. The basic idea of shiftable windows is to examine several windows that include the pixel of interest, not just the window centered at that pixel. This strategy has been shown to eﬀectively deal with occlusions [11] [3]. We extend this idea by excluding pixels detected as specular in the previous section. This results in shiftable windows with ﬂexible shapes, and we show it improves the matching of pixels that are specular in some images and non-specular in others. This is furthermore eﬀective in dealing with pixels near the boundary between specular and diﬀuse regions. In the formation of these ﬂexible windows, we use the specular detection Sk from (1) for respective images Ik . For correspondence between image pairs I0 and Ik , we modify the n×n shiftable window Wn×n (x, y) that includes (x, y) into the window Wf (x, y) whose support is ﬂexibly shaped to exclude specularity: (4) Wf (x, y) = (u, v)|(u, v) ∈ Wn×n (x, y), S0 (u, v) = 0, Sk (u, v) = 0 . In the case where the number of valid pixels in a window falls below a threshold, we progressively increase the original window size. Over this ﬂexible and shiftable window, we aggregate the raw matching cost to compute the SSD: w(u, v)Eraw (u, v, d, k) ESSD (x, y, d, k) =

(u,v)∈Wf (x,y)

(u,v)∈Wf (x,y)

,

w(u, v)

where w(u, v) is the support weight of each pixel in Wf (x, y) for (x, y), which we set to the constant 1 to get the mean. 3.4

Temporal Selection

Rather than summing the match costs over all views, a better approach would be to dynamically select a subset of views where the support window is believed to be mostly diﬀuse and unoccluded. Towards this end, we can formulate from the specular detection results a temporally selective aggregated matching error: wt(k)ESSD (x, y, d, k)

ESSSD (x, y, d) =

k=0 , C(x,y)>T

k=0 , C(x,y)>T

where

wt(k)

,

Diﬀuse-Specular Separation and Depth Recovery from Image Sequences

C(x, y) =

[1 − Sk (u, v)].

219

(5)

(u,v)∈Wf (x,y)

The constraint C(x, y) > T ensures that in the selected views the correlation window includes an appropriate number of diﬀuse points, where T is a percentage of pixels in the original n × n shiftable window. The factors wt(k) are weights of ESSD (x, y, d, k) which could normalize for the number of temporally selected views. We instead use these weights to deal with occlusions in the selected views. Views with a lower local SSD error ESSD (x, y, d, k) are more likely to have visible corresponding pixels, so we set wt(k) = 1 for the best 50% of images satisfying constraint (5), and wt(k) = 0 for the remaining 50%. This temporal selection rule is similar to that described in [6]. Finally, we adopt a winner selection strategy to compute the ﬁnal disparity: d(x, y) = arg

min

d∈D(x,y)

ESSSD (x, y, d),

where D(x, y) is the candidate interval for disparity given in (3). 3.5

Separation and Depth Estimate

Upon corresponding a specular point PS to a diﬀuse point PD in any other view, we can directly compute depth from the disparity and can theoretically take the color of PD as the diﬀuse component of PS , because of the Lambertian diﬀuse reﬂection assumption. The specular component can simply be computed as the diﬀerence in color between PS and PD . In reality, we must also account for noise and mismatches. So we ﬁnd all possible corresponding diﬀuse points in the light ﬁeld for PS , and then compute the mean value of the color measurements to obtain PD .

4

Experimental Results

In this section, we present results on synthetic and real image sequences to validate our approach. For both sequences, our algorithm settings are a multiCHD voting threshold of 50% and a CHD diﬀerencing distance threshold equal to one standard deviation of the image noise. The depth interval values d4 in (3) are 1.4 for synthetic images and 2.2 for real images. 4.1

Synthetic Image Sequence

Experiments on synthetic images are used for comparison to ground truth data. Our synthetic sequence consists of 57 images generated using Phong shading. The baseline distance between consecutive views is 3.125mm. Fig. 4 displays images of our results. An image taken from our sequence is shown in (a). For this image, ground truth and our own detection images are exhibited in (b) and (c) respectively. When the scene patch for a pixel contains more than one color, color blending occurs because of pixel integration. The

220

S. Lin et al.

balance of this blending along color boundaries will change from view to view, so these pixels are mistakenly detected as specular by CHD. In (d-f), an image comparison of depth recovery is shown. Our depth estimation (f) more closely resembles the ground truth (d), because of our consideration of specular reﬂections. When specularities are disregarded as in (e), incorrect virtual depths are computed instead. The images (g-j) show the ground truth separation and our own. Our result is similar to the ground truth, even though several pixels of the fruit in the bowl contain specularity throughout our image sequence. When this happens, our method tends to match with neighboring pixels, which gives a reasonable approximation of the diﬀuse component.

4.2

Real Image Sequence

For a real sequence, we use 64 images with a baseline distance between consecutive views of 7.8125mm and a focal length of 60mm. For SSD matching, we use a window size of 9x9 and only eleven consecutive images for depth recovery, to avoid non-constant disparities that can result from large baselines. Fig. 5(a) displays an image from this sequence, and (b) exhibits our detection result by multi-CHD. There exist some false detections mainly due to color blending, but the true specular areas are mostly found. For specular reﬂections on the wooden bowl, the watermelon and the leaves, the depth recovery without consideration of specular reﬂections in (c) gives an inaccurate virtual depth, while our estimation in (d) appears to approximate the true depth. The apple presents diﬃculties due to lack of texture, which is problematic for all stereo methods. Our diﬀuse and specular separation results are shown in (e) and (f), respectively. Notice that although depth recovery is inaccurate for textureless regions like the apple, its separation result is not bad because correspondence is made with some other points in the textureless region, whose diﬀuse component is often close to being correct. Diﬀerences between tri-view CHD and standard (full-image) CHD are demonstrated in Fig. 6. In (a), we show the result of standard CHD between the reference image and the image taken 7.8125cm to the left. Because of color occlusions due to diﬀerences in scene visibililty, standard CHD detects many false positives on the right side of the image. Moreover, for the large specular area on the wooden bowl, there are many missing detections that result from histogram crowding. For standard CHD with the image taken 7.8125cm to the right, it is seen in (b) that color occlusions cause many false positives on the left side, and there are missing detections on both the wooden bowl and the watermelon. Triview CHD using these three images produces the result in (c), in which the color occlusion and histogram crowding problem is better handled. A combination of (a) and (b) cannot generate the results in (c), and we improve upon this result using all views in multi-CHD to get the detection shown in Fig 5(b).

Diﬀuse-Specular Separation and Depth Recovery from Image Sequences

221

Fig. 4. Experimental results for synthetic scene: (a) original image; (b) ground truth detection of specular reﬂection, with white points representing specular and black ones representing diﬀuse; (c) our detection (d) ground truth depth; (e) depth estimation without special processing for specularities; (f) depth estimation using our approach; (g) ground truth diﬀuse component; (h) our separated diﬀuse component; (i) ground truth specular component; (j) our separated specular component

222

S. Lin et al.

Fig. 5. Experimental results for real scene: (a) original image; (b) detection of specular reﬂection, with white points representing specular and black ones representing diﬀuse; (c) depth estimation without special processing for specularities; (d) depth estimation using our approach; (e) separated diﬀuse component; (f) separated specular component

Fig. 6. Comparison of tri-view CHD with standard CHD (a) standard CHD between reference image and 10th image in sequence to the left; (b) standard CHD between reference image and 10th image in sequence to the right; (c) tri-view CHD

Diﬀuse-Specular Separation and Depth Recovery from Image Sequences

5

223

Discussion

In previous separation works, there is a strong dependence on segmentation to provide constraints on the diﬀuse component of specular reﬂections. Rather than relying on segmentation to get this information, we transfer this burden to stereo, for which this problem is more tractable. Separation methods require segmentation to group pixels into uniform-reﬂectance regions that contain both specular and diﬀuse reﬂections. Because of the large and often sharp intensity diﬀerences between the two forms of reﬂection, such a segmentation is diﬃcult to compute. In stereo, the problem becomes one of matching equal-intensity diﬀuse pixels under the constraint of the specularity position, which should be easier than matching diﬀuse and specular pixels. In the experimental results, color mixing was mentioned as an obstacle to detection, but was nevertheless processed correctly for the separation result. Other image eﬀects that can disrupt our algorithm include color saturation, which can lead to crowding in a small part of the color histogram. Saturation can be substantially reduced or eliminated by capturing high dynamic range images [4]. Another obstacle is image noise, which can reduce accuracy in both detection and correspondence, but this can be remedied by averaging multiple images for each view. Problems can also arise when our viewing assumptions are not held. Diﬀuse reﬂection from very rough surfaces may not follow the Lambertian model, and this is a hindrance for specularity detection and stereo correspondence in general. Our second and third assumptions, that specular color changes among viewpoints and that a specular scene point in one view is diﬀuse in some other views, are ordinarily broken when specular reﬂections exhibit little displacement among the stereo views. This may occur in areas of very high curvature and on surfaces with large roughness, because single pixels contain a broad range of surface normals in these cases. Although separation is diﬃcult in such areas, the eﬀect on stereo correspondence might not be substantial, since specularities behave similarly to ﬁxed scene features.

6

Conclusion

We have described a new approach for identifying and separating specular components from an input image sequence. It integrates color analysis and multibaseline stereo in a single framework to produce accurate separation and dense true depth. The color analysis, which identiﬁes specular regions, is in the form of color histogram diﬀerencing with three new enhancements for increased robustness: (1) Extension to three views, (2) Use of epipolar constraints, and (3) Use of multiple triplets in a voting scheme. Once the specular pixels have been identiﬁed, we apply a dense stereo algorithm on the remaining pixels in order to extract the true depth. This stereo algorithm uses both shiftable windows and temporal selection in order to avoid the occlusion problem associated with depth discontinuity. Currently the image sequences were acquired by moving the camera along a linear path with constant velocity. We plan to extend this work by using the

224

S. Lin et al.

algorithm on sequences acquired with arbitrary camera motions. In addition, it would be interesting to learn how scene shape and lighting conditions aﬀect the optimal camera motion for capture and subsequent specular extraction.

References 1. D.N. Bhat and S.K. Nayar. Binocular stereo in the presence of specular reﬂection. In ARPA, pages II:1305–1315, 1994. 2. D.N. Bhat and S.K. Nayar. Stereo in the presence of specular reﬂection. In ICCV, pages 1086–1092, 1995. 3. A.F. Bobick and S.S. Intille. Large occlusion stereo. IJCV, 33(3):1–20, Sept. 1999. 4. P.E. Debevec and J. Malik. Recovering high dynamic range radiance maps from photographs. Computer Graphics (SIGGRAPH), 31:369–378, 1997. 5. H. Jin, A. Yezzi, and S. Soatto. Variational multiframe stereo in the presence of specular reﬂections. Technical Report TR01-0017, UCLA, 2001. 6. S.B. Kang, R. Szeliski, and J. Chai. Handling occlusions in dense multi-view stereo. In CVPR, pages 103–110, Dec. 2001. 7. G.J. Klinker, S.A. Shafer, and T. Kanade. A physical approach to color image understanding. IJCV, 4(1):7–38, Jan. 1990. 8. L. Lam and C.Y. Suen. Application of majority voting to pattern recognition: An analysis of its behaviour and performance. IEEE Trans. on Systems, Man, and Cyberbetics, 27(5):553–568, 1997. 9. S.W. Lee and R. Bajcsy. Detection of specularity using color and multiple views. Image and Vision Computing, 10:643–653, 1992. 10. D.C. Marr and T. Poggio. A computational theory of human stereo vision. In Lucia M. Vaina, editor, From the Retina to the Neocortex: Selected Papers of David Marr, pages 263–290. Birkh¨ auser, Boston, MA, 1991. 11. Y. Nakamura, T. Matsura, K. Satoh, and Y. Ohta. Occlusion detectable stereo occlusion patterns in camera matrix. In CVPR, pages 371–378, 1996. 12. S.K. Nayar, X. Fang, and T.E. Boult. Removal of specularities using color and polarization. In CVPR, pages 583–590, 1993. 13. M. Okutomi and T. Kanade. A multiple baseline stereo. IEEE PAMI, 15, 1993. 14. Y. Sato and K. Ikeuchi. Temporal-color space analysis of reﬂection. J. of the Opt. Soc. of America A, 11, 1994. 15. S. Shafer. Using color to separate reﬂection components. Color Research and Applications, 10, 1985. 16. F. Tong and B. V. Funt. Specularity removal for shape from shading. In Proc. Vision Interface, pages 98–103, 1988. 17. L.B. Wolﬀ and T.E. Boult. Constraining object features using a polarization rﬂectance model. IEEE PAMI, 13, 1991.

Shape from Texture without Boundaries D.A. Forsyth Computer Science Division U.C. Berkeley Berkeley, CA 94720 [email protected]

Abstract. We describe a shape from texture method that constructs a maximum a posteriori estimate of surface coeﬃcients using only the deformation of individual texture elements. Our method does not need to use either the boundary of the observed surface or any assumption about the overall distribution of elements. The method assumes that texture elements are of a limited number of types of ﬁxed shape. We show that, with this assumption and assuming generic view and texture, each texture element yields the surface gradient unique up to a two-fold ambiguity. Furthermore, texture elements that are not from one of the types can be identiﬁed and ignored. An EM-like procedure yields a surface reconstruction from the data. The method is deﬁned for othographic views — an extension to perspective views appears to be complex, but possible. Examples of reconstructions for synthetic images of surfaces are provided, and compared with ground truth. We also provide examples of reconstructions for images of real scenes. We show that our method for recovering local texture imaging transformations can be used to retexture objects in images of real scenes. Keywords: Shape from texture, texture, computer vision, surface ﬁtting

There are surprisingly few methods for recovering a surface model from a projection of a texture ﬁeld that is assumed to lie on that surface. Global methods attempt to recover an entire surface model, using assumptions about the distribution of texture elements. Appropriate assumptions are isotropy [15] (the disadvantage of this method is that there are relatively few natural isotropic textures) or homogeneity [1,2]. Current global methods do not use the deformation of individual texture elements. Local methods recover some diﬀerential geometric parameters at a point on a surface (typically, normal and curvatures). This class of methods, which is due to Garding [5], has been successfully demonstrated for a variety of surfaces by Malik and Rosenholtz [9,11]; a reformulation in terms of wavelets is due to Clerc [3]. The method has a crucial ﬂaw; it is necessary either to know that texture element coordinate frames form a frame ﬁeld that is locally parallel around the point in question, or to know the diﬀerential rotation of the frame ﬁeld (see [6] for this point, which is emphasized by the choice of textures displayed in [11]; the assumption is known as texture stationarity). A. Heyden et al. (Eds.): ECCV 2002, LNCS 2352, pp. 225–239, 2002. c Springer-Verlag Berlin Heidelberg 2002

226

D.A. Forsyth

There is a mixed method, due to [4]. As in the local methods, image projections of texture elements are compared yielding the cosine of slant of the surface at each texture element up to one continuous parameter. A surface is interpolated using an extremisation method, the missing continuous parameter being supplied by the assumption that the texture process is Poisson (as in global methods) — this means that quadrat counts of texture elements are multinomial. This method has several disadvantages: ﬁrstly, it requires the assumption that the texture is a homogenous Poisson process; secondly, it requires some information about surface boundaries; thirdly, one would expect to extract more than the cosine of slant from a texture element.

1

A Texture Model

We model a texture on a surface as a marked point process, of unknown spatial properties; deﬁnitions appear in [4]. In our model, the marks are texture elements (texels or textons, as one prefers; e.g. [7,8] for automatic methods of determining appropriate marks) and the orientation of those texture elements with respect to some surface coordinate system. We assume that the marks are drawn from some known, ﬁnite set of classes of Euclidean equivalent texels. Each mark is deﬁned in its own coordinate system; the surface is textured by taking a mark, placing it on the tangent plane of the surface at the point being marked, translating the mark’s origin to lie on the surface point being marked, and rotating randomly about the mark’s origin (according to the mark distribution). We assume that these texture elements are suﬃciently small that they will, in general, not overlap, and that they can be isolated. Furthermore, we assume that they are suﬃciently small that they can be modelled as lying on a surface’s tangent plane at a point.

2

Surface Cues from Orthographic Viewing Geometry

We assume that we have an orthographic view of a compact smooth surface and the viewing direction is the z-axis. We write the surface in the form (x, y, f (x, y)), and adopt the usual convention of writing fx = p and fy = q. Texture imaging transformations for orthographic views: Now consider one class of texture element; each instance in the image of this class was obtained by a Euclidean transformation of the model texture element, followed by a foreshortening. The transformation from the model texture element to the particular image instance is aﬃne. This means that we can use the center of gravity of the texture element as an origin; because the COG is covariant under aﬃne transformations, we need not consider the translation component further. Furthermore, in an appropriate coordinate system on the surface and in the image, the foreshortening can be written as 1 0 Fi = 0 cos σi where σi is the angle between the surface normal at mark i and the z axis.

Shape from Texture without Boundaries

227

The transformation from the model texture element to the i’th image element is then TM →i = RG(i) Fi RS(i) where RS (i) rotates the texture element in the local surface frame, Fi foreshortens it, and RG (i) rotates the element in the image frame. From elementary considerations, we have that 1 p q RG (i) = p2 + q 2 −q p The transformation from the model texture element to the image element is not a general aﬃne transformation (there are only three degrees of freedom). We have: Lemma 1: An affine transformation T can be written as RG FRS , where RG , RS are arbitrary rotations and F is a foreshortening (as above) if and only if det(T )2 = tr(T T T ) − 1 and 0 ≤ det(T )2 ≤ 1 If these conditions hold, we say that this transformation is a local texture imaging transformation. Proof: If the conditions are true, then T T T has one eigenvalue 1 and the other between zero and one. By choice of eigenvectors, we can diagonalize T T T to be RG F 2 RTG , meaning that T = RG FRS for arbitrary RS . The other direction is obvious. Notice that, given an aﬃne transformation A that is a texture imaging transformation, we know the factorization into components only up to a two-fold ambiguity. This is because A = RG FRS = RG (−I)F(−I)RS = A where I is the identity. The other square roots of the identity are ruled out by the requirement that cos σi be positive. Now assume that the model texture element(s) are known. We can then recover all transformations from the model texture elements to the image texture elements. We now perform an eigenvalue decomposition of Ti TiT to obtain RG (i) and Fi . From the equations above, it is obvious that these yield the value of p and q at the i’th point up to a sign ambiguity (i.e. (p, q) and (−p, −q) are both solutions). The Texture Element is Unambiguous in a Generic Orthographic View: Generally, the model texture element is not known. However, an image texture element can be used in its place. We know that an image texture element is within some (unknown) aﬃne transformation of the model texture element, but this transformation is unknown. Write the transformation from image element j to image element i as Tj→i

228

D.A. Forsyth

and recall that this transformation can be measured relatively easily in principle (e.g. [4,9,11]). If we are using texture element j as a model, there is a unique (unknown) aﬃne transformation A such that TM →i = Tj→i A for every image element i. The rotation component of A is of no interest. This because rotating a model texture element simply causes the rotation on the surface, RS , to change, and this term oﬀers no shape information. Furthermore, A must have positive determinant, because we have excluded the possibility that the element is ﬂipped by the texture imaging transformation. Finally, A must have a positive element in the top left hand corner, because we exclude the case where A is −I because this again simply causes the rotation on the surface, RS to change without aﬀecting the element shape. Assume that we determine A by searching over the lower diagonal aﬃne transformations to ﬁnd transformations such that TM →i = Tj→i A is a texture imaging transformation for every i. It turns out that there is no ambiguity. Lemma 2: Given TM →i for i = 1, . . . , N is a texture imaging transformation arising from a generic surface, then TM →i B is a texture imaging transformation for i = 1, . . . , N , for B a lower-diagonal matrix of positive determinant and with B00 > 0, if and only if B is the identity. Proof: Recall that only lower diagonal B are of interest, because a rotation in place does not aﬀect the shape of the texture element. Recall that det(M) = det(MT ) and trace(M) = trace(MT ). This means that, for M a texture imaging transformation, both M and MT satisfy the constraints above. We can assume without loss of generality that TM →1 = RG F1 (because we have choice of coordinate frame up to rotation on the model texture element). This T means that TM →1 TM →1 is diagonal — it is the square of the foreshortening — T T and so B TM →1 TM →1 B must also be diagonal and have 1 in the top left hand corner. This means B must be diagonal, and that B00 = 1 (the case B00 = −1 is excluded by the constraint above). So the only element of interest is B11 = λ. Now for some arbitrary j — representing the transform of another element — we can write ab T TM T = →j M →j bc where (a+c−1) = (ac−b2 ). Now if TM →j B is a texture imaging transformation, then we have a λb T T B TM →j TM →j B = λb λ2 c Now this matrix must also meet the constraints to be a texture imaging transformation, so that we must have that (a + λ2 c − 1) = λ2 (ac − b2 ) as well as (a + c − 1) = (ac − b2 ). Rearranging, we have the system (a − 1) = ((a − 1)c − b2 )

Shape from Texture without Boundaries

229

and (a − 1) = λ2 ((a − 1)c − b2 ) which has solutions when λ2 = 1 (or when (a, b) = (1, 0), which is not generic). If λ = −1, then the transformation’s determinant is negative, so it is not a texture imaging transformation, so λ = 1. Notice the case where λ = −1 corresponds to ﬂipping the model texture element in its frame, and incorporating a ﬂip back into the texture imaging transformation. Lemma 2 is crucial, because it means that, for orthographic views, we can recover the texture element independent of the surface geometry (whether we should is another matter). We have not seen lemma 2 in the literature before, but assume that it is known — it doesn’t appear in [10], which describes other repeated structure properties, in [12], which groups plane repeated structures, or in [13], which groups aﬃne equivalent structures but doesn’t recover normals. At heart, it is a structure from motion result. Special cases: The non-generic cases are interesting. Notice that if (a, b) = (1, 0) for all image texture elements, then 0 ≤ λ ≤ min(1/ cos σj ), where the minimum is over all texture elements. The easiest way to understand this case is to study the surface gradient ﬁeld. In particular, at each texture element there is an iso-depth direction, which is per−1 pendicular to the normal. Now apply TM →j to this direction for the j’th texture element, yielding a direction in the frame of the model texture element. This direction, and this alone, is not foreshortened. In the case that (a, b) = (1, 0), for all texture elements the same direction is not foreshortened. There are two cases. In the ﬁrst, this eﬀect occurs as a result of an unfortunate coincidence between view and texture ﬁeld. It is a view property, because the gradient of the surface (which is determined by the view), is aligned with the texture ﬁeld; this case can be dismissed by the generic view assumption. The more interesting case occurs when the texture element is circular; this means that the eﬀect of rotations in the model frame is null, so that texture imaging transformations for circular spots are determined only up to rotation in the model frame. In turn, any distribution of circular spots on the surface admits a set of texture imaging transformations such that (a, b) = (1, 0). This texture is ambiguous, because one cannot tell the diﬀerence between a texture of circular spots in a generic view and a texture of unfortunately placed ellipses; furthermore, the fact that λ is indeterminate means that the surface may consist of ellipses with a high aspect ratio viewed nearly frontally, or ones with a low aspect ratio heavily foreshortened. Again, a generic view assumption would allow only the ﬁrst interpretation, so when a texture element appears like an ellipse, we ﬁx its shape as a circle. Reasoning about the iso-depth direction in the model texture element’s frame allows us to understand lemma 2 in somewhat greater detail. In eﬀect, the reason B is generically unique is that it must ﬁx many diﬀerent directions in the model texture element’s frame. The only transformations that do this are the identity and a ﬂip. Recovering geometric data for orthographic views of unknown texture elements: Now assume that the texture element is unknown, but each texture imaging transformation is known. Then we have an excellent estimate of the texture element. In the simplest case, we assume the image represents the

230

D.A. Forsyth

Fig. 1. The reconstruction process, illustrated for a synthetic image of a sphere. Left, an image of a textured sphere, using a texture element that is not circular. Center, the height of the sphere, rendered as an intensity ﬁeld, higher points being closer to the eye; right shows the reconstruction obtained using the EM algorithm using the same map of height to intensity.

albedo (rather than the radiosity), and simply apply the inverse of each texture imaging transformation to its local patch and average the results over all patches. In the more interesting case, where we must account for shading variation, we assume that the irradiance is constant over the texture element. Write Iµ for the estimate of the texture element, and Ii for the patch obtained by applying Ti−1 to the image texture element i. Then we must choose Iµ and some set of constants λi to minimize Σi || λi Iµ − Ii ||2 Now assume that we have an estimate of the model texture element, and an estimate of the texture imaging transformation for each image texture element. In these circumstances, it is possible to tell whether an image texture element represents an instance of the model texture element or not — it will be an instance if, by applying the inverse texture imaging transformation to the image texture element, we obtain a pattern that looks like the model texture element. This suggests that we can insert a set of hidden variables, one for each image texture element, which encode whether the image texture element is an instance or not. We now have a rather natural application of EM. Practical details: For the i’th texture element, write θgi for the rotation angle of the in-image rotation, σi for the foreshortening, θsi for the rotation angle of the on-surface rotation and Ti for the texture imaging transformation encoded by these parameters. Write δi for the hidden variable that encodes whether the image texture element is an instance of the model texture element or not. Write Iµ for the (unknown) model texture element. To compare image and model texture elements, we must be careful about domains. Implicit in the deﬁnition of Iµ is its domain of deﬁnition D— say a nxn pixel grid — and we can use this. Write Ti−1 I for the pattern obtained by applying Ti−1 to the domain Ti (D). This is most easily computed by scanning D, and for each sample point s = (sx , sy ) evaluating the image at Ti−1 s.

Shape from Texture without Boundaries

231

We assume that imaging noise is normally distributed with zero mean and standard deviation σim . We assume that image texture elements that are not instances of the model texture element arise with uniform probability. We have that 0 ≤ σi ≤ 1 for all i, a property that can be enforced with a prior term. To avoid the meaningless symmetry where illumination is increased and albedo falls, we insert a prior term that encourages λi to be close to one. We can now write the negative log-posterior 1 1 || λi Iµ − Ti−1 I ||2 δi + (1 − δi )K + 2 (λi − 1)2 + L 2 2σim i 2σlight i where L is some unknown normalizing constant of no further interest. The application of EM to this expression is straightforward, although it is important to note that most results regarding the behaviour of EM apply to maximum likelihood problems rather than maximum a posteriori problems. We are aware of no practical diﬃculties that result from this observation. Minimisation of the Q function with respect to δi is straightforward, but the continuous parameters require numerical minization. This minimisation is unusual in being eﬃciently performed by coordinate descent. This is because, for ﬁxed Iµ , each Ti can be obtained by independently minimizing a function of only three variables. We therefore minimize by iterating two sweeps: ﬁx Iµ and minimize over each Ti in turn; now ﬁx all the Ti and minimize over Iµ .

Fig. 2. An image of a running cheetah, masked to suppress distractions, and of a spotted dress, ditto. These images were used to reconstruct surfaces shown in ﬁgures below.

3

Surface Cues from Perspective Viewing Geometry

Shape from texture is substantially more diﬃcult from generic perspective views than from generic orthographic views (unless, as we shall see, one uses the highly restrictive homogeneity assumption). We use spherical perspective for simplicity, imaging onto a sphere of unit radius. Texture imaging transformations for perspective views: Because the texture element is small, the transformation from the model texture element to the particular image instance is aﬃne. This means that we can again use the center of gravity of the texture element as an origin; because the COG is covariant under aﬃne transformations, we need not consider the translation

232

D.A. Forsyth

Fig. 3. Left, the gradient information obtained by the process described in the text for a synthetic image of a textured sphere. Since the direction of the gradient is not known, we illustrate gradients as line elements, do not supply an arrow head and extend the element forward and backward. The gradient is relatively small and hard to see on the many nearly frontal texture elements — look for the black dot at the center of the element. Gradients are magniﬁed for visibility, and overlaid on texture elements; gradients shown with full lines are associated with elements with a value of the hidden data ﬂag (is this a texture element or not) greater than 0.5; the others are shown with dotted lines. The estimated element is shown in the top left hand corner of the image. Right, a similar plot for the image of the spotted dress.

component further. The transformation from the model texture element to the i’th image element is then (p)

TM →i =

1 RG (i)Fi RS (i) ri

where ri is the distance from the focal point to the COG of the texture element, RS (i) rotates the texture element in the local surface frame, Fi foreshortens it, and RG (i) rotates the element in the image frame. We have superscripted this transformation with a (p) to indicate perspective. Again, RG (i) is a function only of surface geometry (at this point, the form does not matter), and RS (i) is a property of the texture ﬁeld. This transformation is a scaled texture imaging transformation. This means it is a general aﬃne transformation. This is most easily seen by noting that any aﬃne transformation can be turned into a texture imaging transformation by dividing by the largest singular value. The Texture Element is Ambiguous for Perspective Views: Recall we know the model texture element up to an aﬃne transformation. If we have an orthographic view, the choice of aﬃne basis is further restricted to a two-fold discrete ambiguity by lemma 2 — we need to choose a basis in which each TM →i is a texture imaging transformation, and there are generically two such bases. There is no result analogous to lemma 2 in the perspective case. This means

Shape from Texture without Boundaries

233

0.45

0.4

rms error in reconstruction

0.35

0.3

0.25

0.2

0.15

0.1

0.05

0

1

1.5

2

2.5

3 run

3.5

4

4.5

5

Fig. 4. The root mean square error for sphere reconstruction for ﬁve examples each of four diﬀerent cases, as a percentage of the sphere’s radius. The horizontal axis gives the run, and the vertical axis gives the rms error. Note that in most cases the error is of the order of 10% of radius. Squares and diamonds correspond to the case where the texture element must be estimated (in each run, the image is the same, so we have ﬁve diﬀerent such images), and circles and crossed circles correspond to the case where it is known (ditto). Squares and circles correspond to the symmetric error metric, and diamonds and crossed circles correspond to EM based reconstruction. Note ﬁrstly that in all but three cases the RMS error is small. Accepting that the large error in the ﬁrst three runs using the symmetric error metric for the estimated texture element may be due to a local minimum, there is very little to choose between the cases. This suggests that (1) the process works well (2) estimating the texture element is successful (because knowing it doesn’t seem to make much diﬀerence to the reconstruction process) and (3) either interpolation mechanism is probably acceptable.

that any choice of basis is legal, and so, for perspective views, we cannot recover the texture element independent of the surface geometry. This does not mean that shape from texture in perspective views is necessarily ambiguous. It does mean that, for a generic texture, we cannot disconnect the process of determining the texture element (and so measuring a set of geometric data about the surface) from the process of surface reconstruction. This implies that reconstruction will involve a fairly complicated minimisation process. We demonstrate shape from texture for only orthographic views here; this is because the process of estimating texture imaging transformations and the ﬁtting process can be decoupled. An alternative approach is to “eliminate” the shape of the model texture element from the problem. This was done by Garding [5], Malik and Rosenholtz [9, 11] and Clerc [3]; it can be done for the orthographic case, too [4], but there is less point. This can be done only under the assumption of local homogeneity.

4

Recovering Surface Shape from Transform Data

We assume that the texture imaging transformation is known uniquely at each of a set of scattered points. Now this means that at each point we know p and q up

234

D.A. Forsyth

to sign, leaving us with an unpleasant interpolation problem. We expect typical surface ﬁtting procedures to be able to produce a surface for any assignment of signs, meaning that we require some method to choose between them — essentially, a prior.

Fig. 5. The reconstructed surface for the cheetah of ﬁgure 2 is shown in three diﬀerent textured views; note that the curve of the barrel chest, turning in toward the underside of the animal and toward the ﬂank is represented; the surface pulls in toward the belly, and then swells at the ﬂank again. Qualitatively, it is a satisfactory representation of the cheetah.

The surface prior: The question of surface interpolation has been somewhat controversial in the vision community in the past — in particular, why would one represent data that doesn’t exist? (the surface components that lie between data points). One role for such an interpolate is to incorporate spatial constraints — the orientation of texture elements with respect to the view is unlikely to change arbitrarily, because we expect the scale of surface wiggles to be greater than the inter-element spacing. It is traditional to use a second derivative approximation to curvature as a prior; in our experience, this is unwise, because it is very

Shape from Texture without Boundaries

235

badly behaved at the boundary (where the surface is nearly vertical). Instead, we follow [4] and compute the norm of the shape operator and sum over the surface. This yields 1 2 2 π(θ) ∝ exp −( 2 ) (κ1 + κ2 )dA 2σk R where the κi are the extremal values of the normal curvatures. Robustness: We expect signiﬁcant issues with outliers in this problem. In practice, the recovery process for texture imaging transformations tends to be unreliable near boundaries, because elements are heavily foreshortened and so have lost some detail. As ﬁgure 3 indicates, there are occasional large gradient estimates at or near the boundary. It turns out to be a good idea to use a robust estimator in this problem. In particular, our log-likelihoods all use φ(x; %) = x/(x+%). We usually compose this function with a square, resulting in something proportional to the square for small argument and close to constant for large x. The data approximation problem: Even with a prior, we have a diﬃcult approximation problem. The data points are scattered and so it is natural to use radial basis functions. However, we have only the gradient of the depth, but, which is worse, we do not know the sign of the gradient. We could either ﬁt using only to p2 + q 2 — which yields a problem rather like shape from shading, which requires boundary information, which is often not available — or use the orientation of the gradient as well. To exploit the orientation of the gradient, we have two options. Option 1: The symmetric method We can ﬁt using only p2 and q 2 — in which case, we are ignoring information, because this data implies a four-fold ambiguity and we have only a two-fold ambiguity. A natural choice of ﬁtting error (or negative log-posterior, in modern language) is

2 1 2 2 2 2 2 − (z (x , y ; θ)) + q − (z (x , y ; θ)) ; %) + π(θ) φ( p x i i y i i i i 2σf2 i We call this the symmetric method because the negative log-posterior is invariant to the transformation (pi , qi ) → (−pi , −qi ) Option 2: The EM method The alternative is to approach the problem as a missing data problem, where the missing data is the sign of the gradient. It is natural to try and attack this problem with EM, too. The mechanics are straightforward. Write the depth function as z(x, y; θ), and the Gaussian curvature of the resulting surface as K(θ). The log-posterior is  

2 2 i − zx (xi , yi ; θ)] + [qi − zy (xi , yi ; θ)] ) (1 − δi )+ 1  φ([p

 + π(θ)

2 2 2σf2 i φ([pi + zx (xi , yi ; θ)] + [qi + zy (xi , yi ; θ)] ) (δi ) where δi (the missing variables) are one or zero according as the sign of the data is correct or not. The log-posterior is linear in the hidden variables, so the Q function is simple to compute, but the maximization must be done numerically,

236

D.A. Forsyth

Fig. 6. The reconstructed surface for the dress of ﬁgure 2 is shown in two diﬀerent textured views. In one view, the surface is superimposed on the image to give some notion of the relation between surface and image. Again, the surface appears to give a fair qualitative impression of the shape of the dress.

and is expensive because the Gaussian curvature must be integrated for each function evaluation (meaning that gradient evaluation is particularly expensive). Approximating functions: In the examples, we used a radial basis function approximation with basis functions placed at each data point. We therefore have z(x, y; θ) =

i

ai (x − xi )2 + (y − yi )2 + ν

where ν sets a scale for the approximating surfaces and ai are the parameters. The main disadvantage with this approach is that the number of basis elements — and therefore, the general diﬃculty of the ﬁtting problem — goes up with the number of texture elements. There are schemes for reducing the number of basis elements, but we have not experimented with their application to this problem.

Shape from Texture without Boundaries

237

Fig. 7. The texture on the surfaces of ﬁgure 6 implies that the surface can follow scale detail at a scale smaller than the elements on the dress; the ﬁgure on the left, which is a height map with lighter elements being closer to the eye, indicates that this is not the case — the reconstruction smooths the dress to about the scale of the inter-element spacing, as one would expect. On the right, texture remapping; because we know the texture imaging transformation for each detected element, we can remap the image with diﬀerent texture elements. Note that a reconstruction is not required, and the ambiguity in gradient need not be resolved. Here we have replaced spots with rosettes. Missing elements, etc. are due to the relatively crude element detection scheme.

4.1

Experimental Results

We implemented this method for orthographic views in Matlab on an 867Mhz Macintosh G4. We used a crude template matcher to identify texture elements In the examples where the texture element is a circular spot, the element was ﬁxed at a circular spot; for other examples, the element was estimated along with the texture imaging transformations. Determining texture imaging transformations for of the order of 60 elements takes of the order of tens of minutes. Reconstruction is achingly slow, taking of the order of hours for each of the examples shown; this is entirely because of the expense of computing the prior term, a computation that in principle is easily parallelised. Estimating transformations: Figure 1 shows input images with gradient orientations estimated from texture imaging transformations superimposed. Notice that there is no arrow-head on these gradient orientations, because we don’t know which direction the gradient is pointing. Typically, elements at or near the rim lead to poor estimates which are discarded by the EM — a poor transformation estimate leads to a rectiﬁed element that doesn’t agree with most others, and so causes the probability that the image element is not an instance of the model element to rise. Reconstructions compared with ground truth: In ﬁgure 4, we compare a series of runs of our process under diﬀerent conditions. In each case, we used

238

D.A. Forsyth

synthetic images of a textured sphere, so that we could compute the root mean square error of the radius of the reconstructed surface. We did ﬁve runs each of four cases — estimated (square) texture element vs. known circular texture element and symmetric reconstruction vs EM reconstruction. In each run of the known (resp. unknown) texture element case, we used the same image for symmetric vs EM reconstruction, so that we could check the quality of the gradient information. In all but three of the 20 runs, the RMS error is about 10% of radius. This suggests that reconstruction is successful. In the other three runs, the RMS error is large, probably due to a local minimum. All three runs are from the symmetric reconstruction algorithm applied to an estimated element. We know that the gradient recovery is not at fault in these cases, because the EM reconstruction algorithm recovered a good ﬁt in these cases. This means that estimating the texture element is not signiﬁcantly worse than knowing it. Furthermore, it suggests that the reconstruction algorithms are roughly equivalent. Reconstructions from Images of Real Scenes: Figures 5 and 6 show surfaces recovered from images of real textured scenes. In these cases, lacking ground truth, we can only argue qualitatively, but the reconstructed surface appears to have a satisfactory structure. Typically, mapping the texture back onto the surface makes the reconstruction look as though it has ﬁne-scale structure. This eﬀect is most notable in the case of the dress, where the reconstructed surface looks as though it has the narrow folds typical of cloth; this is an illusion caused by the shading, as ﬁgure 7 illustrates. Texture Remapping: One amusing application is that, knowing the texture imaging transformation and an estimate of shading, we can remap textures, replacing the model element in the image with some other element. This does not require a surface estimate. Quite satisfactory results are obtainable (ﬁgure 7).

5

Comments

Applications: Shape from texture has tended to be a core vision problem — i.e. interesting to people who care about vision, but without immediate practical application. It has one potential application in image based rendering — shape from texture appears to be the method with the most practical potential for recovering detailed deformation estimates for moving, deformable surfaces such as clothing and skin. This is because no point correspondence is required for a reconstruction, meaning that shape estimates are available from spotty surfaces relatively cheaply — these estimates can then be used to condition point tracking, etc., as in [14]. However, shape from texture has the potential advantage over Torresani et al.’s method that it does not require feature correspondences or constrained deformation models. SFT=SFM: There is an analogy between shape from texture and structure from motion that appears in the literature — see, for example, [5,9,10,11,13] — but it hasn’t received the attention it deserves. In essence, shape from texture is about one view of multiple instances of a pattern, and structure from motion is (currently) about multiple views of one instance of a set of points. Lemma

Shape from Texture without Boundaries

239

2 is, essentially, a structure from motion result, and if it isn’t known, this is because it treats cases that haven’t arisen much in practice in that domain. However, the analogy has the great virtue that it oﬀers attacks on problems that are currently inaccessible from within either domain. For example, one might consider attempting to reconstruct a textured surface which does not contain repeated elements by having several views; lemma 2 then applies in structure from motion mode, yielding estimates of each texture element; these, in turn, yield normals, and a surface results. Acknowledgments. Much of the material in this paper is in response to several extremely helpful and stimulating conversations with Andrew Zisserman.

References 1. Y. Aloimonos. Detection of surface orientation from texture. i. the case of planes. In IEEE Conf. on Computer Vision and Pattern Recognition, pages 584–593, 1986. 2. A. Blake and C. Marinos. Shape from texture: estimation, isotropy and moments. Artiﬁcial Intelligence, 45(3):323–80, 1990. 3. M. Clerc and S. Mallat. Shape from texture through deformations. In Int. Conf. on Computer Vision, pages 405–410, 1999. 4. D.A. Forsyth. Shape from texture and integrability. In Int. Conf. on Computer Vision, pages 447–452, 2001. 5. J. Garding. Shape from texture for smooth curved surfaces. In European Conference on Computer Vision, pages 630–8, 1992. 6. J. Garding. Surface orientation and curvature from diﬀerential texture distortion. In Int. Conf. on Computer Vision, pages 733–9, 1995. 7. T. Leung and J. Malik. Detecting, localizing and grouping repeated scene elements from an image. In European Conference on Computer Vision, pages 546–555, 1996. 8. J. Malik, S. Belongie, J. Shi, and T. Leung. Textons, contours and regions: cue integration in image segmentation. In Int. Conf. on Computer Vision, pages 918– 925, 1999. 9. J. Malik and R. Rosenholtz. Computing local surface orientation and shape from texture for curved surfaces. Int. J. Computer Vision, pages 149–168, 1997. 10. J.L. Mundy and A. Zisserman. Repeated structures: image correspondence constraints and 3d structure recovery. In J.L. Mundy, A. Zisserman, and D.A. Forsyth, editors, Applications of invariance in computer vision, pages 89–107, 1994. 11. R. Rosenholtz and J. Malik. Surface orientation from texture: isotropy or homogeneity (or both)? Vision Research, 37(16):2283–2293, 1997. 12. F. Schaﬀalitzky and A. Zisserman. Geometric grouping of repeated elements within images. In D.A. Forsyth, J.L. Mundy, V. diGesu, and R. Cipolla, editors, Shape, contour and grouping in computer vision, pages 165–181, 1999. 13. T.Leung and J. Malik. Detecting, localizing and grouping repeated scene elements from an image. In European Conference on Computer Vision, pages 546–555, 1996. 14. L. Torresani, D. Yang, G. Alexander, and C. Bregler. Tracking and modelling non-rigid objects with rank constraints. In IEEE Conf. on Computer Vision and Pattern Recognition, 2001. to appear. 15. A.P. Witkin. Recovering surface shape and orientation from texture. Artiﬁcial Intelligence, 17:17–45, 1981.

Statistical Modeling of Texture Sketch Ying Nian Wu1 , Song Chun Zhu2 , and Cheng-en Guo2 1

2

Dept. of Statistics, Univ. of California, Los Angeles, CA 90095, USA [email protected] Dept. of Comp. and Info. Sci., Ohio State Univ., Columbus, OH 43210, USA {szhu, cguo}@cis.ohio-state.edu

Abstract. Recent results on sparse coding and independent component analysis suggest that human vision ﬁrst represents a visual image by a linear superposition of a relatively small number of localized, elongate, oriented image bases. With this representation, the sketch of an image consists of the locations, orientations, and elongations of the image bases, and the sketch can be visually illustrated by depicting each image base by a linelet of the same length and orientation. Built on the insight of sparse and independent component analysis, we propose a two-level generative model for textures. At the bottom-level, the texture image is represented by a linear superposition of image bases. At the top-level, a Markov model is assumed for the placement of the image bases or the sketch, and the model is characterized by a set of simple geometrical feature statistics. Keywords: Independent component analysis; Matching pursuit; Minimax entropy learning; Sparse coding; Texture modeling.

1

Introduction

As argued by Mumford (1996) and many other researchers, the problem of vision can be posted in the framework of statistical modeling and inferential computing. That is, top-down generative models can be constructed to represent a visual system’s knowledge in the form of probability distributions of the observed images as well as variables that describe the visual world, then visual learning and perception become a statistical inference (and model selection) problem that can be solved in principle by computing the likelihood or posterior distribution. To guide reliable inference, the generative models should be realistic, and this can be checked by visually examining random samples generated by the models. Recently, there has been some progress on modeling textures. Most of the recent models involve linear ﬁlters for extracting local image features, and the texture patterns are characterized by statistics of local features. In particular, inspired by the work of Heeger and Bergen (1996), Zhu, Wu, and Mumford (1997) and Wu, Zhu, and Liu (2001) developed a self-consistent statistical theory for texture modeling; borrowing results from statistical mechanics, they showed that a class of Markov random ﬁeld models is a natural choice under the assumption A. Heyden et al. (Eds.): ECCV 2002, LNCS 2352, pp. 240–254, 2002. c Springer-Verlag Berlin Heidelberg 2002

Statistical Modeling of Texture Sketch

241

that texture impressions are decided by histograms of ﬁlter responses. Meanwhile, Portilla and Simoncelli (2000), using steerable pyramid, made a thorough investigation of feature statistics for representing texture patterns. These models appear to be very successful in capturing stochastic textures, but seem to be less eﬀective for textures with salient local structures and regular spatial arrangements. Meanwhile, there have been major developments in sparse coding (Olshausen and Field, 1996) and independent component analysis (Bell and Sejnowski, 1997). These methods represent natural images (actually image patches) by linear superposition of image bases. By asking for sparseness or independence of the coeﬃcients in the linear decomposition, localized, elongate and oriented image bases can be learned from natural images. The learned bases constitute a general and eﬃcient vocabulary for image representation. The idea of sparseness has also been studied in wavelet community (e.g., Mallat and Zhang, 1993; Candes and Donoho, 2000). This inspired us to re-consider the role of linear ﬁlters in texture models. The ﬁlter responses are inner products between the image and localized image bases, and this is a bottom-up operation. However, generative models are for top-down representation, and therefore it appears to be more appropriate for the images bases to play a representational role as in sparse and independent component analysis. This consideration promoted us to construct texture models based on linear representation instead of linear operation. The resulting model is a twolevel top-down model. At the bottom level, the texture image is represented by linear superposition of image bases, and at the top level, the spatial arrangements of image bases and their coeﬃcients are to be modeled. The sparse and independent component analysis suggests that with well-designed image bases, the coeﬃcients should become much sparser and less dependent than the raw image intensities, and thus more susceptible to modeling. Guided by the top-down model, local structures and their organizations can be more sharply captured, because variable selection or explaining-away eﬀect can be easily achieved in model-based context. To be more speciﬁc, the spatial model for the image bases and their coefﬁcients incorporate sparseness in the sense that only a small number of image bases have non-zero coeﬃcients or are active, so that the locations, orientations, and elongations of the active image bases capture the geometrical aspect of the texture image, or they tell us how to sketch the image, while the coeﬃcients of these active bases capture the photometrical aspect of the image, or they tell us how to paint the image given the sketch. To visually illustrate the sketch, we can depict each active image base by a small line segment or a linelet of the same location, orientation, and length, as ﬁrst proposed by Bergeaud and Mallat (1996). The sparse decomposition in general achieves about 100 folds of dimension reduction. The two-level top-down model is a hidden Markov model, with the coeﬃcients of image bases being latent variables or missing data. While rigorous model ﬁtting typically involves EM-type algorithms (Dempster, Laird, and Rubin, 1977),

242

Y.N. Wu, S.C. Zhu, and C.-e. Guo

in this article, we shall isolate the problem of modeling the sketch of the texture images, while assuming that the sketch can be obtained from the image. Our sketch model is a causal Markov chain model whose conditional distributions are characterized by a set of simple geometrical feature statistics automatically selected from a pre-deﬁned vocabulary. Embellished versions of our model can be useful in the following regards. In computer vision, it can be used for image segmentation and perceptual grouping. In computer graphics, as the sketch captures the geometrical essence of images, it may be used for non-photo realistic rendering. For understanding human vision, it provides a model-based representational theory for Marr’s primal sketch (Marr 1982).

2 2.1

Sketching the Image Sparse Coding and Independent Component Analysis

The essential idea of sparse coding (Olshausen and Field, 1996), independent component analysis (Bell and Sejnowski, 1999), and their combination (Lewiki and Olshausen, 1999) is the assumption that an image I can be represented as the superposition of a set of image bases. The bases are selected from an overcomplete basis (vocabulary) {b(x,y,l,θ,e) }, where (x, y) is the central position of the base on the image domain, and (l, θ, e) is the type of the base. l is the scale or length of the base, θ the orientation, and e the indicator for even/odd bases. The DC components of the bases are 0, and the l2 norm of the bases are 1. Therefore,

c(x,y,l,θ,e) b(x,y,l,θ,e) + N (0, σ 2 ),

(1)

c(x,y,l,θ,e) ∼ p(c) = ρδ0 + (1 − ρ)N (0, τ 2 ), independently.

(2)

I=

(x,y,l,θ,e)

The base coeﬃcients c(x,y,l,θ,e) is assumed to be independently distributed according to a distribution p(c), which is the mixture of a point mass at 0 (then the base is said to be inactive) and a Gaussian distribution with a large variance (for active state). In general, the basis (vocabulary) can be learned from natural images by minimizing the number of active bases (i.e. sparse coding) or by maximizing a measure of independence between the coeﬃcients. In our experiments, we select a set of bases for vocabulary b(x,y,l,θ,e) , as shown in Figure 1. We use 128 diﬀerence of oﬀset Gaussian (DOOG) ﬁlters (Malik and Perona, 1989) at various scales and orientations, even and odd. There are 5 scales, and we adopt a curvelet design of Candes and Donoho (2000), that is, for smaller scales, the bases become more elongate and there are more orientations. We also use the center-surround Diﬀerence of Gaussian (DOG) ﬁlters at 8 scales to account for variations that cannot be eﬃciently captured by the DOOG bases.

Statistical Modeling of Texture Sketch

243

Fig. 1. Image bases used in representation and modeling.

2.2

Matching Pursuit and Its MCMC Version

Since the bases are over-completed, the coeﬃcients have to be inferred by sampling a posterior probability. In this section, we extend the heuristic matching pursuit algorithm of Mallat and Zhang (1993) to a more principled Markov chain Monte Carlo (MCMC) algorithm. We sample the posterior distribution of the coeﬃcients p({c(x,y,l,θ,e) } | I) in Model (1) in order to ﬁnd a symbolic representation or a sketch of image I. We assume the parameters (ρ, τ 2 , σ 2 ) are known for this model. First, let’s consider the Gibbs sampler for posterior sampling. For simplicity, we use j or i to index (x, y, l, θ, e) and we deﬁne zj = 1 if bj is active, i.e., cj = 0, andzj = 0 otherwise. Then the algorithm is as follows: 1. Randomly select a base bj . Compute R = I − i=j ci bi , i.e., the residual image. Let rj =< R, bj > /(1 + σ 2 /τ 2 ), and σ 2 = 1/(1/σ 2 + 1/τ 2 ). 2. Compute the Bayes factor by integrating out cj , γj =

rj2 p(I | zj = 1; {ci , ∀i =j}, I) = exp{ 2 } σ 2 /τ 2 . p(I | zj = 0; {ci , ∀i =j}, I) 2σ

244

Y.N. Wu, S.C. Zhu, and C.-e. Guo

Then the posterior probability p(zj = 1 | {ci , ∀i =j}, I) =

(1 − ρ)γj . ρ + (1 − ρ)γj

3. Sample zj according to the above probability. If zj = 0, then set cj = 0, otherwise, sample cj ∼ N (rj , σ 2 ). Go back to [1]. The problem with this Gibbs sampler is that if σ 2 is small, the algorithm is too willing to activate the base bj even though the response rj is not that large, or in other words, the algorithm is too willing to jump into a local energy minimum. The idea of matching pursuit of Mallat and Zhang (1993) can be used to remedy this problem. That is, instead of randomly selecting a base bj to update, we randomly choose a window W on the image domain, and look at all the inactive bases within this window W . Then we sample one base by letting these bases compete for an entry. So we have the following windowed-Gibbs sampler or a Metropolized matching pursuit algorithm: 1. Randomly select a window W on the image domain, let A be the number of active bases within W , and let B be the number of inactive bases within W . With probability pbirth (a pre-designed number), go to [2]. With probability pdeath = 1 − pbirth , go to [4]. 2. For each inactive base j = (x, y, l, θ, e) with (x, y) ∈ W and zj = 0, compute γj as described in [1] and [2] of the Gibbs sampler. Then with probability pdeath j:zj =0; (x,y)∈W (1 − ρ)γj /ρ paccept = , pbirth × (A + 1) go to [3], with probability 1 − paccept go back to [1]. 3. Among all the inactive bases j with (x, y) ∈ W and zj = 0, sample a base j with probability proportional to γj , then let zj = 1 and sample cj as described in [3] of the Gibbs sampler. Go back to [1]. 4. If A > 0, then randomly select an active base bk with zk = 1 and (x, y) ∈ W . Then temporarily turn oﬀ bk , i.e., set ck = 0, and A → A − 1 temporarily. Then for all the inactive bases j with zj = 0 and (x, y) ∈ W , including base k, do the same computation as [2] (as if ck = 0), and compute paccept as in [2]. 5. With probability 1/paccept , accept the proposal of deleting the base k, i.e., set ck = 0. Go back to [1]. With probability 1 − 1/paccept , reject the proposal of deleting base k, i.e., recover the original ck . Go back to [1]. This is a Metropolis algorithm with two types of proposals, adding an inactive base and deleting an active base, and this is a birth and death move. In addition to that, we can easily incorporate updating schemes, including 1) perturbing the coeﬃcient of an active base, i.e., updating cj ; 2) moving an active base to a diﬀerent position, i.e., updating (x, y); 3) shortening or stretching an active base by changing its length l; 4) rotating an active base to a diﬀerent orientation, i.e., updating θ.

Statistical Modeling of Texture Sketch

245

For the simple model (1), if we let σ 2 → 0, then the Metropolis algorithm described above goes to the windowed version of the matching pursuit algorithm. In this paper, we shall just use the latter algorithm for simplicity. We feel that the MCMC version of matching pursuit is useful in two aspects. Conceptually, it helps us to understand matching pursuit as a limiting MCMC for posterior sampling. Practically, we believe that the MCMC version, especially with the moves for updating coeﬃcients, positions, lengths, orientations of the active bases, is useful for us to re-estimate the image sketch after we ﬁt a better prior model for sketch.

a

b

c

d

Fig. 2. Sparse decomposition by overcomplete basis and the symbolic sketch.

Experiments I. Figure 2 shows some results of sparse decomposition with overcomplete basis, and more results are shown in Figures 3 and 4. Figure 2.a are observed images, Figure 2.b are reconstructed images with a small number of bases whose coeﬃcients are larger than 5, |cj | > 5, the rate of compression is usually about 100 to 150 folds, e.g., for a 200× 200 image, we only need about 300 bases. Figure 2.c is a ﬁner reconstruction with all bases whose |cj | > 2, the rate of compression is about 30. For each selected base b(x,y,l,θ,e) , we illustrate it symbolically by a linelet of the same length l and orientation θ, while ignoring the brightness c(x,y,l,θ,e) and the odd/even indicator e. All the linelets are of the same width. Figure 2.d shows the sketches of the images.

3

Modeling the Image Sketch

Now we shall improve the simple prior model (1) by a sophisticated sketch model that accounts for the spatial arrangements of image bases.

246

Y.N. Wu, S.C. Zhu, and C.-e. Guo

We may use the following two representations interchangeably for the sketch of a texture image I, and let’s denote the sketch by S. 1. A list: let n be the number of active bases, then we have S = {st = (xt , yt , ct , lt , θt , et ), t = 1, ..., n}. 2. A bit-map: let δx,y be the indicator of whether there is an active base at (x, y) or not, then S = {sx,y = (δx,y , cx,y , lx,y , θx,y , ex,y ), ∀(x, y)}. If δx,y = 0, i.e., there is no active base, then all the (c, l, θ, e) take null values. Using the ﬁrst representation, the two-level top-down model is of the following form: S = {st = (xt , yt , ct , lt , θt , et ), t = 1, ..., n} ∼ sketch model(Λ), ct bxt ,yt ,lt ,θt ,et + . I= st ∈S

Clearly, the sparse and independent component analysis is a special case of the above model, where at the top level, (xt , yt ) follow Poisson point process, (lt , θt , et ) follow independent uniform distributions, and ct follows a normal distribution with a large variance. The sketch S is the latent variable or missing data, and Λ is the unknown parameter. So the overall model should be ﬁtted by the EM algorithm or its stochastic versions. The general form of such a model ﬁtting procedure is to iterate the following two steps. 1. Scene reconstruction: Estimate S from I, conditional on Λ. 2. Scene understanding: Estimate Λ from S. In our current work, we isolate the problem of modeling S, assuming that S can be inferred from I by matching pursuit algorithm and its MCMC version with the simple sparse and independent prior model on S. To further simplify the problem, we ignore c and e, i.e., the photometrical aspects, and only concentrate on the modeling of (l, θ), i.e., the geometrical aspects of the sketch S. 3.1

Previous Models on Textures

1. MRF and FRAME models. The FRAME model by Zhu, et al. (1997) incorporates the idea of large ﬁlters and histogram[10] into Markov random ﬁelds. p(I | Λ) =

1 exp{ λl,θ,e (< I, b(x,y,l,θ,e) >)}, Z(Λ) x,y

(3)

l,θ,e

where Λ = {λl,θ,e ( ), ∀(l, θ, e)} is a collection of one-dimensional functions, and Z(Λ) is the normalizing constant. This model is of the Gibbs form. The λ functions can be quantitized into step functions. This model is derived and justiﬁed

Statistical Modeling of Texture Sketch

247

as the maximum entropy distribution that reproduces the marginal distributions (or histograms in the quantitized case) of < Iobs , b(x,y,l,θ,e) > where Iobs is the observed image. If we compare model (3) with model (1), we can see that in FRAME model (3), the b(x,y,l,θ,e) are used for bottom-up operation, and the model is based on features < I, b(x,y,l,θ,e) >. In contrast, in model (1), the b(x,y,l,θ,e) are used for top-down representation, and the model is based on latent variables c(x,y,l,θ,e) . We believe that the feature-based models and latent-variable models are two major classes of models, and the former can be modiﬁed into the latter if we replace features by latent variables. Of course, this is in general not a trivial step. As to the spatial model for point process S = {st = (xt , yt , lt , θt ), t = 1, ..., n}, Guo, et al. (2001) proposed a Gestalt model of the Gibbs form p(S | Λ) =

1 λ1 (st ) + λ2 (st1 , st2 )}, exp{λ0 n + Z(Λ) t t ∼t 1

(4)

2

where t1 ∼ t2 means that st1 and st2 are neighbors. So this model is a pairpotential Gibbs point process model (e.g., Stoyan, Kendall, Mecke, 1985). Guo et al. (2000) further parameterized the model by introducing a small set of Gestalt features (Koﬀka, 1935) for spatial arrangement. Again, the model can be justiﬁed by the maximum entropy principle. The Gibbs models (3) and (4) are analytically intractable because of the intractability of the normalizing constant Z(Λ). Sampling and maximum likelihood estimation (MLE) have to be done by MCMC algorithms. 2. Causal Markov models. One way to get around this diﬃculty is to use a causal Markov model. The causal methods have been used extensively in early work on texture generation, most notably, the causal model of Popat and Picard (1993). In the causal model, the joint distribution of the image I is factorized into a sequence of conditional distribution by imposing a linear order on all the pixels (x, y), e.g., m n p(I) = p(Ix,y | IN (x,y) ), x=1 y=1

where x, y index the pixel, and N (x, y) is the neighboring pixels within a certain spatial distance that are scanned before (x, y) The causal model is analytically tractable because of the factorization form. The causal plan has been successfully incorporated in example-based texture synthesis by Efros and Leung (1999), Liang et al. (2001), Efros and Freeman (2001). Hundred of realistic textures have been synthesized. 3.2

Modeling the Sketch Patterns

With the above preparation, we are ready to describe our model for image sketch. Let S be the sketch of I, and let SN (x,y) be the sketch of the causal neighborhood

248

Y.N. Wu, S.C. Zhu, and C.-e. Guo

of (x, y). Recall that both S and SN (x,y) have two representations. Our model is of the following causal form p(S) =

m n

p(sx,y | SN (x,y) ).

x=1 y=1

The conditional distribution is

1 exp{λ0 δx,y + λ1 (lx,y , θx,y ) Z(Λ | SN (x,y) ) λ2 (lx,y , θx,y ; lt , θt ; xt − x, yt − y)}, +

p(sx,y | SN (x,y) ) =

st ∈SN (x,y)

where Z is the normalizing constant, and if lx,y and θx,y take on the null values when δx,y = 0, λ1 ( ) and λ2 ( ) are 0. If λ1 ( ) and λ2 ( ) are always 0, then the model reduces to the simple Poisson model (1). Like in the FRAME model (Zhu, et al. 1997) and the Gestalt model (Guo, et al. 2001), this conditional distribution can be derived or justiﬁed as the maximum entropy distribution that reproduces the probability that there exists a linelet, the distribution of the orientation and length of the linelet, and the pair-wise conﬁguration made up by this linelet and a nearby existing linelet. In this model, the probability that we sketch a linelet at (x, y) depends on the attributes of this linelet, and more important, how this linelet lines up with existing linelets nearby, for instance, whether the proposed linelet connects with a nearby existing linelet, or whether the proposed linelet is parallel with a nearly existing linelet, etc. One can envisage this conditional model as modeling the way an artist sketches a picture by adding one stroke at a time. Similar maximum entropy models are also used in language[3]. We can also write the conditional model in a log-additive form log

p(δx,y = 1, lx,y , θx,y | SN (x,y) ) = λ0 + λ1 (lx,y , θx,y ) p(δx,y = 0 | SN (x,y) ) λ2 (lx,y , θx,y ; lt , θt ; xt − x, yt − y). + st ∈SN (x,y)

One may argue that a causal model for spatial point process is very contrived. We agree. A causal order is physically nonsensical. But for the purpose of representing visual knowledge, it has some conceptual advantages because of its analytical tractability. The situation is very similar to view-based methods for objective recognition. Moreover, the model is also suitable for the purpose of coding and compression. Mathematically, one may view this model as a causal (or factorization) approximation to the Gestalt model of Guo et al. (2001). It is expected that the causal approximation loses some of the expressive power of the non-causal model, but this may be compensated by making the causal model more redundant. The current form of the model only accounts for pair-wise conﬁgurations of the linelets. We can easily extend the model to account for conﬁgurations that involves more than two linelets.

Statistical Modeling of Texture Sketch

3.3

249

Feature Statistics, Model Fitting, and Feature Pursuit

Because the length and orientation has been taken care of by λ1 (ls,y , θx,y ), we choose to parameterize pair-wise conﬁguration function λ2 ( ) as λ2 (D(sx,y , st ), A(sx,y , st )), for st ∈ SN (x,y) , where D( ) is the smallest distance between the two linelets, and A( ) the angle between the two linelets. Note that there is some loss of information by this parameterization, in particular, it cannot distinguish between two crossing linelets and two touching linelets if the angles are the same, because in both cases, the smallest distance is 0. But still, we would like to see how far we can go with this simple parameterization. We could further express λ1 (l, θ) = λ11 (l) + λ12 (θ) + λ13 (l, θ), i.e., decompose λ1 into the main eﬀects for l and θ respectively, and the interaction or combination between l and θ. This is the ANOVA (analysis of variance) decomposition, which is redundant unless we add some constraints. We do not need to worry about this issue because we use this ANOVA decomposition for the sake of feature selection to be described later, and the feature selection method will not select redundant feature statistics. Similarly, we can decompose λ2 ( ) by λ2 (D, A) = λ21 (D) + λ22 (A) + λ23 (D, A). After that we choose to quantize l, θ, D, and A, i.e., each of these variables can only take a ﬁnite set of values. Then the functions can be reduced to vectors of function values. For instance, for a function λ(x), if x ∈ {x1 , ..., xn }, we can represent λ( ) by λ = (λ1 , ..., λn ), so that λj = λ(xj ). Therefore λ(x) = n j=1 λj 1x=xj =< λ, H(x) >, where H(x) is the vector of indicators (1x=xj , j = 1, ..., n). Therefore, we can write the conditional distribution in our sketch model as p(sx,y | SN (x,y) ) =

1 exp{λ0 δx,y + < λ11 , H11 (lx,y ) > Z(Λ | SN (x,y) )

+ < λ12 , H12 (θx,y ) > + < λ13 , H13 (lx,y , θx,y ) > < λ21 , H21 (D(sx,y , st )) > + < λ22 , H22 (A(sx,y , st )) > + st ∈SN (x,y)

+ < λ23 , H23 (D(sx,y , st ), A(sx,y , st )) >} 1 = exp{< Λ, H(sx,y | SN (x,y) ) >}, Z(Λ | SN (x,y) )

where Λ = (λ0 , λ11 , λ12 , λ13 , λ21 , λ22 , λ23 ), and H(sx,y | SN (x,y) ) = (δx,y , H11 (lx,y ), H12 (θx,y ), H13 (lx,y , θx,y ), H21 (D(sx,y , st )), H22 (A(sx,y , st )), SN (x,y)

SN (x,y)

H23 (D(sx,y , st ), A(sx,y , st ))).

SN (x,y)

In our work, H(sx,y | SN (x,y) ) have one to two hundred components. It can be easily shown that p(sx,y | SN (x,y) ) achieves maximum entropy under the constraints on < H(sx,y | SN (x,y) ) >p(sx,y |SN (x,y) ) , where < >p means expectation with respect to distribution p. The full model is

250

Y.N. Wu, S.C. Zhu, and C.-e. Guo

p(S | Λ) =

n m

p(sx,y | SN (x,y) )

x=1 y=1 m n

={

m n 1 H(sx,y | SN (x,y) ) >}. } × exp{< Λ, Z(Λ | SN (x,y) ) x=1 y=1 x=1 y=1

Now, let’s consider model ﬁtting. Let Sobs be the observed sketch of an image. Then we can estimate Λ by maximizing the log-likelihood obs obs {< Λ, H(Sobs l(Λ | Sobs ) = x,y | SN (x,y) ) > − log Z(Λ | SN (x,y) )}. x,y

Statistical theories of exponential family models tell us that ∂ obs obs l(Λ | Sobs ) = {H(sobs ,Λ) }, x,y | SN (x,y) )− < H(sx,y | SN (x,y) ) >p(sx,y |Sobs N (x,y) ∂Λ x,y and

∂2 l(Λ | S ) = − {Varp(sx,y |Sobs ,Λ) [H(sx,y | Sobs obs N (x,y) )]}. N (x,y) ∂Λ2 x,y

Therefore, the log-likelihood is concave, and the model can be ﬁtted by the Newton-Raphson or equivalently in this case, the Fisher scoring algorithm, Λt+1 = Λt − [

∂ ∂2 l(Λ | Sobs ) |Λt . l(Λ | Sobs )] |−1 Λt 2 ∂Λ ∂Λ

The convergence of Newton-Raphson is very fast; usually 5 iterations can already give very good ﬁt. Both the ﬁrst and second derivatives of the log-likelihood are of the form x,y g(x, y). For each pixel (x, y), we need to evaluate the probabilities of all possible sx,y . So the computation is still quite costly, although much more manageable compared to MCMC type of algorithms. To further increase the eﬃciency, we choose to sample a small number of pixels instead of going through all of them. More speciﬁcally, for each (x, y), let πx,y ∈ [0, 1] be the probability that pixel (x, y) will be included in the sample. Then after we collect a sample of (x, y) by independent coin-ﬂipping according to πx,y , we can approximate x,y

g(x, y) ≈

g(x, y)/πx,y ,

(x,y)∈sample

where the right hand side is the Hovitz-Thompson unbiased estimator of the left hand side. As to the choice of πx,y , if there is a linelet at (x, y) on Sobs , then we always let πx,y = 1. For other empty pixels (x, y), we can set πx,y according to our need for speed. Usually, even if we take πx,y = .01 for empty pixels, the algorithm can still give satisfactory ﬁt.

Statistical Modeling of Texture Sketch

251

obs It is often the case that some components of x,y H(sobs x,y | SN (x,y) ) are 0, and if we implement the usual Newton-Raphson procedure, then the corresponding components of Λ will go to −∞, thereby creating hard constraints. While this is theoretically the right result, it can make the algorithm unstable. We choose to components as long as the corresponding components of stop ﬁtting such obs < H(s | S x,y ) drop below a threshold, e.g., .5. x,y N (x,y) ) >p(sx,y |Sobs N (x,y) For a speciﬁc observed image, we do not want to use all the one to two hundred dimensions in our model. In fact, we can just select a small number of components of H(sx,y | SN (x,y) ) using some model selection methods such as Akaike information criterion (AIC). While best-set selection is too time-consuming, we can consider a feature pursuit scheme studied by Zhu, et al. (1997), i.e., we start with only λ0 and δx,y in our model. Then we repeatedly add one component of H at a time, so that the added component leads to the maximum increase in log-likelihood. Although the log-likelihood is analytically tractable for the causal model, the computation of the increase of log-likelihood for each candidate component of H is still quite costly. So as an approximation, we choose the component Hk so that obs obs 2 { x,y Hk (sobs x,y | SN (x,y) )− < Hk (sx,y | SN (x,y) ) >pˆ} gk = , obs x,y Varpˆ[Hk (sx,y | SN (x,y) )] is the largest, where pˆ is the currently ﬁtted model. Intuitively, we choose a component that is worst ﬁtted by the current model. Then we stop after a number of steps. Again, the Horvitz-Thompson estimator can be used for faster computation of gk . Experiments II. Figures 3 and 4 show some experiments. For each input image (left), we ﬁrst compute its sketch (middle), and then a sketch model is learned with 40-60 feature statistics selected by feature pursuit. The angles as well as orientations are quantitized into 12 bins, and the distances are quantitized into 12 bins too. Of course, the model fails to capture some features that human vision is sensitive, which is expected of a pair-wise model. More sophisticated features should be used, and we leave this to future work.

4

Discussion

There are two major loose ends in our work. One is that the coeﬃcients of the active bases are not modeled. The other is that the model ﬁtting is not rigorous. The model can be further extended to incorporate more sophisticated local structures, such as local shapes and lighting, as well as more sophisticated organizations such as ﬂows and graphs. The key is that these concepts should be understood in the context of a top-down generative model. For some stochastic textures, sparse decomposition may not be achievable, and therefore, we might have to stay with models built on pixel values such as the FRAME model. We would like to stress that our goal in this work is to ﬁnd a top-down generative model for textures. We are not merely trying to come up with a linedrawing version of the texture image by some edge detector, and then synthesize

252

Y.N. Wu, S.C. Zhu, and C.-e. Guo

a). input

b). image sketch

c). a sketch synthesized.

Fig. 3. Examples of learning sketch model. a) is an input image, b) is its computed sketch, and c) is the synthesized sketch as a random sample from the sketch model learned from b).

similar line-drawings. We would also like to point out that our work is inspired by Marr’s primal sketch (Marr, 1982). Marr’s method is bottom-up procedurebased, that is, there does not exist a top-down model to guide the bottom-up procedure. Our eventual goal is to ﬁnd the top-down model as a visual system’s conception of primal sketch, so that the largely bottom-up procedure will be modelbased. The hope is that the model is uniﬁed and explicit like a language, with rich vocabularies for local image structures as well as their spatial organizations. When ﬁtted to a particular image, an automatic model selection procedure will identify a low-dimensional sub-model as the most meaningful words. The model should lie between physics-based models (that are not explicit and uniﬁed) and

Statistical Modeling of Texture Sketch

a). input

b). image sketch

253

c). a sketch synthesized.

Fig. 4. Examples of learning sketch model. a) is an input image, b) is its computed sketch, and c) is the synthesized sketch as a random sample from the sketch model learned from b).

image-based synthesis (that does not involve dimension reduction). Needless to say, our current eﬀort is merely a modest ﬁrst step towards this goal.

254

Y.N. Wu, S.C. Zhu, and C.-e. Guo

References 1. A. J. Bell and T.J. Sejnowski, “An information maximization approach to blind separation and blind deconvolution”, Neural Computation, 7(6): 1129-1159, 1995. 2. F. Bergeaud and S. Mallat, “Matching pursuit: adaptive representation of images and sounds.” Comp. Appl. Math., 15, 97-109. 1996. 3. A. Berger, V. Della Pietra, and S. Della Pietra, “A maximum entropy approach to natural language processing”, Computational Linguistics, vol.22, no. 1 1996. 4. E. J. Cand`es and D. L. Donoho, “Curvelets - A Surprisingly Eﬀective Nonadaptive Representation for Objects with Edges”, Curves and Surfaces, L. L. Schumaker et al. (eds), Vanderbilt University Press, Nashville, TN. 5. A. P. Dempster, N.M. Laird, and D. B. Rubin, “Maximum likelihood from incomplete data via the EM algorithm”, Journal of the Royal Statistical Society series B, 39:1-38, 1977. 6. A. A. Efros and T. Leung, “Texture synthesis by non-parametric sampling”, ICCV, Corfu, Greece, 1999. 7. A. A. Efros and W. T. Freeman, “Image Quilting for Texture Synthesis and Transfer”, SIGGRAPH 2001. 8. S. Geman and D. Geman. “Stochastic relaxation, Gibbs distributions and the Bayesian restoration of images”. IEE Trans. PAMI 6. pp 721-741. 1984. 9. C. E. Guo, S. C. Zhu, and Y. N. Wu, “Visual learning by integrating descriptive and generative methods”, ICCV, Vancouver, CA, July, 2001. 10. D. J. Heeger and J. R. Bergen, “Pyramid-based texture analysis/synthesis”, SIGGRAPHS, 1995. 11. M. S. Lewicki and B. A. Olshausen, “A probabilistic framework for the adaptation and comparison of image codes”, JOSA, A. 16(7): 1587-1601, 1999. 12. L. Liang, C. Liu, Y. Xu, B.N. Guo, H.Y. Shum, “Real-Time Texture Synthesis By Patch-Based Sampling”, MSR-TR-2001-40, March 2001. 13. J. Malik, and P. Perona, “Preattentive texture discrimination with early vision mechanisms”, J. of Optical Society of America A, vol 7. no.5, May, 1990. 14. S. G. Mallat, “A theory for multiresolution signal decomposition: the wavelet representation”, IEEE Trans. on PAMI, vol.11, no.7, 674-693, 1989. 15. S. Mallat and Z. Zhang, “Matching pursuit in a time-frequency dictionary”, IEEE trans. on Signal Processing, vol.41, pp3397-3415, 1993. 16. D. Marr, Vision, W.H. Freeman and Company, 1982. 17. D. B. Mumford “The Statistical Description of Visual Signals” in ICIAM 95, edited by K.Kirchgassner, O.Mahrenholtz and R.Mennicken, Akademie Verlag, 1996. 18. B. A. Olshausen and D. J. Field, “Emergence of simple-cell receptive ﬁeld properties by learning a sparse code for natural images” Nature, 381, 607-609, 1996. 19. K. Popat and R. W. Picard, “Novel Cluster-Based Probability Model for Texture Synthesis, Classiﬁcation, and Compression.” Proc. of the SPIE Visual Comm. and Image Proc., Boston, MA, pp. 756-768, 1993. 20. J. Portilla and E. P. Simoncelli, “A parametric texture model based on joint statistics of complex wavelet coeﬃcients”, IJCV, 40(1), 2000. 21. Y. Wu, S. C. Zhu, and X. Liu, (2000)“Equivalence of Julesz texture ensembles and FRAME models”, IJCV, 38(3), 247-265. 22. S. C. Zhu, Y. N. Wu and D. B. Mumford, “Minimax entropy principle and its application to texture modeling”, Neural Computation Vol. 9, no 8, Nov. 1997.

Classifying Images of Materials: Achieving Viewpoint and Illumination Independence Manik Varma and Andrew Zisserman Robotics Research Group Department of Engineering Science University of Oxford Oxford, OX1 3PJ {manik,az}@robots.ox.ac.uk

Abstract. In this paper we present a new approach to material classiﬁcation under unknown viewpoint and illumination. Our texture model is based on the statistical distribution of clustered ﬁlter responses. However, unlike previous 3D texton representations, we use rotationally invariant ﬁlters and cluster in an extremely low dimensional space. Having built a texton dictionary, we present a novel method of classifying a single image without requiring any a priori knowledge about the viewing or illumination conditions under which it was photographed. We argue that using rotationally invariant ﬁlters while clustering in such a low dimensional space improves classiﬁcation performance and demonstrate this claim with results on all 61 textures in the Columbia-Utrecht database. We then proceed to show how texture models can be further extended by compensating for viewpoint changes using weak isotropy. The new clustering and classiﬁcation methods are compared to those of Leung and Malik (ICCV 1999), Schmid (CVPR 2001) and Cula and Dana (CVPR 2001), which are the current state-of-the-art approaches.

1

Introduction

The objective of this paper is the classiﬁcation of materials from their imaged appearance. Texture is a material property of which we have only an intuitive, and not a sound mathematical, understanding. Therefore, classifying materials by their textural appearance in single images photographed under unknown viewing and illumination conditions is still quite an outstanding problem, though signiﬁcant progress has been made recently [2,3,9,12,13,17,20]. In particular, Leung and Malik [13] made an important innovation in giving an operational deﬁnition of a texton. They deﬁned a 2D texton as a cluster centre in ﬁlter response space. This not only enabled textons to be generated automatically from an image, but also opened up the possibility of a universal set of textons for all images. In the same paper Leung and Malik also made a serious attempt on the problem of classifying textures under varying viewpoint and illumination. Their solution was a 3D texton which is a cluster centre of ﬁlter responses over a stack of images with representative viewpoints and lighting. A. Heyden et al. (Eds.): ECCV 2002, LNCS 2352, pp. 255–271, 2002. c Springer-Verlag Berlin Heidelberg 2002

256

M. Varma and A. Zisserman

Fig. 1. The change in image appearance of the same texture (# 30, Plaster B) with variation in imaging conditions. Top row: constant viewing angle and varying illumination. Bottom row: constant illumination and varying viewing angle. There is a considerable diﬀerence in the appearance across images.

The need to seriously address “3D eﬀects” – where the appearance varies considerably with viewpoint and lighting – is illustrated in ﬁgure 1. The importance of such eﬀects for classiﬁcation [1,4,5] and synthesis [15,19] has also been noted by other researchers. Our approach to the classiﬁcation problem is to model a texture as a distribution over textons, and learn the textons and texture models from training images. Classiﬁcation of a novel image then proceeds by mapping the image to a texton distribution and comparing this distribution to the learnt models. Consequently this is quite a standard approach, but the originality is at two points: First, our texton clustering is in an extremely low dimensional space and is also rotationally invariant. The second innovation is to classify the texture from a single image, and we achieve this while representing each 3D texture by a small set of models. This is a considerable diﬀerence in approach to that of [13] to which we compare our results, and it is worth elaborating on the diﬀerence at this point. The main thrust of Leung and Malik’s work is towards classifying images using 3D textons. In the learning stage, 20 images of each texture are geometrically registered and mapped to a 48 dimensional ﬁlter response space. The registration is necessary because the clustering that deﬁnes the texton is in the stacked 20 × 48 = 960 dimensional space (i.e. the textons are 960-vectors), and it is important that each ﬁlter is applied to the same texture point as viewpoint and illumination vary. In the classiﬁcation stage, 20 novel images of the same texture are presented. However, again these images must be registered and more signiﬁcantly must have the same order as the original 20 (i.e. they must be taken from images with similar viewpoint and illumination to the original). In essence, the viewpoint and lighting are being supplied implicitly by this ordering. Leung and Malik also use an MCMC algorithm for classifying a single image under known imaging conditions. However, the classiﬁcation rate of this method is 87%, far inferior to the 95.6% achieved by the multiple image method.

Classifying Images of Materials

257

Fig. 2. Textures from the Columbia-Utrecht database. All images are converted to monochrome in this work, so colour is not used in discriminating diﬀerent textures.

Here we present a texture classiﬁcation method with superior performance to [13], but requiring only a single image as input and with no information (implicit or explicit) about the illumination and viewing conditions. Our approach is most closely related to that of Schmid [18] and Cula and Dana [2,3]. Schmid’s approach is rotationally invariant but the invariance is achieved in a diﬀerent manner and texton clustering is in a higher dimensional space than here. Cula and Dana’s approach is not rotationally invariant and the method of model selection diﬀers from here. The paper is organized as follows: section 2 is concerned with rotationally invariant clustering and the classiﬁcation rates of four ﬁlter sets are compared. The sets include those used by Schmid [18] and Leung and Malik [13]. Then, in section 3, we describe how the number of models for each texture can be substantially reduced. We present results on the Columbia-Utrecht database [6], the same database used by [2,13]. The database contains 61 textures, and each texture has 205 images obtained under diﬀerent viewing and illumination conditions. The variety of textures in this database is illustrated in ﬁgure 2.

258

M. Varma and A. Zisserman

Fig. 3. The LM ﬁlter bank has a mix of edge, bar and spot ﬁlters at multiple scales and orientations. It has a total of 48 ﬁlters - 6 oriented ﬁlters at 3 scales and 2 phases, 8 Laplacian of Gaussian ﬁlters and 4 Gaussian ﬁlters.

2

Rotationally Invariant Filters and Their Eﬀect on Textons

In this section we investigate the advantages and disadvantages of clustering in a rotationally invariant ﬁlter response space. The use of such ﬁlters has already been espoused by Schmid [18]. To be deﬁnite, we will compare four ﬁlter sets: those of Leung and Malik which are not rotationally invariant; those of Schmid which are; and two reduced sets of ﬁlters proposed here using maximum response (which are again rotationally invariant). We will assess the ﬁlter sets by the classiﬁcation rate they achieve based on textons clustered in their response space. 2.1

Description of the Four Filter Sets

The Leung-Malik (LM) set: consists of 48 ﬁlters, partitioned as follows: Second derivative of Gaussians at 6 orientations, 3 scales and 2 phases (where phase refers to even and odd symmetry) making a total of 36; 8 Laplacian of Gaussian ﬁlters; and 4 Gaussians. The scale of the ﬁlters range between σ = 1 and σ = 10. They are shown in ﬁgure 3. The Schmid (S) set: consists of 13 rotationally invariant ﬁlters of the form πτ r r2 F (r, σ, τ ) = F0 (σ, τ ) + cos e− 2σ2 σ where F0 (σ, τ ) is added to obtain a zero DC component with the (σ, τ ) pair taking values (2,1), (4,1), (4,2), (6,1), (6,2), (6,3), (8,1), (8,2), (8,3), (10,1),

Fig. 4. The S ﬁlter bank is rotationally invariant and has 13 isotropic, “Gabor-like” ﬁlters.

Classifying Images of Materials

259

(10,2), (10,3) and (10,4). The ﬁlters are shown in ﬁgure 4. As can be seen all the ﬁlters have rotational symmetry. The Maximum Response (MR) sets: The MR4 ﬁlter bank consists of 14 ﬁlters but only four ﬁlter responses. The ﬁlter bank contains ﬁlters at multiple orientations but their outputs are “collapsed” by recording only the maximum ﬁlter response across all orientations. This achieves rotation invariance. In a similar spirit to [11,12] we have chosen the scale of our ﬁlters as that which gives the strongest response to the textures in the database, and only use a single scale for each ﬁlter (we return to the issue of scale below). The ﬁlter bank is shown in ﬁgure 5 and consists of a Gaussian and a Laplacian of Gaussian both with σ = 10 (these ﬁlters have rotational symmetry), an edge ﬁlter (σx = 4, σy = 12), and a bar ﬁlter (σx = 4, σy = 12). The latter two ﬁlters are oriented. Measuring only the maximum response reduces the number of responses from 14 (six orientations for two oriented ﬁlters, plus two isotropic) to 4. The MR8 ﬁlter bank is similar to the MR4 ﬁlter bank but the oriented edge and bar ﬁlters occur at 3 scales (σx ,σy )={(1,3), (2,6), (4,12)} thereby giving a total of six responses, three for the edge ﬁlter and three for the bar. The remaining two ﬁlters are the Gaussian and Laplacian of Gaussian with σ = 10 taken from the MR4 ﬁlter bank. The motivation for introducing these MR ﬁlters sets is twofold. First, we achieve rotation invariance, but are also able to record the angle of maximum response. This angular information is lost in the S ﬁlters. Thus, with the MR ﬁlters we are able to compute co-occurrence statistics on orientation, and such statistics may prove useful in discriminating isotropic from anisotropic textures. We return to this point in section 4. The second motivation concerns the low dimensionality of the ﬁlter response space. We would like to use the entire ﬁlter response of a texture for classiﬁcation (in the manner of [12]), rather than its vector quantization into textons. Again, we return to this issue in section 4. 2.2

Pre- and Post-processing

When computing the ﬁlter responses the following three steps are followed:

Fig. 5. The MR4 ﬁlter bank consists of four ﬁlters - two rotationally symmetric ones (a Gaussian and a Laplacian of Gaussian), and two oriented ones (an edge and a bar ﬁlter, at six diﬀerent orientations).

260

M. Varma and A. Zisserman

First, before any of the ﬁlters are applied, all the images are converted to grey scale and are intensity normalised to have zero mean and unit standard deviation. This normalization gives partial invariance to linear transformations in the illumination conditions of the images. Second, all 4 ﬁlter banks are L1 normalised so that the ﬁlter responses of each ﬁlter lie roughly in the same range. In more detail, each ﬁlter Fi in the ﬁlter bank is divided by Fi 1 so that the ﬁlter has unit L1 norm. This helps vector quantization, when using Euclidean distances, as the scaling for each of the ﬁlter response axes becomes the same. Third, following [8] (and motivated by Weber’s law), the ﬁlter response at each pixel x is (contrast) normalized as F(x) ← F(x) log (1 + L/0.03)/L where L = F(x)2 is the magnitude of the ﬁlter response vector at that pixel. 2.3

Textons by Clustering

We now consider clustering the ﬁlter responses in order to generate a texton dictionary. This dictionary will subsequently be used to deﬁne texture models based on learning from training images. For each ﬁlter set, we adopt the following procedure for computing a texton dictionary: For each texture, we select 13 images randomly (these images sample the variations in illumination and viewpoint), the ﬁlter responses over all these images are aggregated, and 10 texton cluster centres computed using the standard K-Means algorithm [7]. The learnt textons for each texture are then aggregated into a single dictionary. For example, if 5 textures are used then the dictionary will contain 50 textons. However, the classiﬁcation accuracy achieved is not very sensitive to variations in the procedure for dictionary generation (such as the particular textures

Fig. 6. The ﬁrst 100 textons recovered from 20 training textures using 13 images per texture: (a) S textons. (b) LM textons. (c) MR8 textons. Note that the S textons are rotationally symmetric.

Classifying Images of Materials

261

and method of choosing images etc), see subsection 3.3. Examples of the textons for each ﬁlter set are shown in ﬁgure 6. Our clustering task is considerably simpler than that of Leung and Malik, and Cula and Dana (who use the same ﬁlter bank) as we are able to cluster in low, 4 and 8, dimensional spaces. This compares to 13 dimensional for S, and 48 dimensional for LM (we are not considering 3D textons at this point where the dimensionality is 960). Concerning the rotation properties of the LM and MR textons, consider a texture and an (in plane) rotated version of the same texture. Corresponding features in the original and the rotated texture will map to the same point in MR ﬁlter space, but to a diﬀerent point in LM. It is therefore expected that more signiﬁcant clusters will be obtained in the rotationally invariant case. Secondly, for the LM ﬁlter set, which is not rotationally invariant, it would be expected that its textons can not classify a rotated version of a texture unless the rotated version is included in the training set (both of these points are demonstrated in ﬁgures 7 and 8). This establishes that there is an advantage in rotation invariance in that rotated versions of the same texture can be represented by one histogram, whilst several are required for the LM textons. However, there is still the possibility that rotation invariance has the disadvantage that two diﬀerent textures (which are not rotationally related) have the same histogram. We address this point next, where we compare classiﬁcation rates over a variety of textures.

Fig. 7. Textons learnt from original and rotated textures: (a) textons learnt from the LM ﬁlter bank for an original texture image of ribbed paper and a version rotated by 90 degrees (shown in ﬁgure 8). (b) depicts the textons learnt by the MR4 ﬁlter bank. In each case, the 10 textons learnt from the original image are shown in the top 2 rows while the 10 learnt from the rotated image are shown in the bottom 2 rows. Note that the LM ﬁlter bank learns two sets of textons (where one set is essentially the other set rotated by 90 degrees) and therefore will have two very diﬀerent histograms for the same texture. Conversely, the MR4 textons are virtually identical.

262

M. Varma and A. Zisserman

Fig. 8. Classiﬁcation of rotated textures. Column (a) shows three images of the ribbed paper texture (the ﬁrst two are from # 38B, the last from # 38). Column (b) shows the texton histograms using the LM ﬁlter bank. Column (c) shows the texton histograms obtained using the MR ﬁlter bank. Note that in the LM case, neither of the top two histograms matches the third (as seen through the eyes of the χ2 distance where small values in a bin are signiﬁcant), i.e. the description is rotationally variant. In the MR case all the histograms are, more or less, similar.

2.4

Classiﬁcation Method and Results

Here we perform three experiments to assess texture classiﬁcation rates over 92 images for each of 20, 40 and 61 textures respectively. The ﬁrst experiment, where we classify images from 20 textures, corresponds to the setup employed by Cula and Dana [2]. The second experiment, where 40 textures are classiﬁed, is modeled on the setup of Leung and Malik [13]. In the third experiment, we classify all 61 textures present in the Columbia-Utrecht database. The 92 images

Classifying Images of Materials

263

are selected as follows: for each texture in the database, there are 118 images where the viewing angle θv is less than 60 degrees. Out of these, only those 92 are chosen for which a suﬃciently large region could be cropped across all texture classes. Each experiment consists of three stages: texton dictionary generation; model generation, where models are learnt for the textures from training images; and, classiﬁcation of novel images. The 92 images for each texture are partitioned into two sets. Images in the ﬁrst (training) set are used for dictionary and model generation, classiﬁcation accuracy is only assessed on the 46 images for each texture in the second (test) set. Each of the 46 images per texture deﬁnes a model for that texture as follows: the image is mapped (vector quantized) to a texton distribution (histogram). Thus, each texture is represented by a set of 46 histograms. An image from the test set is then classiﬁed by choosing the closest model histogram, and hence the corresponding texture class. The distance function used to deﬁne closest is the χ2 signiﬁcance test [16]. In all three experiments we follow both [2] and [13], and learn the texton dictionary from 20 textures (using the procedure outlined before in subsection 2.3). The particular textures used are speciﬁed in ﬁgure 7 of [13]. In the ﬁrst experiment, 20 novel textures are chosen (see ﬁgure 5 in [2] for a list of the novel textures) and 20 × 46 = 920 novel images are classiﬁed in all. In the second experiment, the 40 textures speciﬁed in ﬁgure 7 of [13] are chosen. Here, a total of 40 × 46 = 1840 novel images are classiﬁed. Finally, in the third experiment, all the textures in the Columbia-Utrecht database are classiﬁed using the same procedure. The results for all three experiments are presented in table 1. Table 1. Comparison of the classiﬁcation rates for varying number of texture classes for each of the four ﬁlter sets. In all cases, 46 models are used for each texture class. ﬁlters S LM MR4 MR8

# of 20 96.30% 96.08% 94.13% 97.50%

texture classes 40 61 95.16% 94.54% 93.75% 93.44% 92.06% 90.76% 96.30% 96.07%

Discussion: Two points are notable in these results. First, the MR8 and S ﬁlters out perform the LM ﬁlters. This is a clear indicator that a rotationally invariant description is not a disadvantage (i.e. salient information for classiﬁcation is not lost). Second, the fact that MR8 does better than S and LM is also evidence that clustering in a lower dimensional space is advantageous. The MR4 ﬁlter bank loses out because it only contains ﬁlters at a single scale and hence can’t extract such rich features. What is also very encouraging with this classiﬁcation

264

M. Varma and A. Zisserman

method is that as the number of texture classes increases there is only a small decrease in the classiﬁcation rate.

3

Reducing the Number of Models

In this section, our objective is to reduce the number of models required for each texture class. In the previous section, the number of models was the same as the number of training images (and in eﬀect [13] used 20 models/images for every texture). Here we want to reduce the number of models to that appropriate for each texture, independent of the number of training images. As is demonstrated in ﬁgure 9 some textures may require far fewer models than others. We will investigate two approaches for reducing the number of models. The ﬁrst is selection of the histograms to determine the “natural” models for a particular texture. The second is pose normalization. The classiﬁcation experiments in this section are slightly modiﬁed so as to maximise the total number of images classiﬁed. If only M models per texture are used for training, then the rest of the 46 − M training images are added to the test set so that they too may be classiﬁed. Thus, for example when classifying 61 textures, if only M = 10 models are used then a total of 82 images per texture are classiﬁed giving a total of 82 × 61 = 5002 images. The texton dictionary used is the same as in the previous section and has 200 textons.

Fig. 9. Models per texture: the top row shows four images of the same texture, ribbed paper, photographed under diﬀerent viewing and lighting conditions. The images look very diﬀerent. The bottom row shows images of rough paper taken under the same conditions as the images in the ﬁrst row. These images don’t diﬀer so markedly because the texture doesn’t exhibit surface normal eﬀects. The consequence is that fewer models are required to represent rough paper over all viewpoints and lighting than for ribbed paper.

Classifying Images of Materials

3.1

265

Selection to Determine Texture Models

We would expect that the number of diﬀerent models that a texture requires is a function of how much the texture changes in appearance with imaging conditions, i.e. it is a function of the material properties of the texture. For example, if a texture is isotropic then the eﬀect of varying the lighting azimuthal angle will be less pronounced than for one that is anisotropic. Thus other parameters (such as relief proﬁle) being equal, fewer models would be required for the isotropic texture (than the anisotropic) to cover the changes due to lighting variation. However, if we are selecting models for the express purpose of classiﬁcation, then another parameter, the inter class image variation, also becomes very important in determining the number of models. For example, even if a texture varies considerably with changing imaging conditions it can be classiﬁed accurately using just a few models if all the other textures look very diﬀerent from it. Conversely, if two textures look very similar then many models may be needed to distinguish between them even if they do not show much variation individually. Here, we investigate two schemes for model reduction, both of which take into account the inter and intra class image variation. K-Medoid algorithm: Each histogram may be thought of as a point in Rn , where n is the number of bins in the histogram, so that the models for a particular texture class simply consist of a set of points in the Rn space. Therefore, given a distance function between two points, in our case χ2 , the K-Medoid algorithm may be used to cluster the histograms of all the training images (i.e. all the training images taken from all the texture classes) into representative centres (models). K-Medoid is a standard clustering algorithm [10] where the update rule always moves the cluster centre to the nearest data point in the cluster, but does not merge the points as in K-means. In fact, K-Meanscan only be applied to each texture class locally. It can not be applied here as it merges data points and thus the resultant cluster centres can not be identiﬁed distinctly with individual textures. Table 2a lists the results of classifying 20 textures using the four diﬀerent ﬁlter banks with K = 60, 120 and 180, resulting in an average of 3, 6 and 9 models per texture. Table 2. Classiﬁcation rates for each of the four ﬁlter sets when the models are automatically selected by (a) the K-Medoid algorithm, and (b) the Greedy algorithm. In all cases there are 20 texture classes. ﬁlters S LM MR4 MR8

Average # 3 74.66% 74.16% 69.94% 79.10%

of models 6 86.40% 86.69% 80.93% 89.59% (a)

per texture 9 90.12% 90.36% 85.36% 93.49%

ﬁlters S LM MR4 MR8

Average # 3 88.37% 86.69% 85.00% 90.67%

of models 6 97.21% 95.99% 93.66% 98.14% (b)

per texture 9 98.01% 97.83% 96.39% 98.61%

266

M. Varma and A. Zisserman

The classiﬁcation rate with 9 K-Medoid selected models per texture is almost as good as using all 46 models (see column 1 in table 1). It should be noted here, that due to the increase in the size of the test set, many more novel images have to be classiﬁed correctly to achieve the same classiﬁcation rate as in the previous section. Nevertheless, there is still signiﬁcant room for improvement. Greedy algorithm: An alternative to the K-Medoid clustering algorithm is a greedy algorithm designed to maximise the classiﬁcation accuracy while minimising the number of models used. The algorithm is initialized by setting the number of models equal to the number of training images. Then, at each iteration step, one model is discarded. This model is chosen to be the one for which the classiﬁcation accuracy decreases the least when it is dropped. This iteration is repeated until no more models are left. Figure 10, and table 2, show the resultant classiﬁcation accuracy versus number of models for the four ﬁlter banks when classifying 20, 40 and 61 textures. Figure 11 shows the 9 textures that were assigned the most models as well as the 9 textures that were assigned the least models while classifying all 61 textures. A very respectable classiﬁcation rate of over 97% correct is achieved for an average of 9 models per texture, even when all 61 texture classes are included. Furthermore, for most of the images that were misclassiﬁed, the correct texture class was ranked within the 5 most probable texture classes. It can therefore be hoped that incorporating extra information in the form of second order statistics or the co-occurrence of textons will lead to almost all the images being classiﬁed correctly (see future work). These results compare very favourably with those reported in [2] and [13]. Cula and Dana obtain a classiﬁcation rate of 96% while eﬀectively using 11 models per texture. Using 5 models per texture, they achieve a classiﬁcation 20 Textures

Classification accuracy %

100

40 Textures

100

95

95

95

90

90

90

85

85

85

80

80

80

75

70

75

S LM MR4 MR8 2

4 6 8 Models per texture

70

61 Textures

100

75

S LM MR4 MR8 2

4 6 8 10 Models per texture

70

S LM MR4 MR8 2

4 6 8 10 Models per texture

Fig. 10. Classiﬁcation rates for models selected by the Greedy algorithm for 20, 40 and 61 textures.

Classifying Images of Materials

267

Fig. 11. Models selected by the Greedy algorithm while classifying all 61 textures: The top row shows the 9 texture classes, and the corresponding number of models, that were assigned the most number of models by the Greedy algorithm while the bottom row shows the 9 classes that were assigned the least number of models. Moving from left to right, the textures and the number of models assigned to it are: Artiﬁcal grass (17), Sandpaper (15), Velvet (13), Plaster B (13), Rug A (13), Terrycloth (12), Aluminium Foil (12), Quarry Tile (12), White Bread (12), Lettuce Leaf (4), Sponge (4), Cracker A (3), Peacock Feather (3), Corn Husk (3), Straw (3), Painted Spheres (3), Roof Shingle (3) and Limestone (2).

rate of roughly 87% (see ﬁgure 6 and table 2 in [2]). In contrast, again for 20 textures, the MR8 ﬁlter bank achieves a classiﬁcation rate of 96% using 5 models per texture and achieves 98.4% using, on average, 10.4 models per texture. The main reason why the Greedy algorithm uses a fewer number of models for an equivalent classiﬁcation rate is because the models that Cula and Dana learn are general models and not geared speciﬁcally to classiﬁcation. They ignore the inter class variability between textures and concentrate only on the intra class variability. The models for a texture are selected by ﬁrst projecting all the training and test images into a low dimensional space using PCA. A manifold is ﬁtted to these projected points, and then reduced by systematically discarding those points which least aﬀect the “shape” of the manifold. The points which are left in the end correspond to the model images that deﬁne the texture. Since the models for a texture are chosen in isolation from the other textures, their algorithm ignores the inter class variation between textures. For 40 textures, Leung and Malik report an accuracy rate of 95.6% for classifying multiple (20) images and 87% for single images using an algorithm which eﬀectively needs 20 models per texture. The MR8 ﬁlter bank achieves 95.6% accuracy on the same textures using only 6.05 models per texture, and furthermore achieves 98.01% accuracy using, on average, 9.07 models per texture. 3.2

Pose Normalization

In [17] it was demonstrated that, provided a texture has suﬃcient directional variation, it can be pose normalized by maximizing weak isotropy of the second moment gradient matrix (a method originally suggested in [14]). The method is applicable in the absence of solid texture eﬀects. Here we investigate if this normalization can be used to at least reduce the eﬀects of changing viewpoint,

268

M. Varma and A. Zisserman Plaster A

Rough Plastic 0.3 0.7 0.25

0.6

0.5

0.2

0.4

0.15

0.3 0.1 0.2 0.05

0

0.1 Raw Pose Normalised 2

4

6

8

10

12

0

Raw Pose Normalised 2

4

6

8

10

12

Fig. 12. The eﬀect of pose normalization on a set of 12 images for two textures: “Rough Plastic” and “Plaster A”. The 12 images have been sorted according to increasing viewing angle and this is represented on the X axis. The Y axis is the χ2 distance between the model image and the given image. The pose normalized images consistently have a reduced χ2 distance which translates into better classiﬁcation.

and hence provide tighter clusters of the ﬁlter responses, or better still reduce the number of models needed to account for viewpoint change. In detail, if the normalization is successful, then for moderate changes in the viewing angle, two such “pose normalized” images of the same texture should diﬀer from each other by only a similarity transformation. If there are no major changes in scale, the responses of a rotationally invariant ﬁlter bank (MR or S) to these images should be much the same. A preliminary investigation shows that this is indeed the case for suitable textures. Figure 12 shows results for two textures - “Plaster A” and “Rough Plastic”. Twelve images of each texture are selected to have similar photometric appearance, but monotonically varying viewing angle. The graph shows the χ2 distance between the texton histogram of one of the images (selected as the model image) and the rest, before and after pose normalization. As can be seen, the χ2 distance is reduced for the pose normalized images. This in turn translates to better classiﬁcation as well. On experiments on 4 textures, using the same 12 image set and 1 model per texture, the classiﬁcation rate increased from 81.81% before pose normalization to 93.18% afterwards. 3.3

Generalization and Scale

In this subsection, we investigate how various factors aﬀect our algorithm’s classiﬁcation rate. We ﬁrst calculate a benchmark classiﬁcation rate and then vary the images in the training set and also the size of the texton dictionary to see how performance is aﬀected.

Classifying Images of Materials

269

Fig. 13. Scaling the data results in new models: The histogram of texton labelings of (a) the original image (b) the image scaled up by a factor of 3 and (c) the image scaled down by a factor of 3. All three models are substantially diﬀerent indicating that the texton vocabulary is suﬃcient, and it is the model that must be extended.

Initially, the texton dictionary is built by learning 10 textons from each of the 61 textures (using the procedure described in subsection 2.3) to have a total of 610 textons. Also, the 46 training images per texture from which the models will be generated are chosen by selecting every alternate image from the set of 92 available. Under these conditions, the MR8 ﬁlter bank achieves a classiﬁcation accuracy rate of 98.3% when classifying all 61 textures using, on average, 8 models per texture. To see if this choice of training images is important for our algorithm, the classiﬁcation experiment was repeated using 46 training images that were chosen randomly from the 92 available. The average classiﬁcation accuracy, computed over 10000 trial runs, was 98.16% and had stabilized to that value within 84 runs. We also evaluated the “expressive power” of our texton dictionary by performing the experiment having compiled the textons from only 31 randomly chosen texture classes, giving a total of 310 textons. The classiﬁcation rate decreased only slightly to 98.19% signifying that our textons are suﬃciently versatile. The number of textons in the dictionary can further be reduced by merging textons which lie very close to each other. We pruned down the size of the dictionary from 310 to 100 textons by selecting 80 of the most distinct textons (i.e. those textons that didn’t have any other textons lying close by) and ran KMeans, with K = 20, on the rest. This procedure entailed another slight decrease in the classiﬁcation accuracy to 97.38%. Thus, while classifying 61 textures, the best classiﬁcation rate achieved was 98.3% obtained when all 610 textons were used and the worst rate was 97.38% when only 100 textons wree used. We can therefore conclude that our algorithm is robust to the choice of training image set and texton vocabulary with the classiﬁcation rate not being aﬀected much by changes in these parameters. Finally, a word about scale. It may be of concern that the MR4 ﬁlter bank does not have ﬁlters at multiple scales and hence will be unable to handle scale changes successfully. To test this 25 images from 14 texture classes were artiﬁcially scaled, both up and down, by a factor of 3. The classiﬁcation experiment was repeated using the original, normal sized, ﬁlter banks and texton dictionar-

270

M. Varma and A. Zisserman

ies. We found that as long as models from the scaled images were included as part of the texture class deﬁnition, classiﬁcation accuracy was virtually unaffected and classiﬁcation rates of over 97% were achieved. However, if the choice of models was restricted to those drawn from the original sized images, then the classiﬁcation rate dropped to 17%. It is evident from this that ﬁlter bank and texton vocabulary are suﬃcient, and it is the model that must be extended.

4

Conclusions and Future Work

We have demonstrated that with a handful of models per texture, classiﬁcation rates superior to [2,13] can be achieved. In particular, only a single image is required for classiﬁcation, not twenty with registration, and there is no need to supply the imaging conditions, either implicitly or explicitly. By developing the MR set, we are now in a position where we will be able to compare the “texton clustering and distribution” method of discriminating textures with the Bayesian approach proposed by Konishi and Yuille [12]. Their method stores the joint PDF of ﬁlter responses (in suitably sized bins), and is completely infeasible if the dimension of the ﬁlter space is large – [12] use a six dimensional space. For example, it would not be possible to use this approach with the 48 dimensional ﬁlter space used by [13]. Konishi and Yuille make strong use of colour, but achieve impressive results in classifying outdoor scenes into a small number of texture classes. We also plan to investigate how the co-occurrence of textons (as in [18]) can lead to further improvements. Traditional rotation invariant ﬁlter banks are not very successful at measuring anisotropic features. However, the MR ﬁlter banks can detect such features accurately. Furthermore, recording the relative orientation at which the features occur will allow additional discriminability while preserving rotational invariance. Finally, one other interesting point to investigate is whether the dimensionality of the MR ﬁlter response space can be reduced still further by not only taking the maximum response over orientations but also over all scales as well. Acknowledgements. We are grateful to Oana Cula, Tomas Leung and Cordelia Schmid for supplying details of the algorithms used in their papers. We are very grateful to Frederick Schaﬀalitzky for numerous discussions and suggestions. Financial support was provided by a University of Oxford Graduate Scholarship in Engineering, an ORS award and the EC project CogViSys.

References 1. M. J. Chantler, G. McGunnigle, and J. Wu. Surface rotation invariant texture classiﬁcation using photometric stereo and surface magnitude spectra. In Proc. BMVC., pages 486–495, 2000. 2. O. G. Cula and K. J. Dana. Compact representation of bidirectional texture functions. In Proc. CVPR, 2001.

Classifying Images of Materials

271

3. O. G. Cula and K. J. Dana. Recognition methods for 3d textured surfaces. Proceedings of the SPIE, San Jose, Jan 2001. 4. K. Dana and S. Nayar. Histogram model for 3d textures. In Proc. CVPR, pages 618–624, 1998. 5. K. Dana and S. Nayar. Correlation model for 3D texture. In Proc. ICCV, pages 1061–1067, 1999. 6. K. J. Dana, B. van Ginneken, S. K. Nayar, and J. J. Koenderink. Reﬂectance and texture of real world surfaces. ACM Trans. Graphics, 18,1:1–34, 1999. 7. R. O. Duda and P. E. Hart. Pattern Classiﬁcation and Scene Analysis. Wiley, 1973. 8. C. Fowlkes, D. Martin, X. Ren, and J. Malik. Detecting and localizing boundaries in natural images. Technical report, University of California at Berkeley, 2002. 9. G. M. Haley and B. S. Manjunath. Rotation-invariant texture classiﬁcation using a complete space-frequency model. IEEE PAMI, 8(2):255–269, 1999. 10. L. Kaufman and P. J. Rousseeuw. Finding Groups in Data. John Wiley & Sons, 1990. 11. S. Konishi, A. Yuille, J. Coughlan, and S. Zhu. Fundamental bounds on edge detection: An information theoretic evaluation of diﬀerent edge cues. In Proc. CVPR, pages 573–579, 1999. 12. S. Konishi and A. L. Yuille. Statistical cues for domain speciﬁc image segmentation with performance analysis. In Proc. CVPR, 2000. 13. T. Leung and J. Malik. Representing and recognizing the visual appearance of materials using three-dimensional textons. IJCV, Dec 1999. 14. T. Lindeberg and J. G˚ arding. Shape-adapted smoothing in estimation of 3-d depth cues from aﬃne distortions of local 2-d brightness structure. In Proc. ECCV, pages 389–400, May 1994. 15. X. Liu, Y. Yu, and H. Shum. Synthesizing bidirectional texture functions for realworld surfaces. In Proc. ACM SIGGRAPH, pages 117–126, 2001. 16. W. Press, B. Flannery, S. Teukolsky, and W. Vetterling. Numerical Recipes in C. Cambridge University Press, 1988. 17. F. Schaﬀalitzky and A. Zisserman. Viewpoint invariant texture matching and wide baseline stereo. In Proc. ICCV, Jul 2001. 18. C. Schmid. Constructing models for content-based image retrieval. In Proc. CVPR, 2001. 19. A. Zalesny and L. Van Gool. A compact model for viewpoint dependent texture synthesis. volume 2018 of Lecture Notes in Computer Science, pages 124–143. Springer, July 2000. 20. S.C. Zhu, Y. Wu, and D. Mumford. Filters, random-ﬁelds and maximum-entropy (FRAME): Towards a uniﬁed theory for texture modeling. IJCV, 27(2):107–126, March 1998.

Estimation of Multiple Illuminants from a Single Image of Arbitrary Known Geometry Yang Wang and Dimitris Samaras Computer Science Department, State University of New York at Stony Brook, NY 11794, USA {yangwang, samaras}@cs.sunysb.edu

Abstract. We present a new method for the detection and estimation of multiple illuminants, using one image of any object with known geometry and Lambertian reﬂectance. Our method obviates the need to modify the imaged scene by inserting calibration objects of any particular geometry, relying instead on partial knowledge of the geometry of the scene. Thus, the recovered multiple illuminants can be used both for image-based rendering and for shape reconstruction. We ﬁrst develop our method for the case of a sphere with known size, illuminated by a set of directional light sources. In general, each point of such a sphere will be illuminated by a subset of these sources. We propose a novel, robust way to segment the surface into regions, with each region illuminated by a diﬀerent set of sources. The regions are separated by boundaries consisting of critical points (points where one illuminant is perpendicular to the normal). Our region-based recursive least-squares method is impervious to noise and missing data and signiﬁcantly outperforms a previous boundary-based method using spheres[21]. This robustness to missing data is crucial to extending the method to surfaces of arbitrary smooth geometry, other than spheres. We map the normals of the arbitrary shape onto a sphere, which we can then segment, even when only a subset of the normals is available on the scene. We demonstrate experimentally the accuracy of our method, both in detecting the number of light sources and in estimating their directions, by testing on images of a variety of synthetic and real objects.

1. Introduction In this paper we consider the problem of multiple illuminant direction detection from a single image, i.e. using an image of an object or scene to recover all real light sources in that environment. Knowledge of illuminant directions is necessary both in computer vision for shape reconstruction, and in image based computer graphics, in order to realistically manipulate existing images. This problem is particularly hard for diﬀuse (Lambertian) surfaces and directional light sources and cannot be solved using local information only. Lambertian reﬂectance is the most common type of reﬂectance. Previous methods that estimate multiple light sources require images of a calibration object of given shape (typically spheres) A. Heyden et al. (Eds.): ECCV 2002, LNCS 2352, pp. 272–288, 2002. c Springer-Verlag Berlin Heidelberg 2002

Estimation of Multiple Illuminants from a Single Image

273

which needs to be removed from the scene and might cause artifacts. Instead, our method relies instead on partial knowledge of the geometry of the scene and can be used on objects of arbitrary shape. This allows us to possibly use any diﬀuse object of the scene for illumination calibration. The interest in computing illuminant directions ﬁrst arose from shape from shading applications, and was initially focused on recovering a single light source, [5,12,9,22,17]. However, illumination in most real scenes is more complex and it is very likely to have a number of co-existing light sources in a scene. In the context of shape estimation, the ability to automatically detect the light sources allows for more practical methods. For example, there exist methods that use stereo images to compute a partially correct geometry, and then reﬁne it by Shape from Shading under a single light source [18]. These methods can now be extended to multiple light sources, using the partial geometry obtained by stereo to compute the parameters of the light sources with the methods proposed in this paper. More recently, with the advent of image based rendering methods, it is crucial to have knowledge of the light. An early attempt to recover a more general illumination description [6], modeled multiple light sources as a polynomial distribution. A discussion of the various types of light sources can be found in [8]. More recently, with the advent of image based methods in computer graphics, estimation of illumination parameters from images is necessary, in order to compensate for illumination artifacts, and also to allow super-imposition of synthetic images of new objects into real scenes. Most such methods need to use a calibration object of ﬁxed shape, typically a sphere. In [3] a specular sphere is used as a light probe to measure the incident illumination at the location where synthetic objects will be placed in the scene. Such a sphere though might have strong inter-reﬂections with other objects of the scene, especially if they are close to the it. Using the Lambertian shading model, Yang and Yuille [20] observed that multiple light sources can be deduced from boundary conditions, i.e., the image intensity along the occluding boundaries and at singular points. Based on this idea, Zhang and Yang [21] show that the illuminant directions have a close relationship to critical points on a Lambertian sphere and that, by identifying most of those critical points, illuminant directions may be recovered if certain conditions are satisﬁed. Conceptually, a critical point is a point on the surface such that all its neighbors are not illuminated by the same light sources. However, because the detection of critical points is sensitive to noise, the direction of extracted real lights in not very robust to noisy data. In [13] a calibration object that comprises of diﬀuse and specular parts is proposed. Inserting calibration objects in the scene complicates the acquisition process, as they either need to be physically removed before re-capturing the image, which is not always possible, or they need to be electronically removed as a post processing step, which might introduce artifacts in the image. Our proposed method can be applied to objects of known arbitrary geometry, as long as that shape contains a fairly complete set of normals for a least-squares evaluation of the light sources. Thus, it would be possible to estimate the illuminants from the image of a scene, using geometry that is part of the scene. Although estimating

274

Y. Wang and D. Samaras

the object geometry from the image would be more convenient, current single image computer vision methods do not provide the necessary accuracy for precise illuminant estimation. Such accuracy can be achieved only by more complex multi-image methods [10] that require precise calibration. The idea of using arbitrary known shape, can also be found in the approach of Sato et al. [19], which exploits information of a radiance distribution inside shadows cast by an object of known shape in the scene. Recently, under a signal processing approach [15,1] a comprehensive mathematical framework for evaluation of illumination parameters through convolution is described. Unfortunately, this framework does not provide a method to estimate high-frequency illumination such as directional light sources when the BRDF is smooth as in the Lambertian case. Convolution is a local operation and the problem is ill-posed when only local information is considered [2]. Our method uses global information to overcome this problem, and in this sense, it is complementary to the methods of [15], [1], [11]. In this paper, we propose an illuminant direction detection process that minimizes global error (by a recursive least-squares algorithm [4]). In general, each point of a surface is illuminated by a subset of all the directional light sources in the scene. We propose a novel, robust way to segment the surface into regions (virtual “light patches”), with each region illuminated by a diﬀerent set of sources. The regions are separated by boundaries consisting of critical points (where one illuminant is perpendicular to the normal of the surface [21]). We extract real lights based on the segmented virtual “light patches” instead of critical points that are relatively sensitive to noise. Since there are more points in a region than on the boundary, the method’s accuracy does not depend on the exact extraction of the boundary and can tolerate noisy and missing data better. Furthermore, we can further adjust and merge light patches to minimize the least-squares errors. The number of critical boundaries detected is sensitive to the threshold used in the Hough transform, especially when there are noisy or incomplete data. Fewer boundaries will cause lights to be missed, more boundaries will result in spurious lights to be detected. This is not a problem for our method since the spurious lights will be eliminated during the merging stage that follows the Hough transform. The ability of our method to perform well when the data is not perfect is crucial for the extension of the method from spheres (for which we initially develop it) to arbitrary shapes. When the observed shape is not spherical we map its normals to a sphere, although a lot of normals will be missing. However our method works well even for incomplete spheres, as long as there are enough points inside each light patch for the least-squares method to work correctly. Our method to detect multiple illuminant directions consists of the following ﬁve steps: 1. We identify critical points (deﬁned in the same manner as [21]) in the image, using two consecutive windows using a standard recursive least-squares algorithm[4]. 2. We group critical points into diﬀerent critical boundaries by the Hough Transform.

Estimation of Multiple Illuminants from a Single Image

275

3. These critical boundaries segment the image into diﬀerent separate light patches and we calculate a virtual light for each light patch. 4. We adjust and merge critical boundaries by minimizing the average leastsquares error of all light patches. If two critical boundaries are too close to each other, i.e. the angle between them is less than a threshold angle, then we merge these two critical boundaries. 5. We extract real lights based on these segmented virtual light patches instead of critical points that are relatively sensitive to noise. The absolute intensities of real lights can be determined only if the albedo is known. We performed a series of experiments which demonstrate the superior accuracy of our method. Experiments with synthetic data from a sphere and a vase model allow us to recover accurately up to ﬁfteen light sources from a sphere image, with a 0.5 degree average error. We also show that our method performs well even with added noise (error less than 1 degree for ﬁve lights). We successfully detect ﬁve lights from the image of a real ball made of rubber, even though the surface is rather rough and at various parts deviates from the Lambertian reﬂectance assumption. Our results compare favorably with the boundary-based method in [21] for the sphere experiments. We perform experiments on two diﬀerent nonspherical objects, the synthetic images of a vase and the real images of a toy duck under four illuminants. The algorithm performs accurately, despite the increased levels of noise, both in the reﬂectance and in the acquisition of the normals. The rest of this paper is structured as follows: Section 2 describes the notion of critical points and their properties as they pertain to our problem. Section 3 describes our basic algorithm and extensions that make it robust to noise and missing data. These properties of our algorithm allow us to introduce a mapping of normals from arbitrary geometry to the shape of a sphere. This is the key to applying our algorithm to objects of arbitrary shape in Section 4. We present our experiments on synthetic and real data in Section 5 and conclude with future working Section 6.

2. Critical Points Deﬁnition 1. Given an image, let Li , i = 1, 2, . . ., be the light sources of the image. A point in the image is called a critical point if the surface normal at the corresponding point on the surface of the object is perpendicular to some light source Li . 2.1. Observation Model We assume that images are formed by perspective or orthographic projection and the object in the image has a Lambertian surface with constant albedo, that is each surface point appears equally bright from all viewing directions: L = ρI · n

(1)

where L is the scene radiance of an ideal Lambertian surface, ρ is the albedo, I represents the direction and amount of incident light, and n is the normal to the surface.

276

Y. Wang and D. Samaras

2.2. Sphere Model Initially, we develop our algorithm using a sphere model. We will show later in this paper how to extend this to objects of arbitrary shape. – We assume the observed object is a sphere with Lambertian reﬂectance properties whose physical size is already known. – For light sources whose direction is co-linear with the lens axis of the camera and the angle with OS is less than ω (Fig. 1(a)) depending on the resolution, the best possible result is their equivalent frontal light source Lf rontal .

S

O

Light ✮ ω

τ P

α S

β

Camera

θ

✕L ✯ φ LP q N

A (a)

(b)

Fig. 1. (a) minimum detectable angle ω. (b) L and its projection LP onto plane P.

It has been proven in [21] that it is not possible to recover the exact value of the intensity of any individual light source among four (or more) pairs of antipodal light sources. However, this kind of situation, i.e. an object illuminated by antipodal light sources, happens rarely, so for simplicity in the rest of this paper, we will make an additional assumption that there are no antipodal light sources. 2.3. Sphere Cross Section with a Plane P Let P be an arbitrary plane such that S, the center of the sphere, lies on it (Fig. 1(b)), Li , i = 1, 2, . . ., be the light sources of the image and (L)iP their projections on P. A point on the arc τ can be speciﬁed by its corresponding angle parameter in [α, β] using the following proposition [21]: Proposition 1. Consider an angle interval [α, β] of a sphere cross section (Fig. 1(b)). We can always find a partition θ0 = α < θ1 < . . . < θn = β of the interval [α, β] such that in each [θi−1 , θi ] we have E(θ) = bi sin θ + ci cos θ for some constants bi and ci , 1 ≤ i ≤ n (Fig. 2(a)), where E(θ) is the intensity function along the arc τ . Intuitively, (bi−1 , ci−1 ) represents the virtual light source of the [θi−2 , θi−1 ] part, and (bi , ci ) of the neighboring [θi−1 , θi ] part. These two light sources will be diﬀerent, if each of these two parts is lit by a diﬀerent illuminant conﬁguration. More formally, Proposition 2 (from [21]) describes the diﬀerence between (bi−1 , ci−1 ) and (bi , ci ):

Estimation of Multiple Illuminants from a Single Image

Lr = L1 − L2 ✻R

✻Y

✠ L1

L1

X ✲ θ ✲

L2

L2

(a)

277

✼ Lr = L2 − L1 (b)

Fig. 2. (a) the xy-space and the θR-space for the case with two sources. (b) Illustration of real light pre-direction. Lr is the real light direction.

Proposition 2. In the configuration of Proposition 1, for any 1 ≤ i ≤ n, we define Λi as the index set of real light sources contributed to the [θi−1 , θi ] part of the arc τ . Then the Euclidean distance between two (bi , ci ) pairs is (bi − bi−1 )2 + (ci − ci−1 )2 = (Lj )P (2) j∈Λ ∪Λ

where (Lj )P is the Euclidean norm, Λ = Λ i−1 \ Λi (the index set of elements in Λi−1 but not in Λi ), Λ = Λi \ Λi−1 , and j∈Λi Lj is the virtual light source corresponding to [θi−1 , θi ]. Propositions 1 and 2 show that the diﬀerence between (bi−1 , ci−1 ) and (bi , ci ) will be maximized at a critical point for these two virtual light sources. So, possible critical points can be detected by thresholding (Lj )P . 2.4. Properties of Critical Points Let Σ be the set of all critical points and Ω be the space of the sphere image. Then intuitively Σ will cut Ω into a decomposition, i.e. (3) Ω = ( ui ) Σ i∈I

where each ui ⊂ Ω is a subset of R2 which does not contain any critical points and I is an index set. Proposition 3. Given a decomposition (3), for any ui there exists an equivalent light source L such that the image formed by the corresponding surface part of ui illuminated by L is exactly the same as the image on ui . Proposition 2 already provides us with a criterion to detect critical points on the sphere based on the distance between (bi , ci ) pairs. Unfortunately, this criterion greatly depends on the intensities of virtual light sources, j∈Λ ∪Λ (Lj )P , which are projected on the plane with respect to each different cross section. To locate the critical points more accurately, we provide

278

Y. Wang and D. Samaras

another way to detect critical points on each cross section. Instead of using the distance between (bi , ci ) pairs, we can use the tangent angles deﬁned on the intensity curve (Fig. 3(a)). Proposition 4. Along a sine curve, at a critical point θc , the inner angle γ between two tangent lines of each side (T1 , T2 ) will be larger than 180 degrees:

✻ R T1 }

✍

✻ R T1 }

T2

✍

A

γ

γ

θ ✲

θc

T2 γ

(a)

~ T1

θc

θ ✲

P

(b)

τ

θ

S C

W B

(c)

Fig. 3. (a) inner angle γ. (b) Angles between two tangent lines. (c) a part of arc τ , AB, is covered by two consecutive windows AW and WB.

Proof. We deﬁne the sine curve contributing to this critical point θc is x(θ) = κ sin(ξ(θ − θc )), (κ, ξ > 0) (Fig. 3(b)).Then the combined curve function will be sin(θ) (θ ≤ θc ) f (θ) = (4) sin(θ) + κ sin(ξ(θ − θc )) (θ > θc ) We therefore get the tangent lines: T1 : fθ − = cos θc c

(5)

T2 : fθ + = cos θc + κ cos(ξ(θc − θc )) · ξ = cos θc + κξ c

So the angle between T1 and T2 is γ = arctan(cos θc + κξ) − arctan(cos θc ). Since κξ > 0, arctan(cos θc + κξ) > arctan(cos θc ) ⇒ γ