Computer Vision - ACCV 2007, 8 conf

Lecture Notes in Computer Science Commenced Publication in 1973 Founding and Former Series Editors: Gerhard Goos, Juris Hartmanis, and Jan van Leeuwen

Editorial Board David Hutchison Lancaster University, UK Takeo Kanade Carnegie Mellon University, Pittsburgh, PA, USA Josef Kittler University of Surrey, Guildford, UK Jon M. Kleinberg Cornell University, Ithaca, NY, USA Friedemann Mattern ETH Zurich, Switzerland John C. Mitchell Stanford University, CA, USA Moni Naor Weizmann Institute of Science, Rehovot, Israel Oscar Nierstrasz University of Bern, Switzerland C. Pandu Rangan Indian Institute of Technology, Madras, India Bernhard Steffen University of Dortmund, Germany Madhu Sudan Massachusetts Institute of Technology, MA, USA Demetri Terzopoulos University of California, Los Angeles, CA, USA Doug Tygar University of California, Berkeley, CA, USA Moshe Y. Vardi Rice University, Houston, TX, USA Gerhard Weikum Max-Planck Institute of Computer Science, Saarbruecken, Germany

4844

Yasushi Yagi Sing Bing Kang In So Kweon Hongbin Zha (Eds.)

Computer Vision – ACCV 2007 8th Asian Conference on Computer Vision Tokyo, Japan, November 18-22, 2007 Proceedings, Part II

13

Volume Editors Yasushi Yagi Osaka University The Institute of Scientific and Industrial Research 8-1 Mihogaoka, Ibaraki, Osaka, 567-0047, Japan E-mail: [email protected] Sing Bing Kang Microsoft Corporation 1 Microsoft Way, Redmond WA 98052, USA E-mail: [email protected] In So Kweon KAIST School of Electrical Engineering and Computer Science 335 Gwahag-Ro Yusung-Gu, Daejeon, Korea E-mail: [email protected] Hongbin Zha Peking University Department of Machine Intelligence Beijing, 100871, China E-mail: [email protected]

Library of Congress Control Number: 2007938408 CR Subject Classification (1998): I.4, I.5, I.2.10, I.2.6, I.3.5, F.2.2 LNCS Sublibrary: SL 6 – Image Processing, Computer Vision, Pattern Recognition, and Graphics ISSN ISBN-10 ISBN-13

0302-9743 3-540-76389-9 Springer Berlin Heidelberg New York 978-3-540-76389-5 Springer Berlin Heidelberg New York

This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, re-use of illustrations, recitation, broadcasting, reproduction on microfilms or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer. Violations are liable to prosecution under the German Copyright Law. Springer is a part of Springer Science+Business Media springer.com © Springer-Verlag Berlin Heidelberg 2007 Printed in Germany Typesetting: Camera-ready by author, data conversion by Scientific Publishing Services, Chennai, India Printed on acid-free paper SPIN: 12183685 06/3180 543210

Preface

It is our great pleasure to welcome you to the Proceedings of the Eighth Asian Conference on Computer Vision (ACCV07), which held November 18–22, 2007 in Tokyo, Japan. ACCV07 was sponsored by the Asian Federation of Computer Vision. We received 640 abstracts by the abstract submission deadline, 551 of which became full submissions. This is the largest number of submissions in the history of ACCV. Out of these 551 full submissions, 46 were selected for oral presentation and 130 as posters, yielding an acceptance rate of 31.9%. Following the tradition of previous ACCVs, the reviewing process was double blind. Each of the 31 Area Chairs (ACs) handled about 17 papers and nominated five reviewers for each submission (from 204 Program Committee members). The final selection of three reviewers per submission was done in such a way as to avoid conflict of interest and to evenly balance the load among the reviewers. Once the reviews were done, each AC wrote summary reports based on the reviews and their own assessments of the submissions. For conflicting scores, ACs consulted with reviewers, and at times had us contact authors for clarification. The AC meeting was held in Osaka on July 27 and 28. We divided the 31 ACs into 8 groups, with each group having 3 or 4 ACs. The ACs can confer within their respective groups, and are permitted to discuss with pre-approved “consulting” ACs outside their groups if needed. The ACs were encouraged to rely more on their perception of paper vis-a-vis reviewer comments, and not strictly based on numerical scores alone. This year, we introduced the category “conditional accept;” this category is targeted at papers with good technical content but whose writing requires significant improvement. Please keep in mind that no reviewing process is perfect. As with any major conference, reviewer quality and timeliness of reviews varied. To minimize the impact of variation of these factors, we chose highly qualified and dependable people as ACs to shepherd the review process. We all did the best we could given the large number of submissions and the limited time we had. Interestingly, we did not have to instruct the ACs to revise their decisions at the end of the AC meeting—all the ACs did a great job in ensuring the high quality of accepted papers. That being said, it is possible there were good papers that fell through the cracks, and we hope such papers will quickly end up being published at other good avenues. It has been a pleasure for us to serve as ACCV07 Program Chairs, and we can honestly say that this has been a memorable and rewarding experience. We would like to thank the ACCV07 ACs and members of the Technical Program Committee for their time and effort spent reviewing the submissions. The ACCV Osaka team (Ryusuke Sagawa, Yasushi Makihara, Tomohiro Mashita, Kazuaki Kondo, and Hidetoshi Mannami), as well as our conference secretaries (Noriko

VI

Preface

Yasui, Masako Kamura, and Sachiko Kondo), did a terrific job organizing the conference. We hope that all of the attendees found the conference informative and thought provoking. November 2007

Yasushi Yagi Sing Bing Kang In So Kweon Hongbin Zha

Organization

General Chair General Co-chairs

Program Chair Program Co-chairs

Workshop/Tutorial Chair Finance Chair Local Arrangements Chair Publication Chairs Technical Support Staff

Area Chairs

Katsushi Ikeuchi (University of Tokyo, Japan) Naokazu Yokoya (NAIST, Japan) Rin-ichiro Taniguchi (Kyuushu University, Japan) Yasushi Yagi (Osaka University, Japan) In So Kweon (KAIST, Korea) Sing Bing Kang (Microsoft Research, USA) Hongbin Zha (Peking University, China) Kazuhiko Sumi (Mitsubishi Electric, Japan) Keiji Yamada (NEC, Japan) Yoshinari Kameda (University of Tsukuba, Japan) Hideo Saito (Keio University, Japan) Daisaku Arita (ISIT, Japan) Atsuhiko Banno (University of Tokyo, Japan) Daisuke Miyazaki (University of Tokyo, Japan) Ryusuke Sagawa (Osaka University, Japan) Yasushi Makihara (Osaka University, Japan) Tat-Jen Cham (Nanyang Tech. University, Singapore) Koichiro Deguchi (Tohoku University, Japan) Frank Dellaert (Georgia Inst. of Tech., USA) Martial Hebert (CMU, USA) Ki Sang Hong (Pohang University of Sci. and Tech., Korea) Yi-ping Hung (National Taiwan University, Taiwan) Reinhard Klette (University of Auckland, New Zealand) Chil-Woo Lee (Chonnam National University, Korea) Kyoung Mu Lee (Seoul National University, Korea) Sang Wook Lee (Sogang University, Korea) Stan Z. Li (CASIA, China) Yuncai Liu (Shanghai Jiaotong University, China) Yasuyuki Matsushita (Microsoft Research Asia, China) Yoshito Mekada (Chukyo University, Japan) Yasuhiro Mukaigawa (Osaka University, Japan)

VIII

Organization

P.J. Narayanan (IIIT, India) Masatoshi Okutomi (Tokyo Inst. of Tech., Japan) Tomas Pajdla (Czech Technical University, Czech) Shmuel Peleg (The Hebrew University of Jerusalem, Israel) Jean Ponce (Ecole Normale Superieure, France) Long Quan (Hong Kong University of Sci. and Tech., China) Ramesh Raskar (MERL, USA) Jim Rehg (Georgia Inst. of Tech., USA) Jun Sato (Nagoya Inst. of Tech., Japan) Shinichi Sato (NII, Japan) Yoichi Sato (University of Tokyo, Japan) Cordelia Schmid (INRIA, France) Christoph Schnoerr (University of Mannheim, Germany) David Suter (Monash University, Australia) Xiaoou Tang (Microsoft Research Asia, China) Guangyou Xu (Tsinghua University, China)

Program Committee Adrian Barbu Akash Kushal Akihiko Torii Akihiro Sugimoto Alexander Shekhovtsov Amit Agrawal Anders Heyden Andreas Koschan Andres Bruhn Andrew Hicks Anton van den Hengel Atsuto Maki Baozong Yuan Bernt Schiele Bodo Rosenhahn Branislav Micusik C.V. Jawahar Chieh-Chih Wang Chin Seng Chua Chiou-Shann Fuh Chu-song Chen

Cornelia Fermuller Cristian Sminchisescu Dahua Lin Daisuke Miyazaki Daniel Cremers David Forsyth Duy-Dinh Le Fanhuai Shi Fay Huang Florent Segonne Frank Dellaert Frederic Jurie Gang Zeng Gerald Sommer Guoyan Zheng Hajime Nagahara Hanzi Wang Hassan Foroosh Hideaki Goto Hidekata Hontani Hideo Saito

Hiroshi Ishikawa Hiroshi Kawasaki Hong Zhang Hongya Tuo Hynek Bakstein Hyun Ki Hong Ikuko Shimizu Il Dong Yun Itaru Kitahara Ivan Laptev Jacky Baltes Jakob Verbeek James Crowley Jan-Michael Frahm Jan-Olof Eklundh Javier Civera Jean Martinet Jean-Sebastien Franco Jeffrey Ho Jian Sun Jiang yu Zheng

Organization

Jianxin Wu Jianzhuang Liu Jiebo Luo Jingdong Wang Jinshi Cui Jiri Matas John Barron John Rugis Jong Soo Choi Joo-Hwee Lim Joon Hee Han Joost Weijer Jun Sato Jun Takamatsu Junqiu Wang Juwei Lu Kap Luk Chan Karteek Alahari Kazuhiro Hotta Kazuhiro Otsuka Keiji Yanai Kenichi Kanatani Kenton McHenry Ki Sang Hong Kim Steenstrup Pedersen Ko Nishino Koichi Hashomoto Larry Davis Lisheng Wang Manabu Hashimoto Marcel Worring Marshall Tappen Masanobu Yamamoto Mathias Kolsch Michael Brown Michael Cree Michael Isard Ming Tang Ming-Hsuan Yang Mingyan Jiang Mohan Kankanhalli Moshe Ben-Ezra Naoya Ohta Navneet Dalal Nick Barnes

Nicu Sebe Noboru Babaguchi Nobutaka Shimada Ondrej Drbohlav Osamu Hasegawa Pascal Vasseur Patrice Delmas Pei Chen Peter Sturm Philippos Mordohai Pierre Jannin Ping Tan Prabir Kumar Biswas Prem Kalra Qiang Wang Qiao Yu Qingshan Liu QiuQi Ruan Radim Sara Rae-Hong Park Ralf Reulke Ralph Gross Reinhard Koch Rene Vidal Robert Pless Rogerio Feris Ron Kimmel Ruigang Yang Ryad Benosman Ryusuke Sagawa S.H. Srinivasan S. Kevin Zhou Seungjin Choi Sharat Chandran Sheng-Wen Shih Shihong Lao Shingo Kagami Shin’ichi Satoh Shinsaku Hiura ShiSguang Shan Shmuel Peleg Shoji Tominaga Shuicheng Yan Stan Birchfield Stefan Gehrig

Stephen Lin Stephen Maybank Subhashis Banerjee Subrata Rakshit Sumantra Dutta Roy Svetlana Lazebnik Takayuki Okatani Takekazu Kato Tat-Jen Cham Terence Sim Tetsuji Haga Theo Gevers Thomas Brox Thomas Leung Tian Fang Til Aach Tomas Svoboda Tomokazu Sato Toshio Sato Toshio Ueshiba Tyng-Luh Liu Vincent Lepetit Vivek Kwatra Vladimir Pavlovic Wee-Kheng Leow Wei Liu Weiming Hu Wen-Nung Lie Xianghua Ying Xianling Li Xiaogang Wang Xiaojuan Wu Yacoob Yaser Yaron Caspi Yasushi Sumi Yasutaka Furukawa Yasuyuki Sugaya Yeong-Ho Ha Yi-ping Hung Yong-Sheng Chen Yoshinori Kuno Yoshio Iwai Yoshitsugu Manabe Young Shik Moon Yunde Jia

IX

X

Organization

Zen Chen Zhifeng Li Zhigang Zhu

Zhouchen Lin Zhuowen Tu Zuzana Kukelova

Additional Reviewers Afshin Sepehri Alvina Goh Anthony Dick Avinash Ravichandran Baidya Saha Brian Clipp C´edric Demonceaux Christian Beder Christian Schmaltz Christian Wojek Chunhua Shen Chun-Wei Chen Claude P´egard D.H. Ye D.J. Kwon Daniel Hein David Fofi David Gallup De-Zheng Liu Dhruv K. Mahajan Dipti Mukherjee Edgar Seemann Edgardo Molina El Mustapha Mouaddib Emmanuel Prados Frank R. Schmidt Frederik Meysel Gao Yan Guy Rosman Gyuri Dorko H.J. Shim Hang Yu Hao Du Hao Tang Hao Zhang Hirishi Ohno Hiroshi Ohno Huang Wei Hynek Bakstein

Ilya Levner Imran Junejo Jan Woetzel Jian Chen Jianzhao Qin Jimmy Jiang Liu Jing Wu John Bastian Juergen Gall K.J. Lee Kalin Kolev Karel Zimmermann Ketut Fundana Koichi Kise Kongwah Wan Konrad Schindler Kooksang Moon Levi Valgaerts Li Guan Li Shen Liang Wang Lin Liang Lingyu Duan Maojun Yuan Mario Fritz Martin Bujnak Martin Matousek Martin Sunkel Martin Welk Micha Andriluka Michael Stark Minh-Son Dao Naoko Nitta Neeraj Kanhere Niels Overgaard Nikhil Rane Nikodem Majer Nilanjan Ray Nils Hasler

Nipun kwatra Olivier Morel Omar El Ganaoui Pankaj Kumar Parag Chaudhuri Paul Schnitzspan Pavel Kuksa Petr Doubek Philippos Mordohai Reiner Schnabel Rhys Hill Rizwan Chaudhry Rui Huang S.M. Shahed Nejhum S.H. Lee Sascha Bauer Shao-Wen Yang Shengshu Wang Shiro Kumano Shiv Vitaladevuni Shrinivas Pundlik Sio-Hoi Ieng Somnath Sengupta Sudipta Mukhopadhyay Takahiko Horiuchi Tao Wang Tat-Jun Chin Thomas Corpetti Thomas Schoenemann Thorsten Thormaehlen Weihong Li Weiwei Zhang Xiaoyi Yu Xinguo Yu Xinyu Huang Xuan Song Yi Feng Yichen Wei Yiqun Li

Organization

Yong MA Yoshihiko Kawai

Zhichao Chen Zhijie Wang

Sponsors Sponsor Technical Co-sponsors

Asian Federation of Computer Vision IPSJ SIG-CVIM IEICE TG-PRMU

XI

Table of Contents – Part II

Poster Session 4: Face/Gesture/Action Detection and Recognition Palmprint Recognition Under Unconstrained Scenes . . . . . . . . . . . . . . . . . . Yufei Han, Zhenan Sun, Fei Wang, and Tieniu Tan

1

Comparative Studies on Multispectral Palm Image Fusion for Biometrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ying Hao, Zhenan Sun, and Tieniu Tan

12

Learning Gabor Magnitude Features for Palmprint Recognition . . . . . . . . Rufeng Chu, Zhen Lei, Yufei Han, Ran He, and Stan Z. Li

22

Sign Recognition Using Constrained Optimization . . . . . . . . . . . . . . . . . . . . Kikuo Fujimura and Lijie Xu

32

Poster Session 4: Image and Video Processing Depth from Stationary Blur with Adaptive Filtering . . . . . . . . . . . . . . . . . . Jiang Yu Zheng and Min Shi

42

Three-Stage Motion Deblurring from a Video . . . . . . . . . . . . . . . . . . . . . . . . Chunjian Ren, Wenbin Chen, and I-fan Shen

53

Near-Optimal Mosaic Selection for Rotating and Zooming Video Cameras . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Nazim Ashraf, Imran N. Junejo, and Hassan Foroosh

63

Video Mosaicing Based on Structure from Motion for Distortion-Free Document Digitization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Akihiko Iketani, Tomokazu Sato, Sei Ikeda, Masayuki Kanbara, Noboru Nakajima, and Naokazu Yokoya Super Resolution of Images of 3D Scenecs . . . . . . . . . . . . . . . . . . . . . . . . . . . Uma Mudenagudi, Ankit Gupta, Lakshya Goel, Avanish Kushal, Prem Kalra, and Subhashis Banerjee Learning-Based Super-Resolution System Using Single Facial Image and Multi-resolution Wavelet Synthesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Shu-Fan Lui, Jin-Yi Wu, Hsi-Shu Mao, and Jenn-Jier James Lien

73

85

96

XIV

Table of Contents – Part II

Poster Session 4: Segmentation and Classification Statistical Framework for Shot Segmentation and Classification in Sports Video . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ying Yang, Shouxun Lin, Yongdong Zhang, and Sheng Tang

106

Sports Classification Using Cross-Ratio Histograms . . . . . . . . . . . . . . . . . . . Balamanohar Paluri, S. Nalin Pradeep, Hitesh Shah, and C. Prakash

116

A Bayesian Network for Foreground Segmentation in Region Level . . . . . Shih-Shinh Huang, Li-Chen Fu, and Pei-Yung Hsiao

124

Efficient Graph Cuts for Multiclass Interactive Image Segmentation . . . . Fangfang Lu, Zhouyu Fu, and Antonio Robles-Kelly

134

Feature Subset Selection for Multi-class SVM Based Image Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Lei Wang

145

Evaluating Multi-class Multiple-Instance Learning for Image Categorization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Xinyu Xu and Baoxin Li

155

Poster Session 4: Shape TransforMesh: A Topology-Adaptive Mesh-Based Approach to Surface Evolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Andrei Zaharescu, Edmond Boyer, and Radu Horaud

166

Microscopic Surface Shape Estimation of a Transparent Plate Using a Complex Image . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Masao Shimizu and Masatoshi Okutomi

176

Shape Recovery from Turntable Image Sequence . . . . . . . . . . . . . . . . . . . . . H. Zhong, W.S. Lau, W.F. Sze, and Y.S. Hung

186

Shape from Contour for the Digitization of Curved Documents . . . . . . . . Fr´ed´eric Courteille, Jean-Denis Durou, and Pierre Gurdjos

196

Improved Space Carving Method for Merging and Interpolating Multiple Range Images Using Information of Light Sources of Active Stereo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ryo Furukawa, Tomoya Itano, Akihiko Morisaka, and Hiroshi Kawasaki

206

Shape Representation and Classification Using Boundary Radius Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Hamidreza Zaboli and Mohammad Rahmati

217

Table of Contents – Part II

XV

Optimization A Convex Programming Approach to the Trace Quotient Problem . . . . . Chunhua Shen, Hongdong Li, and Michael J. Brooks

227

Learning a Fast Emulator of a Binary Decision Process . . . . . . . . . . . . . . . ˇ Jan Sochman and Jiˇr´ı Matas

236

Radiometry Multiplexed Illumination for Measuring BRDF Using an Ellipsoidal Mirror and a Projector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yasuhiro Mukaigawa, Kohei Sumino, and Yasushi Yagi

246

Analyzing the Influences of Camera Warm-Up Effects on Image Acquisition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Holger Handel

258

Geometry Simultaneous Plane Extraction and 2D Homography Estimation Using Local Feature Transformations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ouk Choi, Hyeongwoo Kim, and In So Kweon

269

A Fast Optimal Algorithm for L2 Triangulation . . . . . . . . . . . . . . . . . . . . . . Fangfang Lu and Richard Hartley

279

Adaptively Determining Degrees of Implicit Polynomial Curves and Surfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Bo Zheng, Jun Takamatsu, and Katsushi Ikeuchi

289

Determining Relative Geometry of Cameras from Normal Flows . . . . . . . Ding Yuan and Ronald Chung

301

Poster Session 5: Geometry Highest Accuracy Fundamental Matrix Computation . . . . . . . . . . . . . . . . . Yasuyuki Sugaya and Kenichi Kanatani

311

Sequential L∞ Norm Minimization for Triangulation . . . . . . . . . . . . . . . . . Yongduek Seo and Richard Hartley

322

Initial Pose Estimation for 3D Model Tracking Using Learned Objective Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Matthias Wimmer and Bernd Radig

332

XVI

Table of Contents – Part II

Multiple View Geometry for Non-rigid Motions Viewed from Translational Cameras . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Cheng Wan, Kazuki Kozuka, and Jun Sato Visual Odometry for Non-overlapping Views Using Second-Order Cone Programming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jae-Hak Kim, Richard Hartley, Jan-Michael Frahm, and Marc Pollefeys

342

353

Pose Estimation from Circle or Parallel Lines in a Single Image . . . . . . . . Guanghui Wang, Q.M. Jonathan Wu, and Zhengqiao Ji

363

An Occupancy – Depth Generative Model of Multi-view Images . . . . . . . Pau Gargallo, Peter Sturm, and Sergi Pujades

373

Poster Session 5: Matching and Registration Image Correspondence from Motion Subspace Constraint and Epipolar Constraint . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Shigeki Sugimoto, Hidekazu Takahashi, and Masatoshi Okutomoi Efficient Registration of Aerial Image Sequences Without Camera Priors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Shobhit Niranjan, Gaurav Gupta, Amitabha Mukerjee, and Sumana Gupta

384

394

Simultaneous Appearance Modeling and Segmentation for Matching People Under Occlusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Zhe Lin, Larry S. Davis, David Doermann, and Daniel DeMenthon

404

Content-Based Matching of Videos Using Local Spatio-temporal Fingerprints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Gajinder Singh, Manika Puri, Jeffrey Lubin, and Harpreet Sawhney

414

Automatic Range Image Registration Using Mixed Integer Linear Programming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Shizu Sakakubara, Yuusuke Kounoike, Yuji Shinano, and Ikuko Shimizu Accelerating Pattern Matching or How Much Can You Slide? . . . . . . . . . . Ofir Pele and Michael Werman

424

435

Poster Session 5: Recognition Detecting, Tracking and Recognizing License Plates . . . . . . . . . . . . . . . . . . Michael Donoser, Clemens Arth, and Horst Bischof

447

Action Recognition for Surveillance Applications Using Optic Flow and SVM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Somayeh Danafar and Niloofar Gheissari

457

Table of Contents – Part II

XVII

The Kernel Orthogonal Mutual Subspace Method and Its Application to 3D Object Recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Kazuhiro Fukui and Osamu Yamaguchi

467

Viewpoint Insensitive Action Recognition Using Envelop Shape . . . . . . . . Feiyue huang and Guangyou Xu

477

Unsupervised Identification of Multiple Objects of Interest from Multiple Images: dISCOVER . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Devi Parikh and Tsuhan Chen

487

Poster Session 5: Stereo, Range and 3D Fast 3-D Interpretation from Monocular Image Sequences on Large Motion Fields . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jong-Sung Kim and Ki-Sang Hong

497

Color-Stripe Structured Light Robust to Surface Color and Discontinuity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Kwang Hee Lee, Changsoo Je, and Sang Wook Lee

507

Stereo Vision Enabling Precise Border Localization Within a Scanline Optimization Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Stefano Mattoccia, Federico Tombari, and Luigi Di Stefano

517

Three Dimensional Position Measurement for Maxillofacial Surgery by Stereo X-Ray Images . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Naoya Ohta, Kenji Mogi, and Yoshiki Nakasone

528

Stereo Total Absolute Gaussian Curvature for Stereo Prior . . . . . . . . . . . . . . . . . . Hiroshi Ishikawa

537

Fast Optimal Three View Triangulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Martin Byr¨ od, Klas Josephson, and Kalle ˚ Astr¨ om

549

Stereo Matching Using Population-Based MCMC . . . . . . . . . . . . . . . . . . . . Joonyoung Park, Wonsik Kim, and Kyoung Mu Lee

560

Dense 3D Reconstruction of Specular and Transparent Objects Using Stereo Cameras and Phase-Shift Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . Masaki Yamazaki, Sho Iwata, and Gang Xu

570

Image and Video Processing Identifying Foreground from Multiple Images . . . . . . . . . . . . . . . . . . . . . . . Wonwoo Lee, Woontack Woo, and Edmond Boyer

580

XVIII

Table of Contents – Part II

Image and Video Matting with Membership Propagation . . . . . . . . . . . . . . Weiwei Du and Kiichi Urahama

590

Temporal Priors for Novel Video Synthesis . . . . . . . . . . . . . . . . . . . . . . . . . . Ali Shahrokni, Oliver Woodford, and Ian Reid

601

Content-Based Image Retrieval by Indexing Random Subwindows with Randomized Trees . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Rapha¨el Mar´ee, Pierre Geurts, and Louis Wehenkel

611

Poster Session 6: Face/Gesture/Action Detection and Recognition Analyzing Facial Expression by Fusing Manifolds . . . . . . . . . . . . . . . . . . . . Wen-Yan Chang, Chu-Song Chen, and Yi-Ping Hung

621

A Novel Multi-stage Classifier for Face Recognition . . . . . . . . . . . . . . . . . . . Chen-Hui Kuo, Jiann-Der Lee, and Tung-Jung Chan

631

Discriminant Clustering Embedding for Face Recognition with Image Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Youdong Zhao, Shuang Xu, and Yunde Jia

641

Privacy Preserving: Hiding a Face in a Face . . . . . . . . . . . . . . . . . . . . . . . . . Xiaoyi Yu and Noboru Babaguchi

651

Face Mosaicing for Pose Robust Video-Based Recognition . . . . . . . . . . . . Xiaoming Liu and Tsuhan Chen

662

Face Recognition by Using Elongated Local Binary Patterns with Average Maximum Distance Gradient Magnitude . . . . . . . . . . . . . . . . . . . . Shu Liao and Albert C.S. Chung

672

An Adaptive Nonparametric Discriminant Analysis Method and Its Application to Face Recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Liang Huang, Yong Ma, Yoshihisa Ijiri, Shihong Lao, Masato Kawade, and Yuming Zhao

680

Discriminating 3D Faces by Statistics of Depth Differences . . . . . . . . . . . . Yonggang Huang, Yunhong Wang, and Tieniu Tan

690

Kernel Discriminant Analysis Based on Canonical Differences for Face Recognition in Image Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Wen-Sheng Vincnent Chu, Ju-Chin Chen, and Jenn-Jier James Lien

700

Person-Similarity Weighted Feature for Expression Recognition . . . . . . . . Huachun Tan and Yu-Jin Zhang

712

Table of Contents – Part II

Converting Thermal Infrared Face Images into Normal Gray-Level Images . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Mingsong Dou, Chao Zhang, Pengwei Hao, and Jun Li Recognition of Digital Images of the Human Face at Ultra Low Resolution Via Illumination Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jen-Mei Chang, Michael Kirby, Holger Kley, Chris Peterson, Bruce Draper, and J. Ross Beveridge

XIX

722

733

Poster Session 6: Math for Vision Crystal Vision-Applications of Point Groups in Computer Vision . . . . . . . Reiner Lenz

744

On the Critical Point of Gradient Vector Flow Snake . . . . . . . . . . . . . . . . . Yuanquan Wang, Jia Liang, and Yunde Jia

754

A Fast and Noise-Tolerant Method for Positioning Centers of Spiraling and Circulating Vector Fields . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ka Yan Wong and Chi Lap Yip

764

Interpolation Between Eigenspaces Using Rotation in Multiple Dimensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Tomokazu Takahashi, Lina, Ichiro Ide, Yoshito Mekada, and Hiroshi Murase Conic Fitting Using the Geometric Distance . . . . . . . . . . . . . . . . . . . . . . . . . Peter Sturm and Pau Gargallo

774

784

Poster Session 6: Segmentation and Classification Efficiently Solving the Fractional Trust Region Problem . . . . . . . . . . . . . . . Anders P. Eriksson, Carl Olsson, and Fredrik Kahl

796

Image Segmentation Using Iterated Graph Cuts Based on Multi-scale Smoothing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Tomoyuki Nagahashi, Hironobu Fujiyoshi, and Takeo Kanade

806

Backward Segmentation and Region Fitting for Geometrical Visibility Range Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Erwan Bigorgne and Jean-Philippe Tarel

817

Image Segmentation Using Co-EM Strategy . . . . . . . . . . . . . . . . . . . . . . . . . Zhenglong Li, Jian Cheng, Qingshan Liu, and Hanqing Lu

827

Co-segmentation of Image Pairs with Quadratic Global Constraint in MRFs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yadong Mu and Bingfeng Zhou

837

XX

Table of Contents – Part II

Shape from X Shape Reconstruction from Cast Shadows Using Coplanarities and Metric Constraints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Hiroshi Kawasaki and Ryo Furukawa

847

Evolving Measurement Regions for Depth from Defocus . . . . . . . . . . . . . . . Scott McCloskey, Michael Langer, and Kaleem Siddiqi

858

A New Framework for Grayscale and Colour Non-Lambertian Shape-from-shading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . William A.P. Smith and Edwin R. Hancock

869

Face A Regularized Approach to Feature Selection for Face Detection . . . . . . . Augusto Destrero, Christine De Mol, Francesca Odone, and Alessandro Verri

881

Iris Tracking and Regeneration for Improving Nonverbal Interface . . . . . . Takuma Funahashi, Takayuki Fujiwara, and Hiroyasu Koshimizu

891

Face Mis-alignment Analysis by Multiple-Instance Subspace . . . . . . . . . . . Zhiguo Li, Qingshan Liu, and Dimitris Metaxas

901

Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

911

Palmprint Recognition Under Unconstrained Scenes Yufei Han, Zhenan Sun, Fei Wang, and Tieniu Tan Center for Biometrics and Security Research National Laboratory of Pattern Recognition, Institute of Automation Chinese Acdamey of Sciences P.O.Box 2728, Beijing, P.R. China, 100080 {yfhan,znsun,fwang,tnt}@nlpr.ia.ac.cn

Abstract. This paper presents a novel real-time palmprint recognition system for cooperative user applications. This system is the first one achieving noncontact capturing and recognizing palmprint images under unconstrained scenes. Its novelties can be described in two aspects. The first is a novel design of image capturing device. The hardware can reduce influences of background objects and segment out hand regions efficiently. The second is a process of automatic hand detection and fast palmprint alignment, which aims to obtain normalized palmprint images for subsequent feature extraction. The palmprint recognition algorithm used in the system is based on accurate ordinal palmprint representation. By integrating power of the novel imaging device, the palmprint preprocessing approach and the palmprint recognition engine, the proposed system provides a friendly user interface and achieves a good performance under unconstrained scenes simultaneously.

1 Introduction Biometrics technology identifies different people by their physiological and behavioral differences. Compared with traditional security authentication approaches, such as key or password, biometrics is more accurate, dependable and difficult to be stolen or faked. In the family of biometrics, palmprint is a novel but promising member. Large region of palm supplies plenty of line patterns which can be easily captured in a low resolution palmprint image. Based on those line patterns, palmprint recognition can achieve a high accuracy of identity authentication. In previous work, there are several successful recognition systems proposed for practical use of palmprint based identity check [1][2][3], and the best-known is developed by Zhang et al [1]. During image capturing, users are required to place hands on the plate with pegs controlling displacement of hands. High quality palmprint images are then captured by a CCD camera fixed in a semi-closed environment with uniform light condition. To alignment captured palmprint images, a preprocessing algorithm [2] is adopted to correct rotation of those images and crop square ROI (regions of interests) with the same size. Detail about this system can be found in [2]. Besides, Connie et al proposed a peg-free palmprint recognition system [3], which captures palmprint images by an optical scanner. Subjects are allowed to place their hand more freely on the platform of the scanner without pegs. As a result, Y. Yagi et al. (Eds.): ACCV 2007, Part II, LNCS 4844, pp. 1–11, 2007. © Springer-Verlag Berlin Heidelberg 2007

2

Y. Han et al.

palmprint images with different sizes, translations and rotation angles are obtained. Similar as in [2], an alignment process is involved to obtain normalized ROI images. However, efficient as they are, there are still some limitations. Firstly, some users may feel uncomfortable with pegs to restrict hands during capturing images. Secondly, even without pegs, subjects’ hands are required to contact plates of devices or platforms of scanners, which is not hygienic enough. Thirdly, semi-closed image capturing devices usually increase volume of recognition systems, which makes them not convenient for portable use. Thus, it’s necessary to improve design of the HCI(human-computer interface), in order to make the whole system easy-to-use. Recently, active near infrared imagery (NIR) technology has received more and more attention in face detection and recognition, as seen in [4]. Given a near infrared light source shining objects in front of cameras, intensity of reflected NIR light is attenuated at a large scale with distance between objects and the light source increasing. This property provides us a promising solution to eliminate affection of backgrounds when palmprint images are captured under unconstrained scenes. Based on the technology, in this paper, we propose a novel real-time palmprint recognition system. It’s designed to localize and obtain normalized palmprint images under clutter scenes conveniently. The main contributions are as followings: First, we present a novel design of portable image capturing device, which mainly consists of two parallel placed web cameras. One is used for active near infrared imagery to localize hand regions. The other one captures corresponding palmprint images in visible light, preparing for further feature extraction. Second, we present a novel palmprint preprocessing algorithm, utilizing color and shape information of hands for fast and effective hand region detection, rotation correction and localization of central palm region. So far as we know, there is no similar work reported in previous literatures. The rest of paper is organized as follows: Section 2 presents a description of the whole architecture of the recognition system. In Section 3, the design of human computer interface of the system is described in detail. Section 4 introduces ordinal palmprint representation briefly. Section 5 evaluates the performance of the system. Finally, in Section 6, we conclude the whole paper.

2 System Overview We adopt a common PC with Intel Pentium4 3.0Ghz and 1G RAM as the computation platform. Based on it, the recognition system is implemented using Microsoft Visual C++ 6.0. It consists of five main modules, as shown in Fig.1. After starting the system, users are required to open their hands in a natural manner and place palm regions toward the imaging device at a certain distance between 35 cm and 50 cm from cameras. Surfaces of palms are approximately orthogonal to the optical axis of cameras. In-plane rotation of hands is restricted between -15 degree to 15 degree deviated from vertical orientation. The imaging device then captures two images for each hand by two cameras placed in parallel respectively. One is a NIR hand image with active NIR lighting, the other is a color hand image with background objects, obtained with normal environment lighting condition. Both of them contain complete hand region, see in Fig.2. After that, an efficient palmprint preprocessing

Palmprint Recognition Under Unconstrained Scenes

3

algorithm is performed on the two captured images to obtain one normalized palmprint image quickly, which makes use of both shape and skin color information of hands. Finally, robust palmprint feature templates are extracted from the normalized image using the ordinal code based approach [5]. Fast hamming distance calculation is applied to measure dissimilarity between two feature templates. An example of the whole recognition process could be seen in the supplementary video of this paper.

3 Smart Human-Computer Interface HCI of the system mainly consists of two parts, image capturing hardware and palmprint preprocessing procedure, as shown in Fig.1. Considering a hand image captured under an unconstrained scene, unlike those captured by devices in [1][2][3], there are not only a hand region containing palmprint patterns, but also background objects of different shapes, colors and positions, as denoted in Fig.2. Even within the hand, there still exits rotation, scale variation and translation of palmprint patterns due to different hand displacements. Thus, before further palmprint feature encoding, HCI should localize the candidate hand region and extract a normalized ROI (region of interest), which contains palmprint features without much geometric deformations. 3.1 Image Capturing Device Before palmprint alignment, it is necessary to segment hand regions from unconstrained scenes. This problem could be solved by background modeling and subtraction or labeling skin color region. However, both methods suffer from unconstrained backgrounds or varying light conditions. Our design of imaging device aims to solve the problem in a sensor level, in order to localize foreground hand regions more robustly by simple image binarization. The appearance of the image capturing device is shown in Fig.2(a). This device has two common CMOS web cameras placed in parallel. We mount near infrared (NIR) light-emitting diodes on the device evenly distributed around one camera, similar as in [4], so as to provide straight and uniform NIR lighting. Near infrared light emit by those LEDs have a wavelength of 850 nm. In a further step, we make use of a band pass optical filter fixed on the camera lens to cut off lights with all the other wavelengths except 850nm. Most of environment lights are cut off because their wavelengths are less than 700nm. Thus, lights received by the camera only consist of reflected NIR LED lights and NIR components in environment lights, such as lamp light and sunlight, which are much weaken than the NIR LED lights. Notably, intensities of reflected NIR LED lights are in the inverse proportion to high-order terms of the distance between object and the camera. Therefore, assuming a hand is the nearest one among all objects in front of the camera during image capturing, intensities of the hand region in the corresponding NIR image should be much larger than backgrounds. As a result, we can further segment out the hand region and eliminate background by fast image binarization, as denoted in Fig.2(b). The other

4

Y. Han et al.

camera in the device captures color scene images, obtaining clear palmprint patterns and reserving color information of hands. An optical filter is fixed on the lens of this camera to filter out infrared components in the reflected lights, which is applied widely in digital camera to avoid red-eye. The two cameras work simultaneously. In our device, resolution of both cameras is 640*480. Fig.2(b) lists a pair of example images, captured by the two cameras at the same time. The upper one is the color image. The bottom one is the NIR image. The segmentation result is shown in the upper row of Fig.2(c). In order to focus on hand regions with a proper scale in further processing, we adopt a scale selection on binary segmentation results to choose candidate foreground regions. The criterion of selection grounds on a fact that area of a hand region in a NIR image is larger if the hand is nearer to the camera. We label all connected binary foreground after segmentation and calculate area of each connected component, then choose those labeled regions with their areas varying in a predefined narrow range as the candidate foreground regions, like the white region shown in the image at the bottom of Fig.2(c).

Fig. 1. Flowcharts of the system

Fig. 2. (a) Image capturing device (b) Pair-wise color and NIR image (c) Segmented fore ground and candidate foreground region selection

Palmprint Recognition Under Unconstrained Scenes

5

3.2 Automated Hand Detection Hand detection is posed as two-class problem of classifying the input shape pattern into hand-like and non-hand class. In our system, a cascade classifier is trained to detect hand regions in binary foregrounds, based on works reported in [6]. In [6], Eng-Jon Ong et al makes use of such classifier to classify different hand gestures. In our application, the cascade classifier should be competent for two tasks. Firstly, it should differentiate shape of open hand from all the other kinds of shapes. Secondly, it should reject open hands with in-plane rotation angle deviating out of the restricted range. To achieve this goals, we construct a positive dataset containing binary open left hands at first, such as illustrated in Fig.3(a). In order to make the classifier tolerate certain in-plane rotation, the dataset consists of left hands with seven discrete rotation angles, sampled every 5 degree from -15 degree to 15 degree deviated from vertical orientation, a part of those binary hands are collected from [11]. For each angle, there are about 800 hand images with slight postures of fingers, also shown in Fig.3(a). Before training, all positive data are normalized into 50*35 images. The negative dataset contains two parts. One consists of binary images containing nonhand objects, such as human head, turtles and cars, partly from [10]. The other contains left hands with rotation angle out of the restricted range and right hands with a variety of displacements. There are totally more than 60,000 negative images. Fig.3(b) shows example negative images. Based on those training data, we use Float AdaBoost algorithm to select most efficient Haar features to construct the cascade classifier, same as in [6]. Fig.3(c) shows the most six efficient Haar features obtained after training. We see that they represent discriminative shape features of left open hand. During detection, rather than exhaustive search across all positions and scales in [6], we perform the classifier directly around the candidate binary foreground regions

Fig. 3. (a) Positive training data (b) Negative training data (c) Learned efficient Haar features (d) Detected hand region

6

Y. Han et al.

to search for open left hands with a certain scale. Therefore, we can detect different hands with a relative stable scale, which reduces influence of scale variations on palmprint patterns. Considering mirror symmetry between left and right hands, to detect right hands, we just perform symmetry transform on the images and apply the classifier by the same way on the flipped images. Fig.3(d) shows results of detection. Obtaining detected hand, all the other non-hand connected regions are removed from binary hand images. The whole detection can be finished within 20 ms. 3.3 Palmprint Image Alignment Palmprint alignment procedure eliminates rotation and translation of palmprint patterns, in order to obtain normalized ROI. Most alignment algorithms calculate rotation angles of hands by localizing key contour points in gaps between fingers [2][3]. However, in our application, different finger displacements may change local contours and make it difficult detect gap regions, as denoted in Fig.4. To solve this problem, we adopt a fast rotation angle estimation based on moments of hand shape. Given R is the detected hand region in a binary foreground image. Its orientation θ can be estimated by calculating its moments [7]: θ=

1

arctan(

2

2μ1,1

μ 2,0 − μ 0,2

(1)

)

μ p ,q (p,q=0,1….) is (p,q) order central moments, which is represented as :

μ p ,q = ∑∑ ( x − x

y

1

1

x) p ( y − ∑∑ y ) q , ( x, y ) ∈ R ∑∑ N N x

y

x

(2)

y

Compared with key point detection, moments are calculated based on the whole hand region rather than only contour points. Thus, it is more robust to local changes in contours. To reduce computation cost, the original binary image is down-sampled to a 160*120 one. Those moments are then calculated on the down-sampled version. After obtaining rotation angle θ , the hand region is rotated by - θ degree to get vertical oriented hands, see in Fig.4. Simultaneously, the corresponding color image is also rotated by - θ , in order to make sure consistency of hand orientations in both two images. In a further step, we locate central palm region in a vertical oriented open hand by analyzing difference of connectivity between the palm region and the finger region. Although shape and size of hands vary a lot, a palm region of each hand should be like a rectangle. Compared with it, stretched fingers don’t form a connective region as palm. Based on this property, we employ an erosion operation on the binary hand image to remove finger regions. The basic idea behind this operation is run length code of binary image. We perform a raster scanning on each row to calculate the maximum length W of connective sequences in the row. Any row with its W less than threshold K1 should be eroded. After all rows are scanned, a same operation is performed on each column. As a result, columns with their maximum length W less than K2 are removed. Finally, a rectangular palm region is cropped from the hand. Coordinates (xp,yp )of its central point is derived as localization result. In order to

Palmprint Recognition Under Unconstrained Scenes

7

cope with varying sizes of different hands, we choose values of K1 and K2 adaptively. Before row erosion, distance between each point in the hand region and nearest edge point is calculated by a fast distance transform. The central point of hand is defined as the one with the largest distance value. Assuming A is the maximum length of connective sequences in the row passing through the central point, K1 is defined as follows: K1 = A * p%

(3)

p is a pre-defined threshold. K2 is defined in the same way: K2 = B * q%

(4)

B is the maximum length of connective sequences in the column passing through the central point after row erosion. q is another pre-defined threshold. Compared with fixed value, adaptive K1 and K2 lead to more accurate location of central palm regions, as denoted in Fig.5(b). Fig.5(a) denotes the whole procedure of erosion. Due to visual disparity between two cameras in the imaging device, we can not use (xp,yp ) to localize ROI in corresponding color images directly. Although visual disparity can be estimated by a process of 3D scene reconstruction, this approach may lead to much computation burden on the system. Instead, we apply a fast correspondence estimation based on template matching. Assuming C is a color hand image after rotation correction, we convert C into a binary image M by setting all pixels with skin color to 1, based on the probability distribution model of skin color in RGB space [8]. Given the binary version of the corresponding NIR image, with a hand region S locating at (xn,yn), a template matching is conducted as in Eq.5, also as denoted in Fig.6:

f ( m, n ) =

∑∑ [ M ( x + m, y + n) ⊕ S ( x, y )], ( x, y ) ∈ S x

(5)

y

⊕ is bitwise AND operator. f(m,n) is a matching energy function. (m,n) is a candidate position of the template. The optimal displacement (xo,yo) of hand shape S in M is defined as the candidate position where the matching energy achieves its maximum. The central point (xc,yc) of palm region in C can be estimated by following equations:

xc = x p + xo − xn yc = y p + yo − y n

Fig. 4. Rotation correction

(6)

8

Y. Han et al.

Fig. 5. (a) Erosion procedure (b) Erosion with fixed and adaptive thresholds

With (xc,yc) as its center, one 128*128 sub-image is cropped from C as ROI, which is then converted to gray scale image for feature extraction.

Fig. 6. Translation estimation

4 Ordinal Palmprint Representation In previous work, the orthogonal line ordinal feature (OLOF) [5] provides a compact and accurate representation of negative line features in palmprints. The orthogonal line ordinal filter [5] F(x,y,θ) is designed as follows: F ( x, y , θ ) = G ( x, y , θ ) − G ( x, y , θ + π / 2)

(7)

Palmprint Recognition Under Unconstrained Scenes

G ( x, y , θ ) = exp[ −(

x cos θ + y sin θ

δx

) −( 2

− x sin θ + y cos θ

δy

2

) ]

9

(8)

G(x,y,θ) is a 2D anisotropic Gaussian filter, and θ is the orientation of the Gaussian filter. The ratio between δx and δy is set to be larger than 3, in order to obtain a weighted average of a line-like region. In each local region in a palmprint image,

three such ordinal filters, with orientations of 0, π/6, π/3 are used in convolution process on the region. The filtering result is then encoded into 1 or 0 according to whether its sign is positive or negative. Thousands of ordinal codes are concatenated into a feature template. Similarity between two feature templates is measured by a normalized hamming distance, which ranges between 0 and 1. Further details can be found in [5].

5 System Evaluation Performance of the system is evaluated in terms of verification rate [9], which is obtained through one-to-one image matching. We collect 1200 normalized palmprint ROI images from 60 subjects using the system, with 10 images for each hand. Fig.7 illustrates six examples of ROI images. During the test, there are totally 5,400 intraclass comparisons and 714,000 inter-class comparisons. Although recognition accuracy of the system lies on effectiveness of both alignment procedure of HCI and the palmprint recognition engine, the latter is not the focus of this paper. Thus we don’t involve performance comparisons between the ordinal code and other state-ofthe-art approaches. Fig.8 denotes distributions of genuine and imposter. Fig.9 shows corresponding ROC curve. The equal error rate [9] of the verification test is 0.54%. From experimental results, we can see that ROI regions obtained by the system are suitable for palmprint feature extraction and recognition. Besides, we also record time cost for obtaining one normalized palmprint image using the system. It includes time for image capturing, hand detection and palmprint alignment. The average time cost is 1.2 seconds. Thus, our system can be competent for point-of-sale identity check.

Fig. 7. Six examples of ROI images

10

Y. Han et al.

Fig. 8. Distributions of genuine and imposter

Fig. 9. ROC curve of the verification test

6 Conclusion In this paper, we have proposed a novel palmprint recognition system for cooperative user applications, which achieves a real-time non-contact palmprint image capturing and recognition directly under unconstrained scenes. Through design of the system, we aim to provide more convenient human-computer interface and reduce restriction on users during palmprint based identity check. The core of HCI in the system consists of a binocular image device and a novel palmprint preprocessing algorithm. The former delivers a fast hand region segmentation based on NIR imaging technology. The latter extracts normalized ROI from hand regions efficiently based on shape and color information of human hands. Benefiting further from the powerful recognition engine, the proposed system achieves accurate recognition and convenient use at the same time. As far as we know, this is the first attempt to solve the problem of obtaining normalized palmprint images directly from clutter backgrounds. However, accurate palmprint alignment has not been well addressed in the proposed system. In our future work, it’s an important issue to improve the performance of the system by reducing alignment error in a further step. In addition,

Palmprint Recognition Under Unconstrained Scenes

11

we should improve the imaging device to deal with influence of NIR component in environment light, which varies much in practical use. Acknowledgments. This work is funded by research grants from the National Basic Research Program (Grant No.2004CB318110), the Natural Science Foundation of China (Grant No.60335010, 60121302, 60275003, 60332010, 69825105,60605008) and the Chinese Academy of Sciences.

References 1. Zhang, D., Kong, W.K., You, J., Wong, M.: Online Palmprint Identification. IEEE Trans on PAMI 25(9), 1041–1050 (2003) 2. Kong, W.K.: Using Texture Analysis in Biometric Technology for Personal Identification, MPhil Thesis, http://pami.uwaterloo.ca/ cswkkong/Sub_Page/Publications.htm 3. Connie, T., Jin, A.T.B., Ong, M.G.K., Ling, D.N.C.: Automated palmprint recognition system. Image and Vision Computing 23, 501–515 (2005) 4. li, S.Z., Chu, R.F., Liao, S.C., Zhang, L.: Illumination invariant Face Recognition using Near- Infrared Images. IEEE Trans on PAMI 29(4), 627–639 (2007) 5. Sun, Z.N., Tan, T.N., Wang, Y.H., Li, S.Z.: Ordinal Palmprint Representation for Personal Identification. Proc. of IEEE CVPR 2005 1, 279–284 (2005) 6. Ong, E., Bowden, R.: A Boosted Classifier Tree for Hand Shape Detection. In: Proc. of International Conference on Automatic Face and Gesture Recognition, pp. 889–894 (2004) 7. Jain, A.K.: Fundamentals of Digital Image Processing, vol. 07458, p. 392. Prentice Hall, Upper Saddle River, NJ 8. Jones, M.J., Rehg, J.M.: Statistical Color Models with Application to Skin Color Detection. International Journal of Computer Vision 46(1), 81–96 (2002) 9. Daugman, J., Williams, G.: A Proposed Standard for Biometric Decidability. In: Proc. of CardTech/SecureTech Conference, Atlanta, GA, pp. 223–234 (1996) 10. http://www.cis.temple.edu/ latecki/TestData mpeg7shapeB.tar.gz

11. UST Hand Image database, http://visgraph.cs.ust.hk/Biometrics/Visgraph_web/ index.html

Comparative Studies on Multispectral Palm Image Fusion for Biometrics Ying Hao, Zhenan Sun, and Tieniu Tan National Laboratory of Pattern Recognition, Institute of Automation, CAS

Abstract. Hand biometrics, including fingerprint, palmprint, hand geometry and hand vein pattern, have obtained extensive attention in recent years. Physiologically, skin is a complex multi-layered tissue consisting of various types of components. Optical research suggests that different components appear when the skin is illuminated with light sources of different wavelengths. This motivates us to extend the capability of camera by integrating information from multispectral palm images to a composite representation that conveys richer and denser pattern for recognition. Besides, usability and security of the whole system might be boosted at the same time. In this paper, comparative study of several pixel level multispectral palm image fusion approaches is conducted and several well-established criteria are utilized as objective fusion quality evaluation measure. Among others, Curvelet transform is found to perform best in preserving discriminative patterns from multispectral palm images.

1

Introduction

Hand, as a tool for human to percept and reconstruct surrounding environment, is most used among body parts in our daily life. Due to its high acceptance by the human beings, its prevalence in the field of biometrics is no surprising. Fingerprint[1], hand geometry[2], palmprint[7][8], palm-dorsa vein pattern[3], finger vein[4] and palm vein[5] are all good examples of hand biometric patterns. These modalities have been explored by earlier researchers and can be divided into three categories: - Skin surface based modality. Examples are fingerprint and palmprint. Both traits explore information from the surface of skin and have received extensive attention. Both of them are recognized as having the potential of being used in highly security scenario; - Internal structure based modality, which extracts information from vein structure deep under the surface for recognition. Although new in biometric family, the high constancy and uniqueness of vein structure make this category more and more active nowadays[3][9]; - Global structure based modality. The only example of this category is hand geometry. Hand geometry is a good choice for small scale applications thanks to its high performance-price ratio. Y. Yagi et al. (Eds.): ACCV 2007, Part II, LNCS 4844, pp. 12–21, 2007. c Springer-Verlag Berlin Heidelberg 2007 

Comparative Studies on Multispectral Palm Image Fusion for Biometrics

13

No matter which category of modality one chooses to work on, a closer look on the skin appearance benefits. Physiologically, human skin consists of many components, such as Cells, fibers, veins and nerves, and they give skin a multilayered structure. At the outermost layer, numerous fine furrows, hair and pores are scattered over the surface of skin, while veins, capillaries and nerves form a vast network inside[6]. Optical study has demonstrated that light with longer wavelength tends to penetrate the skin more deeply, for example, near infrared light from 600nm to 1000nm typically penetrates the skin to about 1-3 mm. Therefore different visual contents, with different optical properties, are detected with incident light of different wavelengths[11]. The uniqueness of human skin, including its micro, meso and macro structures, is a product of random factors during embryonic development. Enlighten by the success and fast development of above mentioned hand based biometrics, each of them reflecting only one aspect of hand, we believe that best potential of biometric feature in the hand region is yet to be discovered. The purpose of this work is to exploit the correlative and complementary nature of multispectral hand images for image enhancement, filtering and fusion. Take palmprint and vein for example, the common characteristics of two modalities is that they both utilize moderate resolution hand imagery and they share similar discriminative information: line-like pattern. On the other hand, the intrinsic physiological nature makes the two traits holding distinctive advantages and disadvantages. More precisely speaking, palmprint is related to outermost skin pattern, therefore, its appearance is sensitive to illumination condition, aging, skin disease and abrasion etc. In contrast, hand vein pattern, as an interior structure, is robust to the above mentioned external factors. However, vein image quality varies dramatically across the population and in case of blood vessel constriction resulting from extremely cold weather. Several advantages can be obtained by fusing the two spectral hand images. First of all, a more user-friendly system can be developed by alternatively combining the two traits or choosing the appropriate one for recognition according to corresponding imaging quality; secondly, forgery is much more difficult for such an intelligent system and hence the system is more secure; and finally, the recognition performance might be boosted. In this work, we designed a device to automatically and periodically capture visible and near infrared spectral images. Two sets of lights are turned on in turn so that palmprint and vein images are captured. With the images at hand, we validated the idea of image fusion. Several pixel level image fusion approaches are performed to combine the original images to a composite one, which is expected to convey more information than its inputs. The rest of this paper is organized as follows. The hardware design of image capture device is presented in Section 2, followed by a brief introduction of the four methods in Section 3. The proposed fusion scheme is introduced in Section 4 and Section 5 includes experimental results as well as performance evaluation. Finally, conclusion and discussion are presented in Section 6.

14

2

Y. Hao, Z. Sun, and T. Tan

Hardware Design

Fig. 1 illustrates the principal design of the device we developed to capture images in both visible (400-700nm) and near infrared (800-1000nm) spectra. The device works under a sheltered environment and the light sources are carefully arranged so that the palm region is evenly illuminated. An infrared sensitive CCD camera is fixed at the bottom of the inner encloser and connected to a computer via USB interface. An integrated circuit plate is mounted near the camera for illumination and different combinations of light wavelengths can be accomplished by replacing the circuit plate. By default, the two sets of lights are turned on in turn so that only expected layer of hand appears to camera. When illuminated with visible light, image of hand skin surface, namely, the palmprint is stored. while when NIR light is on, deeper structure as well as parts of dominant surface features, for example the principal lines, are captured. Manual control of the two lights is also allowed by sending computer instruction to the device. A pair of images captured using the device is shown in Fig. 2(a)(b), where (a) is palmprint image and (b) is vein image. It is obvious that the two images emphasize on quite different components of hand.

Fig. 1. Multispectral palm image capture device, where LEDs of two wavelengths is controlled by computer

3

Image Fusion and Multiscale Decomposition

The concept of image fusion refers to the idea of integrating information from different images for better visual or computational perception. Image fusion sometimes refers to pixel-level fusion, while a broad sense definition also includes feature-level and matching score level fusion. In this work, we focus on pixel level fusion because it features minimum information loss. The key issue in image fusion is to faithfully preserve important information while suppress noises. Discriminative information in palmprint and vein, or more

Comparative Studies on Multispectral Palm Image Fusion for Biometrics

15

specifically principal lines, wrinkle lines, ridges and blood vessels, all takes form of line-like patterns. Therefore, the essential goal is to maximally preserve these patterns. In the field of pixel-level fusion, multiscale decomposition (MSD), such as pyramid decomposition and wavelet decomposition, is often applied because it typically provides better spatial and spectral localization of image information and such decorrelation between pyramid subbands allows for a more reliable feature selection[19]. The methods utilized in this paper also follow this direction of research, while evaluation measures are applied to feature level representation rather than intensity level to accommodate the context of biometrics. We selected four multiscale decomposition methods for comparison. Gradient pyramid can obtained by applying four directional gradient operators to each level of Gaussian pyramid. The four operators correspond to horizontal, vertical and two diagonal directions respectively. Therefore, image features are indexed according to their orientations and scales. Morphological pyramid can be constructed by successive procedure of morphological filtering and sub-sampling. Morphological filters, such as open and close are designed to preserve edges and shapes of objects, which make this approach suitable for the task presented here. Shift invariant digital wavelet transform is a method proposed to overcome the wavy effect normally observed in traditional wavelet transform based fusion. It is accomplished by an over-complete version of wavelet basis and the downsampling process is taken place by dilated analysis filters. In our implementation, Haar wavelet is chosen and decomposition level for the above mentioned three methods is three. Curvelet transform is a bit more complex multiscale transform[12][13][15][14] and is designed to efficiently represent edges and other singularities along curves. Unlike wavelet transform, it has directional parameters and its coefficients have a high degree of directional specificity. Therefore, large coefficients in transform space suggests strong lines on original image. These methods are not new in the field of image fusion[16][17][18][19][21], . However, earlier researchers either focus on remote sensing applications, which involves tradeoff between spectral and spatial resolution, or pursue general purpose image fusion scheme. This work is the one of the first applications that adopt and compared them in the context of hand based biometrics.

4

Proposed Fusion Method

Our fusion method is composed of two steps, namely, a preprocessing step to adjust dynamic ranges and remove noises from vein images and a fusion step to combine information from visible image and infrared image. 4.1

Preprocessing

When illuminated with visible light, images of skin fine structures are captured. Contrast to the behavior in visible wavelength, cameras usually have a much

16

Y. Hao, Z. Sun, and T. Tan

lower sensitivity to infrared lights. Therefore, cameras tend to work at a low luminance circumstance and AGC(Auto Gain Control) feature takes effect to maintain the output level. This procedure amplifies the signal and noises at the same time, producing noisy IR images. The first stage of preprocessing is to distinguish between the two spectra. The relative large difference between camera responses to the two wavelengths makes NIR images constantly darker than visible image, in consequence the separation is accomplished simply via an average intensity comparison. Followed is a normalization step to modify the dynamic range of vein image so that the mean and standard deviation of vein image equals to that of the palmprint image. The underlying reason is that equal dynamic range across source images helps to produce comparable coefficients in transform domain. Finally, bilateral filtering is undertaken to eliminate noises from infrared images. Bilateral filtering is a non-iterative scheme for edge-preserving smoothing [10]. The response at x is defined as an weighted average of similar and nearby pixels, where the weighting function corresponds to a range filter while the domain component closely related to a similarity function between current pixel x and its neighbors. Therefore, desired behavior is achieved both in smooth regions and boundaries. 4.2

Fusion Scheme

According to the generic framework proposed by Zhang et. al.[19], image fusion schemes are composed of (a) Multiscale decomposition, which maps source intensity images to more efficient representations; (b) Activity measurement that determines the quality of each input; (c) Coefficient grouping method to determine whether or not cross scale correlation is considered; (d) Coefficient combining method where a weighted sum of source representations is calculated and finally (e) Consistency verification to ensure neighboring coefficients are calculated in similar manner. As a domain specific fusion scheme, the methods applied in this work can be regarded as examples of this framework. For each of the multiscale decomposition methods mentioned in Section 3, the following scheme is applied: Activity measure - A coefficient-based activity measure is applied to each coefficient, which means that the absolute value of each coefficient is regarded as the activity measure of corresponding scale, position and sometimes orientation; Coefficient combining method - Generally speaking, no matter what kind of linear combination of coefficient is adopted, the basic calculation is weighted sum. We utilized the popular scheme proposed by Burt[20] to high frequency coefficient and average to base band approximation. Consistency verification - Consistency verification is conducted in a blockwise fashion and majority filter is applied in local window of 3 by 3 in case that choose max operation is taken in coefficient combination.

Comparative Studies on Multispectral Palm Image Fusion for Biometrics

5

17

Experimental Results

To evaluate the proposed fusion scheme, we collected a database from 7 subjects. Three pairs of images are captured for both hands, producing a total number of 84 images. 5.1

Subjective Fusion Quality Evaluation

The proposed scheme is applied to each pair of visible and NIR images and the resulting fused images by the four decomposition methods are subjectively examined. Fig. 2 demonstrates such an example. Morphological pyramid, although produces most obvious vein pattern on fused images, sometimes introduced artifacts. Other three methods seem to perform similarly to human eyes, thus objective fusion quality evaluation is necessary for more detailed comparison.

(a) visible image

(b) infrared image

(c) fused image with gradient pyramid (d) fused image with morphological pyramid

(e) fused image with shift-invariant DWT

(f) fused image with Curvelet transform

Fig. 2. Palmprint and vein pattern images captured using the self-designed device as well as the fused images

18

5.2

Y. Hao, Z. Sun, and T. Tan

Objective Fusion Quality Evaluation

Many fusion quality evaluation measures have been proposed[22][23] and we choose four of them for our application. Root Mean Square Error(RMSE) between input image A and fused image F is originally defined in Eq. 1.  N N 2 i=1 j=1 [A(i, j) − F (i, j)] RM SEAF = (1) N2 Mutual information(MI) statistically tells how much information fused image F tells about the input image A. Suppose that pA (x), pF (y) and pAF (x, y) denote marginal distribution from A, F and joint distribution between A and F respectively. Mutual information between A and F is defined as Eq. 2. M IAF =

 x

y

pAF (x, y)log

pAF (x, y) pA (x)pF (y)

(2)

Universal image quality index(UIQI) was proposed to evaluate the similarity between two images, and is defined in Eq. 3. The three components of UIQI denote correlation coefficient, closeness of mean luminance and contrast similarity of two images or image blocks A and F respectively. U IQIAF =

σAF 2μA μF 2σA σF 4σAF μA μF · · 2 = 2 2 + σ2 ) σA σF μ2A + μ2F σA + σF2 (μA + μ2F ) · (σA F

(3)

The above mentioned general purpose criteria are usually applied to intensity image. However, in order to predict the performance of proposed method in the context of biometrics, we apply these measures to feature level representation. In the field of palmprint recognition, the best algorithms reported in literature are those based on binary textual features[7][8]. These methods seek to represent line-like patterns and have been proved to be capable of establishing stable and powerful representation for palmprint. We utilized a multiscale version of Orthogonal Line Ordinal Feature (OLOF) to fused image as well as palmprint image for feature level representation and the average results on collected database are shown in Table 1. Textural features are not suitable for vein image due to the sparse nature of true features and widespread of false features. From Table 1, we can obviously find that Curvelet transform based method outperforms other methods in that it maintains most information available in palmprint. The disadvantage of Curvelet transform is that it takes much longer time in calculating coefficients. We also adopted average local entropy to estimate the information gain from palmprint to fused image and the result is shown in Fig. 3. Curvelet transform based approach is the only one which conveys more information than the original palmprint representation. Thus we can safely draw the conclusion that Curvelet transform based method results in richer representation and is more faithful to source representations. The superior performance of Curvelet transform mainly

Comparative Studies on Multispectral Palm Image Fusion for Biometrics

19

Table 1. Objective Fusion Quality Evaluation RM SEFP alm M IFP alm U IQIFP alm Time Consumption(s) Gradient Pyramid Morphological Pyramid Shift-Invariant DWT Curvelet Transform

0.4194 0.4539 0.4313 0.3773

0.3300 0.2672 0.3083 0.4102

0.5880 0.4800 0.5583 0.7351

0.5742 1.2895 1.9371 18.6979

0.95 0.9 0.85

Average Entropy

0.8 0.75 0.7 0.65 0.6 Original Palmprint Image Fused Image with Curvelet Transform Fused Image with Morphological Pyramid Fused Image with Gradient Pyramid Fused Image with Shift Invariant DWT

0.55 0.5 0.45 4

6

8 10 12 Local Window Size

14

16

Fig. 3. The average local entropy of fused image with regard to window size

results from its built-in mechanism to represent line singularities. Gradient pyramid performs next to Curvelet transform, which suggests its good edge preservation capability and low orientation resolution compared with Curvelet transform. Morphological pyramid method introduces too many artifacts which contribute most to performance degradation.

6

Conclusion and Discussion

In this paper, we proposed the idea of multispectral palm image fusion for biometrics. This concept extends the visual capability of camera and will improve user-friendliness, security and hopeful recognition performance of original palmprint based biometric system. Several image fusion based approaches are evaluated in the context of discriminative features. Experimental results suggest

20

Y. Hao, Z. Sun, and T. Tan

that Curvelet transform outperforms several other carefully selected methods in terms of well established criteria. Further work along proposed direction will include the followings: – Image collection from more spectra. The results presented in Section 5 have ¯ performance of Curvelet transform in combining palmproved the superior print and vein images. To explore the best potential of hand biometrics, we will improve device to capture images from more spectra. Although line-like pattern is dominant on palmprint and vein images, they are not necessarily suitable for other components. Thus more schemes need to be studied based on examination of meaningful physiological characteristics of each skin component. – Recognition based on fused image. Currently, the database is not large enough ¯ to produce convincing recognition performance. A well defined database will be collected in the near future and the proposed method will be tested and also compared with other level fusion; Acknowledgments. This work is funded by research grants from the National Basic Research Pro- gram (Grant No. 2004CB318110), the Natural Science Foundation of China (Grant No. 60335010, 60121302, 60275003, 60332010, 6982510560605008) and the Chinese Academy of Sciences.

References 1. Maltoni, D., Maio, D., Jain, A.K., Prabhakar, S.: Handbook of Fingerprint Recognition. Springer, Heidelberg (2003) 2. Bolle, R., Pankanti, S., Jain, A.K.: Biometrics: Personal Identification in Networked Society. Springer, Heidelberg (1999) 3. Lin, C.-L., Fan, K.-C.: Biometric Verification Using Thermal Images of Palm-Dorsa Vein Patterns. IEEE Trans. on Circuits and Systems for Video Technology 14(2), 199–213 (2004) 4. Finger Vein Authentication Technology, http://www.hitachi.co.jp/Prod/comp/finger-vein/global/ 5. Fujitsu Palm Vein Technology, http://www.fujitsu.com/global/about/rd/200506palm-vein.html 6. Igarashi, T., Nishino, K., Nayar, S.K.: The Appearance of Human Skin. Technical Report CUCS-024-05, Columbia University (2005) 7. Wai-Kin Kong, A., Zhang, D.: Competitive Coding Scheme for Palmprint Verification. In: Intl. Conf. on Pattern Recognition, vol. 1, pp. 520–523 (2004) 8. Sun, Z., Tan, T., Wang, Y., Li, S.Z.: Ordinal Palmprint Recognition for Personal Identification. Proc. of Computer Vision and Pattern Recognition (2005) 9. Wang, L., Leedham, G.: Near- and Far- Infrared Imaging for Vein Pattern Biometrics. In: Proc. of the IEEE Intl. Conf. on Video and Signal Based Surveillance (2006) 10. Tomasi, C., Manduchi, R.: Bilateral Filtering for Gray and Color Images. In: Proc. of Sixth Intl. Conf. on Computer Vision, pp. 839–846 (1998) 11. Anderson, R.R., Parrish, J.A.: The Science of Photomedicine. In: Optical Properties of Human Skin. ch. 6, Plenum Press, New York (1982)

Comparative Studies on Multispectral Palm Image Fusion for Biometrics

21

12. Donoho, D.L., Duncan, M.R.: Digital Curvelet Transform: Strategy, Implementation and Experiments, available http://www-stat.stanford.edu/∼ donoho/Reports/1999/DCvT.pdf 13. Cand`es, E.J., Donoho, D.L.: Curvelets – A Surprisingly Effective Nonadaptive Representation for Objects With Edges. In: Schumaker, L.L., et al. (eds.) Curves and Surfaces, Vanderbilt University Press, Nashville, TN (1999) 14. Curvelet website, http://www.curvelet.org/ 15. Starck, J.L., Cand`es, E.J., Donoho, D.L.: The Curvelet Transform for Image Denoising. IEEE Transactions on Image Processing 11(6), 670–684 (2002) 16. Choi, M., Kim, R.Y., Nam, M.-R., Kim, H.O.: Fusion of Multispectral and Panchromatic Satellite Images Using the Curvelet Transform. IEEE Geoscience and Remote Sensing Letters 2(2) (2005) 17. Nencini, F., Garzelli, A., Baronti, S., Alparone, L.: Remote Sensing Image Fusion Using the Curvelet Transform. Information Fusion 8(2), 143–156 (2007) 18. Zhang, Q., Guo, B.: Fusion of Multisensor Images Based on Curvelet Transform. Journal of Optoelectronics Laser 17(9) (2006) 19. Zhang, Z., Blum, R.S.: A Categorization of Multiscale-Decomposition-Based Image Fusion Schemes with a Performance Study for a Digital Camera Application. Proc. of IEEE 87(8), 1315–1326 (1999) 20. Burt, P.J., Kolczynski, R.J.: Enhanced Image Capture Through Fusion. In: IEEE Intl. Conf. on Computer Vision, pp. 173–182. IEEE Computer Society Press, Los Alamitos (1993) 21. Sadjadi, F.: Comparative Image Fusion Analysais. IEEE Comptuer Vision and Pattern Recognition 3 (2005) 22. Petrovi´c, V., Cootes, T.: Information Representation for Image Fusion Evaluation. In: Intl. Conf. on Information Fusion, pp. 1–7 (2006) 23. Petrovi´c, V., Xydeas, C.: Objective Image Fusion Performance Characterisation. In: Intl. Conf. on Computer Vision, pp. 1868–1871 (2005) 24. Wang, Z., BovikA, A.C.: Univeral Image Quality Index. IEEE Signal Process Letter 9(3), 81–84 (2002)

Learning Gabor Magnitude Features for Palmprint Recognition Rufeng Chu, Zhen Lei, Yufei Han, Ran He, and Stan Z. Li Center for Biometrics and Security Research & National Laboratory of Pattern Recognition Institute of Automation, Chinese Academy of Sciences {rfchu,zlei,yfhan,rhe,szli}@nlpr.ia.ac.cn http://www.cbsr.ia.ac.cn

Abstract. Palmprint recognition, as a new branch of biometric technology, has attracted much attention in recent years. Various palmprint representations have been proposed for recognition. Gabor feature has been recognized as one of the most effective representations for palmprint recognition, where Gabor phase and orientation feature representations are extensively studied. In this paper, we explore a novel Gabor magnitude feature-based method for palmprint recognition. The novelties are as follows: First, we propose an illumination normalization method for palmprint images to decrease the influence of illumination variations caused by different sensors and lighting conditions. Second, we propose to use Gabor magnitude features for palmprint representation. Third, we utilize AdaBoost learning to extract most effective features and apply Local Discriminant Analysis (LDA) to reduce the dimension further for palmprint recognition. Experimental results on three large palmprint databases demonstrate the effectiveness of proposed method. Compared with state-of-the-art Gabor-based methods, our method achieves higher accuracy.

1 Introduction Biometrics is an emerging technology by using unique and measurable physical characteristics to identify a person. The physical attributes include face, fingerprint, iris, palmprint, hand geometry, gait, and voice. Biometric systems have been successfully used in many different application contexts, such as airports, passports, access control, etc. Compared with other biometric technologies, palmprint recognition has a relatively shorter history and has received increasing interest in recent years. Various techniques have been proposed for palmprint recognition in the literature [1,2,3,4,5,6,7,8,9,10]. They can be mainly classified into three categories according to the palmprint feature representation method. The first category is based on structure features, such as line features [1] and feature points [2]. The second one is based on holistic appearance features, such as PCA [3], LDA [4] and KLDA [5]. The third one is based on local appearance features, such as PalmCode [7], FusionCode [8], Competitive Code [9] and Ordinal Code [10]. Among these representation methods, Gabor feature is one of the most efficient representations for palmprint recognition. Zhang et al. [7] proposed a texture-based method for online palmprint recognition, where 2D Gabor filter was used to extract the Y. Yagi et al. (Eds.): ACCV 2007, Part II, LNCS 4844, pp. 22–31, 2007. c Springer-Verlag Berlin Heidelberg 2007 

Learning Gabor Magnitude Features for Palmprint Recognition

23

phase information (called PalmCode) from low-resolution palmprint images. Kong and Zhang [8] improved the efficiency of PalmCode method by fusing the codes computed in four different orientations (called FusionCode). Multiple Gabor filters are employed to extract phase information on a palmprint image. To further improve the performance, Kong and Zhang [9] proposed another Gabor based method, namely competitive code. The competitive coding scheme uses multiple 2D Gabor filters to extract orientation information from palm lines based on the winner-take-all competitive rule [9]. Combined with angular matching, promising performance has been achieved. Gabor phase and orientation features have been extensively studied in existing works [7,8,9]. In this paper, we attempt to explore Gabor magnitude feature representation for palmprint recognition. First, to increase the generalization capacity and decrease the influence of illumination variations due to different sensors and lighting environments, we propose an illumination normalization method for palmprint images. Second, multi-scale, multi-orientation Gabor filters are used to extract Gabor magnitude features for palmprint representation. The original feature set is of high dimensionality. Then, we utilize AdaBoost learning to select most effective features from the large number of candidate feature set, followed by Local Discriminant Analysis (LDA) for further dimensionality reduction. Experimental results demonstrate the good performance of proposed method. Compared with state-of-the-art Gabor-based method, our method achieves higher accuracy. Moreover, the processing speed of the method is very fast. In the testing phase, the execution time for the illumination normalization, feature extraction, feature space to LDA subspace projection and matching for one image are 30ms, 20ms, 1.5ms and 0.01ms, respectively. The rest of this paper is organized as follows. In Section 2, we introduce the illumination normalization method. In Section 3, we describe the Gabor magnitude features for palmprint representation. Section 4 gives the details of statistical learning for feature selection and classifier. Experimental results and conclusions are presented in Section 5 and Section 6, respectively.

2 Illumination Normalization Due to different sensors and lighting environments, the palmprint images are varied significantly, as shown in the top row of Fig. 1. A robust illumination preprocessing method will help to diminish the influence of illumination variations and increase the robustness of recognition method. In general, an image I(x, y) is regarded as product I(x, y) = R(x, y)L(x, y), where R(x, y) is the reflectance and L(x, y) is the illuminance at each point (x, y). The reflectance R depends on the albedo and surface normal, which is the intrinsic representation of an object. The luminance L is the extrinsic factor. Therefore, the illumination normalization problem reduces to how to obtain R given an input image I. However, estimating the reflectance and the illuminance is an ill-posed problem. To solve the problem, a common assumption is that the illumination L varies slowly while the reflectance R can change abruptly. In our work, we introduce an anisotropic approach to compute the estimate of the illumination field L(x, y), which has been used

24

R. Chu et al.

Fig. 1. Examples of the palmprint images from different sensors before and after illumination normalization. Top: Original palmprint images. Bottom: Corresponding processed palmprint images. The images are taken from the PolyU Palmprint Database [12] (first two columns), UST Hand Database [13] (middle two columns) and CASIA Palmprint Database [14] (last two columns).

for face recognition [11]. Then, we estimate the reflectance R(x, y) as the ratio of the image I(x, y) and L(x, y) for palmprint image, The luminance function was estimated as an anisotropically smoothed version of the original image, which can be carried out by minimizing the cost function:     ρ(x, y)(L − I)2 dxdy + λ (L2x + L2y )dxdy (1) J(L) = y

x

y

x

where the first is the data term while the second term is a regularization term which imposes a smoothness constraint. The parameter λ controls the relative importance of the two terms. ρ is Weber’s local contrast between a pixel a and its neighbor b in either the x or y directions [11]. The space varying permeability weight ρ(x, y) controls the anisotropic nature of the smoothing constraint. By Euler-Lagrange equation, Equ. (1) transforms to solve the following partial differential equation (PDE): λ (2) L + (Lxx + Lyy ) = I ρ The PDE approach is easy to implement. By the regularized approach, the influence of the illumination variations is diminished, while the edge information of the palmprint image is preserved. Fig. 1 show some examples from several different palmprint database before and after processing with the method. In section 5, we will further evaluate the effectiveness of the illumination normalization method on a large palmprint database.

3 Gabor Magnitude Features for Palmprint Representation Gabor features exhibit desirable characteristics of spatial locality and orientation selectively, and are optimally localized in the space and frequency domains. The Gabor kernels can be defined as follows [15]: ψμ,v =

2 2 kμ,v kμ,v z2 σ2 )] exp( )[exp(ik z) − exp(− μ,v σ2 2σ 2 2

(3)

Learning Gabor Magnitude Features for Palmprint Recognition

25

where μ and v define the orientation and scale of the Gabor kernels respectively, z = (x, y), and the wave vector kμ,v is defined as follows: kμ,v = kv eiφμ (4) √ where kv = kmax /f v , kmax = π/2, f = 2, φμ = πμ/8. The Gabor kernels in Equ. 3 are all self-similar since they can be generated from one filter, the mother wavelet, by scaling and rotating via the wave vector kμ,v . Each kernel is a product of a Gaussian envelope and a complex plane wave, while the first term in the square brackets in Equ. (3) determines the oscillatory part of the kernel and the second term compensates for the DC value. Hence, a bank of Gabor filters is generated by a set of various scales and rotations. In our experiment, we use Gabor kernels at five scales v ∈ {0, 1, 2, 3, 4} and eight orientations μ ∈ {0, 1, 2, 3, 4, 5, 6, 7} with the parameter μ = 2π to derive the Gabor representation by convoluting palmprint image with corresponding Gabor kernels. Let I(x, y) be the gray level distribution of an palmprint image , the convolution of image I and a Gabor kernel ψμ,v is defined as: Fμ,v (z) = I(z) ∗ ψμ,v (z)

(5)

where z = (x, y), ∗ denotes the convolution operator. Gabor magnitude feature is defined as  Mμ,v (z) = Im(Fμ,v (z))2 + Re(Fμ,v (z))2 (6) where Im(·) and Re(·) denote the imaginary and real part, respectively. For each pixel position (x, y) in the palmprint image, 40 Gabor magnitudes are calculated to form the feature representation.

4 Statistical Learning of Best Features and Classifiers The whole set of Gabor magnitude features is of high dimension. For a palmprint image with size of 128 × 128 , there are about 655,360 features in total. Not all of them are useful or equally useful, and some of them may cause negative effect on the performance. Straightforward implementation is both computationally expensive and exhibits a lack of efficiency. In this work, we utilize AdaBoost learning first to select the most informative features and then apply linear discriminant analysis (LDA) on the selected Gabor magnitude features for further dimension reduction. 4.1 Feature Selection by AdaBoost Learning Boosting can be viewed as a stage-wise approximation to an additive logistic regression model using Bernoulli log-likelihood as a criterion [16]. AdaBoost is a typical instance of Boosting learning. It has been successfully used on face detection problem [17] as an effective feature selection method. There are several different versions of AdaBoost algorithm [16], such as Discrete AdaBoost, Real AdaBoost, LogitBoost and Gentle AdaBoost. In this work, we apply Gentle AdaBoost learning to select most discriminative Gabor magnitude features and remove the useless and redundant features. Gentle AdaBoost is a modified version of the Real AdaBoost algorithm and is defined in Fig. 2.

26

R. Chu et al.

Input: Sequence of N weighted examples: {(x1 , y1 , w1 ), (x2 , y2 , w2 ), . . . , (xN , yN , wN )}; Initialize: wi =

1 N

, i = 1, 2, ..., N, F (x) = 0

Integer T specifying number of iterations; For t = 1, . . . ,T (a) Fit the regression function ft (x) by weighted least squares of yi to xi with weights wi . (b) Update F (x) ← F (x) + ft (x)



(c) Update wi ← wi e−yi ft (xi ) and renormalize. 3. Output the classifier sign[F (x)] = sign[

T t=1

ft (x)]

Fig. 2. Algorithm of Gentle AdaBoost

Empirical evidence suggests that Gentle AdaBoost is a more conservative algorithm that has similar performance to both the Real AdaBoost and LogitBoost algorithms, and often outperforms them both, especially when stability is a crucial issue [16]. While the above AdaBoost procedure essentially learns a two-class classifier, we convert the multi-class problem into a two-class one using the idea of intra- and extraclass difference [18]. However, here the difference data are derived from each pair of Gabor magnitude features at the corresponding locations rather than from the images. The positive examples are derived from pairs of intra-personal differences and the negative from pairs of extra-personal differences. In this work, the weak classifier in AdaBoost learning is constructed by using a single Gabor magnitude feature. Therefore, AdaBoost learning algorithm can be considered as a feature selection algorithm [17,19]. With the selected feature set, a series of statistical methods can be used to construct effective classifier. In the following, we introduce LDA for dimension reduction further and use cosine distance for palmprint recognition and expect it can achieve better performance. 4.2 LDA with Selected Features LDA is a famous method for feature extraction and dimension reduction that maximizes the extra-class distance while minimized the intra-class distance. Let the sample set be X = {x1 , x2 , ..., xn }, where xi is the feature vector for the i-th sample. The withinclass scatter matrix Sw and the between-class scatter matrix Sb are defined as follows: Sw =

L  

(xj − mi )T (xj − mi )

(7)

i=1 xj ∈Ci

Sb =

L 

ni (mi − m)T (mi − m)

(8)

i=1

 L  where mi = n1i xj ∈Ci xj is the mean vector in class Ci , and m = n1 i=1 xj ∈Ci xj is the global mean vector.

Learning Gabor Magnitude Features for Palmprint Recognition

27

LDA aims to find projection matrix W so that the following object function is maximized: J=

tr(WT Sb W) tr(WT Sw W)

(9)

The optimal projection matrix Wopt can be obtained by solving the following generalized eigen-value problem (10) S−1 w Sb W = WΛ where Λ is a diagonal matrix whose diagonal elements are the eigenvalues of S−1 w Sb . Given two input vectors x1 and x2 , their subspace projections are calculated as v1 = WT x1 and v2 = WT x2 , and the following cosine distance is used for the matching: H(v1 , v2 ) =

v1 · v2 v1  v2 

(11)

where . denotes the norm operator. In the test phase, the projections v1 and v2 are computed from two input vectors x1 and x2 , one for the input palmprint image and another for an enrolled palmprint image. By comparing the score H(v1 , v2 ) with a threshold, a decision can be made whether x1 and x2 belong to the same person.

5 Experiments To evaluate the performance of the proposed palmprint recognition method, three large palmprint databases are adopted, including PolyU Palmprint Database [12], UST Hand Image Database [13] and CASIA Palmprint Database [14]. These databases are among the largest in size in the public domain. We train the classifiers and evaluate the effectiveness of illumination normalization method on PolyU Palmprint Database. To explore the generalization of the classifier, we further evaluate the performance of proposed palmprint recognition method on the other two databases, and compare with the state-of-the-art Gabor-based recognition methods [7,8,9]. 5.1 Evaluate on PolyU Palmprint Database PolyU Palmprint Database[12] contains 7752 images corresponding to 386 different palms. Around twenty samples from each of these palms were collected in two sessions. There are some illumination variations between the two sessions. We select 4000 images from 200 different palms collected in two sessions as the testing set, with 20 images per palm. The rest 3752 images from 186 different palms are used for training. All the input palmprint images are normalized to 128 × 128 using the method proposed in [7]. In the training phase, the training set of positive samples were derived from intraclass pairs of Gabor features, the negative set from extra-class pairs. Two Gabor magnitude feature-based classifiers are trained. One is an AdaBoost learning based classifier, and another is an LDA based classifier using AdaBoost-selected features. These two methods are named “GMBoost” and “GMBoostLDA”, respectively. Moreover, to

28

R. Chu et al.

evaluate the effectiveness of the illumination normalization method, we also train two classifiers and test the performance on the palmprint images without illumination normalization. The first two classifiers are trained using the palmprint images without illumination normalization. 882 most effective features are selected by the AdaBoost procedure from the original 655,360 Gabor magnitude features with the training error rate of zero on the training set. For LDA, the feature dimension retained is 181, which is optimal in the test set. The other two classifiers are trained using the palmprint images with illumination normalization. 615 most effective features are selected with the training error rate of zero on the training set. The optimal feature dimension for LDA is 175 found in the test set. The first 5 most effective features learned by Gentle AdaBoost are shown in Fig. 3, in which the position, scale and orientation of corresponding Gabor kernels are indicated on an illumination normalized palmprint image.

Fig. 3. The first 5 features and associated Gabor kernel selected by AdaBoost learning

In the testing phase, we match palmprints from different sessions. Each image from the first session is matched with all the images in the second sessions. This generated 20,000 intra-class (positive) and 380,000 extra-class (negative) pairs. Fig. 4 shows the ROC curves derived from the scores for the intra- and extra-class pairs. From the result, we can see that all these Gabor magnitude feature-based methods achieve good verification performances. The performance of “GMBoostLDA” methods are better than that of “GMBoost” methods. This indicates applying LDA with AdaBoost-selected features is a good scheme for palmprint recognition. Among these classifiers, “GMBoostLDA with Illumination Normalization” performs the best, which demonstrates the effectiveness of the proposed illumination normalization method. The processing speed of proposed method is very fast. In the testing phase, only the features selected by the AdaBoost learning need to be extracted with the Gabor filter, which largely reduce the computational cost. On a P4 3.0GHz PC, the execution time for the illumination normalization, feature extraction, feature space to LDA subspace projection and matching for one image are 30ms, 20ms, 1.5ms and 0.01ms, respectively. In the next subsection, we will further evaluate the performance of our best classifier on the other two databases to explore the generalization capacity and compare with the state-of-the-art Gabor-based recognition methods. 5.2 Evaluate on UST Hand Image Database and CASIA Palmprint Database UST hand image database [13] contains 5,660 hand images corresponding to 566 different palms, 10 images per palm. All images are captured using a digital camera with

Learning Gabor Magnitude Features for Palmprint Recognition

29

Fig. 4. Verification performance comparison on PolyU Palmprint Database

resolution of 1280 × 960 (in pixels) and 24-bit colors. There are totally 25,470 intraclass (genuine) samples and 15,989,500 extra-class (impostor) samples generated from the UST database. CASIA palmprint database [14] contains 4,796 images corresponding to 564 different palms. All images are captured using a CMOS camera with resolution of 640x480 (in pixels) and 24-bit colors. There are 8 to 10 samples in each of these palms. There are totally 18,206 intra-class (genuine) samples and 11,480,204 extra-class (impostor) samples generated from the test set. Fig. 5 shows the ROC curves derived from the scores for the intra- and extra-class samples. According to the ROC curves, the performance of the proposed method is better than that of the state-of-the-art Gabor-based recognition methods in both the two databases. Note that our classifier is trained on the PolyU database and tested on the UST and CASIA palmprint databases. Two accuracy measurements are computed for further comparison in Table 1. One is the equal error rate (EER) and the other is the d (d-prime) [20]. d is a statistical measure of how well a biometric system can discriminate between different individuals defined as

Fig. 5. Comparative results with state-of-the-art Gabor-based recognition methods. Left: ROC curves on UST Hand Image Database. Right: ROC curves on CASIA Palmprint Database.

30

R. Chu et al.

|m1 − m2 | d =  2 (12) (δ1 + δ22 )/2 where m1 and δ1 denote the mean and variance of intra-class feature vector respectively, while m2 and δ2 denote the mean and variance of extra-class feature vector. The larger the d value is, the better a biometric system performs [20]. Table 1. Comparison of accuracy measures for different classifiers on UST and CASIA databases Algorithm Palm Code (θ = 45o ) [7] Fusion Code [8] Competitive Code [9] Proposed method

Results on UST database EER (%) d 1.77 3.39 0.75 3.40 0.38 3.51 0.35 5.36

Results on CASIA database EER (%) d 0.95 3.58 0.57 3.80 0.19 3.81 0.17 5.57

From the experimental results, we can see that both the EER and the discriminating index of proposed method achieve good performance (in bold font). This also suggests the good generalization capacity of proposed method, which can work well on different types of palmprint images.

6 Conclusions In this paper, we have proposed a Gabor magnitude feature based learning method for palmprint recognition. To decrease the influence of illumination variations, we introduced an illumination normalization method for palmprint images. Then, multi-scale, multiorientation Gabor filters are used to extract Gabor magnitude features. Based on the Gabor magnitude features and statistical learning, a powerful classifier is constructed. The experimental results show that Gabor magnitude features with statistical learning can be powerful enough for palmprint recognition. Compared with state-of-the-art Gabor-based method, our method achieves better performance on two large palmprint database.

Acknowledgements This work was supported by the following funding resources: National Natural Science Foundation Project #60518002, National Science and Technology Supporting Platform Project #2006BAK08B06, National 863 Program Projects #2006AA01Z192 and #2006AA01Z193, Chinese Academy of Sciences 100 people project, and the AuthenMetric Collaboration Foundation.

References 1. Zhang, D., Shu, W.: Two novel characteristics in palmprint verification: Datum point invariance and line feature matching. Pattern Recognition 32, 691–702 (1999) 2. Duta, N., Jain, A., Mardia, K.: Matching of palmprint. Pattern Recognition Letters 23, 477–485 (2001)

Learning Gabor Magnitude Features for Palmprint Recognition

31

3. Lu, G., Zhang, D., Wang, K.: Palmprint recognition using eigenpalms features. Pattern Recognition Letters 24, 1463–1467 (2003) 4. Wu, X., Zhang, D., Wang, K.: Fisherpalms based palmprint recognition. Pattern Recognition Letters 24, 2829–2838 (2003) 5. Wang, Y., Ruan, Q.: Kernel fisher discriminant analust-hand-databaseysis for palmprint recognition. In: Proceedings of International Conference Pattern Recognition, vol. 4, pp. 457–461 (2006) 6. Kumar, A., Shen, H.: Palmprint identification using palmcodes. In: ICIG 2004. Proceedings of the Third International Conference on Image and Graphics, Hong Kong, China, pp. 258– 261 (2004) 7. Zhang, D., Kong, W., You, J., Wong, M.: On-line palmprint identification. IEEE Transactions on Pattern Analysis and Machine Intelligence 25, 1041–1050 (2003) 8. Kong, W., Zhang, D.: Feature-Level Fusion for Effective Palmprint Authentication. In: Zhang, D., Jain, A.K. (eds.) ICBA 2004. LNCS, vol. 3072, pp. 761–767. Springer, Heidelberg (2004) 9. Kong, W., Zhang, D.: Competitive coding scheme for palmprint verification. In: Proceedings of International Conference Pattern Recognition, vol. 1, pp. 520–523 (2004) 10. Sun, Z., Tan, T., Wang, Y., Li, S.: Ordinal palmprint represention for personal identification. In: Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 279–284. IEEE Computer Society Press, Los Alamitos (2005) 11. Gross, R., Brajovic, V.: An image preprocessing algorithm for illumination invariant face recognition. In: Proc. 4th International Conference on Audio- and Video-Based Biometric Person Authentication, Guildford, UK, pp. 10–18 (2003) 12. PolyU Palmprint Database, http://www.comp.polyu.edu.hk/biometrics/ 13. UST Hand Image database, http://visgraph.cs.ust.hk/biometrics/Visgraph web/index.html 14. CASIA Palmprint Database, http://www.cbsr.ia.ac.cn/ 15. Daugman, J.G.: Complete discret 2d gabor transforms by neural networks for image analysis and compression. IEEE Trans. ASSP 36, 1169–1179 (1988) 16. Friedman, J., Hastie, T., Tibshirani, R.: Additive logistic regression: a statistical view of boosting. Technical report, Department of Statistics, Sequoia Hall, Stanford Univerity (1998) 17. Viola, P., Jones, M.: Rapid object detection using a boosted cascade of simple features. In: Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Kauai, Hawaii, IEEE Computer Society Press, Los Alamitos (2001) 18. Moghaddam, B., Nastar, C., Pentland, A.: A Bayesian similarity measure for direct image matching. Media Lab Tech Report No.393, MIT (1996) 19. Shan, S., Yang, P., Chen, X., Gao, W.: AdaBoost gabor fisher classifier for face recognition. In: Proceedings of IEEE International Workshop on Analysis and Modeling of Faces and Gestures, Beijing, China, pp. 279–292. IEEE Computer Society Press, Los Alamitos (2005) 20. Daugman, J., Williams, G.: A Proposed Standard for Biometric Decidability. In: Proc. CardTech/SecureTech Conference, pp. 223–234 (1996)

Sign Recognition Using Constrained Optimization Kikuo Fujimura1 and Lijie Xu2 1

Honda Research Institute USA 2 Ohio State University

Abstract. Sign recognition has been one of the challenging problems in computer vision for years. For many sign languages, signs formed by two overlapping hands are a part of the vocabulary. In this work, an algorithm for recognizing such signs with overlapping hands is presented. Two formulations are proposed for the problem. For both approaches, the input blob is converted to a graph representing the finger and palm structure which is essential for sign understanding. The first approach uses a graph subdivision as the basic framework, while the second one casts the problem to a label assignment problem and integer programming is applied for finding an optimal solution. Experimental results are shown to illustrate the feasibility of our approaches.

1

Introduction

There have been many approaches in sign recognition [1]. Among many important elements for sign recognition, components which are of basic importance are hand tracking and hand shape analysis. For hand detection, many approaches use color or motion information [6,7]. It turns out, however, that hand tracking using color is a non-trivial task except for well-controlled environments, as various lighting changes pose challenging conditions [3,10,15,17]. Making use of special equipment such as data gloves is one solution to overcome this difficulty. When the hand is given in a magnified view, hand shape analysis becomes a feasible problem [16], although body and arm posture information might be lost. Successful results are reported by using multiple cameras for extracting 3D information [9,13]. Even though redundancy makes the problem more approachable, handling a bulk of data in real-time poses another challenge for efficient computation. Model fitting for 3D data is known to be a computationally demanding task. Stereo vision is a popular choice in many tasks including man-machine interaction and robotics [9]. However, it still fails to provide sufficient details in depth maps for some tasks such as counting fingers in a given hand posture. Stereo images have been useful in some applications such as large-scale gesture recognition such as pointing motions. In contrast, coded lights or recent techniques such as space-time stereo in general provide a depth resolution much better than a traditional stereo. Such a device is expected to provide a high quality image Y. Yagi et al. (Eds.): ACCV 2007, Part II, LNCS 4844, pp. 32–41, 2007. c Springer-Verlag Berlin Heidelberg 2007 

Sign Recognition Using Constrained Optimization

33

sequence even for posture analysis. Time-of-flight sensors also provide a resolution that is sufficient for hand shape analysis [5,12,13,18]. For our work, we opt to use this type of device as a suitable alternative for stereo. Much of the work in hand shape and motion analysis primarily deal with a single hand shape or two non-overlapping hands [4,8,11,13,16]. Whereas analysis of independent hand shapes is a basic task, analysis of overlapping hands presents another level of challenge in gesture recognition. Natural gestures (including sign languages as in Fig. 1) often use signs formed by two overlapping hands. Motivated in this manner, we present approaches at analyzing various hand patterns formed by two overlapping hands. In particular, we focus on how to separate two hands from one blob representing overlapping hands. In addition, we also present a real-time non-intrusive sign recognition system that can recognize signs by one hand, two non-overlapping hands, and two overlapping hands. The rest of the paper is organized as follows. In section 2, we outline our solution approaches. Sections 3 and 4 present two formulations to the problem and Section 5 contains a description of the entire system. Experimental results are presented in Section 6 and Section 7 contains concluding remarks.

Fig. 1. Examples of signs formed by overlapping hands

2

Flow of the Algorithm

We present two algorithms to address the problem of overlapping hands. The two algorithms share a common basic part, namely, steps 1 - 3 of the following procedure. 1. 2. 3. 4. 5.

Extracting the arm blob(s) from the image Overlapping hand detection and palm detection Graph formation Separation of overlapped hands (two methods are proposed) Sign recognition

Hand segmentation is an important step for most sign recognition methods. In this work, we use depth streams for hand blob segmentation. Even though

34

K. Fujimura and L. Xu

it is conceptually simple to segment hands in depth images, especially when hands are in front of the body, it still requires work to identify palm and finger parts from the arm blob. This part is described in Section 5. (When the blob is determined to contain only one hand, Steps 3 and 4 are skipped.) For the second step, we make use of the observation that for overlapping hands, the blob’s area is larger than a blob containing a single hand and the convex hull of the blob is significantly larger than the blob itself. Next, a graph structure is extracted from the hand blob. For this operation, the medial axis transform (roughly corresponding to the skeleton of the blob) is generated from the blob by a thinning operation (Fig. 2). The skeleton represents the general shape of the hand structure, while it has some short fragmental branches that do not represent fingers. Also, for two fingers that are connected as in Fig 1(c), the connecting point may not become a node (branching point) in the skeleton. Thus, we create an augmented graph (which we call as G hereafter) from the skeleton. This is accomplished by removing short fragments (e.g., ‘dangling branch’ in Fig. 2) and dividing long edges into shorter pieces (Fig. 2), in particular, at a high curvature point. After a connectivity graph G is formed, the remaining task is to determine parts of G that come from the right and the left hands, respectively.

Fig. 2. Example of the skelton structure (left). Example of the augmented graph structure (right).

Two methods are presented. The first algorithm is based on a tree search paradigm, while the second one is formulated by using constrained optimization.

3

Tree Search Framework

The first algorithm for hand disambiguation uses tree search. Given an augmented graph G, we form two subgraphs H1 and H2 such that G = H1 ∪ H2 . Moreover, each Hi is required to satisfy a few contions so that it represents a proper hand shape. A connected subgraph H1 (H2 ) is formed such that it contains the palm corresponding to the right (left) hand, respectively. Our strategy is to generate H1 and H2 systematically and pick the pair that best matches our definition of ‘hands’. The outline of the algorithm is summarized as follows.

Sign Recognition Using Constrained Optimization

35

1. Create a DAG (directed acyclic graph) from G. 2. Separate the tree G into two parts: (a) Do DFS (depth first search) from the source node (top node of the tree). (b) Let H1 be a connected subgraph from by the scan. This forms a possible hand structure. (c) For each H1 , do reduction on graph G to obtain the remaining part which, in turn, forms the second hand structure H2 . 3. Evaluate the given hand structure pair H1 and H2 . 4. The one with the best evaluation is selected as the final answer.

Fig. 3. Example of tree search

3.1

Evaluation Function

After each scan of the tree, we are left with a pair of H1 and H2 . The evaluation of this pair consists of a few criteria. 1. Each of H1 and H2 must be connected. This comes from the natural requirement that each hand is a connected entity. 2. Distance of any part of H from the palm must be within a certain limit. This discourages to have a finger that is longer than a certain limit. 3. For two segments within a subgraph, the angle formed by the segments cannot be small. This condition discourages to have a finger that bends at an extremely sharp angle. Likewise, fingers bending outward are discouranged. 4. A branching node in H must stay within a certain distance from the palm. This condition discourages to form a finger with branches at the tip of the finger. The above criteria are encoded in the decision process and ones that have the best evaluation are considered.

4

Optimization Framework

The second framework reduces the problem to a labeling problem which is formulated as the following optimization problem. We continue to use graph G. A segment (or edge) si in G is to be assigned a label by a function f (that is, f (si ) is either Right or Left). Each si has an estimate of its likelihood of having

36

K. Fujimura and L. Xu

labeling f (si ). This comes from a heuristic in sign estimation. For example, if s is near the left hand palm, its likelihood of being a part of the right hand is relatively low. For this purpose, a non-negative cost function c(s, f (s)) is introduced to represent the likelihood. Further, we consider two neighboring segments si and sj to be related, in the sense that we would like si and sj to have the same label. Each edge e in graph G has a nonnegative weight indicating the strength of the relation. Moreover, certain pairs of labels are more similar than others. Thus, we impose a distance d() on the label set. Larger distance values indicate less similarity. The total cost of a labeling f is given by :   c(s, f (s)) + we d(f (si ), f (sj )) Q(f ) = s∈S

In our problem, the following table (4) is to be completed, where binary variable Aij indicates if segment si belongs to hand pi . For c(i, j), the Euclidean distance from segment si to the palm of hand j is used. Since each (thin) segment si belongs to only one hand, j Aij = 1 hold. In addition to this constraint, a number of related constraints are considered. 1. Neighboring segments should have a similar label. 2. Thick parts represent more than one finger. 3. Thin parts represent one finger.

Fig. 4. Assignment table

It turns out that this is an instance of the Uniform Labeling Problem, which can be expressed as the following integer program by introducing auxiliary variables Ze for an edge e to express the distance between the labels and we use Zej to express the absolute value |Apj − Aqj |. Following Kleinberg and Tardos [2], we can rewrite our optimization problem as follows: ⎛ ⎞ N  M   min ⎝ c(i, j)Aij + we Xe ⎠ i=1 j=1

subject to  j

Aij = 1,

eE

i = 1, 2, 3, · · · , N, if the ith segment is thin.

Sign Recognition Using Constrained Optimization



Aij = 2,

37

i = 1, 2, 3, · · · , N, if the ith segment is thick.

j

Ze =

1 Zej , e ∈ E 2 j

Zej ≥ Apj − Aqj , e = (p, q);

j = 1, · · · , M

Zej ≥ Aqj − Apj , e = (p, q);

j = 1, · · · , M

Aij ∈ {0, 1},

i = 1, 2, 3, , · · · , N ;

j = 1, 2, · · · , M

length(s1 ) + length(s2 ) + · · · < M AXLEN ; for each hand. Here, c(i, j) represents cost (penalty) for segment i to belong to hand j, where j can be either Right or Left. If segment i is far from Left, then c(i, Left) is large. For time-varying image sequences, previous c(i, Left) may be used. This factor is a very powerful factor, assuming palm locations are correctly detected. Terms involving Ze and Zej come from constraint (1). The weight we represents strength between graph node a and b (where e is the edge connecting a and b, where a and b represent segments. For example, if a and b make a sharp turn we is a high number, since a and b are likely to belong to different hands. The weight is given by we = e−αde , where de is the depth difference between two adjacent segments and α is selected based on experiments. For an additional constraint, Aij + Akj < 2 holds for all j, if segments si and sk make a sharp turn. If a segment is thick, an additional constraint may be added. Finally, the total length of fingers must not exceed a certain limit. In general, solving an integer program optimally is NP-hard. However, we can relax the above problem to linear programming with Aij ≥ 0, and this can be solved efficiently by using a publicly available library. Kleinberg and Tardos [2] describe a method for rounding the fractional solution so that the expected objective function Q(f ) is within a factor of 2 from the optimal solution. In our experiments we find that this relaxed linear programming always returns an integer solution.

5 5.1

Sign Recogntion System and Components Palm Detection

To build an entire system for sign recognition, the hand shape analysis module has to be integrated with many other modules such as a motion tracker and pattern classifier. Here, a description is given for modules that are highly related to shape analysis, namely, finger and palm detection. For other related modules, see [18]. To locate fingers within the arm blob, each branch of the skeleton is traced at a regular interval, while measuring width at each position. When the width is smaller than a certain limit, we consider these pixels to belong to a finger. For palm detection, the following steps work well for our experiment.

38

K. Fujimura and L. Xu

1. Trace all branches of the skeleton. At a certain inteval, shoot rays enamating from a point on the skeleton. Pick up the chord that has the shortest distance and we call it the width of the point (Fig. 7 (right)). 2. From all widths, pick up the top few that have the widest widths. Choose the one that is closest to finger positions as the palm center. 3. For the chord at the selected point, pick the center point of the chord and define this as the palm center to be used for the rest of sign recognition. 5.2

Experimental Results

The algorithm has been implemented in C and tested by using alphabet letters used in JSL (Japanese Sign Language). Our experiments show that the second framework has more successful results. The primary reason is that defining the evaluation function that works uniformly for all words is difficult. For example, for ‘X’, we want sharp turns to be minimal, while for ‘B’, we want to keep some of the sharp turns in the pattern. This requires case-based analysis or substantial work is to be done as post-processing. Currently, it takes approximately 0.5 second to resolve overlapping hand cases as in Fig. 5. The second framework also fails at times, for example, when unncecesary fragments remain after pruning in the tree structure. Fig. 6 shows an example of a JSL sentence consisting of three words. ‘convenient store’, ‘place’, and ‘what’, illustrating a sign consisting of a non-overlapping2-hand signs (convenient store) and 1-hand-signs (place and what). Currently, our system has approximately 50 recognizable words. For 1-hand-signs and nonoverlapping-2-hand-signs, recognition speed is less than 0.2 sec for a word (after the total word has been seen) on a laptop computer.

Fig. 5. Examples of two overlapping hand separation. From the top left, signs represent ‘A’, ‘K’, ‘G’, ‘B’, ‘X’, ‘Well’, ‘Meet’, and ‘Letter’ in JSL. Palm centers are marked by circles and hand separation is shown by hands with different shades.

Sign Recognition Using Constrained Optimization

39

Fig. 6. Example of sign recognition. The gray images represent depth profiles obtained by the camera, while the black and white images show processing results. Palm centers are marked as circles. Top row : ‘Convenient store (24 hours open)’. The right and left hands represent ‘2’ and ‘4’, respectively while the hands draws a circular trajectory in front of the body to represent ‘open’. Middle row: ‘place’. The open right hand is moved downward. Bottom row: ‘what’. Only the pointing finger is up and it is moved left and right. The total meaning of the sequence is ‘Where is a convenient store?’. Currently, the system performs word-by-word recognition.

The module has also been incorporated in the word recognition module that takes hand shape, movement, and location into consideration and shows near real-time speed performance.

40

K. Fujimura and L. Xu

Fig. 7. Snapshot of the recognition system (left). Illustration for palm detection (right). For a point on the skeleton, a chord is generated to measure the width at that point. The one that gives rise to the maximum chord length is a candidate for locating the palm center.

6

Concluding Remarks

We have presented an algorithm to separate two overlap hands when fingers of the right and left hands may touch or cross. This is a step toward recognizing various signs formed by two hand formations. Two algorithms have been proposed and examined. Algorithm performance has been experimentally illustrated by showing some overlapping hand patterns taken from JSL words. Some work attempt at identifying fingers in the hand (in addition to counting the number of fingers) from hand appearance. Our work has analyzed finger formation and classify the pattern into several types such as ‘L’ shape and ‘V’ shape. No attempt has been made at this point to identify fingers. We have also been able to integrate our algorithm in a word-based sign recognition system. As compared with existing approaches, salient features of the system constructed as ours are as follows. (i) Signs are recognized without using any special background, marks, or gloves. (ii) Each computation module is computationally inexpensive, thereby achieving fast sign recognition. (iii) Due to depth analysis, the method is less susceptible to illumination changes. The focus of the present work has been to distinguish a given arm blob into two separate parts representing left and right hands. The problem is formulated by a graph division and labelling pardigm and experimental results have been shown that the method has a reasonable performance. Our algorithm requires that much of the fingers is clearly visible. Challenges remain since sign languages usually use overlapping patterns that are complex than those presented in this paper (e.g., ones involving occlusion). We leave this as our future research subject.

Sign Recognition Using Constrained Optimization

41

References 1. Ong, S., Ranganath, S.: Automatic sign language analysis: A survey and the future beyond lexical meaning. IEEE Pattern Analysis and Machine Intelligence 27, 873– 891 (June 2005) 2. Kleinberg, J., Tardos, E.: Approximation algorithms for classfication problems with pairwise relationships: Metric partitioning and Markov random fields. Journal of the ACM 49(5) (2002) 3. Hamada, Y., Shimada, N., Shirai, Y.: Hand shape estimation under complex backgrounds for sign language recognition. In: Proc. of Symposium on Face and Gesture Recognition, Seoul, Korea, pp. 589–594 (2004) 4. Starner, T., Weaver, J., Pentland, A.: Real-time American sign language using desk and wearable computer based video. IEEE Pattern Analysis and Machine Intelligence 20, 12, 1371–1375 (1998) 5. Mo, Z., Neumann, U.: Real-time hand pose recognition using low-resolution depth images. In: Int. Conf. on Computer Vision and Pattern Recognition, New York City (2006) 6. Imagawa, K., Lu, S., Igi, S.: Color-based hand tracking system for sign language recognition. Proc. Automatic Face and Gesture Recognition, 462–467 (1998) 7. Polat, E., Yeasin, M., Sharma, R.: Robust tracking of human body parts for collaborative human computer interaction. Computer Vision and Image Understanding 89(1), 44–69 (2003) 8. Wilson, A., Bobick, A.: Parametric hidden markov models for gesture recognition. IEEE Trans. on Pattern Anal. Mach. Intel. 21(9), 884–900 (1999) 9. Jojic, N., Brumitt, B., Meyers, B., Harris, S., Huang, T.: Detection and estimation of pointing gestures in dense disparity maps. In: Proc. of the 4th Intl. Conf. on Automatic Face and Gesture Recognition, Grenoble, France (2000) 10. Bretzner, L., Laptev, I., Lindeberg, T.: Hand gesture recognition using multi-scale colour features, hierarchical models and particle filtering. In: Proc. of the 5th Intl. Conf. on Automatic Face and Gesture Recognition, Washington D.C., May 2002, pp. 423–428 (2002) 11. Pavlovic, V., et al.: Visual interpretation of hand gestures for human-computer interaction: A review. IEEE Trans. on Pattern Anal. Mach. Intel. 19(7), 677–695 (1997) 12. Iddan, G.J., Yahav, G.: 3D imaging in the studio. In: SPIE, vol. 4298, p. 48 (2000) 13. Malassiotis, S., Aifanti, N., Strintzis, M.G.: A gesture recognition system using 3D data. In: 1st Intl. Symp. on 3D Data Processing, Visualization, and Transmission, Padova, Italy (June 2002) 14. Vogler, C., Metaxas, D.: ASL recognition based on a coupling between HMMs and 3D motion analysis. In: Proc. Int. Conf. Computer Vision, Bombay (1998) 15. Zhu, Y., Xu, G., Kriegman, D.J.: A real-time approach to the spotting, representation, and recognition of hand gestures for human-computer interaction. Computer Vision and Image Understanding 85(3), 189–208 (2002) 16. Athitsos, V., Sclaroff, S.: An appearance-based framework for 3D handshape classfication and camera viewpoint estimation. In: Proc. of the 5th Intl. Conf. on Automatic Face and Gesture Recognition, Washington D.C., May 2002, pp. 45–50 (2002) 17. Zhu, X., Yang, J., Waibel, A.: Segmenting hands of arbitrary color. In: Proc. of the 4th Intl. Conf. on Automatic Face and Gesture Recognition, Grenoble, March 2000, pp. 446–453 (2000) 18. Fujimura, K., Liu, X.: Sign recognition using depth image streams. In: Proc. of the 7th Symposium on Automatic Face and Gesture Recognition, Southampton, UK (May 2006)

Depth from Stationary Blur with Adaptive Filtering Jiang Yu Zheng and Min Shi Department of Computer Science Indiana University Purdue University Indianapolis (IUPUI), USA

Abstract. This work achieves an efficient acquisition of scenes and their depths along long streets. A camera is mounted on a vehicle moving along a path and a sampling line properly set in the camera frame scans the 1D scene continuously to form a 2D route panorama. This paper extends a method to estimate depth from the camera path by analyzing the stationary blur in the route panorama. The temporal stationary blur is a perspective effect in parallel projection yielded from the sampling slit with a physical width. The degree of blur is related to the scene depth from the camera path. This paper analyzes the behavior of the stationary blur with respect to camera parameters and uses adaptive filtering to improve the depth estimation. It avoids feature matching or tracking for complex street scenes and facilitates real time sensing. The method also stores much less data than a structure from motion approach does so that it can extend the sensing area significantly. Keywords: Depth from stationary blur, route panorama, 3D sensing.

1 Introduction For pervasive archiving and visualization of large-scale urban environments, mosaicing views from a translating camera and obtaining depth information has become an interesting topic in recent years. Along a camera path, however, overlapping consecutive 2D images perfectly is impossible due to the inconsistent motion parallax from drastically changed depths in urban environments. Approaches to tackle this problem so far include (1) 1D-2D-3D approach [1] that collects slit views continuously from a translating camera under a stable motion on a vehicle. The generated route panorama (RP) [2][3][4] avoids image matching and stitching. Multiple route panoramas from different slits are also matched to locate 3D features in the 3D space [1][4][5][6]. (2) 2D-3D-2D approach that mosaics 2D images through matching, 3D estimation [7], and re-projection to a 2D image. If scenes are close to a single depth plane, photomontage can select scenes seamlessly [8] to result in a multiperspective projection image. Alternatively, images at intermediate positions can also be interpolated to form a parallel-perspective image [9]. (3) 3D-1D-2D approach obtains a long image close to perspective images at each local position. The 1D sampling slit is shifted dynamically [10] according to a dominant depth measured by laser [11][12] or the image velocity in a video volume [13][14][15][16]. This work aims to scan long route panoramas and the depth with a 1D slit as it is the simplest approach without image matching. We analyze the stationary blur [2][24] ⎯ a perspective effect in the parallel projection due to using a non-zero width Y. Yagi et al. (Eds.): ACCV 2007, Part II, LNCS 4844, pp. 42–52, 2007. © Springer-Verlag Berlin Heidelberg 2007

Depth from Stationary Blur with Adaptive Filtering

43

slit. The degree of blurring is related to the scene depth from the camera path as well as the camera parameters [17]. By using differential filters to evaluate the contrast in the RP against the original contrast in the image, we can obtain depth measure at strong spatial-temporal edges. This paper further adjusts camera parameters such as the vehicle speed, camera focal length and resolution to increase the blur effect, which gains the sensitivity of the method and improve the depth estimation. Adaptive filtering for various depths is implemented to reduce the depth errors. In the next section, we first extend the path to a general curve to obtain a geometric projection of the route panoramas. Then we analyze the physical model of the slit scanning and introduce the stationary blur in Section 3. Section 4 is devoted to a depth calculation method and Section 5 develops a filtering approach adaptive to various depths. Section 6 introduces the experiments followed with a conclusion.

2 Acquisition of Route Panoramas Along Streets in Urban Areas We define the slit-scanning scheme model along a smooth camera path on a horizontal plane. A video camera is mounted on a four-wheeled vehicle moving at a speed V. Denoting the camera path by S(t) in a global coordinate system where t is the scanning time in frame number, such a path is an envelope of circular segments with changing curvature κ(t), where κ(t)=0 for a straight segment. The vehicle keeps V=|V| as constant as possible and the variation can be obtained from GPS. In order to produce good shapes in a route panorama, a vertical Plane of Scanning (PoS) is set in the 3D space through the camera focal point as the camera moves along a path. This ensures that vertical lines in the 3D space appear vertically in the route panorama even if the camera is on a curved path.

Fig. 1. A section of 2D RP from slit scanning. Different horizontal contrasts appear at different depths.

To create the route panorama, we collect temporal data continuously from the slit of one pixel width, which generates an RP image with time t coordinate horizontally and the slit coordinate y vertically (Fig. 1). A fixed sampling rate m (frame/sec), normally selected as the maximum reachable rate of the camera, is used for the scanning. At each instance or position on the path, a virtual camera system O-XYZ(t) can be set such that the image frame is vertical and the X axis is aligned with the moving direction V. Within each PoS, we can linear-transform data on the slit l to a vertical slit l’ in O-XYZ(t). This converts a general RP to a basic one that has a vertical and smooth image surface along the camera path. A 3D point P(X,Y,Z) in O-XYZ(t) has the image projection as

44

J.Y. Zheng and M. Shi

I ( x, y, t ) : x = Xf Z , y = Yf Z

(1)

where f is the camera focal length. The projection of P in the basic RP is then I (t , y ) = I ( x, y, t ) ' , calculated by x∈l

I (t , y ) : t = S r , y = Yf Z , r = V m

(2)

where V=|V|, S=|S|, and r (meter/frame) is the camera sampling interval on the path. We define a path-oriented description of the camera rather than using viewpoint orientated representation. As depicted in Fig. 2, the camera moves on a circular path with a radius of curvature R=1/κ, where κ<0, κ=0, and κ>0 for convex, linear, and concave paths, respectively. The camera translation and rotation velocities are V(V,0,0) and Ω(0,β,0), where β is a piece-wised constant related to the vehicle steering status and is estimated from GPS output. Because V is along the tangent of the path, a four-wheeled vehicle has a motion constraint as V =

∂ S (t ) = Rβ ∂t

(3)

where R and β have the same sign.

Fig. 2. Relation of circular path segments and the camera coordinate systems. (a) Convex path where R<0 and β<0, and (b) Concave path where R>0 and β>0.

On the other hand, the relative velocity of a scene point P(X, Y, Z) to the camera is ∂ P (t ) = −V + Ω × P (t ) ∂t

∂( X (t) Y(t) Z(t)) = −(V 0 0) + (0 β 0) × ( X (t) Y(t) Z(t)) ∂t

(4) (5)

When the point is viewed through the slit at time t, i.e., the point is on PoS, we have ∂ X (t ) = −V + β Z ( t ) ∂t

∂ Y (t ) =0 ∂t

β Z (t ) ∂ Z (t ) = − β X (t ) = − ∂t tan α

using tanα=X/Z. Taking temporal derivative of (1), the image velocity v is

(6)

Depth from Stationary Blur with Adaptive Filtering

v=

∂X ∂t ∂X ∂t ∂ Z (t ) ∂ t ∂x ∂ Z (t ) / ∂ t = f − fX = f −x 2 ∂t Z (t ) Z (t ) Z (t ) Z (t )

45

(7)

Filling in the results from (5) and (6) into (7), we obtain the image velocity on slit l as v=−

f (V − β Z ( t )) xβ fV V x + =− + (f + ) Z (t ) Z (t ) R tan α tan α

(8)

From (8), the depth Z(t) and the 3D point can be obtained by Z (t ) =

f 2V

fV = V x V (f + )− v (f R tan α R

2

X (t ) =

+ x ) − fv 2

Z (t ) tan α

, Y (t ) = Z (t ) y

(9)

f

where image velocity v<0 and slit position x=f× tanα. For a linear motion where β=0 (R=∞), (9) yields X (t ) = − f

V v tan α

Y (t ) = −

Vy v

Z (t ) = − f

V v

(10)

which is a traditional formula for depth calculation. These equations clarify the depth related to the image velocity at the sampling slit.

3 Stationary Blur and Different Sampling Ranges In the route panorama, we observed an image blur along the t direction on distant scenes, scenes on the concave side of a curved path, and scenes captured when the vehicle slows down. This blur, not appearing in a perspective projection image, can be seen in Fig. 1 on a distant house and backyard trees. The ideal projection model discussed in Sec. 2 cannot interpret this blur because the projection assumes a zerowidth slit; the PoS through the slit is extremely thin and the sampling positions are infinitively dense along a camera path. In a real situation as depicted in Fig. 3(a), the slit has a physical width and a RP is generated from a series of narrow perspective ’ Overlapped -sampling range

256

Justsampling Undersampling range

Zj Slit

Camera Path

S

0m Zj =8m

16m

32m

64m

Fig. 3. Real projection of a route panorama (top view) and the stationary blur. (a) Different ranges of sampling with consecutive cones on a camera path. (b) Simulation of projecting an ideal step edge from all depths (0-256m) to short RPs (15 pixels wide) for Zj = 8, 16, 32, 64m. The edge intensity distributions are piled with respect to depth.

46

J.Y. Zheng and M. Shi Table 1. Depth related image properties Depth

Sampling range

Image velocity

Trace slope in EPI

Scene sampling

Blurs

Feature width in RP

Z>Zj

Overlapped sampling Justsampling

|v|<1

>S/4

|v|=1

=S/4

Stationary blur in RP No blur

wider than in image Same as in image

|v|>1

<S/4

Overlapped sampling neither missing nor overlapping scenes Missing some details

Motion blur in images

Shorter than in image

Z=Zj Z
Undersampling

projections. Different depths, classified as just-sampling depth, under-sampling range, and overlapped-sampling range, have different sampling characteristics [17]. For scenes at the just-sampling depth, Zj, their slit views are connected in the RP without scene overlapping just as in normal perspective projection. In the undersampling range (Z
4 Depth Estimation from Blurs in Route Panoramas The stationary blur visible in the RP is sufficient to separate different depth layers by humans. Assuming the same vehicle speed and camera sampling rate, a simulation in Fig. 3(b) also shows the stationary blur related to the depth. One can notice that an edge becomes more blurred as the depth increases, even with different just-sampling depths. A question then arises as to whether the depth, or at least depth layers, can be computed inversely from the degree of the blur. Theoretically, the intensity formation of the RP undergoes two phases: the first is the convolution of a function G(0,W) of the cone with scenes and the second is the sampling at discrete locations. Width W of the cone is proportional to scene depth Z (Fig. 3(a)). The degree of blurring is thus related to the depth in terms of W. We first examine depth estimation from the stationary blur. Although the stationary blur is related to the depth, the contrast distribution in the RP is insufficient to determine the depth independently if the original spatial contrast is unknown. To obtain spatial contrast for points in the RP, we calculate differential value ∂I(x)/∂x=Ix around the slit in the images as the route panorama is scanned. Ix(t,y) is obtained at slit l if we widen the sampling slit to include several neighboring pixels. We also obtain temporal contrast within the route panorama by calculating ∂I(t)/∂t=It, which reflects the contrast change after the stationary blurring. The ratio of the spatial and temporal differentials can be related to the image velocity v because

Depth from Stationary Blur with Adaptive Filtering

v=

I (t , y ) ∂x ∂I ∂t =− =− t I x (t , y ) ∂t ∂I ∂x

47

(11)

Thus the depth at point (t,y) can be computed locally from two blurs (or equivalently contrasts) and vehicle translational and rotational parameters according to (10). The measure of v from spatial and temporal differentials is similar to depth from optical flow but is performed along the RP rather than in the images. Because Ix and It are unreliable at low contrast points, two additional feature selection steps are added. (a) To avoid disturbance from features at a different height due to the vehicle shaking and waving, we avoid near-horizontal features in the depth estimation, i.e.,|∂I(t,y)/∂y|<δy where δy is a threshold. (b) The original contrast level affects Ix and It and then their ratio. We select reliable edges with high contrasts either in temporal or in spatial domain for depth estimation. A spatial-temporal gradient g(t,y) influenced neither from motion blur nor stationary blur is calculated as

g (t , y) = ( I x (t , y)) 2 + ( I t (t , y)) 2

(12)

Feature points satisfying g(t,y)>δ are selected for depth estimation. Using local data to yield depth instantly can avoid many complex issues such as occlusion, dynamic objects, and lack of feature in sensing outdoor scenes. On the other hand, the local image velocities may still produce unreliable depth because of the measure in a narrow region around the slit and the digitalized errors in intensities. One can also implement a global method to evaluate blurs of a region for depth if an RP can be segmented successfully [17], because a global approach uses more evidence and the result will be more robust.

5 Increasing Sensitivity of Depth Measure and Adaptive Filtering How to improve the estimation of blur with more stable filters? It is noticeable in Fig. 3(b) that the degree of blurring or contrast may not be sensitive to the depth change because of the nature of convolution with the sampling cone. The sensitivity of depth from stationary blur also depends on the just-sampling depth (Fig. 3(b)). A large Zj makes the projection close to an ideal parallel-perspective projection, while a small Zj increases the perspective effect in the RP and the stationary blur. The just-sampling depth is further determined from the camera focal length f and the image resolution according to Fig. 3(a). A large f leads to a large Zj, and yields insignificant contrast changes with respect to the depth variation. All of these can be quantified as: W=

rZ Zj = rf, thus W = Z / f or Z = fW Zj

(13)

Although a large f is preferred for an ideal parallel-perspective projection image without the stationary blur (Fig. 3(b)), a parallel projection contains less distance information. Hence, depth measured under a small f can improve the depth accuracy. We should enhance perspective effect of the physical slit by reducing Zj as follows: (1) selecting a wide angle lens optically, which also fits the goal of route scanning to take high buildings into the RP; (2) summarize several pixel lines into one line that

48

J.Y. Zheng and M. Shi

reduces the slit resolution. In general, the vehicle speed, camera sampling rate, and path curvature cannot be changed freely onsite due to the path (street) and device limitations.

Fig. 4. Relation of the camera focal length and the image velocity when viewing the same scene. The figure illustrates two camera focal lengths resulting in different FOVs as well as cones in (a) and (b) respectively. Top, RPs, and EPI images are depicted in order. Traces of two objects in the EPI have orientations corresponding to the image velocities.

As depicted in the Epipolar plane image (EPI) of Fig. 4, the trace of a feature at Zj has angle π/4 in penetrating the RP according to the definition of just-sampling depth. If we increase focal length f under a fixed image resolution, the image velocity also increases. If Zj is close to infinity, the cone approaches a line of sight on the PoS. All feature traces become parallel and horizontal in the EPI (|v|→∞ according to (10)); the perspective effect disappears. Inversely, if f is reduced, the image velocity is lowered under the same vehicle speed V (see (10)). In Fig. 4(b), a shorter f results in a larger cone and a wider image FOV. This yields more vertical traces in the EPI than in Fig. 4(a), because the FOV in Fig. 4(a) is now reduced in Fig. 4(b). Eventually, the shorter the f, the more the stationary blur appears in the RP. The widths of features at distant depths are also extended. To obtain stable temporal contrast It(t, y), we also examine the data coherence in the RP. Under the thin perspective projection model of slits, a distant edge is wider in the RP than a close edge. If scene depth is far, a small size filter applied horizontally on the RP cannot detect subtle changes for It(t, y); it may result in zero or only few levels due to the digitized error. This affects the depth accuracy at distant range. We thus utilize wide filters in the temporal domain and the outputs are scaled by the filter sizes. Such a filter catches detailed changes of It(t, y) in a longer duration for finer levels of depth. At a close range, however, a large filter may include some distant background when it filters a close occluding edge. Hence, a small size filter is better in locating depth of close features in the RP.

Depth from Stationary Blur with Adaptive Filtering

49

We prepare a set of differential filters, ∂G(0,Wi)/∂t, to compute It(t, y) adaptively, where G is Gaussian distribution and Wi=5, 9, 13 pixels. Starting with the largest size filter, we detect It(t, y) at distant edges. Only strong edges at both distant and close ranges will respond. Then, we apply a smaller filter and then the smallest one on the RP consecutively. These filters will no longer respond to distant features but response to close features that are narrow and sharp in the RP. If the new output at a point becomes lower, the point must be far away and the previous value is kept. If the output is higher than the previous one, the feature is a close or detailed one and It(t,y) is updated accordingly.

6 Systems and Experiments Our strategy for the depth estimation is based on the local stationary blur in the RP against the contrast captured in the images, along with the vehicle motion V, R (or β) from GPS that records the path information independently. Using separate sensors for motion parameters are crucial for a sustainable sensing system; this prevents failures in the image-based motion estimation due to lack of features or existence of complex features (e.g., full of occlusions from trees, poles and buildings on the street). We have driven a vehicle through a number of streets to record route panoramas. We keep the calculation of the spatial differential within a stripe around at the slit that is much smaller than the image patches used in feature matching and stitching by other approaches. The motion blur may affect feature matching in the structure from motion. With the route panorama, however, no additional processing of the motion blur is required. Our filtering method uses g(t,y) that is invariant to the motion blur in feature selection. To increase the FOV and stationary blur, and reduce under-sampling at close scenes as well, we use wide angle lenses (30mm conversion lens attached to 44mm camcorder lens) during the scanning. We have also tried a fish-eye lens to create maximum stationary blur optically, however, the image quality of RP is not guaranteed for visualization (scenes getting very small). If the sampling speed of the slit can be maintained, we can average pixels in a wider image stripe (3, 5, 9, … pixels wide) to obtain the slit data for depth estimation in real time, keeping the fine slit of one pixel wide for generating the RP for visualization. The slit sampling rate is 60 fps for a video camera and the vehicle speed is about at 15~36 km/hr. Along with the short focal length of camera, a slow move of the vehicle determines a short justsampling depth Zj. The just-sampling depth is roughly set at the front face of buildings and houses, but it is hardly controllable onsite dynamically. The vehicle can move faster at 45 km/hr if side scenes are predicted to have large depths, because the image velocities of scenes can still be low. The maximum height of the RP is 640 pixels and can be increased if a high definition camera is used. An example of the depth is shown in Fig. 5, where three depth layers such as front trees and parked cars, front face of houses, and backyard trees and houses are included. The contributions of the depth from adaptive filters of three sizes for It are also displayed partially in colors. We can notice in Fig. 6 the depths of distant scenes are more calculated from a wide filter, while close and detailed scenes are from the small filters (see large images in [25]).

50

J.Y. Zheng and M. Shi

Fig. 5. Depth estimation by an adaptive filter. Bright points are closer than dark ones. Houses in the continuous RP show their depth differences from backyard trees and front trees.

Fig. 6. Depths calculated from three filters of different sizes are displayed in R, G, B channels. Red and yellow points (on front trees and brick details) are calculated from small and median size filters. Blue and cyan points (back trees) indicate their depths are from large and median filters. See enlarged color figure in web [25].

To verify the obtained depth data at strong temporal-spatial edges in Fig. 5, we fill empty areas as much as possible to form depth regions (Fig. 7) such that houses and distant trees identifiable. Note that this filling is different from building a complete model and may be less precise than the measure because no segmentation has been implemented. In the result, many glass windows have a more distant depth measured (darker than house fronts), which is correct because scenes reflected from the mirror windows have more than doubled distances from the path. Some variation of the depth is given in Fig. 8. We can notice that the house fronts with solid edges have relatively small error while distant trees may have a large variation in their depths.

Fig. 7. Depth map with empty points filled up for verification. According to the measured depths at points with strong values of spatial-temporal gradient, horizontal and then vertical interpolation are carried out in order.

Depth from Stationary Blur with Adaptive Filtering

51

Fig. 8. Estimating depths of edges and their variances in vertical planes. (a) Regions outlined from Fig. 7 to examine depth values, (b) Means and variances of depth values in the specified regions, (c) Sorted means (the horizontal axis is region index).

7 Conclusion This work developed a depth estimation approach for urban streets based on the stationary blur in the route panoramas. Through an elaborate analysis of blurs, motion, and image sampling, we proposed an algorithm that generates depth layers efficiently according to the sharpness in the route panorama compared to the original image sharpness. The adaptive filtering of the route panorama avoids feature matching and tracking, which is less influenced from occlusion, motion blur, and other complex situations confronted in complex urban environments. From the data storage perspective, the stored spatial differential image and route panorama are much smaller in size than the image sequence for stitching and the EPIs for tracking; this compactness benefits system development.

References [1] Zheng, J.Y., Tsuji, S.: Panoramic representation for route recognition by a mobile robot. IJCV 9(1), 55–76 (1992) [2] Zheng, J.Y.: Digital route panorama. IEEE Multimedia 10(3), 57–68 (2003) [3] Seitz, S., Kim, J.: Multiperspective imaging. IEEE CGA 23(6), 16–19 (2003) [4] Gupta, R., Hartley, R.: Linear push-broom cameras. IEEE PAMI 19(9), 963–975 (1997) [5] Seitz, S., Kim, J.: The space of all stereo images. IJCV 48(1), 21–38 (2002) [6] Zhu, Z., Hanson, A.R.: 3D LAMP: a new layered panoramic representation. In: ICCV 2001, vol. 2, pp. 723–730 (2001) [7] Wang, A., Adelson, E.H.: Representing moving images with layers. IEEE Trans. Image Processing 3(5), 625–638 (1994)

52

J.Y. Zheng and M. Shi

[8] Agarwala, A., et al.: Photographing long scenes with multi-viewpoint panoramas. ACM Trans. Graphics 25(3), 853–861 (2006) [9] Zhu, Z., Hanson, A.R., Riseman, E.M.: Generalized parallel-perspective stereo mosaics from airborne video. IEEE Trans. PAMI 26(2), 226–237 (2004) [10] Roman, A., Garg, G., Levoy, M.: Interactive design of multi-perspective images for visualizing urban landscapes. In: IEEE Conf. Visualization 2004, pp. 537–544 (2004) [11] Zhao, H., Shibasaki, R.: A Vehicle-borne urban 3D acquisition system using single-row laser range scanners. IEEE Trans. on SMC B33(4) (2003) [12] Frueh, C., Zakhor, A.: Constructing 3D city models by merging ground-based and airborne views. In: IEEE CVPR 2003, pp. 562–569 (2003) [13] Baker, H., Bolles, R.: Generalizing epipolar-plane image analysis on the spatial-temporal surface. In: Proc. CVPR 1988, pp. 2–9 (1988) [14] Li, Y., Shum, H.Y., Tang, C.-K., Szeliski, R.: Stereo Reconstruction from Multiperspective Panoramas. IEEE PAMI 26(1), 45–62 (2004) [15] Zhu, Z., Xu, G., Lin, X.: Efficient Fourier-based approach for detecting orientations and occlusions in Epipolar plane images for 3D scene modeling. IJCV 61(3), 233–258 (2004) [16] Zomet, D., Feldman, D., Peleg, S., Weinshall, D.: Mosaicing new views: the crossed-slits projection, IEEE Trans. PAMI, 741–754 (2003) [17] Shi, M., Zheng, J.Y.: A slit scanning depth of route panorama from stationary blur. IEEE CVPR (2005) [18] Potmesil, M., Chakravarty, I.: Modeling motion blur in computer-generated images. In: SIGGRAPH 1983, pp. 389–400 (1983) [19] Fox, J.S.: Range from translational motion blurring. In: IEEE CVPR 1988, pp. 360–365 (1988) [20] Ben-Ezra, M., Nayar, S.K.: Motion deblurring using hybrid imaging. In: IEEE CVPR 2003, pp. 657–665 (2003) [21] Aliaga, D.G., Carlbom, I.: Plenoptic Stitching: A scalable method for reconstructing 3D interactive walkthroughs. In: SIGGRAPH 2001 (2001) [22] Zheng, J.Y.: Stabilizing route panorama. In: 17th ICPR, vol. 4, pp. 348–351 (2004) [23] Ikeuchi, K., Sakauchi, M., Kawasaki, H., Sato, I.: Constructing virtual cities by using panoramic images. IJCV 58(3), 237–247 (2004) [24] Zheng, J.Y., Shi, M.: Removing temporal stationary blur in route panorama. In: 18th ICPR, vol. 3, pp. 709–713 (2006) [25] http://www.cs.iupui.edu/ jzheng/RP/IJCV

Three-Stage Motion Deblurring from a Video Chunjian Ren1 , Wenbin Chen2 , and I-fan Shen3 School of Computer Science and Technology, Donghua University, P.R. China 2 School of Mathematical Sciences, Fudan University, P.R. China Department of Computer Science and Engineering, Fudan University, P.R. China 1

3

Abstract. In this paper, a novel approach is proposed to remove the motion blur from a video, which is degraded and distorted by fast camera motion. Our approach is based on the image statistics rather than the traditional motion estimation. The image statistics has been successfully applied for blind motion deblurring for a single image by Fergus et al [3] and Levin [10]. Here a three-stage method is used to deal with the video. First, the ”unblurred” frames in the video can be found based on the image statistics. Then the blur functions can be obtained by comparing the blurred frames with the unblurred ones. Finally a standard deconvolution algorithm is used to reconstruct the video. Our experiments show that our algorithms are efficient.

1

Introduction

Motion blur is caused by the relative motion between the camera and the scene during exposure time of each frame, which degrades and distorts the video frames. A video is always expected to be of high-quality so that people feel comfortable when they enjoy it, but the motion blur will spoil this. An unblurred video content is needed in many applications, such as the tracking and surveillance system. The video camera, especially the consumer digital cameras which are designed for hand-hold, is usually not fixed by the stabilization systems. One approach that reduces the degree of blur is to increase the temporal resolution, which raises the frame-rate. But this approach can cause other problems such as expensive hardware and sensor noise. An alternative effective solution is to adopt image deconvolution with the point spread function (PSF, also know as blur function). However, the PSF is not usually known, and then the process becomes a blind deconvolution. Under this circumstance, the estimation of unknown blur function is required. Direct estimation of PSF from a single frame is very difficult and inefficient. Traditionally, in order to enhance the video, the motion estimations are performed and then used to reconstruct the video frames. Such methods are complicated and computationally intensive when every frame is taken into account. Moreover, if the blur is large and objects in the video are distorted, motion estimation in a blurred video is sometimes incorrect and even infeasible. The resulting frames can be degraded significantly due to the incorrect PSFs in the deconvolution stage. Y. Yagi et al. (Eds.): ACCV 2007, Part II, LNCS 4844, pp. 53–62, 2007. c Springer-Verlag Berlin Heidelberg 2007 

54

C. Ren, W. Chen, and I.-f. Shen

In this paper we propose a new approach to remove the motion blur from a video. We first divide the video into shots and extract unblurred frames in terms of the specific statistical property. Then we search for a one-dimensional PSF and further adopt the Genetic Algorithm (GA) to refine the PSF. The key novelty of our method is enumerated as follows: (1) The distributions of blurred frames is compared with that of unblurred ones so that there is no need of motion estimation. (2) Blurred frames are contrasted with adjacent unblurred ones so that the PSFs estimation is effective. (3) Genetic Algorithm is adopted to refine the PSFs so that the results are better. 1.1

Related Works

The motion deblurring problem involves two main parts: PSF estimation and image deconvolution. PSF estimation is an active area of research in computer vision and image analysis. Early deblurring methods usually assume that the blur function has a simple parametric form such as a Gaussian component. Recent some breakthrough progresses have been made in this area. Ben-Ezra and Nayar [1] exploit the fundamental trade off between spatial resolution and temporal resolution to construct a hybrid camera. This special prototype system can measure its own motion during image integration. Fergus et al [3] compute the complex blur kernel with maximum marginal probability, using derivatives distribution in an unblurred image. Our approach also relies on image statistics, but the difference is that our approach uses the distribution of unblurred frames as known distribution and avoids the need of estimating the latent unblurred image in every step. Deconvolution algorithms have been developed extensively [7, 11, 13, 17]. In [16], Rav-Acha and Peleg estimate two one-dimensional PSFs simultaneously by using two motion blurred images, which have different blur directions. All other approaches for video enhancement [18, 20] need to do complicated motion estimation. Thus, our approach avoids the need of estimating the complex motion in these blurred frames. Another similar research on this subject is motion deblurring of moving objects from a single image [10, 15]. Genetic algorithms are now widely applied in science and engineering as adaptive algorithms for optimizing practical problems [5, 9]. In [4,12], the literature is expounded in detail. The GA is used to refine the PSF to reduce the visible artifacts in our method.

2

Model of Image Statistics

Natural image statistics typically obey specific distribution of gradients. The shape of gradients distribution can be influenced significantly by motion blur. This property has been used for deblurring a single image [3, 10], which sheds light on our approach. For comparing distributions, the approach needs known distributions for reference. These known distributions must be those of the unblurred image since they are critical for identifying the blur function.

Three-Stage Motion Deblurring from a Video

0

Log probability density

Log probability density

0

using [1 −1] using [1 −1]’

−2

−4

−6

−8

−10

−12 −0.8

55

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

0.8

Gradient

using [−1 0;0 1] using [0 −1;1 0]

−2

−4

−6

−8

−10

−12 −0.8

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

0.8

Gradient

Fig. 1. Oblique direction of motion blur can be distinguished by distributions of gradient computed with Roberts operators. Left: A frame with oblique motion blur. Middle: Distributions of Horizontal gradient and vertical gradient can not distinguish the blur clearly. Right: Distributions computed with Roberts operators obviously distinguish the blur.

Observing the histograms of the adjacent frames, it is evident that the distributions of two frames are very close when they are captured within the same scene and both are unblurred. The moving objects with low velocity do not affect the evidently close distribution, so the distributions of unblurred frames can be taken as known distributions. Comparing the histograms of the vertical gradient with the horizontal gradient in a single image, the horizontal or vertical blur can be identified easily and effectively, as in [10]. In the other case, how does the shape of histogram change if the direction of motion is diagonal? As in Fig. 1 (middle), the shape of histograms is so similar that it is hard to tell the direction of the blur. In order to solve this problem, two other directional derivatives are added by using Roberts operators. As demonstrated in Fig. 1 (right), the Roberts operators are chosen because the histograms of gradient computed by them can distinguish the oblique blur. Please note that, all the histograms illustrated here perform logarithmic on distributions. By employing these four histograms, the unblurred frames in a video can be found in section 3 and the blur functions can be estimated in section 4.

3

Extracting Unblurred Frames

A video usually contains many shots [14] and the frames in a shot show consecutive content. The comparison of two frames within two shots may lead to failure because of the inconsecutiveness of their content. In our experiments, the video is divided into shots by the existing simple method [14]. Given a frame sequence within a shot, we first get four distributions of each frame (see section 2). Let It denote a frame in a video, di means corresponding gradient operator: d1 = [1, −1], d2 = [1, −1]T , d3 = [−1, 0; 0, 1], d4 = [0, −1; 1, 0]. The image gradients are gained through the convolution di ∗ It . The histogram of gradients is normalized to obtain the discrete gradients distributions: P (di ∗ It ) ∝ hist(di ∗ It ) ,

(1)

56

C. Ren, W. Chen, and I.-f. Shen

where P (·) denotes the computing gradient distribution. We assume there are n quantities on the x-axis of the distribution and Pk is the value of the kth quantity. The unblurred frame Is maximizes total value of logarithmic distributions: Is = arg max It

4  n 

log Pk (di ∗ It ) .

(2)

i=1 k=1

Here we use the logarithmic operation to magnify the difference between the two distributions that are compared with each other. It is possible that a lot of frames in one sequence are unblurred due to the irregular camera motion. Therefore, a threshold can be used to extract more than one frame in each sequence. Under some circumstances, there may be no unblurred frames in a sequence but the approach above can also be done. And the frames extracted are relatively ”unblurred”.

4

Estimating the Motion Blur Function

As analyzed in section 2, the blur of the frames can be identified using image statistics. By comparing the distributions of unblurred frame with the distributions of an adjacent blurred frame, the PSF of the blurred one can be estimated. The exposure time of video camera with one frame is mostly very short, so the PSF is always similar to one-dimensional form. Considering the justifiable performance and efficiency, we first estimate a one-dimensional PSF and then refine it to a more complex and suitable one. 4.1

General Estimation

We first treat the PSF as a one-dimensional form, which is similar to video camera motion path in one frame exposure time. This kind of PSF has two parameters: direction (α) and length (). Each entry of PSF equals to 1/. As addressed in [1, 3], PSF can be represented by a convolution kernel (also know as blur kernel) which satisfies an energy conservation constraint (total value of kernel equal to 1). Our method brings the distributions of unblurred frame close to the distributions of blurred frame. The length and direction of PSF can be estimated simultaneously. Let P (·) and Q(·) describe the distribution of unblurred and blurred frames respectively. We minimize a K-L divergence which represents the distance of distributions. The distance can be formulated as: EKL (P ||Q) =

4  n  P (i, k) )) , ( (P (i, k) log Q(i, k) i=1

(3)

k=1

where P (i, k) = Pk (di ∗ K(α, ) ∗ Is ) ,

(4)

Q(i, k) = Qk (di ∗ Ib ) .

(5)

and

Three-Stage Motion Deblurring from a Video

57

Here Ib is a blurred frame and Is is an adjacent unblurred frame, K denotes the blur kernel and it has two parameters α and , and di denotes the gradient operator. The cost function (2) is minimized by the following formula: α,  = arg min EKL (P ||Q) . α,

(6)

The indexes α and  can be determined using an exhaustive search over the angles and length. For each possible values of α and , we compose a blur kernel and compute the cost function. The kernel of blurred frame will be found when the cost function is minimized. In order to be more efficient, we can initialize all the possible blur kernels before we compute the cost function. Moreover, the kernel of α is equivalent to the kernel of π + α, so only α ∈ [0, π) is taken into account. When the length is small, not every angle is needed to be computed. 4.2

Refining the Blur Kernel

The blur frame can be deconvolved with the blur kernel which has been estimated to get a unblurred frame, but sometimes the result contains visible artifacts. As pointed out in [1, 3], the kernel may contain a complex form. The genetic algorithm is adopted to refine the blur kernel, in order to produce a better form.

Fig. 2. Left: The pretreatment of encoding. Right: A example of blur kernel encoding. Zeros in the chromosome are replaced by a nonzero numerical value which is smaller than 1/.

Encoding. A nonzero numerical value is used to replace each zero in the chromosome. The numerical value is much smaller than 1/, as in Fig. 2. And each chromosome in the first generation is initialized: 

ei = randi · ei , 

for i = 1, ..., m . 

ei = ei /

m 



ei .

(7)

(8)

i=1 





So the chromosomes are er (er1 , er2 , ..., erm ), r = 1...50. The population size used here is 50. The equation (6) makes the chromosomes varied. The equation (7) does the normalization to satisfy energy conservation constraint.

58

C. Ren, W. Chen, and I.-f. Shen

Genetic Operators. In the selection stage, 10% of best fitness chromosomes are directly copied to the new population. This way ensures that we can keep the best chromosomes to the next generation at each iteration. The new population is also expected to be of variation, so the crossover rate is set to be 90%. The chromosomes to be crossed are randomly selected. Two crossover operations are used in our approach. Single point crossover is used when  ≤ 5 (: the length of PSF), and two point crossover is used when  > 5. Mutation is important in some applications. It can avoid the local minimum. But it is not essential sometimes, as in [9]. Mutation has little effect in our approach, so we do not use mutation.

5

Experiments

Fig. 3 shows the flowchart of our algorithm.

Fig. 3. Flow chart showing the stages of Motion Deblurring

Given the blur kernels, a non-blind deconvolution algorithm is used to reconstruct the blur frames. Following [3], the Matlab’s implementation of the Richardson-Lucy deconvolution algorithm (deconvlucy) is used here. In practice, we can deconvolve each frame after its blur kernel has been estimated or after all the blur kernels have been estimated. Our algorithm has been applied to the image sequences of some real scenes, and these sequences were reconstructed successfully. As shown in Fig. 5 and Fig. 7. The final kernel produced by our method can be a complex form. In [3], their algorithm can deal with very complicated and large blur, but it is less efficient.

Three-Stage Motion Deblurring from a Video

59

Fig. 4. Left: Blurred frame. Middle: Deblurring using Matlab’s blind deconvolution algorithm deconvblind. The algorithm is initialized with a Gaussian blur kernel. Right: Output of our algorithm.

Fig. 5. Left column: Blurred frames. Right column: Output of our algorithm and the inferred blur kernels.

As reported in [3], with the minimum practical patch (size of 128 × 128), it takes 10 minutes for Matlab implementation. Our algorithm takes less than 1 minute for every blurred frame (size of 320 × 240). Therefore, our method is more suitable for processing video. We also compared our algorithm against Matlab’s blind deconvolution function (deconvblind ). This function provides the implementation of the methods of Biggs and Andrews [2] and Jansson [7]. These methods also estimate the blur kernel and adopt the Richardson-Lucy deconvolution algorithm. We used the Matlab’s deconvblind with a Gaussian blur kernel input, similar in size to blur artifacts, as shown in Fig. 4.

60

C. Ren, W. Chen, and I.-f. Shen

Fig. 6. A synthetic example is shown. The 1st is a ground truth image. The 2nd is a blurred image created by convolution of the 1st image with a known blur kernel. The 3rd image is the result of deconvolution using the known kernel. The 4th image is the output of our algorithm using both the 1st and the 2nd images.

Fig. 7. Left column: Blurred frames. Right column: Output of our algorithm and the inferred blur kernels.

We adopt a synthetic example to run our algorithm for validation. We use a known kernel to blur a natural image, and then reconstruct the blurred image by using both blurred and unblurred images as our algorithm’s input. As shown in Fig. 6, the final blur kernel estimated by our method is similar to the original known kernel. We also observe in experiments that the artifacts may still be introduced in the restored image, even adopting deconvolution that uses the known kernel.

Three-Stage Motion Deblurring from a Video

6

61

Discussion

This paper introduces a method for motion deblurring from a video without performing the motion estimation. For higher efficiency, we first estimate the one-dimensional PSF, but do not limit it to the simple form. We further refine it to one complex form in order to reduce the artifacts. Our work also includes unblurred frames extracting, which is simple and effective. This method is also useful for other applications, like blur detection. We assume a uniform blur over one frame and do not deal with objects motion blur. The problem is more complicated when camera motion deblurring is coupled with objects motion deblurring from a frame. But sometimes it is also worthy of research, for instance, tracking the fast moving objects when both objects and the background are blurry. We use an exhaustive search for finding out the one-dimensional PSF. This method is faster than traditional motion estimation algorithm, but it may still not satisfy other high demand real-time applications. In future work, we should adopt a fast algorithm which can also find the optimal solution. It will also be interesting to perform shape detection or object detection in a blurred image. This may require a method different from the past, if we do not do the deblurring first.

Acknowledgements We thank anonymous reviewers for their suggestions. We are most grateful to Xiaoli Cai, Qian Cui for their helpful discussions and support. This work was supported by NSFC under contract 60473104 and the National Basic Research Program under the Grant 2005CB321701.

References 1. Ben-Ezra, M., Nayar, S.K.: Motion-based motion deblurring. IEEE Trans. on PAMI 26, 689–698 (2004) 2. Biggs, D., Andrews, M.: Acceleration of iterative image restoration algorithms. Applied Optics. 36, 1766–1775 (1997) 3. Fergus, R., et al.: Removing camera shake from a single photograph. In: SIGGRAPH (2006) 4. Goldberg, D.: Genetic algorithms in search, optimization, and machine learning. Addison-Wesley, Reading, MA (1989) 5. Herrera, F., Lozano, M.: Gradual distributed real-coded genetic algorithms. IEEE Trans. on Evolutionary Computation 4, 43–63 (2000) 6. Irani, M., Peleg, S.: Improving resolution by image registration. Graphical Models and Image Processing. 53, 231–239 (1991) 7. Jansson, P.A.: Deconvolution of images and spectra. Academic Press, London (1997) 8. Jia, J., Tang, C.: Image registration with global and local luminance alignment. In: ICCV (2003)

62

C. Ren, W. Chen, and I.-f. Shen

9. Koza, J.: Genetic programming: on the programming of computers by means of natural selection. MIT Press, MA (1992) 10. Levin, A.: Blind motion deblurring using image statistics. In: NIPS (2006) 11. Lucy, L.: Bayesian-based iterative method of image restoration. Journal of Astronomy 79, 745–754 (1974) 12. Mitchell, M.: An introduction to genetic algorithms. MIT Press, Cambridge, MA (1996) 13. Raj, A., Zabih, R.: A graph cut algorithm for generalized image deconvolution. In: ICCV (2005) 14. Rasheed, Z., Shah, M.: Scene detection in Hollywood movies and TV shows. In: CVPR (2003) 15. Raskar, R., Agrawal, A., Tumblin, J.: Coded exposure photography: motion deblurring using fluttered shutter. In: SIGGRAPH (2006) 16. Rav-Acha, A., Peleg, S.: Two motion-blurred images are better than one. Pattern Recognition Letters, 311–317 (2005) 17. Richardson, W.: Bayesian-based iterative method of image restoration. Journal of the Optical Society of America. 62, 55–59 (1972) 18. Shah, N.R., Zakhor, A.: Resolution enhancement of color video sequences. IEEE Trans. on IP 8, 879–885 (1999) 19. Simoncelli, E.P.: Statistical modeling of photographic images. In: Handbook of Image and Video Processing (2005) 20. Tom, B.C., Katsaggelos, A.K.: Resolution enhancement of video sequence using motion compensation. In: ICIP (1996)

Near-Optimal Mosaic Selection for Rotating and Zooming Video Cameras Nazim Ashraf, Imran N. Junejo, and Hassan Foroosh School of Electrical Engineering and Computer Science, University of Central Florida

Abstract. Applying graph-theoretic concepts to solve computer vision problems makes it not only trivial to analyze the complexity of the problem at hand, but also existing algorithms from the graph-theory literature can be used to find a solution. We consider the challenging tasks of frame selection for use in mosaicing, and feature selection from Computer Vision, and Machine Learning, respectively, and demonstrate that we can map these problems into the existing graph theory problem of finding the maximum independent set. For frame selection, we represent the temporal and spatial connectivity of the images in a video sequence by a graph, and demonstrate that the optimal subset of images to be used in mosaicing can be determined by finding the maximum independent set of the graph. This process of determining the maximum independent set, not only reduces the overhead of using all the images, which may not be significantly contributing in building the mosaic, but also implicitly solves the “camera loop-back” problem. For feature selection, we conclude that we can apply a similar mapping to the maximum independent set problem to obtain a solution. Finally, to demonstrate the efficacy of our frame selection method, we build a system for mosaicing, which uses our method of frame selection.

1 Introduction Recently, there has been growing interest in applying graph-based approaches to solve problems in pattern recognition, computer vision, and robotics. This is not surprising owing to the fact that graph-based approaches have a very attractive feature: once the problem at hand has been mapped to an existing graph-theory problem, it not only becomes trivial to analyze the problem, but a number of existing algorithms - from the graph-theory literature - can be used to solve the problem. Therefore, concepts from graph theory have been used in a number of computer vision problems. There are two strands to this paper: First, we focus on the famous problem of finding the maximum independent set of a graph, and discuss its possible uses in computer vision. Namely, two different yet similar problems from computer vision, and machine learning are discussed. These are the (i) Frame selection problem, and (ii) Feature selection problem. We provide complete complexity analysis of the first problem, and arrive at its solution by mapping it into the maximum independent set problem; for the second problem, we give an intuitive understanding of how it can be similarly mapped to the maximum independent set problem and solved. Secondly, in order to prove the efficacy of our frame selection method, we build a system for mosaicing, which uses our method of frame selection. Y. Yagi et al. (Eds.): ACCV 2007, Part II, LNCS 4844, pp. 63–72, 2007. c Springer-Verlag Berlin Heidelberg 2007 

64

N. Ashraf, I.N. Junejo, and H. Foroosh

Frame Selection Problem for Mosaicing: Registering a set of images to form a larger image mosaic has been an active subject of research because of its many different applications [1,7]. A video sequence normally contains a large number of frames with little displacement between consecutive frames. Using all the frames for mosaicing is a time consuming process due to redundant information processing. Hence, we want to use as few frames as possible but we also want to maximize the information contents. Although mosaicing has received tremendous attention from researchers, little research has focused on the problem of selecting the frames for use in mosaicing; and current systems either use all the frames in a video sequence or the frames are manually selected. We approach this problem from a graph-theoretic approach. We demonstrate that we can select a near-optimal subset of images to be used in images by mapping the problem into the existing graph theory problem of finding the maximum independent set of a graph. This scheme not only reduces the overhead of using all the frames but also solves the “camera loop-back” problem. Feature Selection Problem:Feature selection is fundamental in a number of different tasks such as classification [9], tracking [2], image processing [14], conceptual learning and in many others [15,11]. In recent times, the growing importance of knowledge discovery and data-mining approaches in practical applications has made the feature selection problem quite an attractive topic, especially when considering the mining of knowledge from real-world databases or warehouses, containing not only a huge amount of records, but also a significant number of features not always relevant for the task at hand. Graph-theory and Mosaicing: Applying graph theory concepts for mosaicing have been used by many. Of particular interest is the research done by Sawhney et al. [13], and Kang et al.[10]. Both employ graph-theoretic methods to build coherent mosaics. Sawhney et al. [13] uses topology determination, local and global alignment parameters to build consistent mosaics. This technique introduced constraints introduced by nonconsecutive yet spatially neighboring frames. To address the same problem of global consistency in the mosaic, Sawhney et al [10] builds a frame graph representing the temporal and spatial connectivity of the frames. Then for each node of the frame graph, they locate a number of grid-points of the mosaic. Hence, each node has a list of gridpoints, and each grid-point has a list of correspondences to other grid-points located in other nodes. It is then demonstrated that the problem of global consistency translates to finding the optimal path in the resulting graph. The rest of the paper is organized as follows: maximum independent set problem is described in section 2.1. The complexity analysis of the frame selection problem, its mapping into the maximum independent set, and greedy solution is formulated in section 2.1; and an intuitive method of solving feature selection problem is presented in section 2.3. Image mosaicing is described in section 3. Finally we discuss the results in section 4 and conclude in section 5.

2 Graph-Theoretic Approach In this section, we first analyze the complexity of the frame selection problem, and map it to the maximum independent set problem to find possible solutions. Secondly, we

Near-Optimal Mosaic Selection for Rotating and Zooming Video Cameras

65

also provide an intuitive understanding of how the feature selection problem can also be solved in a similar manner. 2.1 Maximum Independent Set Problem Before proceeding let us first define the maximum independent set problem: In graph theory, an independent set in a graph, G = (V, E), is defined as the set of vertices V  (a subset of V ) such that for every two vertices in V  , there is no edge connecting the two. Therefore, each edge in the graph is incident to at most one vertex in the set. The size of an independent set is the number of vertices it contains. A maximum independent set is a largest independent set for a given graph. The problem of finding such a set is called the maximum independent set problem. This problem is proved to be NP-hard [5]. We can find many approximate efficient algorithms for determining the maximum independent set in the literature. Hence, if some problem can be mapped to the maximum independent set problem, then these existing algorithms can be used to solve that problem. 2.2 Frame Selection Problem In this section, we first define the problem in mathematical terms and analyze its complexity. Then, we look at near-optimal solution by mapping our problem to maximum independent set problem. Mathematical Model. In order to analyze the problem of determining which frames in the sequence should be used for mosaicing, we first need to model the problem in mathematical terms. We define a set F of frames, and for each frame we define a set of inter-frame overlaps γ(fi , fj ) ∈ [0, 1], ∀fi , fj ∈ F and i = j. The inter-frame overlap measures the percentage overlap between two frames. In addition, we need a merit which can measure the efficiency of a given frame subset. We can define the merit as: k MF  =  k + (k − 1)¯ γ

(1)

where k is the number of frames in F  , and γ¯ is the average inter-frame overlaps for frames in F  . The above formulation implies that (i) a higher number of frames, k, are more desirable; but (ii) the average inter-feature overlap should be as low as possible. This formulation fulfils our definition of a good frame subset because it would maximize the information content while minimizing the number of frames to use thus reducing the processing time. In other words, we need to find a subset of frames, which gives the highest merit possible. Complexity Analysis. Since, our problem is an optimization problem, we have to analyze the complexity of its corresponding decision problem first. The corresponding decision problem can be formally stated as:

66

N. Ashraf, I.N. Junejo, and H. Foroosh

Frame Selection Problem (FSP): INSTANCE: A finite feature set F , a set of inter-frame overlaps γ(fi , fj ) ∈ [0, 1], ∀fi , fj ∈ F and i = j, an integer k, and merit constraint B ∈ Z + . QUESTION: Does there exist an F  ⊆ F : |F  | = k, and MF  = √ k ≥B k+(k−1)¯ γ

where |F  | is the number of frames in F  , γ¯ is the average inter-frame overlap area for F ? Theorem 1. FSP is NP-Complete in the hard sense Proof. We prove this by (i) showing that the problem is in NP, and (ii) reducing the independent set problem to FSP in polynomial time which implies that FSP is also NP-Complete. Clearly, FSP problem is in NP. If an Oracle claims that an instance F’ is a yes inin polynostance, we can verify the claim by simply calculating, MF  = √ k k+(k−1)¯ γ

mial time, and check to see if MF  ≥ B. To prove that FSP is NP-Complete, we show that we can set up a polynomial reduction from the Independent set problem - which is already known to be NP-Complete [5]- to FSP. The problem of Independent Set (IS) is formally stated as follows: INSTANCE: A graph G = (V, E) and an integer m QUESTION: Does G have an independent set G , |G | ≥ m? We reduce IS to FSP by the following procedure:  1. For an instance I of IS, create √ an instance, I of FSP, such that ∀vi ∈ I’, create a corresponding fi , with B = k, withk = m 2. Set γ(fi , fj ) = 1 ∈ I if vi and vj in I are adjacent or γ(fi , fj ) = 0 otherwise. 2

Since we need n steps to create the corresponding frames in FSP and n2 − n steps to build the inter-frame matrix, the complexity of this reduction is O(n2 ), where n is the number of vertices in graph G for the Independent set problem. To complete the proof, we show that yes instances of IS map to yes instances in FSP and vice versa. Yes instances of IS map to yes instances of FSP: Assume we have a yes instance of IS i.e. a m-vertex independent set V  such that m ≥ m exists for a given graph, G. By our reduction, the corresponding problem in FSP would √ be such that B = k, k = m, and γ(fi , fj ) = 1 if and only if vi and vj were adjacent in G; otherwise γ(fi , fj ) = 0. Now note that the merit constraint B can only be satisfied if = we select the same set as V  because only then will we have merit, MF  = √ k k+(k−1)0 √ k ≥ B. Hence yes instances of IS map to yes instances of FSP. Yes instances of FSP map to yes instances of IS:

Near-Optimal Mosaic Selection for Rotating and Zooming Video Cameras

67

v2 v1

f1 f1 v3

v5

f2

f3

f4

f5

1

0

1

0

1

0

1

1

0

f2

1

f3

0

1

f4

1

0

1

f5

0

1

0

1 1

Inter-frame Overlap Matrix v4

Constraints: k = m, B =

k

Fig. 1. Example reduction of IS to FSP where the frame set constitutes of F {f1 , f2 , f3 , f4 , f5 }

=

Assume there exists a yes instance of FSP such that for F , √ a k-frame subset F  exists k ≥ B = k. This can only be true which satisfies the equation, MF  = √ k+(k−1)¯ γ

if γˆ = 0. This implies that γ(fi , fj ) = 0 for ∀i,j fi , fj ∈ F  . Therefore there exists a corresponding k-vertex independent set V  in IS. Since, m = k, an independent set of size m exists, implying that this is a yes instance of IS. Therefore, yes instances of FSP map to yes instances of IS. Hence, our proof that there is a polynomial reduction from IS to FSP and vice versa is complete implying that FSP is also NP-Complete. Figure 1 gives an example reduction of IS to FSP. Theorem 2. Original Frame Selection Optimization problem is NP-hard Proof. We say that a language is NP-hard if its corresponding decision language is NPComplete. Given that FSP is NP-Complete, we conclude that the original optimization problem of frame selection is NP-hard. Greedy Solution. Having proved that the problem is NP-hard, an optimal polynomial time algorithm cannot be guaranteed; hence, we need an efficient approximate algorithm. Rather, it would be great if we are able to map our problem into an existing, and well-studied problem for which efficient algorithms are available. In the previous section, we saw how the decision problem of both our problem and Maximum Independent Set problem are reducible to each other. Observe that in both cases we are trying to maximize the number of nodes based on some constraint (imposed by merit in our problem, while independence for Maximum Independent Set problem). But while the Maximum Independent Set problem has a graph composed of 0-1 edge values, our problem might have any value for the edge ranging from 0 to 1, signifying the percentage amount of overlap between frames. If we set up a threshold on the values of the overlap so that overlap values larger than or equal to the threshold are mapped to one, and otherwise mapped to zero otherwise, then our problem reduces to the Maximum Independent Set problem. Therefore, the Maximum Independent Set problem is a special case of our problem. Setting up a threshold is intuitively correct because it signifies that for two frames having an overlap value larger than the threshold, only one of the two should be used for mosaicing.

68

N. Ashraf, I.N. Junejo, and H. Foroosh

Hence, the size of subset of frames to be used for mosaicing is dependent on the threshold: having a large threshold value would mean that few frames would be used for mosaicing, while a small value for threshold would account for using a larger number of frames. Normally, in experiments, we have set this value close to sixty percent. Since Maximum Independent Set problem has been studied in depth, we have a range of algorithms to choose from. To approximate the Independent set, we use the simple, yet very efficient method known as Minimum-Degree Greedy algorithm [6]. The Minimum-Degree Greedy operates by a sequence of iterations, where in each iteration, a vertex is selected and added to the solution set - which is initially empty - and that vertex as well as its neighbors are removed from the graph. The iterations continue on the remaining graph and finally stop when the whole graph has been exhausted. Now we present the pseudo-code of the Minimum-Degree Greedy where d(v) represents the degree of vertex v : Require: Minimum-Degree Greedy (G) 1: I ← ∅ 2: while G = ∅ do 3: Choose v such that d(v) = minw∈g d(w) 4: I ←I ∪v 5: G ← G − {v ∪ N (v)} 6: end while The algorithm can be implemented in time linear in the number of edges and vertices. Furthermore, Hallrsson and Radhakrishnan [6] prove that this method surprisingly achieves a performance ratio of Δ+2 3 for approximating independent sets in graphs with ¯ degree bounded by Δ, the maximum degree of the graph; and 2d+3 performance ratio 5 ¯ on graphs where d is the average degree of the graph. Here performance is defined by where A is the algorithm in question, A(G) is the size of the pA = maxG alpha(G) A(G) solution obtained by algorithm A on graph G, and α(G) represents the actual size of the maximum independent set. Hence, in short, our algorithm proceeds by first finding the overlap matrix for the frames, applying a threshold on the values of the matrix, and then using MinimumDegree Greedy algorithm to estimate the subset of frames for use in mosaicing. To demonstrate the usefulness of our method, we have build a mosaicing system which makes use of our method for frame selection. Details can be found in section 3. 2.3 Feature Selection Problem Feature selection is a process commonly used in machine learning, wherein a subset of the features available from the data is selected for application of classification. The objective of feature subset selection in machine learning is to reduce the number of features used to characterize a data-set so as to improve a learning algorithm’s performance on a given task. A good feature subset is one, which contains features highly correlated with the class, but uncorrelated with each other. Irrelevant features should be ignored because they will have low correlation with the class; redundant features should

Near-Optimal Mosaic Selection for Rotating and Zooming Video Cameras

69

be screened out as they will be highly correlated with one or more of the remaining features. The acceptance of a feature will depend on the extent to which it predicts classes in areas of the instance space not already predicted by other features. Thus, the process of feature subset selection involves identifying and removing as much irrelevant and redundant information as possible. This has the effect of reducing the dimensionality of the data, thus allowing learning algorithms to operate faster and more effectively, and improving the accuracy of classification. Since we want to select a subset of features from the available features, which minimize the inter-feature correlations but maximize the feature-class correlation. Intuitively, if we represent the features as nodes of a graph, and mark an edge between two nodes if they have a significantly high inter-feature correlation, then we have reduced this problem into the maximum independent set problem. This is because by finding the maximum independent set of the graph, we have found a subset of maximum number of features, which are un-correlated to each other. Hence, intuitively, the maximum independent set problem can also be used to solve the problem of feature selection.

3 Image Mosaicing In this section, we build a mosaicing system which makes use of the frame selection method derived in section 2.2 in order to demonstrate its efficacy. Our algorithm essentially consists of two steps: the subset selection step, and a reconstruction step based on a maximum likelihood estimator that minimizes accumulating errors. Results are scrutinized using real video sequences. 3.1 Method Description Building the Inter-frame Overlap Matrix: Given a frame set, F , we first build up the inter-frame overlap matrix. This is done by finding the homography based on matched image features. We use SIFT features [12] for this purpose. Since these matches might contain many outliers, giving inaccurate homography, we use RANSAC [4] robust estimation algorithm to simultaneously estimate a homography and to select a set of matches consistent with the homography. Lens distortion is removed by applying the method of Devernay [3]. The estimate is further refined using a non-linear optimization technique [8]. Given a set of image correspondences xi → xi , we need to estimate a corrected set of correspondences, xˆi → xˆi  , which play the role of true measurements. Hence, the maximum likelihood estimate of the homography H, and the set of correspondences, xi → xi is ˆ and corrected set of correspondences, xˆi → xˆi  which minimizes the Homography H C



d2 (xi , xˆi ) + d2 (xi , xˆi )

(2)

i

The cost function is minimized using Levenberg-Marquardt algorithm. Frame Graph: We then build up the frame graph from this matrix. A node in the frame graph represents a frame while an edge between two nodes signifies that the two nodes

70

N. Ashraf, I.N. Junejo, and H. Foroosh

have significant overlap area. In other words, an edge is added between two nodes if the overlap is above a set threshold value. Finding the Maximum Independent Set: Once the frame graph is built, we can find the maximum independent set of graph by using the Minimum-Degree Greedy algorithm, and use this subset for mosaicing. Alignment of frames: Alignment of the frames is done as follows: A frame is chosen as the reference frame, and all the other frames are aligned to this frame by concatenating the intervening homographies. Hence, for instance, frame 0 and frame 4 would be aligned by the homography, H4,0 = H4,3 H3,2 H2,1 H1,0 .

Fig. 2. An example frame graph with its corresponding independent set marked by red vertices

Re-projection Surface: Once the images have been aligned, the frames can be reprojected to any surface. Simple averaging or temporal median filter on each mosaic pixel can be used for the overlapping area. The latter technique is good in that it eliminates independent moving objects which would appear blurry if simple averaging was used. For this reason, we use median filtering. Consistency Issues: Our scheme avoids the camera loop-back problem, where the camera loops-back on itself. Since all the frames are being used for mosaicing in current systems, if the camera loops-back, then the new frames are still used even though they may not contribute significantly to the mosaic. Since the error is being accumulated with each concatenation of the homographies, the old frames and the new frames would be poorly registered. In our case, this simply cannot happen, because the frames resulting from loop-back would be discarded and would not be used for mosaicing.

4 Results Several experiments were performed on real data. As described in 3, we use the method of Devernay [3] to correct lens distortion, and apply median filtering on the overlapping region. But applying median filtering introduces intensity differences in the resulting mosaic, which are otherwise well-registered. We used a threshold value close to sixty

Near-Optimal Mosaic Selection for Rotating and Zooming Video Cameras

71

percent in our experiments. In order to obtain our data, we use a SONY SNC-RZ30N PTZ camera with an image resolution of 320 × 240 where the ground truth rotation angles are known. Image features and correspondences were obtained by using SIFT[12]. Visual inspection and the reduction of the average frame overlap is the only source of measuring the quality of the applied method.

(a)

(b)

Fig. 3. (a) This video sequence contained 27 frames; only 7 frames were chosen for mosaicing. (b) Only eight frames were used for mosaicing; the original video sequence contained eighteen frames.

Fig. 4. This video sequence contained 25 frames; only 8 frames were selected and used for mosaicing. The intensity differences in this mosaic are because of using median filtering on the overlapping regions.

Figure 3 (a) shows an outdoor sequence. The sequences contained a total of 27 frames with mean overlap area of 52.07%. Only 7 frames were selected using the method described in 2 for building the mosaic. The reduced mean overlap area was only 28.49%. A total of 18 frames were captured indoor for another sequence shown in Figure 3 (b). Only 8 frames were selected for mosaic building. The mean overlap area was reduced from 65.82% to 30.44%. Finally, our last sequence, consisting of 25 frames was taken inside a lab. Frame selection method reduced the number of frames to be used for mosaicing to only 8. The mean overlap frame area also reduced from 57.04% to 35.41%. The resultant mosaic is shown in Figure 4. As is clear from these results, our method efficiently selects a good subset for use in mosaicing; and image mosaicing builds globally consistent image mosaics. Based

72

N. Ashraf, I.N. Junejo, and H. Foroosh

on the results, we are able to reduce the number of frames used for mosaicing down to ∼ 40% of the original video frames; and average inter-frame overlap area drops from ∼ 57% to ∼ 30%.

5 Conclusion We have demonstrated that we can map the frame selection problem and feature selection problem into the maximum independent set problem. Through this mapping, we can not only analyze the complexity of the problem at hand, but also use the existing algorithms from the graph-theory literature to solve the problem. In order to demonstrate the efficacy of our solution, we built a system for mosaicing, which uses our method of frame selection. We use real video sequences to test our method. We have found that our method significantly reduces the number of frames for use in mosaicing, hence decreasing the processing time while still building globally consistent mosaics.

References 1. Agapito, L.D., Hayman, E., Reid, I.: Self-calibration of rotating and zooming cameras. Int. J. Comput. Vision 45(2), 107–127 (2001) 2. Comaniciu, D., Meer, P.: Mean shift: A robust approach toward feature space analysis. IEEE Trans. Pattern Anal. Mach. Intell. 24(5), 603–619 (2002) 3. Devernay, F., Faugeras, O.D.: Straight lines have to be straight. Machine Vision and Applications 13(1), 14–24 (2001) 4. Fischler, M., Bolles, R.: Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartology. Communications of the ACM 5. Garey, M.R., Johnson, D.S.: Computers and Intractability: A Guide to the Theory of NPCompleteness. W. H. Freeman & Co, New York (1979) 6. Hallrsson, M., Radhakrishnan, J.: Greed is good: approximating independent sets in sparse and bounded-degree graphs. In: STOC 1994: Proceedings of the twenty-sixth annual ACM symposium on Theory of computing, pp. 439–448. ACM Press, New York (1994) 7. Hartley, R.I.: Self-calibration of stationary cameras. Int. J. Comput. Vision 22(1), 5–23 (1997) 8. Hartley, R.I., Zisserman, A.: Multiple View Geometry in Computer Vision, 2nd edn. Cambridge University Press, Cambridge (2004) 9. Junejo, I., Javed, O., Shah, M.: Multi feature path modeling for video surveillance. In: ICPR 2004. 17th conference of the International Conference on Pattern Recognition (2004) 10. Kang, E.-Y., Cohen, I., Medioni, G.G.: A graph-based global registration for 2d mosaics. In: ICPR, pp. 1257–1260 (2000) 11. Lowe, D.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vision 60(2), 91–110 (2004) 12. Lowe, D.G.: Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision 6(2), 91–110 (2004) 13. Sawhney, H.S., Hsu, S., Kumar, R.: Robust video mosaicing through topology inference and local to global alignment. In: Burkhardt, H., Neumann, B. (eds.) ECCV 1998. LNCS, vol. 1407, pp. 103–119. Springer, Heidelberg (1998) 14. Shi, J., Malik, J.: Motion segmentation and tracking using normalized cuts. In: Proc. IEEE ICCV, IEEE Computer Society Press, Los Alamitos (1998) 15. Shi, J., Tomasi, C.: Good features to track. In: CVPR 1994. IEEE Conference on Computer Vision and Pattern Recognition, Seattle (June 1994)

Video Mosaicing Based on Structure from Motion for Distortion-Free Document Digitization Akihiko Iketani1,2 , Tomokazu Sato1,2 , Sei Ikeda2 , Masayuki Kanbara1,2 , Noboru Nakajima1 , and Naokazu Yokoya1,2 1 2

NEC Corporation, 8916-47 Takayama, Ikoma, Nara 630-0101, Japan Nara Institute of Science and Technology, 8916-5 Takayama, Ikoma, Nara 630-0192, Japan

Abstract. This paper presents a novel video mosaicing method capable of generating a geometric distortion-free mosaic image using a hand-held camera. For a document composed of curved pages, mosaic images of virtually flattened pages are generated. The process of our method is composed of two stages : real-time stage and off-line stage. In the realtime stage, image features are automatically tracked on the input images, and the viewpoint of each image as well as the 3-D position of each image feature are estimated by a structure-from-motion technique. In the offline stage, the estimated viewpoint and 3-D position of each feature are refined and utilized to generate a geometric distortion-free mosaic image. We demonstrate our prototype system on curved documents to show the feasibility of our approach.

1

Introduction

Recently, digital cameras and cellular phones with a built-in camera have become so popular, and there is a strong demand for document digitization using these portable imaging devices, which allows us to scan and send documents anytime, anywhere. Compared with flat-bed scanners, however, these portable imaging devices have several problems to be solved. The most critical problem is the low resolution of the image acquired with these devices. Various techniques have been proposed to solve this low resolution problem. Among them, a video mosaicing technique is one of the most promising solutions. In video mosaicing, partial images of the document are captured as a video sequence, and multiple frame images are stitched seamlessly into one large, high resolution image, called a mosaic image. Szeliski [1] has developed a method using 8-DOF projective image transformation parameters called homography. In this method, for every pair of consecutive frames, the homography which minimizes the sum of squared differences between the two frames is estimated. A mosaic image is constructed by warping all the images to a reference frame (in general, the first frame). After his work, various extensions to this method have been proposed [2,3,4,5,6]. One of the major extensions is the use of image Y. Yagi et al. (Eds.): ACCV 2007, Part II, LNCS 4844, pp. 73–84, 2007. c Springer-Verlag Berlin Heidelberg 2007 

74

A. Iketani et al.

(a)Target document.

(b)Mosaic image with distortion due to curvature.

Fig. 1. Mosaic image with geometric distortion

features instead of all the pixels in images in order to reduce computational cost [3,4,6]. These methods, however, cannot avoid geometric distortion induced by the curvature of the target document. The homography based methods are only applicable when the target is a plane, or the optical center of camera is approximately fixed throughout the video capturing. If the target is a curved surface, the above assumption no longer holds, thus will cause misalignment of images in the resultant mosaic image. A mosaic image for the left page of the book in Figure 1(a) generated by a homography based method is shown in Figure 1(b). Distortion due to the curvature is evident in the curved lines of text and contour lines of the page. In the domain of document analysis, various methods to remove these geometric distortion have been proposed. Cao [7] assume the target document is composed of horizontal text lines, and generates a distortion-free image by warping the captured image so that all the baselines in the image are parallel to one another. Brown [8] assume the contour of the page is captured in the image, and warps the image so that the contour is transformed into a rectangle. Although these methods are capable of removing geometric distortion, they can only be applied to targets which fulfill the underlying assumptions. We present a novel video mosaicing method capable of generating a geometric distortion-free mosaic image. Our work is based on a structure-from motion technique, which recovers camera parameters, pose estimates, and sparse 3-D scene geometry from image sequence in real time. Using the 3-D geometry recovered by the algorithm, unwrapped mosaic images of virtually flattened pages are generated for a document composed of curved pages.

2

Video Mosaicing Based on Structure from Motion

This section describes a method for generating a high-resolution and geometric distortion-free mosaic image from a video sequence. The flow of the proposed method is given in Figure 2.

Video Mosaicing Based on Structure

75

(1) Real-time Stage (a) Camera parameter estimation by tracking features (b) Generation of preview image and instruction for a user Iterate from first frame to last frame

(2) Off-line stage (c) Detection of reappearing features (d) Refinement of estimated camera parameters (e) Surface fitting to 3-D point cloud (if system is in curved mode) Iterate until convergence

(f) Mosaic image generation

Fig. 2. The flow of the proposed method

In the real-time stage, the system carries out 3-D reconstruction processes frame by frame by tracking image features (Figure 2(a)). A coarse preview of the generated mosaic image is rendered in real time (Figure 2(b)). After the real-time stage, the system proceeds to the off-line stage. In this stage, first, reappearing image features are detected in the stored video sequence (Figure 2(c)), and camera parameters and 3-D positions of features estimated in the real-time stage are refined by global optimization (Figure 2(d)). Surface parameters are also estimated by fitting a parameterized 3-D surface to estimated 3-D point clouds (Figure 2(e)). After some iteration, a high-resolution and geometric distortion-free mosaic image is generated (Figure 2(f)). The assumptions made in the proposed method are that intrinsic camera parameters are fixed and calibrated in advance to correct lens distortion. It is also assumed that the curve of the target lies along one direction and its curvature changes smoothly along this direction. In the following sections, first, extrinsic camera parameters and an error function used in the proposed method are defined. The stages (1) and (2) in Figure 2 are then described in detail. 2.1

Definition of Extrinsic Camera Parameter and Error Function

In the proposed method, the coordinate system is defined such that an arbitrary point Sp = (xp , yp , zp ) in the world coordinate system is projected to the coordinate xf p = (uf p , vf p ) in the f -th image plane. Defining 6-DOF extrinsic camera parameters of the f -th image as a 3 × 4 matrix Mf , the relationship between 3-D coordinate Sp and 2-D coordinate xf p is expressed as follows: (auf p , avf p , a)T = Mf (xp , yp , zp , 1)T ,

(1)

where a is a parameter. In the above expression, xf p is regarded as a coordinate on the ideal camera with the focal length of 1 and without radial distortion induced by the lens. In practice, however, Sp is actually projected to the position ˆ f p = (ˆ of x uf p , vˆf p ) in the real image, which is given by transferring xf p using known intrinsic camera parameters including focus, aspect, optical center and

76

A. Iketani et al.

distortion parameters. In the rest of this paper, this transformation from xf p to ˆ f p is omitted for simplicity. x Next, the error function used for 3-D reconstruction is described. In general, the projected position xf p of Sp to the f -th image frame does not coincide with the actually detected position x = (uf p , vf p ), due to errors in feature detection, extrinsic camera parameter and 3-D feature position estimation. In this paper, the squared error Ef p is defined as an error function for the feature p in the f -th frame as follows: (2) Ef p = |xf p − xf p |2 . 2.2

Real-Time Stage for Image Acquisition

As shown earlier in Figure 2, the real-time stage is constructed of two iterative processes for each frame. First, the extrinsic camera parameter is estimated by tracking features (step (a)). A coarse preview of a mosaic image is generated and updated every frame (step (b)). The following describes each process of the real-time stage. Step (a): Camera parameter estimation by tracking features The extrinsic camera parameter and 3-D position Sp of each feature point is estimated by an iterative process. This process is basically an extension of the structure from motion method proposed by Sato [9]. In the first frame, assuming that the image plane in the first frame is approximately parallel to the target, rotation and translation components in Mf are set to an identity matrix and 0, respectively. For each feature point detected in the first frame, its 3-D position Sp is set to (u1p , v1p , 1), based on the same assumption. Note that these are only initial values, which will be corrected in the refinement process (Figure 2(d)). In the succeeding frames (f > 1), Mf is estimated by iterating the following steps toward the last frame. Feature point tracking: All the image features are tracked from the previous frame to the current frame by using a standard template matching with Harris corner detector [10]. The RANSAC approach [11] is also employed to eliminate outliers. Extrinsic camera parameter estimation: Extrinsic camera parameter Mf is estimated using the tracked position (uf p , vf p ) and its corresponding 3-D position Sp  = (xp , yp , zp ). Here, extrinsic camera parameters are obtained by minimizing p Ef p , the sum of the error function defined in Eq. (2), using the Levenberg-Marquardt method. For 3-D position Sp of the feature point p, the estimated result in the previous iteration is used. Estimation of 3-D feature position: For every feature point p in the current frame, its 3-D position Sp = (xp , yp , zp ) is refined by minimizing the error f function i=1 Eip .

Video Mosaicing Based on Structure

77

Addition and deletion of feature points: In order to obtain accurate estimates of camera parameters, good features should be selected. The set of features to be tracked is updated by evaluating the reliability of features [9]. By iterating the above steps, extrinsic camera parameters Mf and 3-D feature positions Sp are estimated. Step (b): Generation of preview image and instruction for user In parallel with the camera parameter estimation, a coarse preview of the mosaic image is rendered. This preview is updated in real time, using captured images and the estimated camera parameters. With this preview, the user can easily recognize which part of the document is still left to be captured, and figure out where to move the camera in the subsequent frames. 2.3

Off-Line Stage for Parameter Refinement and Target Shape Estimation

This section describes the process to globally optimize the estimated extrinsic camera parameters and 3-D feature positions, and to approximate the shape of the target by fitting a parameterized surface. Step (c): Detection of Reappearing Features Due to the camera motion, most of image features come into the image, move across toward the end of the image, and disappear. Some features, however, reappear in the image, as shown in Figure 3. In this step, these reappearing features are detected, and distinct tracks belonging to the same reappearing feature are linked to form a single long track. This will give tighter constraints among camera parameters in temporally distinct frames, and thus makes it possible to suppress cumulative errors in the global optimization step described later. Reappearing features are detected by examining the similarity of the patterns among features belonging to distinct tracks. First, templates of all the features are projected to the fitted surface (described later). Next, feature pairs whose distance in 3-D space is less than a given threshold are selected and tested with the normalized cross correlation function. If the correlation is higher than a threshold, the feature pair is regarded as reappearing features (see Figure 3).

Compensate for Distortion

(a)

(b)

(c)

Matching

(d)

Fig. 3. Detection of re-appearing features. (a) camera path, posture and 3-D feature positions, (b) temporarily distinct frames in the input video, (c) templates of the same feature in different frame images, (d) templates without perspective distortion.

78

A. Iketani et al.

Vmin: direction of minimum principal curvatures of the target

m

Vmax: direction of maximum principal curvatures of the target

n

direction of minimum curvature of Rp direction of maximum curvature of Rp

Sp Rp

N: Normal of projection plane

Plane P V2

V1: first principal direction of projected points

y = f (x )

Projected points

Fig. 4. Target shape estimation by polynomial surface fitting

Step (d): Global Optimization of 3D Reconstruction Since 3-D reconstruction process described in Section 2.2 is an iterative process, its result is subject to cumulative errors. In this method, by introducing bundleadjustment framework [12], the extrinsic camera parameters and 3-D feature positions are globally optimized so as to minimize the sum of re-projection errors Eall = f p Ef p . Step (e): Target Shape Estimation by Surface Fitting In this step, assuming the curve of the target lies along one direction, the target shape is estimated using 3-D point clouds optimized in the previous step (d). First, as shown in Figure 4, the principal direction of curvature is computed from the 3-D point clouds. Next, 3-D position of each feature point is projected to a plane perpendicular to the direction of minimum principal curvatures. Finally, a polynomial equation of variable order is fitted to the projected 2-D coordinates, and the target shape is estimated. Let us consider for each 3-D point Sp a point cloud Rp which consists of feature points lying within a certain distance from Sp . First, the directions of maximum and minimum curvatures are computed for each Rp using local quadratic surface fit. Then, a voting method is applied to determine the dominant direction Vmin = (vmx , vmy , vmz ) of minimum principal curvatures for the whole target. Next, 3-D position Sp for each feature point is projected to a plane whose normal vector N coincides with Vmin ; i.e. P (x, y, z) = vmx x + vmy y + vmz z = 0. The projected 2-D coordinate (¯ xp , y¯p ) of Sp is given as follows:     x ¯p V1 (3) = Sp , y¯p V2 where V1 is a unit vector parallel to the principle axis of inertia of the projected 2-D coordinates (¯ x, y¯), and V2 is a unit vector which is perpendicular to V1

Video Mosaicing Based on Structure

79

and Vmin ; i.e. V2 = V1 × N. Finally, the shape parameter (a0 , a1 , · · · , am ) is obtained by fitting the following variable-order polynomial equation to (¯ x, y¯). q  i y¯ = f (¯ x) = ai x ¯. (4) i=0

Using geometric AIC [13], the optimal order q in the above equation is determined as q which minimize the following criteria: G-AIC = J + 2(N (m − r) + q + 1)2 , (5) where J is the residual, N is the number of points, m is the dimension of observed data, and r is the number of constraint equations in fitting Eq. (4) to the projected 2-D coordinates. , called noise level, is the average error of the estimated feature position along y¯ axis. Here, the order q is independent of m, r, N , thus the actual criteria to be minimized is given as follows: (6) G = J + 2q2 . In our method, the noise level  is approximated as  = Cl, where l is the average of the depth of each feature point in camera coordinate of every frame, and C is a constant, which is empirically set to 0.007. In case of a target with multiple curved surfaces, e.g. a thick bound book shown in Figure 1(a), the target is first divided with a line where the normal vector of the locally fitted quadratic surface varies discontinuously, and the shape parameter is computed for each part of the target. Step (f ): Mosaic Image Generation Finally, a mosaic image is generated by using extrinsic camera parameters and surface shape parameters. Let us consider a 2-D coordinate (m, n) on the unwrapped mosaic image, as shown in Figure 4. Here, the relation between (m, n) and its corresponding 3-D coordinate (¯ x, f (¯ x), z¯) on the fitted surface is given as follows:  x¯  d 1 + { f (x)}2 dx, z¯). (7) (m, n) = ( dx 0 The relationship between coordinate (¯ x, f (¯ x), z¯) and its corresponding 2-D coordinate (uf , vf ) on the f -th image plane is given by the following equation. ⎞ ⎞⎛ ⎛ ⎛ ⎞ x ¯ auf V1 ⎝ avf ⎠ = Mf ⎝ V2 ⎠ ⎝ f (¯ x) ⎠ . (8) a N z¯ The pixel value at (m, n) on the unwrapped mosaic image is given by computing the average of the pixel values at all the corresponding coordinates (uf , vf ) in the input image, given by Eq. (7) and (8). After an unwrapped mosaic image is generated by the above process, the shade induced by the curved shape of the target is removed. In this process, the following assumptions are made: the background of the target page is white, and the target is illuminated by a parallel light source. In the proposed method, the vertical direction in the mosaic image coordinate (m, n) is defined to coincide with the direction of the minimum principle curvature of the target. Thus, under

80

A. Iketani et al.

a parallel light source, the effect of shade is uniform for pixels having the same m coordinate on the mosaic image. If we can assume that, in any column of the mosaic image, there exists at least one pixel belonging to the background, a new pixel value Inew (m, n) after shade removal can be computed as follows: Inew (m, n) =

Imax I(m, n) , max(I(m + u, n + v); ∀(u, v) ∈ W )

(9)

where W is a rectangular window whose height is larger than its width, e.g. a window with the size of 5×500 pixel, and Imax is the maximum possible intensity value of an image (typically 255).

3

Experiments

We have developed a prototype video mosaicing system which consists of a laptop PC (Pentium-M 2.1 GHz, Memory 2GB) and an IEEE 1394 type web-cam (Aplux C104T, VGA, 15fps). Experiments are performed on two curved targets: one on a thick bound book with 2 curved pages and the other on a curved poster on a cylindrical column. In both experiments, the intrinsic camera parameters are calibrated in advance using Tsai’s method [14], and are fixed throughout image capturing. A thick bound book captured in the first experiment is shown in Figure 5. It is composed of 2 pages of curved surfaces: one page with texts and the other with pictures and figures. Plus marks (+) were printed on 40mm grid points on both pages for quantitative evaluation (described later). The target is captured as a video of 200 frames at 7.5fps. Sampled frames of the captured images are shown in Figure 6. Tracked feature points are depicted with cross marks. 3-D reconstruction result is shown in Figure 7. The curved line shows the camera path, pyramids show the camera postures in every 10 frames, and the point cloud shows 3-D positions of feature points. The shape of the target estimated after 3 iterations of reappearing feature detection and surface fitting process is shown in Figure 8(a). The optimal orders of the polynomial surface, which are automatically determined by geometric AIC [13], are 5 and 4 for the left and right pages, respectively. The unwrapped mosaic image is shown in Figure 8(b). The resolution of the mosaic image is 3200 × 2192. We evaluate the distortion in the generated mosaic image quantitatively, using plus marks (+) printed on the target paper at every 40mm grid positions. First, the positions of the plus marks are acquired manually in the generated Table 1. Distances of adjacent grid points on the mosaic image [pixels (percentage from average)] page average maximum minimum std. dev. left 338.5(100.0) 348.0(102.8) 331.0(97.8) 3.77(1.11) right 337.6(100.0) 345.0(102.2) 331.1(98.1) 2.75(0.81)

Video Mosaicing Based on Structure

81

Fig. 5. Thick bound book with curved surface

1st frame

100th frame

200th frame

300th frame

Fig. 6. Sampled frames of input images and tracked features (Book)

(a) frontal view

(b) side view

Fig. 7. Estimated extrinsic camera parameters and 3-D positions of features (Book)

(a) Estimated target shape.

(b) Unwrapped mosaic image.

Fig. 8. Estimated target shape and unwrapped mosaic image (Book)

82

A. Iketani et al.

Fig. 9. Poster on a cylindrical column

1st frame

35th frame

87th frame

142th frame

Fig. 10. Sampled frames of input images and tracked features (Poster)

(a) frontal view

(b) side view

Fig. 11. Estimated extrinsic camera parameters and 3-D positions of features (Poster)

(a) Estimated target shape.

(b) Unwrapped mosaic image.

Fig. 12. Estimated target shape and unwrapped mosaic image (Poster)

Video Mosaicing Based on Structure

83

mosaic image. The distances between adjacent plus marks are then computed in the unit of pixel. The average, maximum, minimum and standard deviation of the distances are shown in Table 1. The percentage of each value from the average distance is also shown in parenthesis. Here, the standard deviation can be considered as the average distortion in the generated image. In this experiment, the average distortions for the left and right pages are only 1.1% and 0.8%, respectively. We can confirm that the geometric distortion has been correctly removed. The other experiment is performed on a poster on a cylindrical column shown in Figure 9. Sampled frames of the captured images with tracked feature points, and 3-D reconstruction result are shown in Figure 10, 11, respectively. The shape of the target estimated after 3 iterations is shown in Figure 12(a). The optimal order of the polynomial surface fitted to the target is 4. The unwrapped mosaic images is shown in Figure 12(b). The resolution of the mosaic image is 3200 × 5161. Although the target has few text features, the distortion has been successfully removed in the resultant image. The performance of our system measured in the first experiment is as follows: 22 seconds for initial 3-D reconstruction, 188 seconds for camera parameter refinement and surface fitting, and 410 seconds for generating the final mosaic image.

4

Conclusion

A novel video mosaicing method for generating a high-resolution and geometric distortion-free mosaic image for curved documents has been proposed. With this method based on 3-D reconstruction, the 6-DOF camera motion and the shape of target document are estimated. Assuming the curve of target lies along one direction, the shape model is fitted to the feature point cloud and unwrapped image is automatically generated. In experiments, a prototype system based on the proposed method has been developed and has been successfully demonstrated. Our future work is to reduce the computational cost, and to implement the algorithm on mobile PC.

References 1. Szeliski, R.: Image Mosaicing for Tele-Reality Applications. In: Proc. IEEE Workshop on Applications of Computer Vision, pp. 230–236. IEEE Computer Society Press, Los Alamitos (1994) 2. Capel, D., Zisserman, A.: Automated Mosaicing with Super-resolution Zoom. In: Proc. IEEE Conf. on Computer Vision and Pattern Recognition, pp. 885–891. IEEE Computer Society Press, Los Alamitos (1998) 3. Chiba, N., Kano, H., Higashihara, M., Yasuda, M., Osumi, M.: Feature-based Image Mosaicing. In: Proc. IAPR Workshop on Machine Vision Applications, pp. 5–10 (1998) 4. Hsu, C.T., Cheng, T.H., Beuker, R.A., Hong, J.K.: Feature-based Video Mosaicing. In: Proc. IEEE Int. Conf. on Image Processing, vol. 2, pp. 887–890. IEEE Computer Society Press, Los Alamitos (2000)

84

A. Iketani et al.

5. Lhuillier, M., Quan, L., Shum, H., Tsui, H.T.: Relief Mosaicing by Joint View Triangulation. In: IEEE Int. Conf. on Computer Vision and Pattern Recognition, vol. 1, pp. 785–790. IEEE Computer Society Press, Los Alamitos (2001) 6. Takeuchi, S., Shibuichi, D., Terashima, N., Tominage, H.: Adaptive Resolution Image Acquisition Using Image Mosaicing Technique from Video Sequence. In: Proc. IEEE Int. Conf. on Image Processing, vol. 1, pp. 220–223. IEEE Computer Society Press, Los Alamitos (2000) 7. Cao, H., Ding, X., Liu, C.: A Cylindrical Surface Model to Rectify the Bound Document Image. In: Proc. Int. Conf. on Computer Vision, vol. 1, pp. 228–233 (2003) 8. Brown, M.S., Tsoi, Y.C.: Undistorting Imaged Print Materials using Boundary Information. In: Proc. Asian Conf. on Computer Vision, vol. 1, pp. 551–556 (2004) 9. Sato, T., Kanbara, M., Yokoya, N., Takemura, H.: Dense 3-D Reconstruction of an Outdoor Scene by Hundreds-baseline Stereo Using a Hand-held Video Camera. Int. J. of Computer Vision 47(1-3), 119–129 (2002) 10. Harris, C., Stephens, M.: A Combined Corner and Edge Detector. In: Proc. Alvey Vision Conf., pp. 147–151 (1988) 11. Fischler, M.A., Bolles, R.C.: Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography. In: Communications of the ACM, vol. 24(6), pp. 381–395. ACM Press, New York (1981) 12. Triggs, B., McLauchlan, P., Hartley, R., Fitzgibbon, A.: Bundle Adjustment a Modern Synthesis. In: Proc. Int. Workshop on Vision Algorithms, pp. 298–372 (1999) 13. Kanatani, K.: Geometric Information Criterion for Model Selection. Int. J. of Computer Vision 26(3), 171–189 (1998) 14. Tsai, R.Y.: An Efficient and Accurate Camera Calibration Technique for 3D Machine Vision. In: Proc. IEEE Conf. on Computer Vision and Pattern Recognition, pp. 364–374. IEEE Computer Society Press, Los Alamitos (1986)

Super Resolution of Images of 3D Scenecs Uma Mudenagudi, Ankit Gupta, Lakshya Goel, Avanish Kushal, Prem Kalra, and Subhashis Banerjee Department of Computer Science and Engineering IIT Delhi, New Delhi, India [email protected],{pkalra,suban}@cse.iitd.ernet.in [email protected]

We address the problem of super resolved generation of novel views of a 3D scene with the reference images obtained from cameras in general positions; a problem which has not been tackled before in the context of super resolution and is also of importance to the field of image based rendering. We formulate the problem as one of estimation of the color at each pixel in the high resolution novel view without explicit and accurate depth recovery. We employ a reconstruction based approach using MRF-MAP formalism and solve using graph cut optimization. We also give an effective method to handle occlusion. We present compelling results on real images.

1

Introduction

In this paper we address the problem of estimating a single high resolution (HR) image of a 3D scene given a set of low resolution (LR) images obtained from different but known viewpoints with no restrictions on either the scene geometry or the camera position. The super resolved image can correspond to one of the known view points in the input set, or even be of a novel view. There have been several different approaches to image super resolution, with estimation of high resolution images from multiple low resolution observations related by small 2D motions being by far the most common one [1,2,3]. Most of these approaches assume that the low resolution images can be accurately registered by a 2D affine transformation or a homography, and attempt to super resolve by reversing the degradation caused by blur or defocus and sampling. There has been very little work on super resolution when the scene is 3D and the input cameras are generally positioned. The super resolution problem becomes considerably harder when, because of the depth variation in the scene and arbitrary placement of the input cameras, there is no simple registration transformation between the input images. For then to super resolve one needs dense depth estimation at each pixel using multiple-views geometry; a problem which has not been entirely solved. Though qualitative estimation of such depth information is possible, high precision depth recoveryrequired for accurate and high resolution rendering of a 3D scene is still a challenging problem. Super resolution rendering of a novel view of a 3D scene is an even harder problem because then one needs to effectively handle occlusion. For super resolution of one of the input views the key problem is to determine which pixels/colors Y. Yagi et al. (Eds.): ACCV 2007, Part II, LNCS 4844, pp. 85–95, 2007. c Springer-Verlag Berlin Heidelberg 2007 

86

U. Mudenagudi et al.

from the other views need to be used for generating each high resolution pixel in the target view. However, for a novel view, the visibility information itself may change significantly - and this needs to be handled. The main contributions of this paper are as follows: 1. We give an MRF-MAP formulation of super resolution reconstruction of 3D scenes from a set of calibrated LR images obtained from arbitrary 3D positions and solve using graph cut optimization. 2. We show that the super resolved novel view synthesis problem can be posed as one of first generating a set of pixels in the source LR views, depending on their color, which are most likely to contribute to a target pixel in the novel view, and then using standard techniques of super resolution to recover the high frequency details. In this sense, our approach is closely related to the novel view synthesis methods of [4,5], which however do not address the issues of super resolution. 3. We give an effective method for handling occlusion using photo-consistency [6]. We demonstrate this with results on real images. Our approach is motivated by [4,5] where also the novel view generation problem have been formulated as one of estimation of color at each generated pixel, without explicit depth recovery. In [4] the virtual cameras are placed within the scene itself, and the visibility information does not change significantly between the input cameras. The color selection is done by geometric analysis of “line of sight”. [5] relies on a probabilistic analysis of the possible colors at a target pixel and uses a database of color patches, computed from the input images themselves, as regularization priors. Occlusion is not explicitly handled. We draw insights from these works to formulate a scheme for super resolution rendering of novel views which can explicitly handle occlusion. In Section 2, we describe the image formation process, and formulation of novel view super resolution using MRF-MAP formalism. We present an effective method to resolve occlusion using photo-consistency in section 3. We give energy minimization using graph cut in section 4. In Section 5, we present results to demonstrate the effectiveness of our method. We conclude in section 6.

2

Novel View Super Resolution

2.1

MRF-MAP Formulation

We are given a set of 2D LR images g1 , . . . gn with their 3 × 4 projective camera matrices P1 , . . . , Pn . Let Ci be the camera center for the camera Pi . Let f be the high resolution image seen from the novel view with projection matrix Pcam . The projection matrix of the k th observed image Pk projects homogeneous 3D points X to homogeneous 2D points xk = λ(x, y, 1)T [7] projectively and xk = Pk X, where the equality is up to scale. We compute Pcam by first multiplying matrix of camera internals with desired magnification, and then multiplying with external parameters of novel view in a standard way. The main task is to assign a color to

Super Resolution of Images of 3D Scenecs

87

a high resolution pixel p which is photo-consistent in the input images, i.e., the corresponding pixels in the input LR reference images have similar color, unless occluded [8,9,6]. We need to compute the expected color for a HR pixel p. Ideally, this would require the computation of a dense depth map from the input images to obtain complete registration, and a reliable solution to this problem is still elusive in the computer vision literature. We compute the expected label (color) in a given image using the photo-consistency constraint. We model f as a Markov Random Field (MRF) [10] and use a maximum a posteriori (MAP) estimate as the final solution. The problem of estimating f can be posed as a labeling problem where each pixel is assigned a color label. We formulate the posterior energy function as n   αk (p, p (z))(h(p) ∗ f (p, z) − gk (p (z)))2+λs Vp,q (f (p), f (q))(1) E(f, z|g) = p∈Sk=1

p,q∈N

where S is the set of sites (pixels) in the novel view, z is a depth along the back-projected ray from site p in the novel view camera, f (p, z) is the label at site p and at depth z, h(p) is the space invariant PSF of the camera for k th image, p (z) = Pk (X(z)) is the projection of back projected 3D point at depth z on to the k th LR image, gk (p (z)) is the expected color label when z along the back-projected ray from p is projected to k th LR image, αk (p, p (z)) is a weight associated with the k th LR image and Vp,q = min(θ, |f(p)−f(q)|) is a smoothness prior. We compute the weights αk (p, p (z)) in the following way. We define a zone HR pixel not influenced by the LR pixel Mapping of pixel in any low resolution image

LR pixel Zone of Influence No Pixles zone

p

X(z)

3D Ray

z

HR pixel

Projected rays

p p

p(x, y) p

HR pixel influenced by the LR pixel

(a)

p

(b)

Fig. 1. (a) Limiting the zone of influence of a LR pixel in space. Only those HR pixels whose centers lie inside the circle are influenced by the LR pixel. (b) Zone of influence of LR pixel.

of influence for every low resolution pixel such that the LR pixel contributes to the reconstruction of only those high resolution pixels whose centers lie within the zone of influence (see Figure 1(a) and 1(b)). Specifically, we define the following functions to determine whether a LR pixel p (z) = (x , y  ) contributes to the reconstruction of a high resolution pixel p = (x, y). Let p (z) = (x , y  ) be the projection of p, in real coordinates, which is projected from z along the back projected ray from the pixel p, in to the k th LR image. We define

88

U. Mudenagudi et al.

αk (p, p (z)) =



1 if d((x , y  ), (x , y  )) < θ1 0 otherwise

(2)

αk (p, p (z)) = 1 signifies that the high resolution pixel p is within the zone of influence of the LR pixel p in the k th LR image. Here d() is the Euclidean distance function and θ1 is a threshold. We set the threshold to the σ values of the low resolution blur function. In most of our experiments the value of spatial blur σ is evaluated to 0.45 of LR pixel. Minimization of the energy function given in Equation 1 over all possible depths and colors is computationally intractable. In what follows we explain a pre-processing step for pruning the potential color labels gk (p (z)) for a site p by considering only those depths which are photo-consistent at p [6]. We first outline a strategy for the simpler problem of super resolution of one of the input reference images and then outline a procedure for novel view super resolution. 2.2

Pruning of Color Labels Using Photo-Consistency

Consider, first, the problem of super resolution of one of the source images. We finely sample the back-projected ray from a site in the target super resolved view between a range Zmin to Zmax . For each z in this range, we project the 3D point X(z) to each of the LR input images and compute c(p, z) = {gk (p (z)) | 1 ≤ k ≤ n}

(3)

as set of possible colors corresponding to depth z. For each z we do a nearest neighbor clustering of c(p, z) in Y CbCr space, with a suitable distance threshold on color differences, and remove all but the top few dominant clusters. If there are no dominant clusters then we remove z from the list of candidate depths at p. For all candidate z values at site p we write   (4) C(p, z) = I(p, z, i) i=1...mpz input images in cluster 1 patch

p(z)

Zmax pair−wise patch based correlation in each cluster

z candidate depth

Zmin

z1

z2 X(z)

B1 B2

p C1

C2

C6

Cn

Ccam C4

C3

input images in cluster 2

(a)

C5

Pcam

(b)

Fig. 2. (a) Clusters at a depth z and patches in each of the input images projected from z along the back projected ray from the pixel p in the super resolved novel view. (b) Cameras C1 to C3 see the color of the surface B1 and rest of the cameras see the color of B2 .

Super Resolution of Images of 3D Scenecs

89

where mpz is the number of dominant clusters, and I(p, z, i) is the set of the image indices that belong to the ith dominant cluster. For each depth z, we define a correlation function mpz corr(patch(I(p, z, i))) (5) f corr(z) = i=1 mpz I(p,z,i)| i=1

2

where patch(I(p, z, i)) is the set of 3 × 3 patches around the LR pixel gk (p ) and corr(patch(I(p, z, i))) is the sum of pair-wise correlations between patches of I(p, z, i). We form a set of candidate depths dp at site p by choosing the local minimas of f corr(z). The intra cluster correlation does not guarantee that the candidate depths are correct, but one can only infer that these depths satisfy the photo-consistency constraint [6,9]. In this process we get the possible color clusters for each pixel and there can be two possibilities: 1. Only one cluster: In this case the pixel is photo consistent in all the views at this particular depth, and hence the dominant color cluster can be chosen unambiguously solely based on photo-consistency. 2. More than one dominant clusters: This suggests that either the 3D point which projects to the pixel is occluded in some of the source images (see Figure 2(b)); or that the high resolution pixel is on an edge so that it projects on to more than one color in the LR images. In view of the above, we write the energy to be minimized as mpz



   β(p, z, i) αk (p, p ) h(p) ∗ f (p, z) − gk (p (z)) 2 min E(f |g) = p∈S

z∈dp

+λs

i=1



k∈I(p,z,i)

Vp,q (f (p), f (q))

(6)

p,q∈N

where αk (p, p ) is given by Equation 2 and β(p, z, i) is the weight given to the cluster i at candidate depth z for pixel p. We set the cluster weight β(p, z, i) as follows. In case there is only one dominant cluster, we set β(p, z, i) = 1. If there are more than one dominant clusters we determine if p is an edge point. This check is easy for super resolution of one of the source views because then the LR reference pixel corresponding to p is available. If p is an edge point we set the weights β(p, z, i) for all the clusters corresponding to neighboring colors of p as exp(b Ni ) β(p, z, i) = mpz k=1 exp(b Nk )

(7)

Number of images in ith the cluster and b is a tuning parameter. where Ni = total number of images If p is not an edge point we look up the color of p in the LR image and set β(p, z, i) = 1 for the corresponding cluster and all others to zero. Note

90

U. Mudenagudi et al.

that the above energy function given in Equation 6 is not in a form that is amenable to minimization by graph cut. Moreover, in case of super resolution reconstruction of a novel view, the problem of choosing the photo-consistent set of colors becomes considerably harder because then one needs an effective strategy for resolving occlusion. In the next section we explain how we can handle occlusion using the photo-consistency constraint [6,9].

3

Occlusion Handling Using Photo-Consistency

Clearly, if the above procedure returns multiple dominant clusters for a high resolution pixel p in the target novel view due to occlusion, there is no way to resolve the correct cluster from an analysis based on a single pixel. Seitz and Dyer [8,9] and Kutulakos and Seitz [6] develop a theory for occlusion analysis using photo-consistency, which gives the ordinal visibility constraint [8,9]. If X(z1 ) and X(z2 ) are any two points on two back projected rays then X(z1 ) can occlude X(z2 ) only if z1 < z2 . This allows us to resolve occlusion in a way similar to the voxel coloring algorithm of [8,9]. We scan the back projected rays from each pixel in a target novel view image in the order of depth (z) values. For a given z value along a ray through p, we project onto the source images and mark source image pixels if they satisfy photo-consistency. For the subsequent z values we consider only the unmarked pixels in the source images. We outline the algorithm to resolve occlusion using photo-consistency in Algorithm 1.

Algorithm 1. Algorithm to resolve occlusion For each z from Zmin to Zmax : for each pixel p in the novel view image that has z as a candidate depth (z ∈ dp from previous analysis in Section 2.2): set C = Φ. for all input LR images gk (indexed by k): a. project X(z) on the ray back projected through p on to gk . Let p (z) be the projected pixel in gk (after round off). b. push the color of p (z) into C if p (z) is not marked. end. if C is photo-consistent then a. retain the clusters with color similar to the photo-consistent set and prune all others from C(z, i). Set the weights β(z, p, i) for the remaining clusters in C(z, i) according to their membership as given in Equation 7. b. retain this z value in dp and remove all others. c. mark all pixels p (z) in gk which are unmarked. endif. end. end.

Super Resolution of Images of 3D Scenecs

(a)

91

(b)

Fig. 3. (a) Row 1: One of the input images (out of 40) and photo-consistent depth map, Row 2: Occlusion-map, the pixels for which more than one clusters are formed, and are marked white and generated novel view. (b) Super resolved novel view using 20 images. Note that though the depth map generated is not accurate, it provides correct photo-consistent color in the novel view.

The space carving theory of [6] guarantees that the algorithm outlined above will converge to a photo-consistent shape. Consequently, the procedure guarantees a unique z value for all p, which though may not be “correct”, nevertheless provides a photo-consistent color assignment, which is all that we require for super resolved novel view generation. We show one of the input images, the depth map using a unique z values, the image where the pixels for which multiple clusters are formed, and are marked white (occlusion-map, with slight abuse of notation) and generated novel view in Figure 3(a), and the super resolved novel view in 3(b). The final energy is then given as pz



2   m        β(p, z , i) αk (p, p (z )) h(p) ∗ f (p, z ) − gk (p (z )) E(f |g) = p∈S

+λs

i=1



k∈I(p,z  ,i)

Vp,q (f (p), f (q))

(8)

p,q∈N

where z  is the unique depth value we get after resolving the occlusion. The energy in Equation 8 is clearly amenable to minimization by graph cut.

4

Energy Minimization Using Graph-Cut

Graph cut [11] can minimize only graph-representable energy functions. An energy function is graph-representable if and only if each term Vp,q satisfies the regularity constraint [12]. The energy function of Equation 8 is not in the graphrepresentable form. The data term of site p also depends on the neighbors of p because of the blurring operator. We approximate the data term of Equation 8 as follows. Consider the blur kernel h(p) with values wpp at the center and wpq at the neighbor q of p. Writing f (p, z  ) as fp , αk (p, p (z)) as αk , β(p, z  , i) as β, I(p, z  , i) as Ii , gk (p (z)) as gk (p ) and denoting as Nsp the set of spatial neighbors of p, then the data term is given by

92

U. Mudenagudi et al.

pz pz 2 m 2 



 m   β αk h(p) ∗ fp −gk (p ) = β αk wpp fp −gk (p )+ wpq fq p∈S i=1 k∈Ii

p∈Si=1 k∈Ii

p∈Nsp

pz

2

2   m   = β αk wpp fp − gk (p ) + wpq fq +2 wpp fp − gk (p ) wpq fq . (9) p∈Si=1 k∈Ii

q∈Nsp

q∈Nsp

We write the overall energy as  

  Dp∗ (fp ) + E(f |g) = φpq (fp , fq ) + λs Vp,q (fp , fq ) p∈S

q∈Nsp

(10)

q∈Nsp

where, collecting all terms that depend on a single pixel in to the data term, we have mpz 2 2 

  ∗  Dp (fp ) = β αk wpp fp − gk (p ) + wpq fq (11) i=1

q∈Nsp

k∈Ii

and



mpz

φpq (fp , fq ) =

i=1

β



αk



 2(wpp fp − gk (p )) (wpq fq ) 

(12)

k∈Ii

The above expression is still not graph representable because of the term fp fq in φpq . We further approximate wpp fp − gk (p ) = Δkp in φpq and hold Δkp constant during a particular α-expansion move during the minimization using graph cut. It is easy to verify that φpq satisfies the regularity condition in [12] with this approximation. Unfortunately, it cannot be guaranteed that every step of energy minimization of the approximate energy using α-expansion also minimizes the original energy. However, in our experiments we find that this is almost always the case. [13] gives a different approximation for spatial deconvolution using graph cut, where they can give such guarantee.

5

Results

In this section we present some results of novel view super resolution. In all our experiments we obtained the reference images from image sequences captured using a hand held video camera (Sony), and the sequences were calibrated using an automatic tracking and calibration software, Boujou [14]. We provide the input and the output images in the supplemental material through index.html link. 5.1

Novel View Generation

In our first example we select 11 frames from the image sequence and use 10 of them as reference images to generate the missing 11th image, so that the reconstructed image can be compared with the ground truth. In Figure 4(a) we

Super Resolution of Images of 3D Scenecs

(a)

93

(b)

Fig. 4. (a) Row 1 : two input images out of 10, Row 2: generated novel view and difference with ground truth. (b) Super resolved novel view with magnification of 2× in each direction.

show two of the reference images, the generated novel view and the difference image with ground truth. In Figure 4(b) we show the super resolved output corresponding to the novel view. In Figure 5 we show a close up of interpolated 11th image, the corresponding super resolved novel view and the difference image. The difference image clearly shows the missing high frequency components in the interpolated image.

Fig. 5. Close ups of the interpolated, the super resolved & difference image of Figure 4

5.2

Occlusion Handling

In this example we show that we can effectively handle occlusion. Four out of 10 input images are shown in Figure 6(a). Note that different colors are visible in between the petals of the flower because of occlusion. The occlusion has been properly handled in the reconstructed super resolved view of one of the source image as shown in Figure 6(b). In Figure 6(c) and Figure 6(d) we show the super resolved novel view using 10 and 20 images. In case of super resolution of one of the source view the cluster is selected correctly and in case of super resolution of novel view, the cluster selection improves when we increase the number of images as seen in the super resolved view using 20 images compared to using 10 images.

94

U. Mudenagudi et al.

(a)

(b)

(c)

(d)

Fig. 6. Resolving occlusion: (a) four of the 10 input images where different colors are visible in between the petals of flower, (b) super resolved image of one of the source view, (c) super resolved novel view using 10 images and (d) super resolved novel view using 20 images

(a)

(b)

Fig. 7. Novel views and the corresponding difference image with ground truth (a) using 10 and 40 images, (b) using FOV 11.5o and 7.5o

5.3

Effect of Number of Views and Field of View

In this example we show the novel view generation using different number of input images keeping same field of view (FOV) and different FOVs keeping same number of input images. In the first experiment we consider image sequence of 61 images and generate the novel view using 10, 20, 30, 40 and 50 images. The corresponding rms errors between ground truth and the novel view are 20.2027, 17.7744, 16.8216, 16.3895 and 16.3895 respectively. Two of the generated novel views using 10 and 40 images and corresponding difference images with ground truth are shown in Figures 7(a). In the second experiment we used 10 images from three input sets with FOVs 11.5o, 9.0o and 7.5o . The rms errors corresponding to threes sets are 30.3203, 20.2027 and 15.6003 respectively. The generated novel view using FOV 11.5o and 7.5o , and the corresponding difference images with the ground truth are shown in Figure 7(b).

6

Conclusions

In this paper we have addressed the problem of super resolved generation of novel views of a 3D scene with the reference images obtained from cameras in general

Super Resolution of Images of 3D Scenecs

95

positions. We have posed novel view super resolution problem in the MRF-MAP framework and proposed a solution using graph cut. We have formulated the problem as one of estimation of the color at each pixel in the high resolution novel view without explicit and accurate depth recovery. We have also presented an effective method to resolve occlusion. We have presented compelling results on real images.

References 1. Irani, M., Peleg, S.: Improving resolution by image registration. CVGIP: Graphical Models and Image Processing 53, 231–239 (1991) 2. Chaudhuri, S., Joshi, M.V.: Motion-Free Super-Resolution. Springer, Heidelberg (2004) 3. Borman, S., Stevenson, R.L.: Linear Models for Multi-Frame Super-Resolution Restoration under Non-Affine Registration and Spatially Varying PSF. In: Computational Imaging II. Proceedings of the SPIE, vol. 5299, pp. 234–245 (2004) 4. Irani, M., Hassner, T., Anandan, P.: What Does the Scene Look Like from a Scene Point? In: Heyden, A., Sparr, G., Nielsen, M., Johansen, P. (eds.) ECCV 2002. LNCS, vol. 2351, pp. 883–897. Springer, Heidelberg (2002) 5. Fitzgibbon, A., Wexler, Y., Zisserman, A.: Image-Based Rendering Using ImageBased Priors. International Journal of Computer Vision 63, 141–151 (2005) 6. Kutulakos, K.N., Seitz, S.M.: A Theory of Shape by Space Carving. International Journal of Computer Vision 38, 199–218 (2000) 7. Hartley, R.I., Zisserman, A.: Multiple View Geometry in Computer Vision. Cambridge University Press, Cambridge (2000) 8. Seitz, S.M., Dyer, C.R.: Photorealistic Scene Reconstruction by Voxel Colouring. In: Proceedings of the IEEE CVPR, pp. 1067–1073. IEEE Computer Society Press, Los Alamitos (1997) 9. Seitz, S.M., Dyer, C.R.: Photorealistic Scene Reconstruction by Voxel Colouring. International Journal of Computer Vision 35, 151–173 (1999) 10. Geman, S., Geman, D.: Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images. IEEE Transactions on Pattern Analysis and Machine Intelligence 6, 721–741 (1984) 11. Boykov, Y., Veksler, O., Zabih, R.: Fast Approximate Energy Minimization via Graph Cuts. IEEE Transactions on PAMI 23, 1222–1239 (2001) 12. Kolmogorov, V., Zabih, R.: What Energy Functions Can Be Minimized via Graph Cuts? IEEE Transactions on PAMI 26, 147–159 (2004) 13. Raj, A., Singh, G., Zabih, R.: MRFs for MRIs: Bayesian Reconstruction of MR Images via Graph Cuts. In: Proceedings of the IEEE CVPR, pp. 1061–1068. IEEE Computer Society Press, Los Alamitos (2006) 14. 2d3 Ltd (2002), http://www.2d3.com

Learning-Based Super-Resolution System Using Single Facial Image and Multi-resolution Wavelet Synthesis Shu-Fan Lui, Jin-Yi Wu, Hsi-Shu Mao, and Jenn-Jier James Lien Robotics Laboratory, Dept. of Computer Science and Information Engineering National Cheng Kung University, No. 1, Ta-Hsueh Road, Tainan, Taiwan {kilo, Curtis, marson, jjlien}@csie.ncku.edu.tw http://robotics.csie.ncku.edu.tw

Abstract. A learning-based super-resolution system consisting of training and synthesis processes is presented. In the proposed system, a multi-resolution wavelet approach is applied to carry out the robust synthesis of both the global geometric structure and the local high-frequency detailed features of a facial image. In the training process, the input image is transformed into a series of images of increasingly lower resolution using the Haar discrete wavelet transform (DWT). The images at each resolution level are divided into patches, which are then projected onto an eigenspace to derive the corresponding projection weight vectors. In the synthesis process, a low-resolution input image is divided into patches, which are then projected onto the same eigenspace as that used in the training process. Modeling the resulting projection weight vectors as a Markov network, the maximum a posteriori (MAP) estimation approach is then applied to identity the best-matching patches with which to reconstruct the image at a higher level of resolution. The experimental results demonstrate that the proposed reconstruction system yields better results than the bi-cubic spline interpolation method. Keywords: Super-resolution, learning-based, reconstruction, Markov network, multi-resolution wavelets, maximum a posteriori.

1 Introduction Super-resolution (SR) is an established technique for expanding the resolution of an image since directly up-sampling a low-resolution image to produce a high-resolution equivalent invariably results in blurring and a loss of the finer (i.e. higher-frequency) details in the image. SR has attracted increasing attention in recent years for a variety of applications, ranging from remote sensing, to military surveillance, face zooming in video sequence, and so forth. Some researches [8], [9] and [15] use the interpolation method, such as bilinear and bi-cubic interpolation, etc., to obtain the high-resolution image. However, it is difficult to interpolate the details well, such as the texture and corner-like regions. Considering that one single low-resolution image may not provide enough information to recover the high-frequency details, multiframe image reconstruction algorithms [5], [13] and [14] were developed to facilitate the reconstruction of single high-resolution images from a sequence of low-resolution Y. Yagi et al. (Eds.): ACCV 2007, Part II, LNCS 4844, pp. 96–105, 2007. © Springer-Verlag Berlin Heidelberg 2007

Learning-Based Super-Resolution System

97

images. Some of the methods are successful to enlarge the images, but we focus on reconstructing the high-resolution image from one single low-resolution image in this paper since it may be more useful in our daily life. Recently, the application of SR to the resolution enlargement of single images has attracted particular attention. For example, Freeman et al. [6], [7] learned the relationship between the corresponding high- and low-resolution pairs from the training data and modeled the images as a Markov network. Although the algorithms presented in these studies were applicable to the resolution scaling of generic images, they were unsuited to the manipulation of facial images characterized by pronounced geometric structures, as described in [1]. Baker and Kanade [1], [2] developed a learning-based facial hallucination method in which a process of Bayesian inference was applied to derive the high-frequency components of a facial image from a parent structure. Meanwhile, Liu et al. [10] presented a two-step procedure for hallucinating faces based on the integration of global and local models. The structure of face is generated in the global model and the local texture is obtained in the local model. However, the methods proposed in [1], [2] and [10] were sometimes difficult to obtain in practice since they were reliant upon the complicated probabilistic models. In general, the objective of SR reconstruction is to preserve the high-frequency details of an image while simultaneously expanding its scale to the desired resolution. Various researchers have demonstrated the successful interpolation and reconstruction of high-resolution images using wavelet algorithms [3], [4], [11]. Such algorithms have a number of fundamental advantages when applied to image reconstruction, including the ability not only to keep the geometric structure of the original image, but also to preserve a greater proportion of the high-frequency components than other methods, such as interpolation. Accordingly, the current study develops a learning-based SR method based upon a multi-resolution wavelet synthesis approach. The SR system consists of two basic processes, namely a training process and a synthesis (reconstruction) process. In the training process, the input image is decomposed into a series of increasingly lowerresolution images using a discrete wavelet transformation (DWT) technique. The image at each resolution level is divided into patches, which are then projected onto an eigenspace to compute the corresponding projection weight vectors. The projection weight vectors generated from all of the patches within the image at each level are grouped into a projection weight database. In the subsequent synthesis procedure, the input image is firstly interpolated to a double-size one (Fig.2, I). By applying three corresponding Haar filters on it, the three initial subbands are created. The three subbands together with the input image are then divided into patches, which are then mapped onto the eigenspace associated with the lowest-resolution input image in the training process. The resulting projection weight vector for each patch is then compared with those within the corresponding projection weight database. The bestfitting vector for each patch is identified by modeling the spatial relationships between patches as Markov network and using the maximum a posteriori (MAP) criterion. The corresponding patch is then mapped back to the input image. Having replaced all of the patches in the three subbands with the best-fitting patches from the training database, the resulting image is synthesized using an inverse wavelet transformation process and used as the input image for an analogous synthesis procedure performed at the next (higher) resolution level. This up-sampling procedure is repeated iteratively until the required image resolution has been obtained.

98

S.-F. Lui et al.

2 Training Process Figure 1 presents a schematic illustration of the training process within the current SR system. As shown, the input image is decomposed into two lower-resolution images, each divided into four sub-bands. At each resolution level, patches from each subband are concatenated to form a patch vector, which is then projected onto the corresponding eigenspace to obtain the corresponding projection weight vector. The vectors associated with each image resolution level are grouped to construct a projection weight database. w

h

level 0

w/2

h/2

Projection Weight Database 1

Haar DWT

w/2

Haar

LL 1

HL HH

…

LH

e1

Projection

N1

level 1

Eigenspace

…

3x3

HL

LL

HH w/4

LL 1

LH

HH

N2

3x3

HL HH

Eigenspace e1

…

HL

LH

…

LL

eN1

Projection Weight Database 2

Haar DWT

h/4 level 2

LH

Projection

eN2

Fig. 1. Flowchart of SR system triaining process. The original input training image (Level 0) is successively transformed using Haar discrete wavelet transformation (DWT) to generate lowerresolution images (Levels 1 and 2, respectively) comprising four sub-bands. At each level, 3x3 pixel patches from each sub-band are concatenated to form a 3×3×4-dimensional vector with a one patch-based eigenspace. These vectors are projected onto the eigenspace to generate a corresponding projection weight vector. The projection weight vectors produced at each level are grouped to form a projection weight database.

2.1 Feature Extraction Using Multi-resolution Wavelet Analysis Discrete wavelet transformation (DWT) is a well-established technique used in a variety of signal processing and image compression / expansion applications. In the

Learning-Based Super-Resolution System

99

current SR system, DWT is applied to decompose the input image into four subbands, namely LL, LH, HL and HH, respectively, where L denotes a low-pass filter and H a high-pass filter. Each of these sub-bands can be thought of as a smaller-scale version of the original image, representing different properties of the image, such as its vertical and horizontal information, for example. The LL sub-band is a coarse approximation of the original image, containing its overall geometrical structure, while the other three sub-bands contain the high-frequency components of the image, i.e. the finer image details. In the synthesis stage of the proposed SR system, the LL sub-band is used as the basis for estimating the best-fitting patches in the remaining three sub-bands. Having identified these patches, the four sub-bands are synthesized using an inverse DWT process to construct an input image for the subsequent higherlevel image reconstruction process. Note that in the current system, the Haar DWT function is employed to carry out wavelet transformation in the training and synthesis procedures since for this function, the direct transform is equal to its inverse. The multi-resolution wavelet approach described above can be repeated as many times as required to achieve the desired scale of enlargement. Assuming that the input image has a 48×64-pixel resolution, the output image should therefore have a resolution of 192x256 pixels. Accordingly, the SR procedure involves the use of two consecutive reconstruction processes, i.e. an initial process to enlarge the input image from 48x64 pixels to 96×128 pixels, and then a second process to produce the final enlarged image with the desired resolution of 192×256 pixels. 2.2 Patch-Based Eigenspace Construction As described in the previous section, four sub-band images are required for the wavelet reconstruction process. However, initially only a single low-resolution image is available. Therefore, the problem arises as to how the coefficients of the other three sub-bands may be estimated. In the current study, this problem is resolved by using a learning-based approach [1], [2] to estimate the missing high-frequency image details on the basis of the image information contained within the training datasets. As shown in Fig. 1, the current SR system incorporates two training datasets, namely projection weight databases 1 and 2, respectively. Database 1 is constructed by applying DWT to each of the original (i.e. Level 0) w×h-pixel training images, resulting in the creation of four sub-band images, i.e. LL, LH, HL and HH, respectively. These sub-band images are then further divided into an array of 3×3pixel patches. Since the image has a total of four sub-bands, a 36-dimensional patchbased vector can be constructed by concatenating the 9 pixels in the corresponding patches of each sub-band. Assuming that a single sub-band image can be divided into M patches and that the training dataset contains a total of N original images, respectively, then an M×N by 36 patch-based matrix can be constructed. There will be over 250000 patch-based vectors if we use 100 192×256-pixel resolution images to train the database, so we use PCA to lower the dimension in order to speed up the computational time of the system. Having constructed projection weight database 1 for the level-1 resolution images, a similar procedure is performed to compute the projection weight vectors for project weight database 2, corresponding to the lowestresolution level images.

100

S.-F. Lui et al.

3 Synthesis Process 3.1 Wavelet Synthesis and Patch-Based Weight Vector Creation As described in Section 2.1, each input image is decomposed into four sub-bands. Subband LL contains the global image structure, but lacks the finer details of the image. Therefore, the LL sub-band is used as the input image in the reconstruction process. The patches within the other three sub-bands are then estimated and used to reconstruct the higher-resolution image, as described below. w h

Inverse wavelet

Output

Bi-cubic interpolation: I1

LL LH HL HH Projection HL

1

3x3

HH

…

LL LH

N1

Markov network: Best Match using MAP

Eigenspace 1 Weight Database 1

Second wavelet reconstruction

Haar filter w/2 Inverse wavelet h/2

LL LH HL HH Projection HL HH

w/4 Input

Haar filter

N2

3x3

…

h/4

LL

Bi-cubic LH interpolation: I2

1

Markov network: Best Match using MAP

Eigenspace 2 Weight Database 2

First wavelet reconstruction

Fig. 2. Flowchart of SR system synthesizing process. The input image with a quarter-resolution of the desired output is interpolated and a Haar filter applied to produce four sub-bands. These sub-bands are then divided into patches and projected onto patch-based eigenspace 2 to generate the corresponding projection weights. Using the maximum a posteriori criterion, the projection weights are then compared with those within projection weight database 2. The bestfitting matches are used to improve the patches in the original input image, which is then synthesized using an inverse wavelet transformation process to construct a new image with twice the resolution of the original input image. Using this new image as an input, the reconstruction process is repeated using patch-based eigenspace 1 and projection weight database 1 to synthesize the required highest-resolution image.

Learning-Based Super-Resolution System

3x3

3x3

3x3

3x3

y

Find a best match from weight database 3x3

3x3

3x3

101

x

3x3

Fig. 3. Each search vector is constructed by concatenating four 3x3-pixel patches from the four sub-band images. These search vectors are then compared with the weight projection vectors within the training database(s) to identify the best-fitting match. The training image patches corresponding to the identified weight projection vector are then mapped back to the input image. Note that the search vector and the training databases are modeled as a Markov network.

As shown in Fig. 2, the input image at the lowest level of resolution, i.e. w/4×h/4 pixels, is interpolated and partitioned into four subbands using a Haar filter. Adopting the same approach as that applied in the training process, the four subbands are divided into patches, and the pixels within corresponding patches in the four different sub-bands are then concatenated to form a search vector (see Fig. 3). All of the search vectors relating to the image are then projected onto eigenspace 2 to create a projection weight set W2i, where i = 1, …, N, in which N is the total number of patches within one sub-band. The search vectors are then compared with the projection weight vectors contained within database 2 to identify the best-matching vectors. The search patches are in raster-scan order. Having found the best-matching vector in the database, the corresponding patches are used to replace the patches in the LH, HL and HH sub-bands in the original input image. Once all of the original patches have been replaced, an inverse wavelet transformation process is performed to synthesize a higher-resolution image with a w/2×h/2-pixel resolution. This image is then used as the input to a second reconstruction process performed using eigenspace 1 and projection weight database 1 to reconstruct the desired w×h-pixel resolution image (see Fig. 2). 3.2 Markov Network: Identifying Best-Matching Vectors Using Maximum a Posteriori Approach In the current SR system, the patches within the four sub-bands are modeled as a Markov network. In this network, the observed nodes, y, represent the transformed patches and the aim is to estimate the hidden nodes, x, from y. In the current network, the hidden nodes represent the fine details of the original images, i.e. the patches in subbands LH, HL and HH. Since the spatial relationship between x and y is modeled as a Markov network (see Fig. 3), the joint probability over the transformed patch y [7] can be expressed as follows:

p ( x1 , x 2 ,..., x N , y 1 , y 2 ,..., y N ) = ∏ Ψ ( xi , x j ) ∏ Φ ( x k , y k ) , (i , j )

k

(1)

102

S.-F. Lui et al.

where Ψ and Φ are the pair-wise compatibility functions learned; i and j are neighboring nodes; and N is the total number of patches in each subband. To estimate xˆ j , which is the best match one of each patch, we adopt the MAP estimation described in [7] to take the maximum over the other variables in the posterior probability.

xˆ jMAP = arg max xj

max p( y1 , y2 ,..., y N , x1 , x2 ,..., x N ) .

[ all xi ,i ≠ j ]

(2)

According to the conditional probability and Bayes’ theorem, the posterior probability can be written as p(x | y) =

1 p(x, y) = p(y) Z

∏Ψ

( xi , x j )Φ ( y k , xk ) ,

(3)

k

(i, j )

where Z is a normalization constant, therefore the probability of any given transformed patch for each node is proportional to the product of the pair-wise compatibility functions. In this Markov network, the relationship between the observation node and the hidden node is modeled by Ψ, and the relationship between the neighboring hidden nodes is modeled by Φ. In our work, we use Ψ to find the best matching one of each patch from training data, and Φ is used to make sure the neighboring patches connect with each other.The compatibility function Φ(x, y) is used to measure the similarity between two patch vectors, and is defined as

φ (x j , y j ) = e

−| y j − x j |2 / 2σ 12 ,

(4)

where yj and xj represent patch vectors. Specifically, yj is the observed node and consists of patches from the transformed images and xj is the hidden node which is to be found in the projection weight database 1. σ1 is a standard deviation which is the corresponding eigenvalue of the eigenvector in our system. The compatibility function between the hidden nodes is defined as

ψ ( xi , x j ) = e

− | d ji − d ij | 2 / 2 σ 22

,

(5)

where dij denotes the pixels in the overlapping region between node i and j, and σ2 is the standard deviation of the overlapping region. The pixels in the overlapping region should agree to guarantee that neighboring nodes are compatible. The best match for each search vector, is obtained by solving these compatibility functions.

4 Experimental Results To evaluate the performance of the proposed SR system, a training database was compiled comprising 100 images. Although the figures presented in this study feature facial images, the current SR system is not limited to the processing of a specific class

Learning-Based Super-Resolution System

103

of images. Hence, a general training database was constructed containing images from the CMU PIE database, a self-compiled database of facial images, and the Internet, respectively. Note that the images were not pre-processed in any way.

(b)

(a)

(1) Input (48×64)

(2) High resolution (192×256)

(1) Input (48×64)

(2) High resolution (192×256)

(4) Current method (3) Bi-cubic spline (192×256) interpolation (192×256)

(3) Bi-cubic spline (4) Current method interpolation (192×256) (192×256)

(5) Result of (2) - (3). (6) Result of (2) - (4). Average difference : Average difference : 3.3 gray value/pixel 9.7 gray value/pixel

(5) Result of (2) - (3). Average difference : 9.5 gray value/pixel

(6) Result of (2) - (4). Average difference : 4.0 gray value/pixel

Fig. 4. Comparison between enlargement results of current SR method and those obtained using a bi-cubic spline interpolation method: (a.1) low-resolution input with 48×64 resolution; (a.2) original high-resolution image with 192×256 resolution; (a.3) bi-cubic spline interpolation results; (a.4) enlargement results obtained using current method; and (a.5) and (a.6) difference images of (a.2) minus (a.3) and (a.2) minus (a.4), respectively. Note that the average difference is calculated as the sum of the gray values in the difference image divided by the total number of pixels in the image. The results show that the proposed SR method achieves a better enlargement performance than the bi-cubic interpolation. Figure 4.(b) presents a corresponding set of images for a different input image.

Increasing the patch size to 5×5 pixels was found to generate no significant improvement in the enlargement results. Typical experimental results obtained when processing facial images are shown in Fig. 3. In general, super-resolution can be

104

S.-F. Lui et al.

(a) Input (48×64)

(c) Current method (d) High resolution (b) Bi-cubic spline (192×256) (192×256) interpolation (192×256)

Fig. 5. Further results

performed using a variety of interpolation techniques, including the nearest-neighbor, bi-linear and bi-cubic spline schemes [10]. In the current study, the enlargement results obtained using the proposed SR system is compared with those obtained from the bi-cubic spline interpolation method. As shown in Figs. 4 and 5, the reconstruction results generated by the SR method are both quantitatively and qualitatively superior to those obtained using the interpolation technique.

5 Conclusions This study has presented a learning-based super-resolution system for the reconstruction of high-resolution images from a single low-resolution image. In the proposed method, a multi-resolution wavelet approach is used to synthesize both the global geometrical structure of the input image and the local high-frequency details. The SR system incorporates a training process and a synthesis process. In the training process, the high-resolution input image is transformed into a series of lowerresolution images using the Haar discrete wavelet transform (DWT). The images at each resolution level are divided into four subbands (LL, LH, HL and HH) and the corresponding patches from each subband are combined to form a search vector. The search vector is then projected onto an eigenspace to derive the corresponding projection weight vector. All of the projection weight vectors associated with the image at its current level of resolution are then grouped into a projection weight vector database. In the reconstruction process, the LL sub-band is utilized as the input image to ensure that the global, low-frequency components of the original image are retained during the enlargement process. A high-resolution image of the desired scale is obtained by performing the reconstruction process iteratively, taking the output image at one resolution level as the input to the synthesis process at the next (higher) resolution level. During each reconstruction process, the high-frequency components

Learning-Based Super-Resolution System

105

of the image are replaced by the best-fitting patches identified when modeling the patches in the subbands as a Markov network and performing a maximum a posteriori search of the projection weight database such that the finer details of the image are preserved as the image resolution is progressively enlarged.

References 1. Baker, S., Kanade, T.: Hallucinating Faces. In: Proc. of Inter. Conf. on Automatic Face and Gesture Recognition, pp. 83–88 (2000) 2. Baker, S., Kanade, T.: Limits on Super-Resolution and How to Break them. IEEE Trans. on Pattern Analysis and Machine Intelligence 24(9), 1167–1183 (2002) 3. Chan, R., Chan, T., Shen, L., Shen, Z.: Wavelet Algorithms for High-Resolution Image Reconstruction. SIAM Journal on Scientific Computing 24(4), 1408–1432 (2003) 4. Chan, R., Riemenschneider, S., Shen, L., Shen, Z.: Tight Frame: An Efficient Way for High-Resolution Image Reconstruction. Applied and Computational Harmonic Analysis 17, 91–115 (2004) 5. Farsiu, S., Robinson, D., Elad, M., Milanfar, P.: Fast and Robust Multi-Frame SuperResolution. IEEE Trans. Image Processing 13(10), 1327–1344 (2004) 6. Freeman, W.T., Jones, T.R., Pasztor, E.C.: Example-Based Super-Resolution. IEEE Computer Graphics and Applications 22(2), 56–65 (2002) 7. Freeman, W.T., Pasztor, E.C., Carmichael, O.T.: Learning Low-Level Vision. IJCV 40(1), 25–47 (2000) 8. Hou, H.S., Andrew, H.C.: Least Squares Image Restoration Using Spline Basis Function. IEEE Transactions on Computers C-26(9), 856–873 (1977) 9. Hou, H.S., Andrews, H.C.: Cubic Splines for Image Interpolation and Digital Filtring. IEEE Transactions Acoust, Speech Signal Processing 26(6), 508–517 (1978) 10. Liu, C., Shum, H., Zhang, C.: A Two-Step Approach to Hallucinating Faces: Global Parametric Model and Local Nonparametric Model. CVPR 1, 192–198 (2001) 11. Nguyen, N., Milanfar, P.: A Wavelet-Based Interpolation-Restoration Method for Superresolution. Circuits, System, Signal Process 19, 321–338 (2000) 12. Pratt William, K.: Digital Image Processing (1991) 13. Schultz, R.R., Stevenson, R.L.: Extraction of High-Resolution Frames from Video Sequences. IEEE Transactions on Image Processing 5(6), 996–1011 (1996) 14. Shekarforoush, H., Chellappa, R.: Data-Driven Multi-Channel Super-Resolution with Application to Video Sequences. JOSA. A 16, 481–492 (1999) 15. Ur, H., Gross, D.: Improved Resolution from Subpixel Shifted Pictures. CVGIP: Graph. Models Image Process 54, 181–186 (1992)

Statistical Framework for Shot Segmentation and Classification in Sports Video Ying Yang1,2 , Shouxun Lin1 , Yongdong Zhang1 , and Sheng Tang1 Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100080, China 2 Graduate University of Chinese Academy of Sciences, Beijing 100085, China {yyang, sxlin, zhyd, ts}@ict.ac.cn

1

Abstract. In this paper, a novel statistical framework is proposed for shot segmentation and classification. The proposed framework segments and classifies shots simultaneously using same difference features based on statistical inference. The task of shot segmentation and classification is taken as finding the most possible shot sequence given feature sequences, and it can be formulated by a conditional probability which can be divided into a shot sequence probability and a feature sequence probability. Shot sequence probability is derived from relations between adjacent shots by Bi-gram, and feature sequence probability is dependent on inherent character of shot modeled by HMM. Thus, the proposed framework segments shot considering the character of intra-shot to classify shot, while classifies shot considering character of inter-shot to segment shot, which obtain more accurate results. Experimental results on soccer and badminton videos are promising, and demonstrate the effectiveness of the proposed framework. Keywords: shot, segmentation, classification, statistical framework.

1

Introduction

In recent years, there has been increasing research interests in sports video analysis due to its tremendous commercial potentials, such as sports video indexing, retrieval and abstraction. Sports video can be decomposed into several types of video shots which are sequences of frames taken contiguously by a single camera. In sports video analysis, shot segmentation and classification play an important role for shots are often basic processing unit and give potential semantic hints. Much work was done on shot segmentation and classification, most of which take shot segmentation and classification as a two-stage successive process, and perform the two stages independently using different features without considering the relationship between them. Many shot segmentation algorithms have 

This research was supported by National Basic Research Program of China (973 Program, 2007CB311100), Beijing Science and Technology Planning Program of China (D0106008040291), and National High Technology and Research Development Program of China (863 Program, 2007AA01Z416).

Y. Yagi et al. (Eds.): ACCV 2007, Part II, LNCS 4844, pp. 106–115, 2007. c Springer-Verlag Berlin Heidelberg 2007 

Statistical Framework for Shot Segmentation and Classification

107

been developed by measuring the similarity of adjacent shots [1]. However, these algorithms get poorer performance on sports video due to frequent panning and zooming caused by fast camera motion. After shot segmentation, shot classification is performed on each segment for higher level video content analysis. Some work classified sports video shots based on domain rules of certain sports game [2,3], such as classifying soccer video shots into long shots, medium shots and close-up shots using dominant color. Others tried a unified method to solve this problem using SVM or HMM [4,5,6,7,8]. For these approaches, since classification is done after segmentation independently, the incorrect segmentation will have a negative effect on shot classification. In this paper, a novel statistical framework is proposed for shot segmentation and classification. Compared with previous work, the proposed framework classifies and segments shot simultaneously using same difference features based on statistical inferences. The task of shot segmentation and classification is taken as finding the most possible shot sequence given feature sequence, and it can be formulated by a conditional probability which can be divided into a shot sequence probability and a feature sequence probability. Shot sequence probability is derived from relations between adjacent shots models modeled by Bi-gram, and feature sequence probability is dependent on inherent characters of shot modeled by HMM(Hidden Markov Model). Therefore, the proposed framework segments shot considering characters of intra-shot, while classifies shot considering characters of inter-shot, which is a global search on all the possible shot sequence for the best shot sequence matching feature sequence. The rest of paper is organized as follows. In Section2, the main idea of the statistical framework is presented. Section 3 gives the details of shot segmentation and classification based on the proposed statistical framework. To evaluate the performance of this framework, two applications in soccer and badminton videos and result analysis are described in Section4. Finally, conclusions are drawn and future work is discussed in Section 5.

2

Main Idea of the Statistical Framework

Suppose that a video stream is composed of a sequence of shots, denoted by H = h1 h2 . . . ht . After feature extraction, the video stream can be viewed and manipulated as a sequence of feature vectors, denoted by O = o1 o2 . . . oT . Hence, the task of shot segmentation and classification can be seen as mapping the sequence of feature vectors to a sequence of shots, and the best mapping is the expected result of shot segmentation and classification. Hence, in the proposed framework, the task is interpreted as finding a shot sequence that maximize the conditional probability of H under condition O, namely, finding ˆ = arg max{P(H|O)} = arg max{P(H) · P(O|H)/P(O)} H (1) H

H

The equation is transformed by applying Bayes’ theorem. Since P(O) is constant for O is a known sequence, the problem can be simplified as following ˆ = arg max{P(H) · P(O|H)} H (2) H

108

Y. Yang et al.

The calculation of above probability involves two types of probability distribution, i.e. P(H) and P(O|H).The former indicates probability of shot sequence without effect of features, and the latter indicates probability of feature sequence under a given shot sequence. So we call them Shot Sequence Probability (SSP) and Feature Sequence Probability (FSP), respectively. 2.1

Shot Sequence Probability

Shot sequence H is composed of successive shots of different categories, so the shot sequence probability is dependent on the transition probability between adjacent shots. Since shot sequence is a temporal sequence, the appearance of the present shot is only related to the appearances of shots prior to it. Therefore, shot sequence can be taken as a Markov process. We suppose the shot sequence is a 1 D Markov, namely, the appearance of present shot is only related to the last shot, which can be formulated by the following equation P(hm |h1 h2 . . . hm−1 ) = P(hm |hm−1 )

(3)

This assumption is reasonable since sports video shots regularly alternate to exhibit certain semantic content according to the play status. Hence, shot sequence probability P(H) can be calculated by P(H) = P(h1 h2 . . . ht−1 ht ) = P(h1 ) · P(h2 |h1 ) · . . . P(ht |ht−1 )

(4)

where P(hi ) denotes the initial distribution probability of shot hi , and P(hj |hi ) denotes transition probability between shot hi and hj , which we called Bi-gram in our framework. So SSP can be deduced according to the Bi-gram. 2.2

Feature Sequence Probability

Feature sequence is also a temporal sequence which indicates a certain shot sequence. Since HMM has been successful used in speech recognition to model the temporal evolution of features in a word [9], we use HMM to model sports video shot for the word and shot have similar temporal structure. Hence, feature sequence of a shot is an observed sequence of a given shot HMM, and each emitting state of HMM produces a feature vector in the feature sequence [9]. So feature sequence probability P(O|H) can be transformed into  P(O, S|H) (5) P(O|H) = S

where S = s1 s2 . . . sT is the state sequence which emits feature sequence O = o1 o2 . . . oT through the link of all the shot HMMs which we called a super HMM. A super HMM is obtained by concatenating the corresponding shot HMMs using a pronunciation lexicon [10]. So P(O, S|H) can be derived from P(O, S|H) =

T  t=1

P(ot , st |st−1 , H) =

T 

[P(st |st−1 , H) · P(ot |st )]

t=1

(6)

Statistical Framework for Shot Segmentation and Classification

109

P(ot |s) is the emission probability distribution of state s, and P(st |st−1 , H) is the transition probability between two states. State transitions of intra-HMM are determined from HMM parameters, and state transitions of inter-HMM are determined by Bi-gram Therefore, these two components can be derived from the parameters of shot HMM and Bi-gram.

3

Shot Segmentation and Classification in Sports Video

In this section, we present the semantic shot segmentation and classification based on the statistical framework presented in Section 2, and shots are classified into three categories including Long Shot (LS), Medium Shot (MS) and Close-up Shot (CS). As described in Section 2, the task of shot segmentation and classification is taken as solving the problem of a maximum conditional probability P(H|O), which is dependent on SSP and FSP derived from Bi-gram and the parameters of shot HMM. Hence, Bi-gram and shot HMM are the two keys to shot segmentation and classification. The whole framework is shown in Fig.1.

Fig. 1. Framework for shot segmentation and classification

Since parameters of HMM are estimated by EM algorithm using the feature vector sequence [11], appropriate features are vital to the HMM construction and can better explain the temporal evolution in a shot. So in section 3.1, feature extraction is introduced. Section 3.2 and Section 3.3 present the shot HMM and Bi-gram constructions, respectively. In section 3.4, the procedure of simultaneous shot segmentation and classification is discussed. 3.1

Feature Extraction

As mentioned in Section 2, feature sequence of a shot is the observed vector sequence of shot HMM. So each shot is partitioned into segments to extract features from each segment, and features of all the segments form the feature vector sequence. Therefore, shot segment can be one or more consecutive frames, which called as Shot Segment Unit (SSU). Given the length of SSU, the Segmenting Rate (SR) is required at frame level to determine the space of two successive SSUs. To remain more information of shot, the size of SR may be smaller than that of SSU, as shown in Fig.2.

110

Y. Yang et al.

After the magnitudes of SSU and SR are set, feature vector is extracted from each SSU. Two classes of color and motion related features are used in our work for they are generic and can be easily computed, which can be extended to most categories of sports game. Features are firstly extracted from each frame, and then feature values of a SSU are set as the mean of corresponding feature values of all the frames in it.

Fig. 2. SSU segmentation

Color Related Features. Since the three categories of shots have different playfield and player sizes, such as LS has the largest playfield view, MS have a smaller playfield and a whole player, frames in each type of shot have a distinct color distribution which is differentiated from that of other types of shots. Hence, color features are derived from L, U and V components for CIE LUV color space is approximately perceptual uniform and computed by the following equations ⎧  ⎪ ⎨ Lf =  all pixels in frame f L(x, y)/numbers of pixels in frame f Uf = all pixels in frame f U (x, y)/numbers of pixels in frame f (7) ⎪ ⎩ Vf =  V (x, y)/numbers of pixels in frame f all pixels in frame f where Lf , Uf and Vf are the 3 basic color features of a frame f , and L(x, y), U (x, y), V (x, y) are the L, U, V components of pixel (x, y) in f , respectively. Motion Related Features. Motion is another important factor to distinguish shot for different shots reflect various camera motions, such as LS usually has relatively much smaller motion than CS. 3 basic motion features are extracted for a frame, which are frame difference Df , compensated frame difference Cf and motion magnitude Mf . Df =

255 

(H(f, i) − H(f − 1, i))2 /number of pixels in frame f

(8)

i=0

where H(f, i) is the number of pixels of color i in frame f . The other two basic motion features are built on block-based motion analysis. For each block Bf (s, t) at location (s, t) in present frame f , a block Bf∗−1 (u∗ , v ∗ ) from the previous frame f − 1 is found to best match it. So Cf and Mf are computed as follows, where Gf is the set of all blocks in frame f .  255 Cf = Bf (s,t)∈Gf ( i=0 (Hf (Bf (s, t), i) − Hf −1 (Bf∗−1 (u∗ , v ∗ ), i))2 )  (9) Mf = Bf (s,t)∈Gf ((s − u∗ )2 + (t − v ∗ )2 )1/2 Difference is widely used in signal processing, which can interpret the trend of transformation. Therefore, differences are used to express the trend of variety

Statistical Framework for Shot Segmentation and Classification

111

of color and motion of different types of shots. Given 6 basic features, the 1st and 2nd differences of basic features are computed as follows ∇1 F eaf = F eaf − F eaf −1

∇2 F eaf = ∇1 F eaf − ∇1 F eaf −1

(10)

where F eaf denotes one of the 6 basic features, ∇k F eaf (k = 1, 2) denotes the k th difference of basic feature F eaf of frame f . As a result, there is a 18-D feature vector for each frame and SSU. 3.2

Shot HMM Construction

Since there are 3 categories of shots to be classified, a general solution is to build 3 HMMs modeling 3 types of shots, respectively. However, this method only considers the temporal evolution of SSUs in a single shot, and doesn’t take into account the temporal evolution of SSUs at the transitions between adjacent shots. In fact, in sports videos, the alternation of various types of shots exhibits certain rules, such as MS is mainly followed by LS, while LS is often followed by CS. Therefore, to better simulate the temporal evolutions of SSUs of intra-shot and inter-shot, a context-dependent shot model is introduced, which is defined as tri-shot HMM. Tri-shot HMM, just as its name indicates, models the temporal evolution of SSUs in three shot including prior shot, present shot and next shot. Compared with an ordinary shot model, tri-shot model is trained use feature vector sequence of the 3 involved shots. Since we have 3 categories of shots, there are 33 = 27 tri-shots in total.

Fig. 3. Topology of HMM structure

For simplicity, we use a left-to-right HMM to represent each tri-shot HMM. The middle states are emitting states with output probability distributions, as shown in Fig.3. Each HMM contains 5 states, and Gaussian Mixture Models are used to express continuous observation densities of the emitting states. 3.3

Bi-gram Construction

As described in Section 2, Bi-gram denotes the transition probability between two adjacent shots, which indicates the possibility of a transit of a shot to another shot. In sports video, various types of shots display different play status and they don’t appear randomly. For example, after LS is shown to exhibit the global game status, MS or CS is often shown to track the player or ball. Therefore, Bi-gram

112

Y. Yang et al.

can be calculated according to the statistics of the appearances of each couple of shots in training sports video. The following formulas embody the basic idea of derivation of Bi-gram.

P(hj |hi ) = αN (hi , hj )/N (hi ), if N (hi ) = 0 (11) otherwise P(hj |hi ) = 1/l, where N (hi , hj ) is the number of times shot hj follows shot hi and N (hi ) is the number of times that shot hi appears. l l is the total number of distinct shot models, and α is chosen to ensure that j=1 P(hj |hi ) = 1. 3.4

Procedure of Shot Segmentation and Classification

With Bi-gram and tri-shot HMM constructed, sports video can be segmented and classified into 3 types of shots. Since log operator can transform product operation into summation operation, in practice, product probability in the equations presented in Section 2 can be transformed into summation of corresponding log probability. As mentioned in Section 2, a super HMM is obtained by concatenated the corresponding shot HMMs using a pronunciation lexicon. So each shot HMM in the super HMM is considered as a node. Hence, the task of shot segmentation and classification is to find a path of the maximum log probability from the start node to the end node in the super HMM, as shown in Fig.1. For an unknown shot sequence of T SSUs whose feature vector sequence is O(O = o1 o2 . . . oT ), each path from start node to end node in the super HMM which passes through exactly T emitting HMM states is a potential recognition hypothesis. The log probability of each path is computed by summing the log probability of each individual transition in the path and the log probability of each emitting state generating the corresponding SSU. Intra-HMM transitions are determined from HMM parameters, and inter-HMM transitions are determined by Bi-gram [12]. Thus, the path of maximum log probability is the best result of shot segmentation and classification. As a result, the nodes (shot HMMs) on the best path are the optimal result of shot classification. Since each state in a shot HMM matches a SSU, the boundaries of each shot are the first and last SSUs of it, which realizes shot segmentation simultaneously.

4

Experimental Results and Analysis

We first evaluated the performance of our proposed statistical framework on soccer and badminton videos, and compared the performance of our method with that of a general two-stage method of shot segmentation and classification using the same features and test videos, and then we analyzed experimental results and parameters. Test video is CIF- 352 × 288 × 25fps got from FIFA World Cup 2006 and 2005 World Badminton Championship including 4 full half-time soccer videos and 4 full game badminton videos. The implementation of the proposed

Statistical Framework for Shot Segmentation and Classification

113

framework is based on HTK 3.3 [12]. The ground-truth of boundaries and type of each shot are labeled manually. Table.1 shows the test corpus, and each shot lasts at least 1s, i.e. 25 frames. Table 1. Test Videos Name

Match(2006)

Soccer1 Soccer2 Soccer3 Soccer4

GER-ITA(7-4) ENG-POR(7-1) GER-ARG(6-30) SUI-UKR(6-26)

Length (min) 46:42 46:38 46:05 47:04

Name

Match(2005-8-21)

Badminton1 Badminton2 Badminton3 Badminton4

INA-THA(Mixed Doubles) MAS-INA(Men’s Singles) CHN-INA(Mixed Doubles) NZL-CHN(Mixed Doubles)

Length (min) 24:11 26:12 26:24 17:09

Results are assessed by recall and precision rate which can be computed by Recall = Correct/Ground truth

Precision = Correct/Detected

(12)

where Detected is the shot number obtained by shot segmentation, and Correct is the number of correctly classified shots. Correct classification denotes not only the shot type is correctly recognized but also the overlap of the ranges of classified shot and actual shot more than 90 percent of the length of actual shot. The proposed framework is tested on soccer and badminton videos since they are complete different types. Experimental results are shown in Table 2. Table 2. Experimental results on soccer and badminton video Ground-truth Detected Correct Shot Type soccer badminton soccer badminton soccer badminton LS 570 156 658 165 559 152 MS 103 121 80 392 474 326 CS 215 266 212 402 395 313 Soccer: Recall = 87.8%; Precision = 78.5% Total Badminton: Recall =93.7% ; Precision = 80.4%

On the average, the proposed framework achieves 87.8% recall and 78.5% precision rate on soccer videos, and 93.7% recall and 80.4% precision rate on badminton videos. The results are promising, which demonstrates the effectiveness of our general framework for shot segmentation and classification. We studied that false alarms are mainly caused by the misclassification of CS and MS and over segmentation of CS and MS, and the performance can be improved by applied more complicated features instead of simple color and motion features. Experiments Using SVM. we applied a general method proposed in [5] for shot segmentation and classification using the same features for comparison. To simplify the procedure, we use SVM to classify manually segmented shots in above soccer videos. We chose C = 21 and γ = 2−10 by cross-validation, and

114

Y. Yang et al.

the experimental results are shown in Table 3. The total precision rate is 70.3%, which is far less than 78.5% precision rate achieved by our method. Note that shot classification using SVM is performed on the manual segmented shots, which avoids occurrences of wrong classifications caused by inaccurate segmentation. Thus, the proposed framework performs much better. Table 3. Experimental results of SVM classification Shot Type Ground-truth Correct Precision(%) Total LS 570 530 92.9 Precision = 70.3% MS 392 181 46.2 CS 402 248 61.7

Experiments Using Various Orders of Difference Features. Another 2 types of feature vectors are tested to demonstrate that difference can improve the performance, and they are 6-D feature vector (without difference feature), 12-D feature vector (with 1st difference feature). As we can see from Table 4, performance is significantly improved with higher order difference features. Table 4. Experimental results using various orders of difference features Name Soccer1 Soccer2 Soccer3 Soccer4

6-D feature vector 12-D feature vector 18-D feature vector Recall : Precision (%) Recall : Precision (%) Recall : Precision (%) 83.5 : 72.5 85.8 : 74.7 86.1 : 76.2 84.1 : 73.2 87.3 : 74.1 87.9 : 78.9 83.2 : 68.8 87.9 : 77.1 90.1 : 79 80.6 : 75.6 85.2 : 78.5 86.7 : 80

Experiments Using Various Combinations of SSU and SR. As described in Section 3.1, the size of SSU and SR determined the fineness and number of feature vector in a shot, respectively. 8 groups of different SSU and SR are tested on soccer video using 18-D feature to find the best combination of SSU and SR.

Fig. 4. Experimental results using various combinations of SSU and SR

Statistical Framework for Shot Segmentation and Classification

115

Results are shown in Fig. 4, where the combination of SSU and SR is denoted by SSU SR (frames). As we can see, performances are much better when SR is more than 10 frames and the ratio of SSU to SR is in [1.5, 2]. So in our work, the sizes of SSU and SR are set as 20 and 10 frames, respectively.

5

Conclusions

A statistical framework for shot segmentation and classification in sports video is presented in this paper. The main idea of the proposed framework is that the task of shot segmentation and classification is taken as a conditional probability which implicates intra-shot and inter-shot information. Thus, the proposed method realizes shot segmentation and classification simultaneously, and achieves much better performance than general two-stage method on soccer videos. Experimental results on badminton videos are also promising, which demonstrate the framework can be extended to other sports video. Furthermore, difference of feature introduced from speech recognition area is applied in our framework and has been proved it’s superiority in improvement of performance. In the further work, we will employ higher semantic features to enhance the performance, and apply the framework to event detection in sports video.

References 1. Hanjalic, A.: Shot-boundary Detection: Unraveled and resolved? IEEE Trans. Circuits and Systems for Video Technology 12, 90–105 (2002) 2. Lexing, X., Peng, X., Chang, S.-F., Divakaran, A., Sun, H.: Structure Analysis of Soccer Video with Domain Knowledge and Hidden Markov Models. Pattern Recognition Letters 24(15), 767–775 (2003) 3. Ekin, A., Tekalp, A.M., et al.: Automatic Soccer Video Analysis and Summarization. IEEE Trans. Image Processing 12, 796–807 (2003) 4. Dahyot, R., Rea, N., Kokaram, A.: Sports Video Shot Segmentation and Classification. In: SPIE Int. Conf. Visual Communication and Image Processing, pp. 404–413 (2003) 5. Wang, L., Liu, X., Lin, S., Xu, G., Shum, H.-Y., et al.: Generic Slow-motion Replay Detection in Sports Video. IEEE ICIP, 1585–1588 (2004) 6. Duan, L.-Y., Xu, M., Tian, Q.: Semantic Shot Classification in Sports Video. In: SPIE Proc. Storage and Retrieval for Media Databases, pp. 300–313 (2003) 7. Duan, L.Y., Xu, M., Tian, Q., et al.: A Unified Framework for Semantic Shot Classification in Sports Video. IEEE Trans. on Multimedia 7, 1066–1083 (2005) 8. Xu, M., Duan, L., Xu, C., Tian, Q.: A fusion scheme of visual and auditory modalities for event detection in sports video. IEEE ICASSP 3, 189–192 (2003) 9. Rabiner, L.R.: A tutorial on hidden Markov models and selected applications in speech recognition. Proceeding of the IEEE 77, 257–286 (1989) 10. Ney, H., Ortmanns, S.: Progress in dynamic programming search for LVCSR. Proceeding of the IEEE 88, 1224–1240 (2000) 11. Bilmes, J.: A Gentle Tutorial on the EM Algorithm and its Application to Parameter Estimation for Gaussian Mixture and Hidden Markov Models. Technical Report of University of Berkeley, ICSI-TR-97-021 (1998) 12. Young, S., Evermann, G., et al.: The HTK book (for HTK version 3.3). Cambridge University Tech Services Ltd. (2005), http://htk.eng.cam.ac.uk/

Sports Classification Using Cross-Ratio Histograms Balamanohar Paluri, S. Nalin Pradeep, Hitesh Shah, and C. Prakash Sarnoff Innovative Technologies Pvt Ltd

Abstract. The paper proposes a novel approach for classification of sports images based on the geometric information encoded in the image of a sport’s field. The proposed approach uses invariant nature of a crossratio under projective transformation to develop a robust classifier. For a given image, cross-ratios are computed for the points obtained from the intersection of lines detected using Hough transform. These cross-ratios are represented by a histogram which forms a feature vector for the image. An SVM classifier trained on aprior model histograms of crossratios for sports fields is used to decide the most likely sport’s field in the image. Experimental validation shows robust classification using the proposed approach for images of Tennis, Football, Badminton, Basketball taken from dissimilar view points.

1

Introduction

The exponential growth during the last decade of photographic content has fueled the requirement for intelligent content management systems. One of the essential part of such systems is an automated classification of image content. In this paper, we address identification of sports based on the sport field in image using a robust classification mechanism. Conventional approaches discussed in the literature for sport identification are primarily related to video, for example Wang et. al [1] distinguish sports videos shot using color and motion features. Takagi et. al [2] proposed a HMM based video classification system using camera motion parameters, Kobla et. al [3] applied replay detection,text and motion features with Bayesian classifiers to identify sports video. Messer et. al [4] employed neural networks and the texture codes on semantic cues for the same purpose. More recently, Wang et. al [5] classified sports videos with pseudo-2D-HMM using visual and audio features. In these approaches, sports classification is dependent on cues like color, texture and spatial arrangement, which are not preferable to use due to the variability of these features between images of the same sport for two different fields and views. Also, none of the currently existing approaches applied to classify sports videos can be directly used for classifying images of sports field as they employ temporal features also. As opposed to existing approaches, we have opted to use the geometric information of a field to identify the sport. The motivation for this approach is attributed to the fact that sports fields have dominant geometric structures on Y. Yagi et al. (Eds.): ACCV 2007, Part II, LNCS 4844, pp. 116–123, 2007. c Springer-Verlag Berlin Heidelberg 2007 

Sports Classification Using Cross-Ratio Histograms

P1’

P1

Δ14

Δ’13

Δ13

P2

Δ23 P3

117

P2’

Δ24 Δ’ 23 P3’

Δ’14

P4

Δ13 Δ24 Δ14 Δ23

=

Δ’13 Δ’24 Δ’14 Δ’23

Δ’24

P4’

Fig. 1. Invariant nature of cross ratio for four collinear points subjected to a projective transformation

a planar surface. These structures are well defined and uniform across different fields for the same sport. Moreover, these structures consist of lines that are in stark contrast, like white over green ground, to the sports fields making it easy to identify such lines using conventional image processing techniques. These observations necessitated the use of geometric information to develop a robust classifier. The idea of using geometric information like projective invariance has been used for object recognition [6,7,8]. But to the best of our knowledge, there has been no prior work done related to the classification of sport’s field images using projective invariance. In the proposed approach, we exploit the idea of invariance of cross ratio under projective transformation. Thus in any view of a sport’s field four corresponding co-linear points have the same cross ratio. Since this is true only for the four corresponding points, we use a histogram of cross-ratio for a given sport’s field image to describe a model for that sport. To ensure selection of the same points in each frame, we limit our consideration to points of intersection of dominant lines only. The paper is organized as follows. Section 2 explains the proposed approach. The results are propounded in section 3. Finally, the section 4 concludes the paper by suggesting the future work.

2

Proposed Approach

The planar geometric structures in images of a sport’s field, i.e. lines to demarcate a sport’s field, taken from different view points, are related with each other under a projective transformation. Such a transformation does not preserve

118

B. Paluri et al.

(a)

(b) 0.18 0.16 0.14 0.12 0.1 0.08 0.06 0.04 0.02 0

0

0.5

(c)

1

1.5

2

2.5

3

(d)

Fig. 2. Interim steps for calculation of the feature vector: (a) Input image (b) Canny edge detections (c) Hough transform based line detections and the points of interesection of the lines (d) Histogram of cross ratios calculated using the intersection points

distances or angles, however incidence and cross-ratios are invariant under it, see [9]. In the following paragraphs, we build upon the invariant property of both these attributes to develop a robust classifier for sports images. 2.1

Cross Ratio Histogram

For a line, incidence relations are : ‘lies on’ between points and lines (as in ‘point P lies on line L’), and ‘intersects’ (as in ‘line L1 intersects line L2’). The cross ratio is a ratio of ratios of distances. Given four collinear points p1 , p2 , p3 , and p4 in P 2 and denoting the Euclidean distance between two points pi and pj as Δij , cross ratio can be defined as shown in figure 1, τp1 p2 p3 p4 =

Δ13 Δ24 . Δ14 Δ23

(1)

Although cross ratio is invariant once the order of the points has been chosen, its value is different depending on this order. Four points can be chosen in

Sports Classification Using Cross-Ratio Histograms

119

4! = 24 different orders, but in fact only six distinct values are produced, which are related by the set 1 τ −1 τ 1 , , }. {τ, , 1 − τ, τ 1−τ τ τ −1

(2)

Thus, for given four points, we use minimum of the above cross ratio value as a representative for them. To classify a given unknown sport’s field image, a canny edge detection [10] is done on the image. The dominant lines in the image are then identified using a Hough Transform [11]. Since in a sport’s field lines drawn on the field are dominant as compared to other edges produced due to noise or player, these dominant lines are are easily identified using the above approach. On each line obtained, we find intersecting points with the rest of the lines. These intersection points are then used for a cross ratio calculation as they will be consistently detected in any view of a sport’s field. Also for all the coplanar lines, such points of intersection will correspond to the same physical point on the field irrespective of the viewing angle of the camera due to the invariance of incidence relations. From the entire set of points obtained in a line, we form subsets of four points each. For each subset we calculate the representative cross-ratio as explained using the set in equation (2). Thus for a given image, we obtain a large number of these cross ratio values which encode the geometric structures present in the field. Ideally, if we are able to consistently reproduce the corresponding points in each input image for a given sport, we can match the cross ratio values individually and obtain a good classification. But this is not the case normally, there are two issues which make the direct comparison of cross ratio values infeasible. Firstly, it is challenging to consistently establish correspondence in points across images of a sport’s field from different view points because some points might not be detected due to noise in the image and as well due to the orientation of the sports fields being significantly different. Secondly, there might be small variation in the cross ratio values due to noise in measurement of lines. To overcome these issues, a histogram based representation of cross ratios is done. For each new image to be classified, the above steps are done to obtain a histogram of cross ratios. This histogram is then used as a feature vector describing the dominant geometric structures in the image. A trained SVM classifier is used to decide the class to which each feature vector and its corresponding image belong to. The entire process with intermediate results is pictorially depicted in figure 2. 2.2

Support Vector Machine Classifier

Since multiple sports are being considered, there is a need for a multi-class SVM classifier. SVM is originally designed for binary classifications only. However, multiple strategies of extending the binary classifier to a multi-class classifier have been discussed in literature such as One-against-all [12], oneagainst-one [13], half-against-half [14] etc. In case of one-against-all(OVA), the

120

B. Paluri et al.

Table 1. Cross ratio values for horizontal and a vertical line, on the outer boundary of the sport field, in each sports. These values are calculated based on the standard field measurements directly. Tennis Badminton Football Basketball

Horizontal line Vertical line 1.0208 1.0989 1.08 1.412 1.333 1.05 0.91 0.96

N class problem is decomposed into series of N binary class problems. In oneaginast-one(OVO), for a N class problem N (N2−1) binary classifiers are trained and the appropriate class is found by a voting scheme. For a Half-againstHalf(HAH) extension, a classifier is built by recursively dividing the training dataset of N classes into two subsets. We have used a modified OVA approach. OVA is one of the earliest approaches for multi-class SVM and gives good results for most of the problems. For an Nclass problem, it constructs N binary SVMs, each binary SVM is trained with all the samples belonging to one class as positive samples and all the samples from the rest of the classes as negative. Given a sample to classify, all the N SVMs are evaluated and the label of the class that has the largest value of the decision function is chosen. One drawback of this method is that when the results from multiple classifiers are combined, each classifier gets the same importance irrespective of its competence. Hence, instead of directly comparing the decisions, we use reliability measures to make the final decision which makes the multi-class framework more robust. The static reliability measure proposed in [15] has been used for our multi-class framework.

3

Experimental Results

Images of Tennis, Basketball, Badminton, and Football fields have been used for experimental evaluation since they possess a good geometric structure in their respective fields. First column of Table 1 shows the cross ratio values considering points of intersection of the vertical lines with the outer boundary horizontal line of the field. Similarly, second column shows the cross ratio value for points of intersection of horizontal lines with the outer vertical line of the field. It can be observed that the difference of geometric structure in a sport’s field is reflected well in the cross ratio values and hence these can be used as a basis for classification of the sport’s field. Cross ratio histogram for images of two different sports fields for the all the four sports considered is shown in figure 3. It can be noted that for the same sport images taken on different fields from different view points generate similar cross ratio histograms. We have collected 200 images of each sport’s field from videos, Internet; and by taking photos of the various sport’s field we ensure that the images for each

Sports Classification Using Cross-Ratio Histograms

121

0.25

0.2

0.15

0.1

0.05

0

0

0.5

1

1.5

2

2.5

3

0

0.5

1

1.5

2

2.5

3

0

0.5

1

1.5

2

2.5

3

0.35

0.3

0.25

0.2

0.15

0.1

0.05

0

0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0

0.35

0.3

0.25

0.2

0.15

0.1

0.05

0

0

0.5

1

1.5

2

2.5

3

Fig. 3. First and second column show two seperate views of the sport field with detected line and points of intersections overlaid on the gray image. The third column shows the cross ratio histogram for both the views (blue represent the histogram for first column image and red for the second column image in each row).

class were taken with different viewpoints and illumination conditions. Out of these images, 50 images from each class were used for generating the cross ratio histogram feature vector to train the SVM classifier. The remaining 150 images from each class were mixed to form a collective database of 600 images which has been used as test set to evaluate the performance of the trained SVM classifier. To test robustness of the proposed system when presented with non-sport images, we introduced 100 random images that did not belong to any of the four sports field. For SVM classifier, we experimented with Polynomial kernel function and Radial basis kernel functions with a variety of parameter values. Among the both, we observed that the second order polynomial function performed with higher accuracy.

122

B. Paluri et al.

Table 2. Classification accuracies in percentage for each class. The overall classification accuracy is 91.857 %. Sport Tennis Badminton Basketball Football Non-sport

Tennis Badminton Basketball Football Non-sport 96.66 2 0.66 0 0.66 2.66 94 0.66 1.33 1.33 0.66 2 91.33 3.33 2.66 0.66 0.66 3.33 90.66 4.66 2 1 5 8 84

The performance metric is defined as the percentage of correct classification per sport. Classification accuracies for the 4 types of sports is given in Table 2. Overall classification error on the test dataset was observed to be 8.14 % . The results clearly indicate that for real world datasets under different conditions and viewpoints, the classification accuracies are very good. Also we noted that most of the non-sport images were also segregated well and the few which were not classified as one of the four sports consisted of images of buildings.

4

Conclusion

The paper proposed an approach for sports image classification based on geometric structure of the sports field. Our experimental validation shows encouraging results and high accuracy of classification even for widely separated views of a sport’s field. One of the observation from experimental results was that the proposed approach performs very well when used for classifying images of the sports fields it was trained for. But the performance drops, from the classification error of 6.83% to 8.14%, with introduction of random images having prominent geometric structures. Thus, currently we are working on to improve the system’s ability to discern between sports and non-sports images using other invariant cues from geometry of the sports fields to improve robustness and accuracy of the proposed approach. To address this, we are exploring a relative spatial distribution of intersection points and invariant measures for lines under projective transformation to further augment the feature vector for the classifier. Also, we are gathering more dataset for sports like Hockey, Baseball, and Lacrosse to extend the classifier.

References 1. Wang, D.H., Tian, Q., Gao, S., Sung, W.K.: News sports video shot classification with sports play field and motion features. In: ICIP, pp. 2247–2250 (2004) 2. Takagi, S., Hattori, S., Yokoyama, K., Kodate, A., Tominaga, H.: Sports video categorizing method using camera motion parameters. In: ICME 2003. Proceedings of the 2003 International Conference on Multimedia and Expo, pp. 461–464. IEEE Computer Society, Washington, DC, USA (2003)

Sports Classification Using Cross-Ratio Histograms

123

3. Kobla, V., DeMenthon, D., Doermann, D.: Identifying sports videos using replay, text, and camera motion features (2000) 4. Messer, K., Christmas, W., Kittler, J.: Automatic sports classification. icpr 02, 21005 (2002) 5. Wang, J., Xu, C., Chng, E.: Automatic sports video genre classification using pseudo-2d-hmm. In: ICPR 2006. Proceedings of the 18th International Conference on Pattern Recognition, pp. 778–781. IEEE Computer Society, Washington, DC, USA (2006) 6. Lei, G.: Recognition of planar objects in 3-d space from single perspectiveviews using cross ratio. IEEE Transactions on Robotics and Automation 6(4), 432–437 (1990) 7. Weiss, I., Ray, M.: Model-based recognition of 3d objects from single images. IEEE Transactions on Pattern Analysis and Machine Intelligence 23(2), 116–128 (2001) 8. Rajashekhar, Chaudhuri, S., Namboodiri, V.: Image retrieval based on projective invariance, pp. I: 405–408 (2004) 9. Mundy, J.L., Zisserman, A.: Geometric invariance in computer vision. MIT Press, Cambridge, MA, USA (1992) 10. Canny, J.: A computational approach to edge detection. IEEE Trans. Pattern Anal. Mach. Intell. 8(6), 679–698 (1986) 11. Hough, P.: Method and means for recognizing complex patterns. In: US Patent 3,069,654 (1962) 12. Vapnik, V.: Statistical Learning Theory. Wiley, Chichester 13. KreBel, U.H.G.: Pairwise classification and support vector machines, pp. 255–268 (1999) 14. Lei, H., Govindaraju, V.: Half-against-half multi-class support vector machines. In: Multiple Classifier Systems, pp. 156–164 (2005) 15. Yi Liu, Z.Y.: One-against-all multi-class svm classification using reliability measures. In: International Joint Conference on Neural Networks, vol. 2, pp. 849–854 (2005)

A Bayesian Network for Foreground Segmentation in Region Level Shih-Shinh Huang1 , Li-Chen Fu2 , and Pei-Yung Hsiao3 Dept. of Computer and Communication Engineering, National Kaohsiung First University of Science and Technology, Kaohsiung, Taiwan, R.O.C Dept. of Computer Science and Information Engineering, National Taiwan University, Taipei, Taiwan, R.O.C. Dept. of Electrical Engineering, National University of Kaohsiung, Kaohsiung, Taiwan, R.O.C Abstract. This paper presents a probabilistic approach for automatically segmenting foreground objects from a video sequence. In order to save computation time and be robust to noise effect, a region detection algorithm incorporating edge information is first proposed to identify the regions of interest. Next, we consider the motion of the foreground objects, and hence utilize the temporal coherence property on the regions detected. Thus, foreground segmentation problem is formulated as follows. Given two consecutive image frames and the segmentation result obtained priorly, we simultaneously estimate the motion vector field and the foreground segmentation mask in a mutually supporting manner. To represent the conditional joint probability density function in a compact form, a Bayesian network is adopted, which is derived to model the interdependency of these two elements. Experimental results for several video sequences are provided to demonstrate the effectiveness of our proposed approach.

1

Introduction

Foreground segmentation is a fundamental element in developing a system that can deal with high-level tasks of computer vision. Accordingly, the objective of this paper is to propose a new approach to automatically segment the objects of interest from the video. 1.1

State of the Art

Generally speaking, techniques for foreground segmentation can be grouped into two categories, namely, background subtraction and motion-based foreground segmentation. For background subtraction, a simple and an intuitive way is to model the background by independent representation of the gray level or color intensity of every pixel in terms of some uni-modal distributions or an uniform distributions [1,2]. If the intensity of each pixel is due to the light reflected from a particular surface under a single source of lighting, a uni-modal distribution will be sufficient to represent the pixel value. However, in the real world, the model Y. Yagi et al. (Eds.): ACCV 2007, Part II, LNCS 4844, pp. 124–133, 2007. c Springer-Verlag Berlin Heidelberg 2007 

A Bayesian Network for Foreground Segmentation in Region Level

125

of a pixel value in most video sequences is characterized by multi-modal distribution, and hence the usage of a mixture of Gaussian distributions is common in it [3,4,5]. For example, Friedman and Russell [4] modeled the pixel intensity as a mixture of three Gaussian distributions respectively corresponding to three hypotheses - road, vehicle, and shadow. In [5], this idea is extended to model the scene by using multiple Gaussian distributions and to develop a fast approximate method for incrementally updating the parameters. However, not all distributions are in Gaussian form [6]. In [7,6], a non-parametric background model based on kernel function is employed to represent the color distribution of each background pixel. This kind of approach is a generalization of the mixture of Gaussians, but unfortunately the resulting computation complexity of this method is high. Moreover, their performance deteriorates when the background exhibits noise or dynamic properties. This is because they carry out the foreground segmentation at pixel level without taking spatial relations among pixels into consideration. The approaches of motion-based foreground segmentation arise from the fact the images of foreground objects are generally accompanied with motion. Accordingly, motion-based foreground segmentation refers to dividing of an image into a set of motion-coherent objects which are associated with different labels. To achieve this, the first step is to iteratively compute the dominant motion and significant areas consistent with the current dominant motion [8]. A drawback of these methods is that their performance deteriorates in lack of well-defined dominant motion. To overcome this problem, color information is introduced to obtain more accurate segmentation results. Color segmentation algorithms, such as watershed or mean-shift ones are first applied to obtain an initial segmentation and then the regions with similar motion and intensity are merged. Finally, spatiotemporal constraint is imposed to classify each segmented region into foreground or background. This kind of approaches is generally referred to as region-merging approaches [9,10,11]. Nevertheless, motion-based foreground segmentation approaches are seriously limited by the accuracy of the motion estimation algorithm. Thus, the segmentation results may become unsatisfactory when the image sequence exhibits severe noise effect or when the foreground object is subjected to large displacement. 1.2

Approach Overview

To reduce the problem dimensionality and to provide more accurate segmentation results, an algorithm incorporates the edge information with intensity to identify the regions of interest and the neighborhood relations among them are expressed by a region adjacency graph (RAG). Let G = {V, E} be a RAG, where V = {si : i ∈ [1, ..., R]} is a set of nodes in graph and each node si corresponds to a region; E is a set of edges and (si , sj ) ∈ E if the regions si and sj are adjacent. Here, R is the number of extracted regions of interest. The set of neighboring regions of si is denoted as N (si ) = {sj : (si , sj ) ∈ E and si = sj }. The main drawback of the methods [12,9] that directly estimate the motion followed by foreground segmentation is that their performance is limited by

126

S.-S. Huang, L.-C. Fu, and P.-Y. Hsiao

the motion estimation. Instead of that, we model the relationship between the motion vector field Dt and the foreground segmentation mask Ft throught a Bayesian network. To impose the temporal coherence, the main idea behind is to reinforce the interdependency between Dt and Ft so that motion estimation and foreground segmentation can be mutually support to each other. The remainder of this paper is organized as follows. In section 2, we present an algorithm to extract regions of interest. Based on the obtained regions, section 3 describes how to model the foregoing interdependency among variables through a Bayesian network and to infer the results of motion vector field and foreground segmentation mask simultaneously. Section 4 gives the probability definition of each term aforementioned. In section 5, we demonstrate the effectiveness of the developed approach by providing some satisfactorily experimental results. Finally, we conclude the paper in section 6 with some insightful discussions.

2

Interested Region Detection

In tradition, color segmentation followed by the region merging algorithm [9,10] is a widely used method for region generation. The approaches based on such a region-merging strategy usually result in over-segmentation, Here, we propose an algorithm that incorporates both intensity and edge information to identify the regions of interest so that noise effect can be successfully removed and the number of regions can be greatly decreased. This algorithm mainly consists of two steps, namely, CDM (change detection mask) detection and region generation. 2.1

CDM Detection

First of all, an initial change detection mask (CDMit ) between two consecutive image frames It−1 and It , is computed by a thresholding operation. Fig. 1(a) shows an example of CDMi between frames 39 and 40 in Hall Monitoring video sequence. However, only taking intensity into account for change detection suffers from the problem that non-changed pixels are easily mis-detected as changed ones. Here we incorporate the moving edge in the CDM detection step for identifying the regions with motion. The moving edge (ME) map is defined as   (1) MEt = e ∈ Et | min ||e − x|| ≤ Tchange , x∈DEt where Et is the edge map of the current frame It , DEt is the difference edge map obtained by applying Canny edge detector on the difference image Dt , and Tchange is a distance threshold. In our implementation, Tchange is set to 2. Fig 1(b) shows the ME of frame 40 in Hall Monitoring video sequence. Obviously, moving edges only result in the boundaries of foreground objects that are invariant to noise effect. Let these pixels in ME serve as initial seed points, we then use the region growing algorithm [13] is used to expand them pixel by pixel in CDMi. The output of the region growing algorithm is denoted

A Bayesian Network for Foreground Segmentation in Region Level

127

as CDMu. Note that the noise effect is almost removed as shown in Fig. 1(c) after incorporating the extracted moving edges. In order to maintain temporal stability which means that pixels belonging to the previous object mask should also belong to the current change mask, we set all pixels in the previous object mask OMt−1 as changed ones in the resulting change detection mask (CDMt ). Fig. 1(d) shows the finally obtained change detection mask.

Fig. 1. An example of the algorithm for interested region detection

2.2

Region Generation

Based on the assumption that the foreground boundaries always coincide with color segmented boundaries, the colors of pixels in the CDM are first quantized by a k-means clustering algorithm in RGB color space followed by the connectedcomponent finding algorithm so as to extract a set of homogeneous color regions. Currently, the number of quantized colors used here is 12. We show all regions with different colors in Fig. 1(e). Finally, a data structure called region adjacency graph (RAG) is used to represent the spatial relations among these regions of interest.

3

Bayesian Foreground Segmentation

In our formulation, the problem of foreground segmentation is described as follows. Given previous foreground segmentation mask Ft−1 and two consecutive image frames It−1 and It , we define the joint conditional probability density function of the two variables: motion vector field Dt and foreground segmentation mask Ft , that is, Pr(Dt , Ft |It−1 , It , Ft−1 ). To be clearer, Dt is a random variable that represents the motion of each region. For simplicity, we use translation motion model to describe the region motion in this research. As for Ft , it is a random variable that assigns a label, such as background (b) or foreground (f ) to each region. From Bayes’ rule, the a posteriori probability density function of Dt and Ft given It−1 , It , and Ft−1 , can be expressed as Pr(Dt , Ft |Ft−1 , It−1 , It ) =

Pr(Dt , , Ft−1 , Ft , It−1 , It ) , Pr(Ft−1 , It−1 , It )

(2)

Because Pr(Ft−1 , It−1 , It ) is constant with respect to the unknowns and the estimation of Dt and Ft is by computation of the maximum a posteriori (MAP), that is, ˜ t , F˜t ) = arg max Pr(Dt , Ft−1 , Ft , It−1 , It ) (D (3) (Dt ,Ft )

In order to represent the joint probability density function in a compact form, we must identify the dependency among these variables. A Bayesian network

128

S.-S. Huang, L.-C. Fu, and P.-Y. Hsiao

in Fig. 2(a) is adopted to model the interrelationships among these variables in three aspects. For explanation, we decompose it into three sub-graphs as shown in Fig. 2(b)(c)(d). Under such modeling, Pr(Dt , Ft−1 , Ft , It−1 , It ) can be decomposed into five terms and can be further expressed as: Pr(Dt , Ft−1 , Ft , It−1 , It ) = Pr(It |Ft ) Pr(It−1 |Dt , It ) × Pr(Ft−1 |Dt , Ft ) Pr(Dt ) Pr(Ft ).

(4)

The first term Pr(It |Ft ) models the observation likelihood and is commonly used in traditional background subtraction approaches. The graphical model expressing this term is shown in Fig. 2(b). The second term Pr(It−1 |Dt , It ) stands for displaced frame difference (DFD), which was widely used in [14] for estimating motion vector field. Fig. 2(c) shows its graphical model. The third term Pr(Ft−1 |Dt , Ft ) represents the temporal coherence [10] which denotes that the foreground segmentation mask Ft should be almost consistent with the Ft−1 through motion compensation using Dt (see Fig. 2(d)). Finally, the last two terms ,Pr(Dt ) and Pr(Ft ), respectively model the prior probabilities of Dt and Ft , respectively.

Fig. 2. Graphical Models

4

Probability Modeling

In this section, we will concentrate our attention on defining the five terms described in (4). The term Pr(It |Ft ) evaluates how well the currently observed image It fits the labels given in the Ft . Here, the label assigned to a region site s in Ft is denoted as Ft (s) and it is either b (background) or f (foreground). Accordingly, the two conditionally probability density functions Pr(It (s)|Ft (s) = b) and Pr(It (s))|Ft (s) = f ) must be defined for modeling background likelihood and foreground likelihood, respectively. The way we define these two functions is the same as [15]. 4.1

Temporal Constraint

Temporal information is a fundamental element to maintain the consistency of the results over time. The usage of motion information across image sequence is a general way to impose the temporal coherence.

A Bayesian Network for Foreground Segmentation in Region Level

129

Displaced Frame Difference. The conditional probability Pr(It−1 |It , Dt ) quantifies how well the estimated motion vector fits the consecutively observed images. As in [14], we model this term by a Gibbs distribution with the following energy function.   UDF D (It−1 |It , Dt ) = λDF D (x, y)2 , (5) si ∈V (x,y)∈si

where (x, y) = ||It (x, y) − It−1 ((x, y) − Dt (si ))||

(6)

is the displaced frame difference (DFD) and λDF D is the parameter for controlling the contribution of this term. Tracked Foreground Difference. The term Pr(Ft−1 |Dt , Ft ) is to model temporal coherence between the segmented foreground masks Ft−1 and Ft , respectively, at two consecutive time instants. It denotes that a region site at the current frame will probably continue having the same label as its corresponding region through motion compensation using Dt in the previous frame. Therefore, we define the energy function of Pr(Ft−1 |Dt , Ft ) as: UT F D (Ft−1 |Dt , Ft ) =   − λT F D δ(Ft−1 (x , y  ) − Ft (x, y))

(7)

si ∈V (x,y)∈si

where δ(x) is the Kronecker delta function and (x , y  ) is the corresponding pixel of (x, y) ∈ si at the previous frame, that is, (x , y  ) = (x, y) − Dt (si ). 4.2

Spatial Constraint

The property of spatial constraint infers that region sites in adjacency tend to have the same label. In other words, it encourages the formation of smooth motion estimation in Dt and contiguous components in Ft . Motion Smoothness. Pr(Dt ) represents the prior probability of the motion vector field and the motion smoothness constraint is imposed to define this term. Suppose that si and sj are two neighboring region sites and bij denotes the length of the common border between si and sj . We define Pr(Dt ) by a Gibbs distribution with the energy function   UMS (Dt ) = λMS bij ||Dt (si ) − Dt (sj )|| (8) si ∈V sj ∈N (si )

where ||.|| is the Euclidean distance. Foreground Smoothness. As for Pr(Ft ), we relate it to the foreground smoothness which means that the two neighboring region sites should be likely

130

S.-S. Huang, L.-C. Fu, and P.-Y. Hsiao

assigned to the same label with high probability. Similar to Pr(Dt ), Pr(Ft ) is defined as follows.   UF S (Ft ) = λF S VF S (si , sj ) (9) si ∈V sj ∈N (si )



where VF S (si , sj ) =

−bij if Ft (si ) = Ft (sj ) +bij otherwise

(10)

Once the energy functions for modeling the interdependency among the aforementioned variables are defined, the next step is to find a solution by maximizing (4). However, due to the nature of the problem, this probability distribution is non-convex and is difficult to estimate Dt and Ft simultaneously. Therefore, we employ an iterative optimization strategy [16] to find the solution as the results of the foreground segmentation.

5

Experiment

This section presents some experiment results of the proposed approach described in this paper. Both subjective evaluation and objective evaluation in comparison with Lee’s approach [17] are introduced to demonstrate the effectiveness of our approach. 5.1

Subjective Evaluation

The first image sequence we use here is the Hall Monitoring MPEG-4 test sequence in CIF format at 10 fps. In Fig. 3, the first row illustrates the original frames and the numbers shown in the top-left corner are the frame number. In particular, frame 10 used here is to demonstrate that our approach can automatically detect the newly introduced objects. On the contrary, in Lee’s approach, the red circles referring to the noisy pixels due to light fluctuation in Fig. 3(b) and shadow areas indicated by blue circles in Fig. 3(b) are both mis-classified as foreground regions. Therefore, it can be seen that performing foreground segmentation in a more semantic region level and using an appropriate scaling factor, both the noise and the shadow effect can be successfully removed in our proposed approach as shown in Fig. 3(c). 5.2

Objective Evaluation

Besides subjective evaluation, we take two videos from the contest held in IPPR (Image Processing and Pattern Recognition), Taiwan, 2006, with manually generated ground-truth images to validate the effectiveness of our approach from objective viewpoint. Here, the error rate ε similar to [1] is adopted for objective evaluation. It is defined as ε = Ne /NI , where NI is the frame size and Ne is the number of pixels that are mis-classified. Two videos with frame size

A Bayesian Network for Foreground Segmentation in Region Level

10

20

40

60

100

131

80

(a) Original Images noise

shadow

shadow

shadow

‘ s Approach (b) Lee ‘

(c) Our Approach

Fig. 3. Hall Monitoring Sequence. (a) are frames 10, 20, 40, 60, and 80. (b) and (c) are the segmentation results of Lee’s approach and our proposed approach, respectively.

Fig. 4. IPPR Contest Sequence One. (a) are original images exhibiting similar appearance and large movement; (b) and (c) are the segmentation results of Lee’s approach and our approach, respectively.

320 × 240 from IPPR contest (http://www.ippr.org.tw/) are unfortunately low quality image sequences and have severe noise effect. The challenges of the first video lie in that a person has similar appearance to the background and the foreground objects have large movements in the scene. However, by modeling the interdependency between Dt and Ft , we can

132

S.-S. Huang, L.-C. Fu, and P.-Y. Hsiao

Fig. 5. Error Rate

obtain the accurate motion estimation to facilitate the foreground segmentation as demonstrated in the right-bottom picture in Fig. 4(c). The error rate curves over 150 frames of the Lee’s approach and our approach are both shown in Fig. 5(a) and the average error rates are 1.1% and 0.5%, respectively. The second IPPR contest sequence shows that two persons go through the gallery. The strong lighting shed through the left windows is blocked by the walking persons and severe shadow areas on the ground and wall are unfortunately formed. Due to space limitation, the results of this video sequence are skipped in this paper. For this video, the average error rates (see Fig. 5(b)) of the Lee’s approach and our approach are 2.3% and 1.0%, respectively.

6

Conclusion

In this paper, we propose a region detection algorithm by incorporating moving edges with intensity difference to extract the regions of interest for further foreground segmentation at the region level. This greatly reduces the influence due to noise effect and at the same time saves the computation complexity. Under the proposed Bayesian network representing the crucial interdependency of the involving variables, the problem of foreground segmentation task is naturally formulated as a kind of inference problem with the imposed spatiotemporal constraints corresponding to the abovementioned interdependency relationship. The solution to this problem is then an iterative optimization strategy which is set to maximize the objective functions of Dt and Ft under the MAP criterion. The main advantage is that both motion estimation and foreground segmentation are simultaneously achieved in a mutually supporting manner so that more accurate results can be obtained.

Acknowledgement This work was supported by the National Science Council under the project NSC 94-2752-E-002-007-PAE and NSC 96-2516-S-390-001-MY3.

A Bayesian Network for Foreground Segmentation in Region Level

133

References 1. Chien, S.Y., Ma, S.Y., Chen, L.G.: Efficient Moving Object Segmentation Algorithm Using Background Registration Technique. IEEE Trans. on Circuits and Systems for Video Technology 12(7), 577–586 (2002) 2. Haritaoglu, I., Harwood, D., Davis, L.S.: W4: Real-Time Surveillance of People and Their Activities. IEEE Trans. on Pattern Analysis and Machine Intelligence 22(8), 809–830 (2000) 3. Benedek, C., Sziranyi, T.: Markovian Framework for Foreground-BackgroundShadow Separation of Real World Video Scenes. In: Narayanan, P.J., Nayar, S.K., Shum, H.-Y. (eds.) ACCV 2006. LNCS, vol. 3851, pp. 898–907. Springer, Heidelberg (2006) 4. Friedman, N., Russell, S.: Image Segmentation in Video Sequence: A Probabilistic Approach. In: Intl. Conf. on Uncertainty in Artificial Intelligence (1997) 5. Stauffer, C., Grimson, W.: Adaptive Background Mixture Models for Real-Time Tracking. IEEE Conf. on Computer Vision and Pattern Recognition 2, 246–252 (1999) 6. Elgammal, A., Harwood, D., Davis, L.S.: Non-parametric Model for Background Subtraction. European Conf. on Computer Vision 2, 571–767 (2000) 7. Sheikh, Y., Shah, M.: Bayesian Modeling of Dynamic Scenes for Object Detection. IEEE Trans. on Pattern Analysis and Machine Intelligence 27(1), 1778–1792 (2005) 8. Wang, J.Y.A., Adelson, E.H.: Representation Moving Images with Layers. IEEE Trans. on Image Processing 3(5), 625–638 (1994) 9. Tsaig, Y., Averbuch, A.: Automatic Segmentation of Moving Objects in Video Sequences: A Region Labeling Approach. IEEE Trans. on Circuits and Systems for Video Technology 12(7), 597–612 (2002) 10. Patras, I., Hendriks, E.A., Lagendijk, R.L.: Video Segmentation by MAP Labeling of Watershed Segmentation. IEEE Trans. on Pattern Analysis and Machine Intelligence 23(3), 326–332 (2001) 11. Altunbasak, Y., Eren, P.E., Tekalp, A.M.: Region-Based Parametric Motion Segmentation Using Color Information. Graphical Models and Image Processing: GMIP 60(1), 13–23 (1998) 12. Chen, P.C., Su, J.J., Tsai, Y.P., Hung, Y.P.: Coarse-To-Fine Video Object Segmentation by MAP Labeling of Watershed Regions. Bulletin of the College of Engineering, N.T.U 90, 25–34 (2004) 13. Adams, R., Bischof, L.: Seeded Region Growing. IEEE Trans. on Pattern Analysis and Machine Intelligence 16(6), 641–647 (1994) 14. Wang, Y., Loe, K.F., Tan, T., Wu, J.K.: Spatiotemporal Video Segmentation Based on Graphic Model. IEEE Trans. on Image Processing 14(7), 937–947 (2005) 15. Huang, S.S., Fu, L.C., Hsiao, P.Y.: A Bayesian Network for Foreground Segmentation. In: IEEE Intl. Conf. on Systems, Man, and Cybernetics, IEEE Computer Society Press, Los Alamitos (2006) 16. Chang, M.M., Tekalp, A.M., Sezan, M.I.: Simultaneous Motion Estimation and Segmentation. IEEE Trans. on Image Processing 6(9), 1326–1333 (1997) 17. Lee, D.S.: Effective Gaussian Mixture Learning for Video Background Subtraction. IEEE Trans. on Pattern Analysis and Machine Intelligence 27(5), 827–832 (2005)

Efficient Graph Cuts for Multiclass Interactive Image Segmentation Fangfang Lu1 , Zhouyu Fu1 , and Antonio Robles-Kelly1,2 Research School of Information Sciences and Engineering, Australian National University National ICT Australia , Canberra Research Laboratory, Canberra, Australia 1

2

Abstract. Interactive Image Segmentation has attracted much attention in the vision and graphics community recently. A typical application for interactive image segmentation is foreground/background segmentation based on user specified brush labellings. The problem can be formulated within the binary Markov Random Field (MRF) framework which can be solved efficiently via graph cut [1]. However, no attempt has yet been made to handle segmentation of multiple regions using graph cuts. In this paper, we propose a multiclass interactive image segmentation algorithm based on the Potts MRF model. Following [2], this can be converted to a multiway cut problem first proposed in [2] and solved by expansion-move algorithms for approximate inference [2]. A faster algorithm is proposed in this paper for efficient solution of the multiway cut problem based on partial optimal labeling. To achieve this, we combine the one-vs-all classifier fusion framework with the expansion-move algorithm for label inference over large images. We justify our approach with both theoretical analysis and experimental validation.

1

Introduction

Image segmentation is a classical problem in computer vision. The purpose is to split the pixels into disjoint subsets so that pixels in the same subset form a region with similar and coherent features. This was usually posed as a problem of pixel level grouping based on certain rules or affinity measures. The simplest way to do this is region growing starting from some initial seeds [3]. More sophisticated clustering methods have been introduced later to solve the grouping problem, characterized by the work of Belongie et. al. on the EM algorithm for segmentation [4] and Shi and Malik on normalized cut [5]. While early work on segmentation is mostly focused on unsupervised learning and clustering, a new paradigm of interactive segmentation has come into fashion recently following the pioneering work of Boykov and Jolly [1]. In [1], pixels are labeled by the user with a brush to represent the foreground and background regions. The purpose is to assign the remaining unlabeled pixels to the foreground/background classes. This can be treated as a supervised version 

National ICT Australia is funded by the Australian Governments Backing Australia’s Ability initiative, in part through the Australian Re- search Council.

Y. Yagi et al. (Eds.): ACCV 2007, Part II, LNCS 4844, pp. 134–144, 2007. c Springer-Verlag Berlin Heidelberg 2007 

Efficient Graph Cuts for Multiclass Interactive Image Segmentation

135

of segmentation, providing more accuracy and robustness with user interaction. A number of methods have been proposed to extend [1] in different ways, including grab cut [6] which presents a more convenient way of user interaction instead of brush labellings, lazy snap [7] which applies an over-segmentation as a preprocessing step. Most of the previous methods for interactive segmentation consider only binary segmentation, i.e. foreground and background. An exception is the random walk segmentation algorithm proposed by Grady in [8], which can cope with multiclass segmentation. However, it is based on solving linear systems of equations and not as fast as graph cuts. To our knowledge, the problem of interactive segmentation with multiple classes or regions under the graph cuts framework has not yet been studied. In this paper, we generalize binary interactive image segmentation to multiclass segmentation based on the Markov random field (MRF) model which can then be treated by the multiway cut framework in [2]. Moreover, we develop a strategy for the efficient computation of the multiway cut cost which combines the one-vs-all classifier fusion framework with the expansion-move algorithm proposed in [9]. Some theoretical analysis are also presented on the optimality of the proposed approach. The remainder of this paper is organized as follows. In Section 2, we briefly present the background material of the MRF model for binary segmentation and the graph cuts algorithm. In Section 3, we proposed the model for multiclass interactive segmentation and the details of the labeling algorithm. Experimental results are provided in Section 5. Conclusions are given in Section 6.

2 2.1

Binary Image Segmentation Using Graph Cuts An MRF Formulation

Markov Random Fields (MRF) are undirected graphical models. They provide a probabilistic framework for solving labeling problems over the graph nodes. Let GV, E denote a graph with node-set V = {V1 , . . . , VN } and edge-set E = {Eu,v |u, v ∈ V}.Each node u ∈ V is associated with a label variable Xu which takes a set of discrete values in {1, . . . , K} where K is the number of label classes. Each Eu,v ∈ E represents a pairwise relationship between node u and node v. The optimal set of labels are obtained by maximizing the following function of joint distributions over the label set maximize P (X) = X

1 Z

 Eu,v ∈E

ψu,v (Xu , Xv )



φu (Xu )

(1)

u∈V

where φu (Xu ) and ψu,v (Xu , Xv ) are unitary and binary potential functions which determine the compatibility of the assignment of nodes in the graph to the label   classes. u ∼ v implies  that u and v are neighbors, i.e. Eu,v ∈ E. Z = X u,v∈E ψu,v (Xu , Xv ) u∈V φu (Xu ) is the unknown normalization factor independent of the assignment of X and thus can be omitted for the purpose of inference.

136

F. Lu, Z. Fu, and A. Robles-Kelly

Taking the negative logarithm of Equation 1, we can convert the MRF formulation into an equivalent problem of energy minimization as follows   minimize log P (X) = cu (Xu ) + wu,v (Xu , Xv ) (2) X

u

u∼v

where cu (Xu ) = − log φu (Xu ) and wu,v (Xu , Xv ) = − log ψu,v (Xu , Xv ). Here the binary weight term wu,v (a, b) is determined by the interactions between neighboring nodes. A special form of binary interaction term is the Potts prior for smoothness constraints, which will be introduced shortly.In this paper, we will focus on the Potts MRF model for multiclass image segmentation. 2.2

Solving Binary MRF with Graph Cuts

We focus on general Potts prior MRF in this section with the following form of the binary interaction term. wu,v (a, b) = λu,v δ(a, b)  1 a = b δ(a, b) = 0 a=b

where a, b ∈ {0, 1}

(3)

This will simply penalize disparate labels of adjacent nodes and enforces smoothness of the labels across local neighborhood. The conventional Potts prior is a special case of the above equation with constant λu,v ’s across all neighboring sites u and v. The MRF cost function in Equation 2 can then be re-written for the Potts model as minimize log P (X) = X

N  u=1

cu (Xu ) +



λu,v δ(Xu , Xv )

(4)

u∼v

The above cost function can be naturally linked with graph cuts. To see this, we first augment the original undirected graph G = V, E defined over the image pixels, where each pixel is treated as a node in the node-set V, by two additional nodes s and t and edge-links between each pixel and the two terminal nodes. Then the graph nodes are split into two disjoint subsets V1∗ = {V1 , s} and V2∗ = {V2 , t } such that V1 ∪ V2 = V. The cut metric C(G) of the graph G is then defined as the weighted sum of edges linking nodes in subset V1∗ and nodes in V2∗ .     C(G) = wu,v = wu,t + wu,s + wu,v (5) u∈V1∗ ,v∈V2∗

u∈V1

u∈V2

u∈V1 ,v∈V2

where wu,t and wu,s define the edge weights between node u and terminal nodes, and wu,v defines the edge weight between node u and node v in different subsets. Suppose s and t represent binary class 0 and 1 respectively, define wu,s = cu (1), wu,t = cu (0) and wu,v = λu,v . Then minimizing the cost function in Equation 4 is equivalent to minimizing the cut metric C(G) in Equation 5. As a result, pixel

Efficient Graph Cuts for Multiclass Interactive Image Segmentation

137

u is assigned to class 1 if it is both linked to the terminal node t representing class 1 and broken from the terminal node s representing class 0. Graph min-cut is a well studied problem and equivalent to the max-flow problem, which can be solve efficiently in polynomial time [10]. For binary classes, we can obtain globally optimal solution for the min-cut. An example of binary cut for a toy problem of 9 nodes is illustrated in Figure 1(a). For the sake of visualization, terminal links that have been severed are not shown here.

(a) Binary Cut

(b) Multiway Cut

Fig. 1. Illustration of Binary and Multiway Cuts

In interactive segmentation, we have to keep fixed the membership of the user specified pixels,i.e. brush labeled foreground pixels must remain in the foreground and so do labeled background pixels. These hard constraints over the assignment of labeled pixels can be naturally enforced with the graph cut framework by manipulating the edge weights of links between nodes of labeled pixels and the terminal nodes. For each labeled foreground (class 1) pixel uf , we can set wuf ,1 to an infinitely large value and wuf ,0 to 0 so that uf always remains linked to the foreground t node otherwise it will incur an infinite cost by assigning uf to the background s node. We can also set the weights of terminal links for background pixels in a similar way. To summarize, we list the rules for setting edge weights for binary interactive segmentation in Table 1(a).

3 3.1

Multiclass Interactive Image Segmentation A Multiway Cut Formulation for Multiclass Segmentation

The graph cut framework proposed for binary segmentation in the last section can be extended to multiclass segmentation in a straightforward manner via multiway cuts. Multiway cuts are the generalization of binary cuts and were first proposed by Boykov et.al. in [2]. For m-way multiway cuts, an extra nodeset of m terminal nodes T = {t1 , . . . , tm } are inserted in the original graph G = V, E and links are established from each t-node tj (j = 1, . . . , m) to each node in the node-set V of graph G. The purpose is to split the nodes of this

138

F. Lu, Z. Fu, and A. Robles-Kelly

Table 1. (a) Edge weights for binary segmentation. (b) Edge weights for multiclass segmentation. (b) weight

(a) edge weight

for

edge

{u, v}

λu,v u, v ∈ V and u ∼ v ∞ u ∈ C0 {u, t} 0 u ∈ C1 cu (1) otherwise

{u, s}

0 ∞ cu (0)

for

{u, v}

λu,v u, v ∈ V and u ∼ v ∞ u ∈ C1 {u, t1 } 0 u ∈ Cj (j = 1) su − cu (1) otherwise ... ∞ u ∈ Cm {u, tm } 0 u ∈ Cj (j = m) su − cu (m) otherwise

u ∈ C0 u ∈ C1 otherwise

∗ augmented graph into m disjoint subsets V1∗ = {t1 , V1 }, . . . , Vm = {tm , Vm } such that each subset contains one and only one t-node from T and V1 ∪. . . ∪Vm = V. An example of 3-way cut is illustrated in Figure 1(b). According to [2], we can establish the link between the Potts MRF cost function in Equation 4 and the multiway cut metric as follows. For arbitrary node u in V, after multiway cut, it only remains linked to a single terminal node and the edges linking to all other terminal nodes are severed. Define

du =

m 

cu (j)

(6)

j=1

the total sum of costs for assigning pixel u to all different classes. We then define the edge weights of t-links wu,tj = du − cu (j) and the edge weights between neighboring nodes in V remain the same as binary cut, i.e. wu,v = λu,v . Then the multiway cut metric becomes C(G) =



wu,v =

∀i,j∈{1,...,m} u∈Vi∗ ,v∈Vj∗

=

 

(du − cu (j)) + 

u∈V

wu,tj +

i=1 u∈Vi j=1 j=i

u∈V j=Xu

= (m − 2)

m   m 

du +

 u∈V







wu,v

(7)

i,j u∈Vi ,v∈Vj

λu,v δ(Xu , Xv )

u∼v

cu (Xu ) +



λu,v δ(Xu , Xv )

u∼v

Since du is independent of label vectors Xu ’s, the multiway cut is equivalent to the multiclass Potts MRF cost defined in Equation 4 up to a constant additive factor. In the case of binary cut where m = 2, the two costs are exactly the same. Thus, inference of a multiclass Potts MRF model can be recast as a multiway cut problem. We can also encode hard constraints for labeled pixels into the edge weights of t-links in exactly the same way as we did for binary segmentation.

Efficient Graph Cuts for Multiclass Interactive Image Segmentation

139

The resulting rules for choosing edge weights for multiclass segmentation tasks are summarized in Table 1(b).

4

Efficient Implementation of Multiway Cuts

Although multiclass interactive image segmentation can be transformed the multiway cut problem, computing the minimum value of multiway cuts is a NP hard problem. An expansion-move algorithm was proposed by Boykov et.al. in [9] to iteratively update the current labels for lower cost. However, a number of expansion and move steps have to be computed alternately to recover a suboptimal labeling for the purpose of multiclass segmentation. Each step involves solving a graph cut over all pixels in the image. This is quite inefficient for images with moderately large sizes. To make efficient inference of multiway cut without compromising on the accuracy, we present an alternative inference algorithm in this section. Strategy. Our inference algorithm makes use of the one-against-all classifier fusion framework to transform the multiway cut problem into a set of binary cut subproblems. Each binary cut subproblem has two classes, where the positive class is chosen from one of the original label classes, and the negative class is constructed by the rest of the classes. The global optimal solution of the binary subproblem can be obtained by using the max flow algorithm in [10]. We can then obtain a partial solution of the multiway cut problem by voting on the binary cut results. Pixels with consistent labeling across all binary cuts are then assigned with the consistent labels. To resolve the ambiguities in the labels of remaining pixels, we build a graph over pixels with ambiguous labels apply the expansion-move algorithm [9] to solve the multiway cut problem for the labels of the ambiguous pixels with the labels of all other pixels fixed, which is more efficient than applying the expansion-move operations to all unlabeled pixels from scratch and more accurate than resolving the ties arbitrarily. The algorithm is summarized in the Table 2. Analysis. Several points deserve further explanation here. To obtain the cost cu (j) of assigning pixel u to each different region j, we first compute the posterior probability bu,j of pixel u in class j, which can obtained via the following Bayes rule P (u|Cj ) P (u|Cj )P (Cj ) = (9) bu,j =  j P (u|Cj )P (Cj ) j P (u|Cj ) 1 is the prior probability of class j, which is assumed to be equal m for every class. P (u|Cj ) the conditional probability of pixel u in the jth region. The class conditional probabilities are obtained by indexing the normalised color histogram built from labeled pixels in the region. The edge weight term λu,v is given by ||I(v) − I(u)||2 ||pv − pu ||2 − ) (10) λu,v = βexp(− σc2 σs2 where p(Cj ) =

140

F. Lu, Z. Fu, and A. Robles-Kelly Table 2. Efficient Graph Cut Algorithm for Multiclass Segmentation

Input: image I with pixels labeled for m different regions. 1. Compute the cost cu (j) for assigning each unlabeled pixel u to the jth region. 2. Repeat the following operations for each region index by j – Establish a binary graph cut problem with the jth region as class 1 and all other 1 regions as class 0. Set c∗u (1) = cu (j) and c∗u (0) = cu (i) as new unary m − 1 i=j 1 λu,v as binary costs for neighboring costs for pixel u, and λ∗u,v = 1 − 2m − 2 pixels u and v. The meaning for the choice of weights will become evident later. – Solve the binary cut problem with new unary and binary costs and recover the (j) label Xu for each pixel u.



(1)

(m)

3. Vote on Xu , . . . , Xu Xu =

j



from the results of m binary cuts

undefined

(j)

(i)

Xu = 1 and Xu = 0 for i = j otherwise

(8)

4. Add the unlabeled pixels with resolved labels into the set of labeled pixels. Run an additional graph cuts optimization over the subset of unresolved pixels to obtain their labels using the neighboring pixels with known labels as hard constraints . Output: labels Xu for each pixel u in image I

where I(u) and pu denote the pixel value and coordinate of pixel v and σc and σs are two bandwidth parameters, β gauges the relative importance of data consistency and smoothness, which is set to 10 in the following experiments. The above choice of binary weight accounts for the image contrast and favors segmentation along the edges as justified in [1]. After obtaining the posterior probability for pixel u, we can compute the cost  of assigning u to class j by cu (j) = i=j P (u|Ci ). That is, the larger the posterior probability of class j, the smaller cost for assigning it to the  same class. Nom tice that the costs for different classes are normalized such that i=1 cu (i) = 1. In Step 2 of the above algorithm, we have to solve m binary cut problems. Notice that any two binary problems share the same graph topology and only differ in the weights of edges linked to the terminal nodes, hence starting from the second binary cut subproblem, we can capitalize on the solution of the previous subproblem for solving the current subproblem. This is achieved by the dynamic graph cuts framework proposed by Kohli and Torr in [11] to update the labels iteratively from previous results instead of starting from scratch. After running binary cuts for all m subproblems, we get m votes for the label of each pixel u. If the votes are consistent, i.e. Equation 8 is satisfied for certain

Efficient Graph Cuts for Multiclass Interactive Image Segmentation

141

region j which gets all m votes from the binary cut results, then we can assign pixel u to the jth region with confidence. We can also prove the following Proposition: The cost function of multiway cut is the aggregation of the costs of the binary cut subproblems up to a scaling and translation factor. Proof: The cost of the multiway cut problem is given by f (X) =





cu (Xu ) +

u

λu,v δ(Xu , Xv ) =

m  

cu (j) +

j=1 u|Xu =j

u,v|u∼v



λu,v

u,v|u∼v Xu =Xv

 where a|b is a notation indicating sum over variable(s) a . . . given condition(s) b . . .. The cost of the jth binary cut is given by 

f (j) (X) = =

 1 cu (i) + m−1

u|Xu =j

u|Xu =j

i|i=j

The sum over all binary costs is then given by

f m

   (X) = m

(j)

j=1

j=1

=

cu (j) +

u|Xu =j



 u|Xu =j

m−2 m−2 N + 1+ m−1 m−1

λ∗u,v

u,v|u∼v



cu (j) +



c∗u (0) +

u|Xu =j

u|Xu =j





c∗u (1) +

1 m−1

 



λ∗u,v

u,v|u∼v Xu =j,Xv =j

 c (i) +   m

u

i|i=j

m



j=1

cu (j) + 1 +

j=1 u|Xu =j

λ∗u,v

u,v|u∼v Xu =j,Xv =j

m−2 m−1

 

λu,v

u,v|u∼v Xu =Xv

where N is the total number of pixels under study. The second row of the above equation holds due to the identity below, which is true for normalized values of cu (j) over j. m  



j=1 u|Xu =j i|i=j

cu (i) = (m − 2)N + (m − 2)

m  

cu (j)

j=1 u|Xu =j

This concludes the proof. The  above proposition states that minimizing f (X) is equivalent with minimizing j f (j) (X) due to their linear relation. Hence we can treat the consistent labels over all binary cuts as the approximate partial optimal labeling and solve the labels for the few remaining inconsistent pixels. This can be done using the expansion-move algorithm for multiway cut in [9]. As a result, we only need to build a much smaller graph for resolving the ambiguities in the labeling of remaining pixels together with their labeled neighbors, as compared to the total number of unlabeled pixels from the beginning.

142

5

F. Lu, Z. Fu, and A. Robles-Kelly

Experimental Results

We present experimental results for multiclass interactive segmentation of 7 realworld color images. Pixels were selected by the user with scribbles to define different regions. The original images overlapped with scribble labellings are shown in the left column of Figure 2, where different colored brushes indicate different regions in the image. First, we compared the efficiency of the proposed algorithm with the conventional expansion-move algorithm for the purpose of multiclass image segmentation. Specifically, we adopted the α expansion algorithm [9] over the alternative α-β move algorithm in our implementation. We runned our experiment on an Intel P4 workstation with 1.7GHz CPU and 1GB RAM. We have used the maxflow code of Boykov and Kolmogorov [10] for solving graph cuts for both binary inference and α expansion which also includes implementation of the dynamic graph cuts algorithm in [11]. The wrapper code is written in matlab. The average execution time over 5 trials for the segmentation of each image is reported in Table 3. Table 3. Comparison of speed for the conventional expansion-move based multiway graph cut algorithm [9] and the proposed efficient graph cut algorithm Image Name Image Size Classes face∗ bush∗ grave∗ garden∗ tower lake beach

132 × 130 520 × 450 450 × 600 450 × 600 800 × 600 800 × 600 1600 × 1200

3 3 3 3 3 4 4

Execution Time (s) α-Expansion Efficient Graph Cuts 0.0976 0.0836 2.2765 0.9999 3.5721 1.1672 2.6871 1.1407 5.5650 2.9180 5.8270 3.4522 26.3098 13.2942

From the table, we can find that the proposed efficient graph cut algorithm is consistently faster than the expansion-move algorithm with a speedup factor of roughly 2 on average. The two methods achieve comparable results for binary labeling problems. And not surprisingly, for the images we tested, the two algorithms under study achieve similar segmentation results. Hence, in the following, we only show segmentation results of our algorithm compared with those produced by maximum likelihood estimation (MLE) based pixel-level segmentation, which is equivalent to the MRF model with zero second-order pairwise terms. The segmentation results of both our method and the pixel-level segmentation are shown in the middle and right columns of Figure 2 respectively. We visualize the segmentation results with an alpha matte αI + (1 − α)L = 0, with α = 0.3. I is the color of image pixel and L is the color of brush label for the region it belongs to. From those results, we can see that the proposed algorithm can capture the structure of the regions labeled using the brush while pixel-level approach is prone to produce small scattered regions.

Efficient Graph Cuts for Multiclass Interactive Image Segmentation

143

Fig. 2. Segmentation results on example images. Left Column: Original color images with brush labellings. Middle Column: Results of pixel based approach. Right Column: Results of multiclass graph cuts.

144

6

F. Lu, Z. Fu, and A. Robles-Kelly

Conclusions

An energy minimization approach is proposed for multiclass image segmentation in this paper based on the Markov random field model. The cost function of the MRF model can be optimized by solving an equivalent multiway graph cut problem. An efficient implementation of multiway cut is developed and experimental results demonstrate encouraging performance of the proposed approach.

References [1] Boykov, Y., Jolly, M.P.: Interactive graph cuts for optimal boundary & region segmentation of objects in n-d images. In: Intl. Conf. on Computer Vision, pp. 105–112 (2001) [2] Boykov, Y., Veksler, O., Zabih, R.: Markov random fields with efficient approximations. In: Intl. Conf. on Computer Vision and Pattern Recognition (1998) [3] Gonzalez, R.C., Wintz, P.: Digital Image Processing. Addison-Wesley Publishing Company Limited, London, UK (1986) [4] Belongie, S., Carson, C., Greenspan, H., Malik, J.: Color- and texture-based image segmentation using em and its application to content-based image retrieval. In: Proc. of Intl. Conf. on Computer Vision (1998) [5] Shi, J., Malik, J.: Normalized cuts and image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence 22(8), 888–905 (2000) [6] Rother, C., Kolmogorov, V., Blake, A.: Grabcut: Interactive foreground extraction using iterated graph cuts. ACM Trans. on Graphics 23(3), 309–314 (2004) [7] Li, Y., Sun, J., Tang, C.K., Shum, H.Y.: Lazy snapping. ACM Trans. on Graphics 23(3) (2004) [8] Grady, L.: Random walks for image segmentation. IEEE Trans. on Pattern Analysis and Machine Intelligence 28(11), 1768–1783 (2006) [9] Boykov, Y., Veksler, O., Zabih, R.: Fast approximate energy minimization via graph cuts. IEEE Trans. on Pattern Analysis and Machine Intelligence 23(11), 1222–1239 (2001) [10] Boykov, Y., Kolmogorov, V.: An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision. IEEE Trans.on Pattern Analysis and Machine Intelligence 26(9), 1124–1137 (2004) [11] Kohli, P., Torr, P.: Efficiently solving dynamic markov random fields using graph cuts. In: Intl. Conf. on Computer Vision (2005)

Feature Subset Selection for Multi-class SVM Based Image Classification Lei Wang Research School of Information Sciences and Engineering The Australia National University Australia, ACT, 0200 Abstract. Multi-class image classification can benefit much from feature subset selection. This paper extends an error bound of binary SVMs to a feature subset selection criterion for the multi-class SVMs. By minimizing this criterion, the scale factors assigned to each feature in a kernel function are optimized to identify the important features. This minimization problem can be efficiently solved by gradient-based search techniques, even if hundreds of features are involved. Also, considering that image classification is often a small sample problem, the regularization issue is investigated for this criterion, showing its robustness in this situation. Experimental study on multiple benchmark image data sets demonstrates the effectiveness of the proposed approach.

1

Introduction

Multi-class image classification often involves (i) high-dimensional feature vectors. An image is often reshaped as a long vector, leading to hundreds of dimensions; (ii) redundant information. Only part of an image, for instance, the foreground, is really useful for classification. In the past few years, multi-class Support Vector Machines (SVMs) have been successfully applied to image classification [1,2]. However, the feature subset selection issue, that is, identifying p important features from the original d ones has not been paid enough attention.1 Classical feature subset selection techniques may be used to find the important dimensions. For instance, in [3], a subset of 15 features is selected by using the wrapper approach from 29 features. However, this approach often has heavy computational load and cannot deal with image classification generally having hundreds of features. Also, the selected features may not fit the SVM classifier well when the selection criteria have no connection with the SVMs. Recent advance in the binary SVMs provides a ray of hope. In [4], a leave-oneout error bound of the binary SVMs (the radius-margin bound) is minimized to optimize the kernel parameters. The minimization is solved by gradient-based search techniques which can handle a large number of free kernel parameters. 1

Please note that this is different from the problem of finding the p optimal combination of the original d features, for instance, in the way of PCA or LDA. In feature subset selection, the features are kept individual and the feature dependence is left to the classifier, for instance, the SVMs, which handles it automatically.

Y. Yagi et al. (Eds.): ACCV 2007, Part II, LNCS 4844, pp. 145–154, 2007. c Springer-Verlag Berlin Heidelberg 2007 

146

L. Wang

Also, it is found that the optimized kernel parameters can reflect the importance of features for classification. These properties are just what we are seeking for. Unfortunately, this error bound cannot be straightforwardly applied to the multiclass scenario because it is rooted in the theoretical result of binary SVMs, for instance, the VC-Dimension theory upon which it is developed. A possible way may be to apply the radius-margin bound to all of the binary SVMs obtained in a decomposition of the multi-class SVMs. However, by doing so, the selected features will be good at discriminating a certain pair of classes only. Also, how to integrate these selection results is another issue. Hence, this paper seeks a one-shot feature selection from the perspective of multi-class classification. To realize this, this paper extends the radius-margin bound to a feature subset selection criterion for the multi-class SVMs. Firstly, the relationship between the binary radius-margin bound and the classical class separability concept is discussed. Enlightened by it, this work redefines the radius and margin to accommodate multiple classes, forming a new criterion. Its derivative with respect to the kernel parameters can also be analytically calculated, making the gradient-based search techniques still applicable. Minimization of this criterion can efficiently optimize hundreds of kernel parameters simultaneously. These optimized parameters are then used to identify the important features. This approach is quite meaningful for practical image classification because it facilitates the reduction of system complexity and the feature discovery from the perspective of multiclass classification. Experimental result on benchmark data sets demonstrates the effectiveness of this approach.

2

Background

Let D denote a set of m training samples and D = (x, y) ∈ (Rd × Y)m , where Rd denotes a d-dimensional input space, Y denotes the label set of x, and the size of Y is the number of classes, c. A kernel is defined as kθ (xi , xj ) = φ(xi ), φ(xj ), where φ(·) is a possibly nonlinear mapping from Rd to a feature space, F , and θ is the kernel parameter set. Due to the limit of space, the introduction of multi-class SVMs is omitted and this information can be found from [1,2]. The radius-margin bound. It is an upper bound of the test error estimated via a leave-one-out cross-validation procedure. Let Lm = L((x1 , y1 ), · · · , (xm , ym )) be the number of errors in this procedure. It is shown in [4] that Lm ≤

4R2 = 4R2 w2 γ2

(1)

where R is the radius of the smallest sphere enclosing all the m training samples in F , γ the margin, w the normal vector of the optimal separating hyperplane, and γ −1 = w. The R2 is obtained by solving   m m R2 = maxβ∈Rm i=1 βi kθ (xi , xi ) − i,j=1 βi βj kθ (xi , xj ) (2) m subject to : i=1 βi = 1; βi ≥ 0 (i = 1, 2, · · · , m)

Feature Subset Selection for Multi-class SVM Based Image Classification

where βi is the i-th Lagrange multiplier. The w2 is obtained by solving   m m 1 1 2 m i=1 αi − 2 i,j=1 αi αj yi yj kθ (xi , xj ) 2 w = maxα∈R m subject to : i=1 αi yi = 0; αi ≥ 0 (i = 1, 2, · · · , m)

147

(3)

where αi is the i-th Lagrange multiplier. The derivative of R2 w.r.t. the t-th kernel parameter, θt , is shown in [4] as m m   ∂kθ (xi , xi ) ∂kθ (xi , xj ) ∂R2 = βi0 − βi0 βj0 ∂θt ∂θt ∂θt i=1 i,j=1

(4)

where βi0 is the solution of Eq. (2). Similarly, the derivative of w2 w.r.t. θt is m  ∂kθ (xi , xj ) ∂w2 = (−1) · α0i α0j yi yj ∂θt ∂θt i,j=1

(5)

where yi ∈ {+1, −1} is the label of xi . Thus, the radius-margin bound can be efficiently minimized by using the gradient search methods. As seen, this bound is rooted in binary SVMs and cannot be directly applied to the multi-class case. An extension of this bound to a feature selection criterion for multi-class SVMs is proposed below.

3

Proposed Approach

Class separability is a concept widely used in pattern recognition. The scatter matrix based measure is often favored thanks to its simplicity. The Between(ST ) are defined as SB = class scatter matrix (SB ) and Total scatter matrix  c c ni   n (m − m)(m − m) and S = (x − m)(x − m) , i i i T ij ij j=1 i=1 i=1 where ni is the size of class i, xij the j-th sample of class i, mi the mean of class i, and m the mean of all c classes. The following derives the class separability in the feature space, F . Let Di be the training samples from class i and D be the set of all training samples. The mφi and mφ are the mean vectors of Di and D in F , respectively. KA,B is the kernel matrix where {KA,B }ij = kθ (xi , xj ) with the constraints of xi ∈ A and xj ∈ B. The traces of SφB and SφT can be expressed as tr(SφB ) =

c  1  KD i=1

where n =

c

i=1

i ,Di

ni

1

−

1 KD,D 1 1 KD,D 1 ; tr(SφT ) = tr(KD,D ) − n n

(6)

ni and 1 is a column vector whose components are all 1. Based

on these, the class separability in F can be written as C = two classes, it can be shown that   n1 n2 φ tr(SB ) = mφ1 − mφ2 2 ; n1 + n2

tr(SφT ) =

ni 2   i=1 j=1

tr(Sφ B) . tr(Sφ T)

In the case of

φ(xij ) − mφ 2

(7)

148

L. Wang

and it can be proven (the proof is omitted) that γ2 ≤

4−



1 n1 +n2 n1 n2



tr(SφB )

R2 ≥

;

1 tr(SφT ) . (n1 + n2 )

(8)

That is, tr(SφB ) measures the squared distance between the class means in F . tr(SφT ) measures the squared scattering radius of the training samples in F if divided by (n1 + n2 ). Conceptually, the mφ1 − mφ2 2 and γ 2 reflect the similar property of data separability and tr(SφT ) positively correlates with R2 . Theoretically, γ 2 is upper bounded by a functional of tr(SφB ) and R2 is lower bounded by tr(SφT )/(n1 + n2 ). When minimizing the radius-margin bound, γ 2 is maximized while R2 is minimized. This requires tr(SφB ) to be maximized and tr(SφT ) to be minimized (although this does not necessarily have γ 2 maximized or R2 minimized). Hence, minimizing the radius-margin bound can be viewed as approximately maximizing the class separability in F . Part of the analysis can also be seen in [5]. Enlightened by this analogy, this work extends the radius-margin bound to accommodate multiple classes. In the multi-class case, tr(SφT )/n measures the squared scattering radius of all training samples in F . Considering the analogy between tr(SφT ) and R2 , the R2 is redefined as the squared radius of the sphere enclosing all the training samples in the c classes, Rc2 =

min

ˆ 2 ,∀ xi ∈D φ(xi )−ˆ c2 ≤R

ˆ2) . (R

(9)

For tr(SφB ) in the multi-class case, it can be proven that  tr(SφB )

=

1≤i<j≤c

ni nj mφi − mφj 2 n

.

(10)

Noting the analogy between mφ1 − mφ2 2 and γ 2 , the margin is redefined as   2 Pi Pj γij = Pi Pj wij −2 (11) γc2 = 1≤i<j≤c

1≤i<j≤c

where γij is the margin between class i and j, and Pi = nni , which is the priori probability of the i-th class estimated from the training data. By doing so, a feature selection criterion for multi-class SVMs is obtained. The optimal kernel parameters is sought by minimizing this criterion,  2 Rc  . (12) θ = arg min θ∈Θ γc2 The derivation of this criterion w.r.t. the t-th kernel parameter, θt , is  2   2 2 Rc ∂ 1 2 ∂Rc 2 ∂γc − Rc = 4 γc . ∂θt γc2 γc ∂θt ∂θt

(13)

Feature Subset Selection for Multi-class SVM Based Image Classification ∂R2

149

∂w 2

The calculation of ∂θtc and ∂θijt follows the Eq. (4) and (5). This criterion can still be minimized by using the gradient-based search techniques. Finally, please note that the obtained criterion is not necessarily an upper bound of the leave-one-out error of the multi-class SVMs. It is only a criterion reflecting the idea of the radius-margin bound and working for the multi-class case. An elegant extension of the radius-margin bound to multi-class SVMs is in [6]. Regularization issue. When optimizing a criterion with multiple parameters, regularization is often needed to avoid over-fitting the training samples, especially when the number of training samples is small. To check whether this is also needed for this new criterion, a regularized version is given as J (θ) = (1 − λ)

Rc2 + λθ − θ0 2 γc2

(14)

where λ (0 ≤ λ < 1) is the regularization parameter which penalizes the deviation of θ from a preset θ0 . Such a regularization imposes a Gaussian prior over the parameter θ to be estimated. When using the ellipsoid Gaussian RBF kernel defined in Section 4, θ0 can be chosen by applying the constraint of θ1 = θ2 = · · · = θd and solving  2

Rc

 . (15) θ0 = arg min

θ∈Θ γc2 θ1 =θ2 =···=θd Since the number of free parameters reduces to one, over-fitting is less likely to occur. It is worth noting that the obtained θ0 has been optimal for the minimization of Rc2 /γc2 with a spherical Gaussian RBF kernel. It is a good starting point for the subsequent multi-parameter optimization.

4

Experimental Result

  d −yi )2 , is used. The elliptical Gaussian RBF kernel, k(x, y) = exp − i=1 (xi2σ 2 i For binary SVMs, the optimized σi can reflect the importance of the feature i for classification [4]. The larger the σi , the less important the feature i. Now, the multi-class case is studied with the proposed criterion. The BFGS Quasi-Newton optimization method is used. For the convenience of optimization, gi (gi = 1/2σi2 ) is optimized instead. A multi-class SVM classifier using the selected features is then trained and evaluated by using [7]. Following [4], the feature selection criteria of “Fisher score”, “Pearson correlation coefficient” and “Kolmogorov-Smirnov test” are compared with the proposed criterion. USPS for Optical digit recognition. This data set contains 7291 training images and 2007 test images from ten classes of digits of “0” to “9”. Each 16 × 16 thumbnail image is reshaped to a 256-dimensional feature vector consisting of its gray values. The proposed feature selection criterion is minimized on the training set to optimize the gi for each dimension. The 256 features are then sorted in a descending order of the g values, and the top k features are used for

150

L. Wang

(a) Optical character recognition

(b) Facial image classification

(c) Textured image classification

Fig. 1. Examples of image classification tasks USPS data set

0

0.055

0.8 0.7 0.6

Test error

0.05

Pearson correlation coefficients Kolmogorov−Smirnov test Fisher criterion score Proposed criterion

2 0.045 4 0.04

0.5

6

0.4

8

0.035 0.03 0.025

0.3

10

0.02

0.2

0.015

12

0.1

0.01 14

0 0

0.005

10

20 30 40 The number of selected feature

50

(a) Feature selection result

60 16

0

2

4

6

8

10

12

14

16

(b) Optimized value of g = 1/2σ 2

Fig. 2. Results for the USPS data set

classification. As shown in Figure 2(a), the test error for the proposed criterion decreases quickly with k. With only 50 features selected, it reaches 6.7% (The reported lowest error is 4.3% when all 256 features are used [8]). Compared with the other three selection criteria, this criterion gives the best feature selection performance. Now, the optimized g1 , · · · , g256 are reshaped back into a 16 × 16 matrix and shown in Figure 2(b). In this map, each block corresponds to one of the 256 g’s and its value is reflected by the gray level. As seen, the blocks with the larger g values scatter over the central part of this map, whereas those with the lower values are mostly at the borders and corners. This suggests that the pixels at the borders and corners are less important for discrimination. This observation well matches the images in Figure 1(a) in that the digits are displayed at the central parts and surrounded by a uniform black background. ORL for Facial image classification. This database contains 40 subjects and each of them has 10 gray-level facial images, as shown in Figure 1(b). Each image is resized to 16 × 16 and a 256-dimensional feature vector is obtained. The 400 images are randomly split into 10 pairs of training/test subsets with the equal size of 200, forming ten 40-class classification problems. For each problem, the g values are optimized on the training subset and the features are sorted accordingly. The feature selection result averaged over the 10 classification problems is reported in Figure 3(a). By using only the top 50 selected features, the

Feature Subset Selection for Multi-class SVM Based Image Classification

151

ORL Face data set 0.7

16 Pearson correlation coefficient Kolmogorov−Smirnov Test Proposed criterion Fisher score

0.6

Test error

0.5

Forehead 0.9

14 12

0.8

Right eye

Left eye

0.7 10

0.4

0.6

8

0.3

0.5

Nose

0.4

6 Mouth

0.2

0.3

4 0.1

0.2 Chin

2 0 0

50

100 150 200 The number of selected features

250

(a) Feature selection result

300

2

4

6

8

10

12

0.1 14

16

(b) Optimized value of g = 1/2σ 2

Fig. 3. Results for the ORL face data set

test error can be as low as 11.10 ± 2.87%, and the test error with all 256 features is 7.95 ± 2.36%. Again, the proposed criterion achieves the best selection performance, especially at the top 50. The optimized g values are also reshaped back and plotted in Figure 3(a). A shading effect is used. As seen, the blocks with larger g values (whiter) roughly present a human face, where the eyes, nose, mouth, forehead and chin are marked. This result is consistent with our daily experience that we recognize a person often by looking at the eyes while the cheek, shown as the darker areas, is less used. The subjects in this database often have different hairstyles at the forehead part, and this may explain why the forehead part is also shown as a whiter region. Brodatz for Textured image classification. This database includes 112 different textured images, as shown in Figure 1(c). Top 10 images are used in this experiment. For each of them, sixteen 128×128 sub-images are cropped, without overlapping, from the original 512 × 512 image to form a texture class. By doing so, a ten-class textured image database is created, including 160 samples. Again, they are randomly split into 10 pairs of training (40 samples)/test (120 samples) subsets. By utilizing a bank of Gabor filters (6 orientations and 4 scales), the mean, variance, skewness and kurtosis of the values of the Gabor filtered images are extracted, leading to a 96 (6 × 4 × 4) dimensional feature vector [9]. Using the proposed criterion, the 96 g values are optimized and the features are sorted. The test error with different number of selected features is plotted in Figure 4(a). This criterion and the “Kolmogorov-Smirnov test” show comparable performance, with the former slightly better at selecting the top 10 features. Using this criterion, the lowest test error of 1.42 ± 1.36% is achieved at the top 50 selected features, whereas the error with all 96 features is 2.08 ± 1.06%. This indicates that half features are actually useless for discrimination. Again, the optimized g values are shown in Figure 4(b). It is found that the features from the 25-th to 48-th (the features of “variance”) and those from the 80-th to 96-th (the features of “kurtosis”) dominate the features with larger g values, suggesting that these two types of features are more useful for discriminating the textured images.

152

L. Wang Brodatz texture data set 0.9

3.5

Pearson correlation coefficient Kolmogorov−Smirnov Test Proposed criterion Fisher score

0.8 0.7

3 2.5

The g value

Test error

0.6 0.5 0.4

2 1.5

0.3 1

0.2 0.5

0.1 0 0

20

40 60 80 The number of selected features

100

(a) Feature selection result

0 0

20

40 60 Feature number

80

100

(b) Optimized value of g = 1/2σ 2

Fig. 4. Results for the Brodatz data set Table 1. Feature selection performance with regularization (ORL data set) Average test error for different number of selected features (In %) λ value

1

2

5

10

20

50

68.85±2.73

51.80±8.16

33.75±3.28

24.65±3.94

16.80±3.28

11.10±2.87

0.1

70.10±4.01

62.85±10.15

40.70±4.74

30.15±2.24

19.60±3.85

11.75±2.20

0.25

70.10±4.01

62.05±11.10

40.30±5.06

28.80±1.46

19.15±3.42

12.20±2.38

0.5

73.50±5.25

57.10±12.12

36.05±5.23

29.15±4.78

20.20±3.12

12.55±2.75

0

0.75

77.40±6.82

53.85±9.92

35.05±4.73

28.80±4.19

20.17±2.72

12.95±3.07

0.9

74.45±6.09

55.40±11.29

35.60±4.40

28.10±3.27

21.25±2.44

12.60±3.18

0.99

79.60±7.40

64.15±8.67

37.15±6.03

28.25±3.43

22.10±2.73

12.85±3.35

Finally, the help of regularization is investigated for the proposed criterion on the ORL and Brodatz data sets. For the ORL, there are only 5 training samples in each class, whereas the number of features is 256. For the Brodatz, the case is 4 versus 96. Both have the small-sized training sets with respect to the number of features. In this experiment, θ0 is obtained by solving Eq. (15). By gradually increasing the regularization parameter, λ, from 0 to 0.99, the g values are optimized and the features are selected. The average test error of a multi-class SVM classifier with the selected features is reported in Table 1. It is found that the lowest error (in bold) is consistently obtained with λ = 0, that is, no regularization is imposed. Simply applying the regularization leads to inferior selection performance. This result suggests that the proposed criterion is quite robust to the small sample problem in this classification task. Similar case is observed on the Brodatz data set from Table 2. These results are, more or less, surprising and further analysis leads to the following experiment. By looking into the ORL facial image classification problem, it is found that the total number of training samples is still comparable to the number of features, that is, 5 × 40 = 200 vs. 256, because there are 40 classes. As defined in the proposed criterion, the radius 2 R2 is estimated by using all of the 200 training samples, although the margin γij is estimated from the 10 training samples from each pair of classes. Hence, such a training set is not a typical small sample for estimating R2 . Believing that regularization is necessary in the presence of a small sample problem, this paper

Feature Subset Selection for Multi-class SVM Based Image Classification

153

Table 2. Feature selection performance with regularization (Brodatz data set) Average test error for different number of selected features (In %) λ value 0

1

2

5

10

15

20

44.42±6.11

21.58±5.01

5.92±4.91

3.33±3.24

2.00±2.19

2.17±1.12

0.1

50.33±6.70

25.33±4.32

7.75±5.31

4.25±2.95

3.17±2.32

2.67±2.07

0.25

50.42±6.67

25.00±4.84

9.42±3.73

3.58±1.80

3.25±2.10

2.67±2.22

0.5

50.08±7.27

23.92±3.49

9.67±3.38

4.42±2.42

2.83±2.01

2.75±1.84

0.75

48.42±7.76

25.08±3.98

9.25±4.09

5.17±2.99

2.92±1.72

3.25±2.10

0.9

47.58±8.04

23.58±3.89

9.42±4.94

5.58±2.99

3.08±2.15

2.50±1.80

0.99

47.00±7.64

24.08±4.87

8.92±3.33

5.92±2.73

3.08±2.19

2.92±1.85

Result for 30 classes

Result for 20 classes

0.7 λ=0 λ=0.1 λ=0.25 λ=0.5 λ=0.75 λ=0.9 λ=0.99

Test error

0.5 0.4

λ=0 λ=0.1 λ=0.25 λ=0.5 λ=0.75 λ=0.9 λ=0.99

0.5

0.4

Test error

0.6

0.3

0.3

0.2

0.2 0.1

0.1 0 0

10

20 30 40 The number of selected features

0 0

50

(a) Class number = 30

10

20 30 The number of selected features

40

50

(b) Class number = 20

Result for 10 classes

Result for 8 classes 0.45 λ=0 λ=0.1 λ=0.25 λ=0.5 λ=0.75 λ=0.9 λ=0.99

Test error

0.4

λ=0 λ=0.1 λ=0.25 λ=0.5 λ=0.75 λ=0.9 λ=0.99

0.35 0.3

Test error

0.5

0.4

0.3

0.2

0.25 0.2 0.15 0.1

0.1 0.05 0 0

10

20 30 40 The number of selected features

(a) Class number = 10

50

0 0

10

20 30 40 The number of selected features

50

(b) Class number = 8

Fig. 5. Results for the ORL data set with less number of classes

further conducts the following experiment: maintain 5 training samples in each class and gradually reduce the number of classes to 30, 20, 10 and 8. After that, following the previous procedures, perform feature selection with the proposed criterion for these four new classification problems, respectively. The result is plotted in Figure 5 and it shows what is expected. The selection performance of the criterion without regularization (labelled by λ = 0 and shown as a thick curve) degrades with the decreasing number of classes. When there are only 8 classes (or, equally, 5 × 8 = 40 training samples), the necessity and benefit of regularization can be clearly seen. Summarily, two points can be drawn from Table 1,2 and Figure 5: (1) thanks to the way of defining R2 , regularization is not a concern for the proposed criterion when the number of training samples in

154

L. Wang

each class is small but the number of classes is large. This case is quite reasonable for multi-class classification; (2) when the total number of training samples is really small, employing the regularized version of the proposed criterion can achieve better feature selection performance.

5

Conclusion

A novel feature subset selection approach has been proposed for multi-class SVM based image classification. It handles the selection with hundreds of features, very suitable for image classification where high-dimensional feature vectors are often used. This approach produces overall better performance than the compared selection criteria and shows robustness to the small sample case in multi-class image classification. These results preliminarily verify its effectiveness. More theoretical and experimental study will be conducted in the future work.

References 1. Kim, K.I., Jung, K., Park, S.H., Kim, H.J.: Support vector machines for texture classification. IEEE Transactions on Pattern Analysis and Machine Intelligence 24(11), 1542–1550 (2002) 2. Chapelle, O., Haffner, P., Vapnik, V.: Support vector machines for histogram-based image classification. IEEE Transactions on Neural Networks 10(5), 1055–1064 (1999) 3. Luo, T., Kramer, K., Goldgof, D., Hall, L., Samson, S., Remsen, A., Hopkins, T.: Recognizing plankton images from the shadow image particle profiling evaluation recorder. IEEE Transactions on Systems, Man and Cybernetics, Part B 34(4), 1753– 1762 (2004) 4. Chapelle, O., Vapnik, V., Bousquet, O., Mukherjee, S.: Choosing multiple parameters for Support Vector Machines. Machine Learning 46(1-3), 131–159 (2002) 5. Wang, L., Chan, K.L.: Learning Kernel Parameters by using Class Separability Measure. In: Presented in the sixth kernel machines workshop, In conjuction with Neural Information Processing Systems (2002) 6. Darcy, Y., Guermeur, Y.: Radius-margin Bound on the Leave-one-out Error of Multi-class SVMs. Technical Report, No. 5780, INRIA (2005) 7. Chang, C.C., Lin, C.J.: LIBSVM: a library for support vector machines (2001), Software available at http://www.csie.ntu.edu.tw/∼ cjlin/libsvm 8. Sch¨ olkopf, B., Smola, A.: Learning with Kernels Support Vector Machines, Regularization, Optimization, and Beyond. MIT Press, Cambridge, MA (2002) 9. Manjunath, B.S., Ma, W.Y.: Texture features for browsing and retrieval of image data. IEEE Transactions on Pattern Analysis and Machine Intelligence 16(8) (1996)

Evaluating Multi-Class Multiple-Instance Learning for Image Categorization Xinyu Xu and Baoxin Li Department of Computer Science and Engineering, Arizona State University [email protected], [email protected]

Abstract. Automatic image categorization is a challenging computer vision problem, to which Multiple-instance Learning (MIL) has emerged as a promising approach. Typical current MIL schemes rely on binary one-versus-all classification, even for inherently multi-class problems. There are a few drawbacks with binary MIL when applied to a multi-class classification problem. This paper describes Multi-class Multiple-Instance Learning (McMIL) to image categorization that bypasses the necessity of constructing a series of binary classifiers. We analyze McMIL in depth to show why it is advantageous over binary MIL when strong target concept overlaps exist among the classes. We systematically valuate McMIL using two challenging image databases, and compare it with state-of-the-art binary MIL approaches. The McMIL achieves competitive classification accuracy, robustness to labeling noise, and effectiveness in capturing the target concepts using smaller amount of training data. We show that the learned target concepts from McMIL conform to human interpretation of the images. Keywords: Image Categorization, Multi-Class Multiple-Instance Learning.

1 Introduction Multiple-Instance Learning (MIL) is a generalization of supervised learning in which training class labels are associated with sets of patterns, or bags, instead of individual patterns. While every pattern may possess an associated true label, it is assumed that pattern labels are only indirectly assessable through bag labels [1]. MIL has been successfully applied to many applications such as drug activity prediction [8], image indexing for content-based image retrieval [1, 13, 21, 5, 2, 14, 6], object recognition [6], and text classification [1]. In many real-world multi-class classification problems, a common treatment adopted by traditional MIL is “one-versus-all” (OVA) approach, which constructs c binary classifiers, fj(x), j=1,…,c, each trained to distinguish one class from the remaining, and labels a novel input with the class giving the largest fj(x), j=1,…,c. Although OVA delivers good empirical performances, it has potential drawbacks. First, it is somewhat heuristic [15]. The binary classifiers are obtained by training in separate binary classification problems, and thus it is unclear whether their realvalued outputs are on comparable scales [15]. This can be a problem, since situations often arise where several binary classifiers acquire identical decision values, making the class prediction based on them very obscure. Second, binary OVA has been Y. Yagi et al. (Eds.): ACCV 2007, Part II, LNCS 4844, pp. 155–165, 2007. © Springer-Verlag Berlin Heidelberg 2007

156

X. Xu and B. Li

criticized for its inadequacy in dealing with rather asymmetric problems [15]. When the number of classes becomes large, each binary classifier becomes highly unbalanced with many more negative examples than positive examples. In the case of non-separable classes, the class with smaller fraction of instances tends to be ignored, leading to degraded performances [18]. Third, with respect to Fisher consistency, it is unclear if OVA targets the Bayes rule in absence of a dominating class [10]. Recently, a Multi-class Multiple-Instance Learning (McMIL) approach has been proposed which considers multiple classes jointly by simultaneously minimizing a Support Vector Machine (SVM) objective function. In this paper we focus on the systematic evaluation of the McMIL using image categorization as a case study, and we show that why it is advantageous over binary OVA approaches.

2 Related Work There has been abundant work in the literature on image categorization. Due to space limit we only review the MIL-based approaches. Andrew et al. [1] proposed MI-SVM and applied it to small-scale image annotation. [12] proposed Diverse Density MIL framework and [13] used it to classify natural scene images. [20] proposed EM-DD, which views the knowledge of which instance corresponds to the label of the bag as a missing attribute. [2] performed an instance-based feature mapping through a chosen metric distance function, and a sparse SVM was adopted to select the target concepts. [6] proposed MILES for image categorization and object recognition. MILES does not impose the assumption relating instance labels to bag labels, and it does not rely on searching local maximizers of the DD functions, which makes MILES quite efficient. [14], within the CBIR setting, presented MISSL that transforms any MI problem into an input for a graph-based single-instance semi-supervised learning method. Unlike most prior MIL algorithms, MISSL makes use of the unlabeled data. Most of these works belong to the binary MIL approaches, and thus potentially suffer from the drawbacks mentioned in the previous section. One work by [5] is mostly related with ours. They proposed DD-SVM. Their major contribution is that the image label is no longer determined by a single region; rather it is determined by some number of Instance Prototypes (IPs) learned from the DD function. A nonlinear mapping is defined using the IPs to map every training image to a point in a bag feature space. Finally, for each category, an SVM is trained to separate that category from all the others. Note that the multi-class classification here is essentially achieved by OVA. The method described in this paper differs from DDSVM in that a multi-class feature space is built that enables us to train the SVM only once to yield the multi-class label, leading to increased classification accuracy.

3 Methodology of McMIL For completeness of the paper, we describe the McMIL approach in this section. Let us first define the problem and the notations. Let D={(x1, y1),…, (x|D|, y|D|)} be the labeled dataset, where xi, i=1,...,|D| is called a bag in the MIL setting, and yi ∈ {1,…,c} denotes the multi-class label. In the MIL model xi={xi1,…,xi|xi|}, where xij ∈ \ d represents the jth instance in the ith bag. Our goal is to induce a classifier from patterns to multi-class labels, i.e. a multi-class separating function f: \ d Æ y.

Evaluating Multi-Class Multiple-Instance Learning for Image Categorization

157

3.1 Learning Instance Prototypes The first step in the training phase of the McMIL is to learn the IPs for each category according to the DD function [12]. Because the DD function is defined based on the binary framework, McMIL learns a set of IPs specifically for every class: the ith DD function uses all of the examples in the ith class with positive labels and some examples from other classes with negative labels. McMIL applies EM-DD to locate the multiple local peaks by using every instance in every positive bag with uniform weighs as the starting point of the Quasi-Newton search. Then the IPs with larger DD values and are distinct from each other are selected to represent the target concepts [5]. 3.2 Multi-class Feature Mapping Let IPm =

{( I

1 m

, I m2 ,..., I mnm

)} , m ∈ {1, 2,..., c} be the set of instance prototypes learned by

EM-DD for the mth category. Each I mk = ⎡ xmk , smk ⎤ , k=1,2,...,nm denotes the kth IP in the ⎣ ⎦ mth class, and nm is the number of IPs in the mth class. The learned IP consists of two parts: the ideal attribute value x and the weight factor s. To facilitate the multi-class classification, a similarity mapping kernel ψ is defined which maps a bag of instances into a multi-class feature space ] based on a similarity metric function h: \ d ×\ d → \ . The mapping ψ is defined as ψ(xi)= ⎡ h( xi , I11 ),..., h( xi , I1n1 ),..., h( xi , I c1 ),..., h( xi , I cnc ) ⎤ ⎣

T

⎦

(1)

The similarity function h is defined as ⎧ ⎩

h( xi , I mk ) = max j =1,..., x ⎨exp ⎡⎢ − xij − xmk i

⎣

2 smk

⎤ ⎫ , k = 1,..., n , m ∈ {1,..., c} m ⎥⎦ ⎬⎭

(2)

Each feature corresponds to the similarity between a bag and one IP, which is the similarity between I mk and the instance in xi that is closest to I mk . Note that while this is similar to the bag feature space mapping that was proposed by [5], an important difference between these two mappings is that, the mapping in our method simultaneously incorporate the similarity between a bag and each IP of every category into one feature matrix, while in binary MIL [5, 2, 6], separate feature matrices are needed, each of which only considers the similarity between a bag and each IP of one category that is under consideration. For a given training set of bags with multi-class labels D={(x1, y1),…, (x|D|, y|D|)} , applying the above mapping yields the following kernel similarity matrix S= n n 1 1 ⎡h1 ⎤ ⎡h(x1, I1 ),...., h(x1, I1 1 ),...., h(x1, Ic ),...., h(x1, Ic c ) ⎤ ⎢ ⎥ ⎢h ⎥ n1 nc 1 1 ⎢ 2 ⎥ = ⎢h(x2 , I1 ),...., h(x2 , I1 ),...., h(x2 , Ic ),...., h(x2 , Ic ) ⎥ ⎥ ⎢M ⎥ ⎢LLLLLLLLLLLLLLLLLL ⎥ ⎢ ⎥ ⎢ n 1 1 n ⎢⎣h|D| ⎥⎦ ⎢⎣h(x|D| , I1 ),..., h(x|D| , I1 1 ),..., h(x|D| , Ic ),..., h(x|D| , Ic c )⎥⎦

(3)

158

X. Xu and B. Li

3.3 Multi-class SVM Training and Classification Given the kernel similarity matrix S, the multi-class classification is achieved by multi-class SVM [16, 19] which simultaneously allows the computation of a multiclass classifier by considering all the classes at once. The formulation is |D| 1 c min ∑ wmT wm + C ∑ ∑ ξim w,b ,ξ 2 m =1 i =1 m ≠ y (4) T s.t. wy ψ ( xi ) + by ≥ wmTψ ( xi ) + bm + 2 − ξim , i

i

i

ξim ≥ 0, i = 1,...,| D |, m ∈ {1,.., c} y . i

where w is a vector orthogonal to the separating hyper-plane, |b|/||w|| is the perpendicular distance from the hyper-plane to the origin, C is a parameter to control the penalty due to misclassification. Nonnegative slack variables ξi are introduced in each constraint to account for the cost of the errors. In the testing phase, we first project the test image into the feature space ] using the learned IPs. The decision function is then given by (5) arg max m=1,...,c ( wmTψ ( x) + bm ) which is the same with the OVA decision making. Note that in multi-class SVM, there are c decision functions but all are obtained by solving one problem Eq. (4). 3.4 The Learned Target Concept Each instance prototype represents one kind of target concept that distinguishes one class from another. Thus it is important to visualize the target concept. In order to do that, in each class, for each feature that is defined by one positive instance prototype,

People

Portrait

Scene

Structure

Fig. 1. The target concepts of four categories in the SIMPLIcity-II data set. The target concepts are represented by conceptual regions. For “People”, the various shapes of human are returned as target concepts. For “Portrait”, the target concepts are mainly skin and hair with varying colors. For “Scene”, the major target concepts are mountain, sky, sea water and plants. For “Structure”, the target concepts are various kinds of building structures like walls, windows and roofs.

we find the training image that maximizes the corresponding feature, then we go further to locate the region that is closest to the corresponding prototype. The target concepts for a category are defined by the regions that maximize the similarity

Evaluating Multi-Class Multiple-Instance Learning for Image Categorization

159

between images and the instance prototypes. Fig. 1 shows the target concepts of four classes in the SIMPLIcity-II dataset (Sec. 4).

4 The Advantage of Direct Multi-Class MIL The McMIL approach has advantage over binary MIL when strong target concept overlaps exist among the classes. To show the advantage of McMIL, let us suppose we have a dataset that has three categories: People, Portrait and Scene. Typical People IPs include “hair”, “face”, “clothes”, “building” and “water”, and typical Portrait IPs include “hair”, “face” and “clothes”. Fig. 2(a) shows the characteristic of the feature vectors for a People image and a Portrait image. We can see that the People IPs are inclusive of Portrait IPs. Now let us simplify the problem by assuming that the target concept of People are summarized by “face” and “non-face”, and the target concept of Portrait is summarized by the “face” IP.

(a)

(b)

(c)

Fig. 2. (a) The characteristic of the feature vectors for a People image and a Portrait image. Top: the features for a People image defined by Portrait IPs. Bottom: the features for a Portrait image defined by People IPs. (b) and (c) illustrate the advantage of McMIL over OVA MIL. (b) The feature space of the People and the Portrait classifier in the OVA MIL approach. (c) The multi-class feature space in the proposed McMIL approach.

Let us first consider the OVA MIL approach. OVA MIL will construct 3 binary classifiers. Fig. 2(b) shows the feature space of the People and the Portrait classifier in the OVA MIL approach. In the People classifier (top part of Fig. 2(b)), the features are defined in terms of People IPs: face and non-face. It is more likely that the three People training images will fall into circle A, because the similarity between a People bag and People IPs (face or non-face) would be large. And we can expect that the three Portrait training images will more likely fall into circle B since the similarity between a Portrait bag and the face IP is large but the similarity between a Portrait bag and a non-face IP would be small. In the same sense, the three Scene images will fall into circle C. Because the bags are clearly separable in the People feature space, we would expect that People binary classier will do a good job in classifying a new test point. Now let us turn to Portrait binary classifier whose feature is only defined by face IP. As shown in the bottom part of Fig. 2(b), the People and Portrait bags will become linearly non-separable in the Portrait feature space in that the feature values defined by Portrait face for the People bags and the Portrait bags are both likely to be

160

X. Xu and B. Li

large, and so People bags and Portrait bags will overlap with each other. As a result, Portrait classier will be more likely to make prediction errors in classifying a new test input. Now let us predict the label for a new test image whose true label is People. We feed it into each of the three binary classifier and we get the following decision values: People classifier 0.3218, Portrait classifier 0.3825, and Scene classifier 0.7726. Note that to classify this new image into the right category, the Portrait classifier should give us a negative decision value. But because of Portrait classifier’s poor job in classification, the new image will be associated with a wrong label Portrait since 0.3825>0.3218>-0.7726. Now let us turn to multi-class SVM classification whose feature space is illustrated in Fig. 2(c). Note that the multi-class feature space is defined by the IPs of all the classes, so McMIL has three features defined by People face (face 1), Portrait face(face 2) and non-face. In this multi-class feature space, the probability that the multi-class SVM makes a prediction error is greatly reduced because the dimension of the feature space is enlarged and thus any linearly non-separable points are likely to become separable in a higher dimensional feature space. This is essentially similar to applying a kernel to make linearly non-separable feature space nonlinearly separable. Actually the multi-class feature matrix S is a kernel similarity matrix, the kernel mapping is defined by Eq. (1).

5 Evaluation of McMIL 5.1 Experiment Design Using image categorization as a case study, we evaluate the performance of McMIL on two challenging image data sets: COREL database [2, 5, 6], and SIMPLIcity-II. The images in SIMPLIcity-II are selected from the SIMPLIcity image library [11, 17]. The COREL image database consists of 10 categories, with 100 images per category. The SIMPLIcity-II data set consists of 5 categories, with 100 images per category. Images in both data sets are in JPEG format of size 384 × 256 or 256 × 384. Using the COREL dataset, we compared the classification accuracy of McMIL with that of a few state-of-the-art methods including MILES [6], 1-norm SVM [2], DDSVM [5], k-means-SVM [7], Hist-SVM [4] and MI-SVM [1]. Using COREL dataset, we also comprehensively evaluate McMIL against DD-SVM. Using SIMPLIcity-II, we compared the robustness of McMIL with that of DD-SVM in classifying images where strong target concept overlaps exist among the classes. To facilitate fair comparison with other methods, we use the same feature matrix for all the methods. In all of the experiments, images within each category were randomly partitioned in half to form a training set and a test set. We repeated each experiment for 5 random splits, and reported the average of the results obtained over 5 random test sets, as done in the compared approaches in the literature. 5.2 Results Classification accuracy on COREL data set. Table 1 illustrates the classification accuracy for each class obtained by McMIL and that of DD-SVM and MILES. For McMIL, we presented the results on its two variants: one is McMIL90, which, instead

Evaluating Multi-Class Multiple-Instance Learning for Image Categorization

161

of using all the examples in all the other classes with negative labels, uses 10 images per negative class to learn the IPs for each class. Another variant is McMIL450 which uses all the examples in all the other classes with negative labels to learn the IPs for each class. We are surprised to know that McMIL90 performs much better than DDSVM using 450 negative examples and MILES. This further implies that McMIL is able to achieve higher classification accuracy by using much smaller amount of training negative examples. The left table in Table 2 illustrates the average classification accuracies of McMIL in comparison with other approaches. McMIL90 performs much better than Hist-SVM, k-means-SVM, MI-SVM, 1-norm SVM. McMIL90 performs slightly better than DD-SVM and MILES. Moreover, McMIL450 performs much better than Hist-SVM, k-means-SVM, MI-SVM and 1-norm SVM with a large margin, and it performs comparably well with MILES. McMIL450 outperforms DD-SVM, though the difference is not statistically significant. Table 1. Classification accuracies (in %) for each class obtained by McMIL90, McMIL450, DD-SVM and MILES on the COREL dataset Class

McMIL90

McMIL450

0 1 2 3 4 5 6 7 8 9

71.6 ± 4.7 68.0 ± 6.8 76.8 ± 2.7 90.8 ± 2.7 100.0 ± 0.0 81.6 ± 6.1 94.4 ± 1.9 94.4 ± 3.4 67.2 ± 4.7 88.8 ± 1.6

70.4 ± 3.8 69.6 ± 4.19 74.8 ± 2.66 91.6 ± 3.37 100.0 ± 0.0 79.6 ± 5.32 95.6 ± 2.6 96.4 ± 2.6 62.8 ± 8.37 88.8 ± 4.04

DDSVM 67.7 68.4 74.3 90.3 99.7 76.0 88.3 93.4 70.3 87.0

MILES 68.8 66.0 75.7 90.3 99.7 77.7 96.4 95.0 71.0 85.4

Table 2. The classification accuracy (in %) obtained by McMIL and other methods (with the corresponding 95% confidence intervals) using the COREL (left table) and SIMPLIcity-II (right table) dataset

162

X. Xu and B. Li

Sensitivity to labeling noise. In multi-class classification, the labeling noise is defined as the probability that an image is misclassified. It is important to show the sensitivity of McMIL to labeling noise because in many practical applications, it is usually impossible to get a “clean” data set and the labeling process is often subjective [6]. In this experiment, we used 500 images from Class 0 to Class 4, with 100 images per class. The training and test sets have equal size. In the case of multiclass classification, training set with different level of labeling noise is generated as follows: for the i-th class, i=0,…,4 we randomly pick d% training images and modify their labels to j, j ≠ i, and the probability of j taking any value other than i is equal; for the i-th class, i=0,…,4, we also randomly pick d% images from among all the other classes (200 images in this case) and modify their labels to i. We compared McMIL with DD-SVM for d=0, 2, 4, …, 20 and report the average classification accuracy over 5 random test sets, as shown in fig. 3 (a). We use 50 positive images and 200 negative images per class to learn the IPs for both McMIL and DD-SVM. We observe that for d=0, 2, 4, 6, 8, the classification accuracies of McMIL are 2% higher than DD-SVM. As we increase the percent of training images with label noise from 10 to 20, the accuracy of McMIL remains roughly the same, but the accuracy of DD-SVM drops very quickly. This indicates that our method is less sensitive to labeling noise than DD-SVM.

(a)

(c)

(b)

(d)

Fig. 3. Compare McMIL with DD-SVM in terms of sensitivity to labeling noise (a), sensitivity to number of negative images (b), sensitivity to number of categories (c), and sensitivity to number of sample images (d)

Sensitivity to number of training negative images. We used 1000 images from Class 0 to Class 9 (training and test sets are of equal size) to analyze the performance of McMIL as the number of negative examples used for learning IPs for each class

Evaluating Multi-Class Multiple-Instance Learning for Image Categorization

163

varies from 90 to 450. That is, in the ith (i = 1,…,5) experiment, we randomly choose 10 × i negative examples from the 50 training images of each of the other classes, and run the experiment for 5 random splits. In all these experiments, the number of positive examples remains 50 in learning the IPs for the jth class, j=1,…, 10. As indicated by fig. 3(b), even when the number of negative images drops to 90, McMIL still performs very well. This is an advantage over binary MIL since the performance of most binary MIL will degrade significantly as the amount of negative training data decreases. Why the performance of direct multi-class MIL does not degrade with the decreasing of the amount of negative training examples? We attempt to answer this question as follows: for the OVA MIL approaches, a series of one-versus-all binary classifiers are obtained, each trained to separate one from all the rest. This requires the IPs learned for each class not only to represent the target concepts of the underlying class but also distinguish the underlying class from the rest. This is because, in the definition of DD [12], the target concepts are those intersection points of all the positive bags without intersecting the negative bags, so the intersection points of all the positive bags which also intersect with negative bags should be excluded by learning from a large number of negative examples. However, in the direct multi-class MIL, the requirement for the IPs to distinguish one class from another is lessened, because the “distinguishing” power of the classifier can be offset by learning in the multi-class feature space, due to the nice property of multi-class feature matrix analyzed in Sec. 4. Therefore in order to accurately capture the target concept of the underlying class (i.e. the instance prototypes), the number of training negative examples used for McMIL can be much smaller than that used in DD-SVM. Sensitivity to number of categories. In this experiment, a total of 10 data sets are used, the number of categories in a data set varies from 10 to 19. A data set with i categories contains 100 × i images from Class 0 to Class i-1. In learning IPs for each class, we use the images in the current class with positive labels and all the images in the other classes with negative labels. As shown by fig. 3(c). McMIL outperforms DD-SVM consistently by a large margin for each data set. Sensitivity to number of training images. We compared McMIL with DD-SVM as the size of training set varies from 100 to 500. 1,000 images from Class 0 to Class 9 are used, with the size of the training sets being 100, 200, 300, 400, and 500 (the number of images from each category is size of the training set/10). As indicated in fig. 3(d), McMIL performs consistently better than DD-SVM as the number of training images increases from 100 to 500. Classification accuracy on SIMPLIcity-II. We have built SIMPLIcity-II that is specifically designed to test the robustness of our method in classifying images where strong target concept overlaps exist among the classes. While being small, the SIMPLIcity-II dataset is in fact a very challenging one for various reasons. First, the images in a semantic category are not “pure”. Therefore target concept overlaps occur more frequently than in the COREL dataset. Second, the data contain noise in that a negative bag may contain positive instances as well. Third, the images are very diverse in the sense that they have various background, colors and combinations of semantic concepts, and images are photographed at a wide-range or close up shots.

164

X. Xu and B. Li

The right table of Table 2 shows the average classification accuracy of McMIL and DD-SVM on the SIMPLIcity-II dataset over 5 random test sets (training and test sets are of equal size). Despite the aforementioned difficulties, McMIL has achieved reasonably good results on this dataset, demonstrating that this is indeed a promising approach. We also found that DD-SVM made much higher classification error than McMIL in classifying People and Portrait images: DD-SVM misclassifies 9.2% of People images into Portrait and 8.6% Portrait images into People; while McMIL only produces 6.4% and 6.8% for the two classes respectively.

6 Concluding Remarks In this paper, we described a multi-class multiple instance learning approach. We have carried out systematic evaluation to demonstrate the benefits of direct McMIL. Our empirical studies show that, in addition to overcoming the drawbacks of OVA MIL approaches, McMIL is able to achieve higher classification accuracy than most of the existing OVA MIL approaches even using only a small amount of training data, and it is less sensitive to labeling noise. These two benefits are significant in practical applications. The ability of McMIL in classifying images with strong target concept overlap is found to be superior to DD-SVM. One drawback of the McMIL is that as the number of classes becomes large, it becomes computationally inefficient since the gradient search for the IPs takes quite a lot of time. One solution to increase the efficiency of McMIL is to use only a small amount of negative examples; another is to bypass searching for the IPs by combining MILES [6] with McMIL. Our future work will explore these possibilities.

References 1. Andrews, S., Tsochantaridis, I., Hofmann, T.: Support vector machines for multipleinstance learning. NIPS 15, 561–568 (1998) 2. Bi, J., Chen, Y., Wang, J.Z.: A sparse support vector machine approach to region-based image categorization. CVPR (1), 1121–1128 (2005) 3. Chang, C.-C., Lin, C.-J.: LIBSVM: a library for support vector machines. Software available at http://www.csie.ntu.edu.tw/ cjlin/libsvm2001 4. Chapelle, O., Haffner, P., Vapnik, V.N.: Support vector machines for histogram-based image classification. IEEE Trans. Neural Networks 10(5), 1055–1064 (1999) 5. Chen, Y., Wang, J.Z.: Image categorization by learning and reasoning with regions. Journal of Machine Learning Research 5, 913–939 (2004) 6. Chen, Y., Bi, J., Wang, J.Z.: MILES: Multiple-Instance Learning via Embedded Instance Selection. IEEE Trans. on PAMI. 28(12), 1931–1947 (2006) 7. Csurka, G., Bray, C., Dance, C., Fan, L.: Visual Categorization with Bags of Keypoints. In: ECCV 2004 Workshop on Statistical Learning in Computer Vision, pp. 59–74 (2004) 8. Dietterich, T.G., Lathrop, R.H., Lozano-Pérez, T.: Solving the multiple-instance problem with axis-parallel rectangles. Artificial Intelligence 89, 31–71 (1997) 9. Hsu, C.-W., Lin, C.-J.: A comparison of methods for multi-class support vector machines. IEEE Transactions on Neural Networks 13(2), 415–425 (2002)

Evaluating Multi-Class Multiple-Instance Learning for Image Categorization

165

10. Lee, Y., Lin, Y., Wahba, G.: Multicategory support vector machines: theory, and application to the classification of microarray data and satellite radiance data. Journal of the American Statistical Association 99, 67–81 (2004) 11. Li, J., Wang, J.Z.: Automatic linguistic indexing of pictures by a statistical modeling approach. IEEE Trans. on PAMI 25(9), 1075–1088 (2003) 12. Maron, O., Lozano-Pérez, T.: A framework for multiple-instance learning. NIPS 10, 570– 576 (1998) 13. Maron, O., Ratan, A.L.: Multiple-instance learning for natural scene classification. ICML, 341–349 (1998) 14. Rahmani, R., Goldman, S.A.: MISSL: multiple-instance semi-supervised learning. ICML, 705–712 (2006) 15. Schölkopf, B., Smola, A.J.: Learning with Kernels-Support Vector Machines, Regularization, Optimization and Beyond, pp. 211–214. MIT Press, Cambridge, MA (2002) 16. Vapnik, V.N.: Statistical Learning Theory. Wiley, New York (1998) 17. Wang, J.Z., Li, J., Wiederhold, G.: SIMPLIcity: Semantics-sensitive Integrated Matching for Picture Libraries. IEEE Trans. on PAMI 23(9), 947–963 (2001) 18. Wang, L., Shen, X., Zheng, Y.F.: On L-1 norm multi-class Support Vector Machines. In: Proc. 5th Intl. Conf. on Machine Learning and Applications (2006) 19. Weston, J., Watkins, C.: Multi-class support vector machines. In: Verleysen, M. (ed.) Proc. ESANN 1999, D. Facto Press, Brussels (1999) 20. Zhang, Q., Goldman, S.A.: EM-DD: An improved multiple-instance learning technique. NIPS 14, 1073–1080 (2002) 21. Zhang, Q., Goldman, S.A., Yu, W., Fritts, J.E.: Content-based image retrieval using multiple-instance learning. In: Proc. ICML, pp. 682–689 (2002)

TransforMesh : A Topology-Adaptive Mesh-Based Approach to Surface Evolution Andrei Zaharescu, Edmond Boyer, and Radu Horaud INRIA Rhone-Alpes, 655 ave de l’Europe, Montbonnot, 38330, France [email protected]

Abstract. Most of the algorithms dealing with image based 3-D reconstruction involve the evolution of a surface based on a minimization criterion. The mesh parametrization, while allowing for an accurate surface representation, suffers from the inherent problems of not being able to reliably deal with selfintersections and topology changes. As a consequence, an important number of methods choose implicit representations of surfaces, e.g. level set methods, that naturally handle topology changes and intersections. Nevertheless, these methods rely on space discretizations, which introduce an unwanted precision-complexity trade-off. In this paper we explore a new mesh-based solution that robustly handles topology changes and removes self intersections, therefore overcoming the traditional limitations of this type of approaches. To demonstrate its efficiency, we present results on 3-D surface reconstruction from multiple images and compare them with state-of-the art results.

1 Introduction A vast number of problems in the area of image-based 3-D modeling are casted as energy minimization problems where 3-D shapes are optimized such that they best explain image information. More specifically, when interested in performing 3-D reconstruction from multiple images, one typically attempts to recover 3-D shapes by evolving a surface with respect to various criteria, such as photometric or geometric consistencies. Meshes, either triangular, polygonal or even tetrahedral, are one of the most used forms of shape representation. Traditionally, an initial mesh is obtained using some well known method, e.g. bounding boxes or visual hulls, and then deformed over time such that it minimizes an energy, based typically on some form of image similarity measure. There exist two main schools of thought on how to cast the above mentioned problem. Lagrangian methods propose an intuitive approach where the surface representation, i.e. the mesh, is directly deformed over time. Meshes present numerous advantages, among which adaptive resolution and compact representation, but raise two major issues of concern when evolved over time: self-intersections and topology changes. It is critical to deal with both issues when evolving a mesh over time. In order to answer to this, McInerney and Terzopoulos [1] proposed topology adaptive deformable curves and meshes, called T-snakes and T-surfaces. However, in solving the intersection problem, the authors use a spatial grid, thus imposing a fixed spatial resolution. In addition, only offsetting motions, i.e. inflating or deflating, are allowed. Lauchaud et al. [2] proposed a heuristic method where merges and splits are perform in near boundary cases: Y. Yagi et al. (Eds.): ACCV 2007, Part II, LNCS 4844, pp. 166–175, 2007. c Springer-Verlag Berlin Heidelberg 2007 

TransforMesh : A Topology-Adaptive Mesh-Based Approach to Surface Evolution

167

when two surface boundaries are closer than a threshold and facing each other, an artificial merge is introduced; a similar procedure is applied for a split, when the two surface boundaries are back to back. Self-intersections are avoided in practice by imposing a fixed edge size. Eulerian methods formulate the problem as time variation over sampled space, most typically fixed grids. In such a formulation, the mesh, also called the interface, is implicitly represented. One of the most successful methods, the level set method [3] [4], represents the interface as the zero set of a function. Such an embedding within an implicit function allows one to automatically handle mesh topology changes, i.e. merges, splits. In addition, such methods allow for an easy computation of geometric properties. Nevertheless, this method does not come without its drawbacks. The amount of space required to store the implicit representation is much larger than when storing a mesh parametrization. In addition, at each step of the evolution, the mesh has be recovered from the implicit function. This operation is limited by the grid resolution. Consequently, a mesh with variable resolution cannot be properly preserved by the embedding function. In light of this, we propose a mesh-based lagrangian approach. To solve for the selfintersections and topology changes issues, we adopt the algorithm proposed by Jung et al. [5] in the context of mesh-offseting and extend it to more general situations, as faced with when modeling from multiple images. We have successfully applied our approach to surface reconstruction using multiple cameras. The 3-D reconstruction literature is vast. Recently, Furukawa et al. [6] provide some of the most impressive results. They use however a mesh based solution [2] mentioned above that imposes equal face sizes. Pons et al. [7] provide an elegant level-set based implementation. Hernandez et al. [8] provide an in-between solution: they choose an explicit mesh representation, however they immerse the mesh within a vector field with forces proportional to the image consistencies at given places within a grid. Self-intersections and topology changes are not handled. Other approaches are also surveyed in the recent overview on multi-view stereo reconstruction algorithms [9]. Our contribution with respect to these methods is to provide a purely mesh based solution that does not constrain meshes and allows faces of all sizes as well as topology changes, with the goal of increasing precision without sacrificing complexity when dealing with surface evolution problems. The rest of the paper is organized as follows. Section 2 introduces the algorithm that handles self-intersection and topology changes. In section 3, we present its application to the 3-D surface reconstruction problem. Section 4 shows results on well known data sets and makes comparisons with state-of-the-art approaches, before concluding in section 5.

2 TransforMesh - A Topology-Adaptive Self-intersection Removal Mesh Solution As stated earlier, the main limitations that prevent many applications from using meshes are self-intersections and topology changes, which frequently occur when evolving meshes. In this paper, we show that such limitations can be overcome using a very intuitive geometrically-driven solution. In essence, the approach preserves the mesh consistency by detecting self-intersections and considering the subset of the original mesh surface that is outside with respect to the mesh orientation. A toy example is il-

168

A. Zaharescu, E. Boyer, and R. Horaud

lustrated in Figure 1. Our method is inspired by the work of [5] proposed in the context of mesh-offsetting or mesh expansions. We extend it to the general situation of identifying a valid mesh, i.e. a manifold, from a self-intersecting one with possible topological issues. The currently proposed algorithm has the great advantage of dealing with topological changes naturally, much in the same fashion as the Level-Set based solutions, casting it as a viable solution to surface evolutions with meshes. The only requirement is that the input mesh is the result of the deformation of an oriented and valid mesh. The following sections detail the sequential steps of the algorithm.

(a) Input

(b) Input

(c) Intersections

(d) Output

(e) Output

Fig. 1. A toy example of TransforMesh at work

2.1 Pre-treatment Most mesh-related algorithms suffer from numerical problems due to degenerate triangles. Such degenerate triangles are mainly triangles having area close to zero. In order to remove those triangles, two operations are performed: edge collapse and edge flip. 2.2 Self-intersections The first step of the algorithm consists of identifying self-intersections, i.e. edges along which triangles of the mesh intersect. In practice, one would have to perform n2 /2 checks to see if two triangles intersect, which can become quite expensive when the number of facets is large. In order to decrease the computational time, we use a bounding box test to determine which bounding boxes (of triangles) intersect, and only for those perform a triangle intersection test. We use the fast box intersection method implemented in [10] and described in [11]. The complexity of the method is O(n log d (n)+k) for the running time and O(n) for the space occupied, where n is the number of triangles, d the dimension (3 in the 3-D case), and k the output complexity, i.e., the number of pairwise intersections of the triangles. 2.3 Valid Region Growing The second step of the algorithm consists in identifying valid triangles in the mesh. To this purpose, a valid region growing approach is used to propagate validity labels on triangles that composed the outside of the mesh. Initialization. Initially, all the triangles are marked as non-visited. This corresponds to Figure 2(a). Three queues are maintained: one named V, of valid triangles, i.e. triangles outside and without intersections; one named P of partially valid triangles, i.e. only part

TransforMesh : A Topology-Adaptive Mesh-Based Approach to Surface Evolution

(a) Init

(b) Seed-triangle (c) Valid Queue (d) Partial Queue

169

(e) End

Fig. 2. Valid Region Growing (2-D simplified view). The selected elements are in bold.

of the triangle is outside, and finally one named G, where all the valid triangles will be stored until stitched together into a new mesh (in section 2.4). All that follows is performed within an outer loop, while there still exists a valid seed triangle. Seed-triangle Finding. A seed-triangle is defined as a non-visited triangle without intersections whose normal does not intersect any other triangle of the same connected component. This corresponds to Figure 2(b). In other words, a seed-triangle is a triangle that is guaranteed to be on the exterior. This triangle is crucial, since we will start our valid region growing from such a triangle. If found, the triangle will be added to V and marked as valid; otherwise, the algorithm will jump to the next stage (section 2.4). The next two-steps are performed within a inner loop until both V and P are empty. Valid Queue Processing. While V is not empty, pop a triangle t from the queue, add it to G and for each neighbouring triangle N (t) perform the following: if N (t) is nonvisited and has no intersections, then add it to V; if N (t) is non-visited and has intersections, then add it to P together with the entrance segment and direction, corresponding in this case to the oriented half-edge. (see Figure 2(c)). Partially-Valid Queue Processing. While P is not empty, pop a triangle t from the queue, together with the entrance half-edge ft . Also, we have previously calculated all the intersection segments between this triangle and all the other triangles. Let St = {sti } represent all the intersection segments between triangle t and the other triangles. In addition, let Ht = {htj |where j = 1..3} represent the triangle half-edges. A constrained 2-D triangulation is being performed in the triangle plane, using [12], to ensure that all segments in both St and Ht appear in the new mesh structure and that propagation can be achieved in a consistent way. A fill-like traversal is performed from the entrance half-edge to adjacent triangles, stopping on constraint edges, as depicted in Figure 3. A crucial aspect to ensure a natural handling of topological changes is on choosing the correct side of continuation of the "fill" like region growing when crossing a partially valid triangle. The correct orientation is chosen such that, if the original

170

A. Zaharescu, E. Boyer, and R. Horaud Intersection Segments

TD3

Entrance exit

Entrance exit

TD2 TD1

Entrance edge

(a) Triangle Intersections

(b) Partial Triangle Traversal

Fig. 3. Partial Triangle Traversal

normals are maintained, the two newly formed sub-triangles would preserve the watertightness constraint of a manifold. This condition can also be casted as following: the normals of the two sub-triangles should be opposing each other when the two subtriangles are "folded" on the common edge. A visual representation of the two cases is shown in Figure 4.

A

D

Incoming

Outgoing

A

C

Incoming

O

O Outgoing

C

B

D

B

Fig. 4. Partial Triangle Crossing Cases (side view)

2.4 Triangle Stitching The region growing algorithm described previously will iterate until all the triangles have been selected. In the chosen demo example, this corresponds to Figure 2(e). At this stage, what remains to do is to stitch together the 3-D triangle soup (G queue) in order to obtain a valid mesh which is manifold. We adopt a method similar in spirit to [13,14]. In most cases this is a straight forward operation, reduced to identifying the common vertices and edges, followed by stitching. However, there are three special cases, in which performing a simple stitching will violate the mesh constraints and produce locally non-manifold structures. The special cases, shown in Figure 5, arise from performing stitching in places where the original structure should have been maintained. We adopt the naming convention from [13], calling them the singular vertex case, the

TransforMesh : A Topology-Adaptive Mesh-Based Approach to Surface Evolution

171

singular edge case and the singular face case. All cases are easy to identify only from local operations and are identified after all the stitching has been performed. In the singular vertex case, in order to detect whether vertex v is singular, we adopt the following algorithm: mark all the facets incident to the vertex v as non-visited. Then start from a facet of v, mark it visited and do the same with its non-visited neighbours that are also incident to v (neighbour as chosen based on half-edges). The process is repeated until all the neighbouring facets are processed. If by doing so we exhausted all the neighbouring facets, vertex v is non singular, otherwise it is singular, so a copy of it is created and added to all the remaining non-visited facets. In order to detect a singular edges e, all we have to do is count the number of triangles that share that edge. If it is bigger than 2, we have a singular edge case and two additional vertices and a new edge will be added to account for it.

(a) Singular Vertex

(b) Singular Edge

(c) Singular Face

Fig. 5. Special case encountered while stitching a triangle soup

(a) Original Mesh.

(b) Resulting mesh.

Fig. 6. An example of how a singular vertex occurs in a typical self-intersection removal situation, due to an inverted triangle (marked in red)

(a) Join Case

(b) Split Case

(c) Inside Carving Case

Fig. 7. Different topological changes examples (2-D simplified view). The outline of the final surface obtained after self-intersection removal is outlined in bold blue.

172

A. Zaharescu, E. Boyer, and R. Horaud

In practice, only the singular vertex case appears in real triangular mesh test cases. Such an example has been created to illustrate such a scenario and it is shown in Figure 6. 2.5 Topological Changes The partial-triangle crossing technique described earlier ensures a natural handling of the topological changes (splits and joins) that plagued the mesh approaches until now. Representative cases are illustrated in Figure 7.

3 Using TransforMesh to Perform Mesh Evolutions Our original motivation in developing a mesh self-intersection removal algorithm was to perform mesh evolutions, in particular when recovering 3-D shapes from multiple calibrated images. As stated earlier, few efforts have been put in mesh-based solutions for such 3-D surface reconstruction problem, mostly due to the topological issues raised by mesh evolutions. However, meshes allow one to focus on the region of interest in space, namely the shape’s surface and, as a result, lower the complexities and lead to better precisions with respect to volumetric approaches. In this section we present the application of TransforMesh to the surface reconstruction problem. Often such a problem is casted as an energy minimization over a surface. We decided to start from exact visual hull reconstructions, obtained using [15] and further improve the mesh using photometric constraints by means of an energy functional described in [7]. The photometric constraints are casted as an energy minimization problem using similarity measure between pairs of cameras that are close to each other. We denote by S ⊂ R3 the 3-D surface. Let Ii : Ωi ⊂ R2 → Rd be the image captured by camera i (d=1 for grayscale and d=3 for color images). The perspective projection of camera i is represented as Πi : R3 → R2 . Since the method uses visibility, consider Si as part of surface S visible in image i. In addition, the back-projection of image from camera i onto the surface is represented as Π−1 : Πi (S) → Si . i Armed with theabovenotation,onecan computeasimilarity measureMij of thesurface S as the the similarity measure between image Ii and the reprojection of image Ij into the othercameraiviathesurfaceS.Summingacrossallthecandidatestereopairs,onecanwrite:  Mij (S) (1) M(S) = i

j=i

  −1 Mij (S) = M |Ωi ∩Πi (Sj ) Ii , Ii ◦ Πi ◦ Πj,S Finally, the surface evolution equation at a point x is given by:   ∂S = − λ1 Esmooth + Eimg N ∂t

(2)

(3)

where Esmooth depends on the curvature H (see [6]), N represents the surface normal and Eimg is a photoconsistency term that is a summation across pairs of cameras which depends upon derivatives of the similarity measure M, of the images I, of the projection matrices Π and on the distance xz (see [7] for more details).

TransforMesh : A Topology-Adaptive Mesh-Based Approach to Surface Evolution

173

In the original paper [7], the surface evolution equation was implemented within the Level-Set framework. We extend it to meshes using the TransforMesh algorithm described in the previous section. The original solution performs surface evolution using a coarse to fine approach in order to escape local minima. Traditionally, in Level-Set approaches, the implicit function that embeds the surface S is discretized evenly on a 3-D grid. As a side-effect, all the facets of recovered surface are of approximately equal triangle size. In contrast, mesh based approaches do not impose such a constraint and allow facets of all sizes on the evolving surface. This is particularly useful when starting from visual hulls, for which the initial mesh contains triangles of all dimensions. In addition, the dimension of visual facets appears to be a relevant information since regions where the visual reconstruction is less accurate, i.e. concave regions on the observed surface, are described by bigger facets on the visual hull. Thus, we adopt a coarse to fine approach in which bigger triangles are moved until they stabilize, followed by dimension reduction via an edge-splits. The algorithm iterates at a smaller scale until the desired smallest edge size is obtained. Therefore, the√algorithm uses a multi-scale approach, starting from scale smax to smin = 1 in λ2 = 2 increments using Δt = 0.001 as the timestep. A vertex can maximally move 10% of the average incoming half-edges. The original surface, obtained from a visual hull, is evolved using equation (3), where the cross correlation was used as a similarity measure and λ1 = 0.3. Every 5 iterations TransforMesh is performed, in order to remove the self-intersections and allow for any topological changes.

4 Results We have tested the mesh evolution algorithm with the datasets provided by the MultiView Stereo Evalutation site [9] (http://vision.middlebury.edu/mview/) and our results are comparable with state-of-the-art: we rank in the top 1-3 (depending on the data set and ranking criteria chosen) and the results are within sub-milimeter accuracy. Detailed results are extracted from the website and presented in Table 1 (consult the website for detailed info and running times). We have also included results by Furukawa et al. [6], Pons et al. [7] and Hernandez et al. [8], considered to be the state of the art. The differences between all methods are very small, ranging between 0.01mm to 0.3mm. Some of our reconstruction results are shown in Figure 8. The algorithm reaches a good solution without the presence of a silhouette term in the evolution equation. In a typical evolution scenario, there are a more self-intersections at the beginning, but, as the algorithm converges, intersections rarely occur. Additionally, in the temple case, we performed a test where we have started from one big sphere as Table 1. 3-D Rec. Results. Accuracy: the distance d in mm that brings 90% of the result R within the ground-truth surface G. Completeness: the percentage of G that lies within 1.25mm of R.

PP PPDataset PP Paper P

Temple Ring Acc. Compl. Pons et al. [7] 0.60mm 99.5% Furukawa et al. [6] 0.55mm 99.1% Hernandez et al. [8] 0.52mm 99.5% Our results 0.55mm 99.2%

Temple Sparse Ring Acc. Compl. 0.90mm 95.4% 0.62mm 99.2% 0.75mm 95.3% 0.78mm 95.8%

Dino Ring Acc. Compl. 0.55mm 99.0% 0.33mm 99.6% 0.45mm 97.9% 0.42mm 98.6%

Dino Sparse Ring Acc. Compl. 0.71mm 97.7% 0.42mm 99.2% 0.60mm 98.52% 0.45mm 99.2%

174

A. Zaharescu, E. Boyer, and R. Horaud

(a) Dino - Input

(e) Temple - Input

(b) Dino - Start

(f) Temple - Start

(c) Dino - Results

(g) Temple-Results

(d) Dino - Closeup

(h) Temple-Closeup

Fig. 8. Reconstruction results obtained in the temple and in the dino case

the startup condition, in order to check whether the topological split operation performs properly. Proper converges was obtained. We acknowledge that TransforMesh was not put to a thorough test using the current data sets, which might leave the reader suspicious about special cases in which the method could fail. We have implemented a mesh morphing algorithm in order to test the robustness of the method. We have successfully morphed meshes with significantly different topology from the surface of departure. Results will be detailed in another publication. Implementation Notes. In our implementation we have made extensive use of CGAL (Computational Geometry Algorithms Library) [16] C++ library, which provides excellent implementations for various algorithms, among which the n-dimentional fast box intersections, 2-D constrained Delaunay triangulation, triangular meshes and support for exact arithmetic kernels. The running times of TransforMesh depend greatly on the number of self-intersections, since more than 80% of the running time is spent performing them. Typically, the running time for performing the self-intersections test is under 1 second for a mesh with 50, 000 facets, where exact arithmetic is used for triangle intersections and the self-intersections are in the range of 100.

5 Conclusion We have presented a fully geometric efficient Lagrangian solution for triangular mesh evolutions able to handle topology changes gracefully. We have tested our method in the

TransforMesh : A Topology-Adaptive Mesh-Based Approach to Surface Evolution

175

context of multi-view stereo 3-D reconstruction and we have obtained top ranking results, comparable with state-of-the-art methods in the literature. Our contribution with respect to the existing methods is to provide a purely geometric mesh-based solution that does not constrain meshes and that allows for facets of all sizes as well as for topology changes. Acknowledgements. This research was supported by the VISIONTRAIN RTN-CT2004-005439 Marie Curie Action within the European Community’s Sixth Framework Programme. This paper reflects only the author’s views and the Community is not liable for any use that may be made of the information contained therein.

References 1. McInerney, T., Terzopoulos, D.: T-snakes: Topology adaptive snakes. Medical Image Analysis 4(2), 73–91 (2000) 2. Lachaud, J.O., Taton, B.: Deformable model with adaptive mesh and automated topology changes. In: Proceedings of the Fourth International Conference on 3-D Digital Imaging and Modeling (2003) 3. Osher, S., Fedkiw, R.: Level Set Methods and Dynamic Implicit Surfaces. Springer, Heidelberg (2003) 4. Osher, S., Senthian, J.: Front propagating with curvature dependent speed: algorithms based on the Hamilton-Jacobi formulation. Journal of computational Physics 79(1), 12–49 (1988) 5. Jung, W., Shin, H., Choi, B.K.: Self-intersection removal in triangular mesh offsetting. Computer-Aided Design and Applications 1(1-4), 477–484 (2004) 6. Furukawa, Y., Ponce, J.: Accurate, dense and robust multi-view stereopsis. CVPR (2007) 7. Pons, J.P., Keriven, R., Faugeras, O.: Multi-view stereo reconstruction and scene flow estimation with a global image-based matching score. International Journal of Computer Vision 72(2), 179–193 (2007) 8. Hernandez, C.E., Schmitt, F.: Silhouette and stereo fusion for 3-D object modeling. Computer Vision and Image Understanding 96(3), 367–392 (2004) 9. Seitz, S.M., Curless, B., Diebel, J., Scharstein, D., Szeliski, R.: A comparison and evaluation of multi-view stereo reconstruction algorithms. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 1, pp. 519–526. IEEE Computer Society Press, Los Alamitos (2006) 10. Kettner, L., Meyer, A., Zomorodian, A.: Intersecting sequences of dD iso-oriented boxes. In: Board, C.E. (ed.) CGAL-3.2 User and Reference Manual (2006) 11. Zomorodian, A., Edelsbrunner, H.: Fast software for box intersection. International Journal of Compational Geometry and Applications (12), 143–172 (2002) 12. Hert, S., Seel, M.: dD convex hulls and delaunay triangulations. In: Board, C.E. (ed.) CGAL3.2 User and Reference Manual (2006) 13. Gueziec, A., Taubin, G., Lazarus, F., Horn, B.: Cutting and stitching: Converting sets of polygons to manifold surfaces. IEEE Transaction on Visualization and Computer Graphics 7(2), 136–151 (2001) 14. Shin, H., Park, J.C., Choi, B.K., Chung, Y.C., Rhee, S.: Efficient topology construction from triangle soup. In: Proceedings of the Geometric Modeling and Processing (2004) 15. Franco, J.S., Boyer, E.: Exact polyhedral visual hulls. In: British Machine Vision Conference, vol. 1, pp. 329–338 (2003) 16. Board, C.E.: CGAL-3.2 User and Reference Manual (2006)

Microscopic Surface Shape Estimation of a Transparent Plate Using a Complex Image Masao Shimizu and Masatoshi Okutomi Graduate School of Science and Engineering, Tokyo Institute of Technology

Abstract. This paper proposes a method to estimate the surface shape of a transparent plate using a reflection image on the plate. The reflection image on a transparent plate is a complex image that consists of a reflection on the surface and on the rear surface of the plate. A displacement between the two reflection images holds the range information to the object, which can be extracted from a single complex image. The displacement in the complex image depends not only on the object range but also on the normal vectors of the plate surfaces, plate thickness, relative refraction index, and the plate position. These parameters can be estimated using multiple planar targets with random texture at known distances. Experimental results show that the proposed method can detect microscopic surface shape differences between two different commercially available transparent acrylic plates.

1

Introduction

Shape estimation of a transparent object is a difficult problem of computer vision that has attracted many researchers. The background changes its apparent position through a moving transparent object according to the refraction of light [1]. Based on that observation, some techniques have been proposed to estimate the shape and the refraction index of a transparent object: using optical flow [5], detecting the positional change of the structured lights [4], and measuring the effects on the shape of a structured light [3] caused by a moving object. Other methods that have been investigated include using polarized light [6]. On the one hand, a monocular range estimation method, named reflection stereo, using a complex image observed as a reflection on a transparent parallel planar plate, has also been proposed [7],[8]. The complex image consists of the reflection at the plate surface (the surface reflection image Is ) and the reflection at the rear surface of the plate, which is then transmitted again through the surface (the rear-surface reflection image Ir ). The object range is obtainable from the displacement in the complex image, that is, the displacement between Is and Ir , which has a unique constraint that is the equivalent of the epipolar constraint in stereo vision. The displacement in the complex image depends not only on the object range, but also the normal vectors of the plate surfaces, plate thickness, relative refraction index, and the plate position. This paper proposes a method to estimate these parameters of the transparent plate through calibration of the reflection stereo system, using multiple planar Y. Yagi et al. (Eds.): ACCV 2007, Part II, LNCS 4844, pp. 176–185, 2007. c Springer-Verlag Berlin Heidelberg 2007 

Microscopic Surface Shape Estimation of a Transparent Plate

177

targets with random texture at known distances. These estimated parameters are available for range estimation; simultaneously, they express the shape and orientation of the plate. The proposed method estimates a microscopic shape of a plate using a differential measurement of Is and Ir , whereas conventional methods have used only light transmitted through a transparent object. The method might be used for an industrial inspection of a planar glass plate with a common camera. This paper is organized as follows. Section 2 describes range estimation by reflection stereo with a perfectly parallel planar transparent plate. Section 3 extends to the case with a non-parallel planar plate, and shows the parameters to be estimated. Section 4 presents a parameter estimation method through the calibration of the reflection stereo using multiple targets. Experimental results are described in Section 5. This paper concludes with remarks in Section 6.

2

Range Estimation from a Single Complex Image

This section describes the reflection stereo with a perfectly parallel planar plate. Figure 1 shows the two light paths from an object to the camera center. A transparent plate reflects and transmits the incident light on its surface. The transmitted light is then reflected on the rear-surface and is transmitted again to the air through the surface. These two light paths have an angle disparity θs , which depends on the relative refractive index n of the plate, the plate thickness d, the incident angle θi , and the object distance Do . The fundamental relation between the angle disparity θs and the distance Do are explainable as the reflection and refraction of light in a plane including the object, the optical center, and the normal vector of the plate. A two-dimensional (2D) ξ-υ coordinate system is set with its origin at the reflecting point on the surface. The following equation can be derived by projecting the object position (−Do sin θi , Do cos θi ) and the optical center position (Dc sin θi , Dc cos θi ) to ξaxis. sin (2 (θi − θs ))  . (1) Do + Dc = d sin θs n2 − sin2 (θi − θs ) The angle disparity θs is obtainable by finding the displacement in the complex image. Then the object distance Do is derived from Eq. (1). The displacement has a constraint which describes a correspondent position in the rear-surface image Object

υ

(− Do sin θ i , Do cosθ i )

Optical center

( Dc sin θ i , Dc cosθ i ) θs Surface reflection

θi

Is Rear-surface reflection

Ir Transparent plate

n

d

ξ

Fig. 1. Fundamental light paths in the reflection stereo

178

M. Shimizu and M. Okutomi

moving along a constraint line according to the image position [7]. The constraint reduces the search to 1D, just as stereo vision with the epipolar constraint. The angle disparity takes the minimum θs = 0 when Do = ∞ if the plate is manufactured perfectly as a parallel planar plate.

3

Reflection Stereo with Non-parallel Planar Plate

The transparent acrylic plate has limited parallelism and has a specific wedge angle, even if it has sufficient surface flatness. In this section, we investigate the range estimation of the reflection stereo using a non-parallel planar plate. 3.1

Range Estimation Using Ray Tracing

Figure 2 depicts the camera coordinate system with its origin at the optical center. The f /δ denotes the lens focal length in the CCD pixel unit. The transparent plate is set with an angle for the Z-axis. We use ray tracing [9] to describe the light paths to estimate the object distance. Figure 3(a) shows that an object in a surface reflection image at an image position A(a) = [ua , va , f /δ] exists in the following line LS with parameter tS : LS (tS ) = qS + vS tS

(2)

qS = q(o, a, mS , n ˆS ) ˆS ), vS = r(a, n ˆS respectively denote a position and the unit normal vector where MS (mS ) and n of the plate surface. Point O(o) represents the origin. Transparent plate Image plane

A( )

Y

v

u a , va

X

u

Z

O

f /δ

Dc

Fig. 2. Transparent plate and camera

ˆS

Image plane

S

O

S

A( )

ˆR

Image plane

R

S

ˆS

S

R

O

R

0 1 0

n (a) Surface reflection image

1

n

(b) Rear-surface reflection image

Fig. 3. Surface and rear-surface reflections

D( )

Microscopic Surface Shape Estimation of a Transparent Plate

179

Similarly, a detected displacement d = d∗p (a)+e∗ (a)Δ of a rear-surface reflection image from the image position a shows that the object in the rear-surface image exists in the following line LR with parameter tR : LR (tR ) = qR + vR tR

(3)

qR = q(q1 , v1 , mS , n ˆS ) ˆR ) q1 = q(q0 , v0 , mR , n

vR = t(v1 , −ˆ nS , 1/n) v1 = r(v0 , n ˆR)

q0 = q(o, a + d, mS , n ˆS )

v0 = t(d, n ˆS , n),

ˆR respectively denote a position and the unit normal vector where MR (mR ) and n of the rear surface of the plate, as shown in Fig. 3(b). In addition, d∗p (a) and e∗ (a) denote the displacement of the rear-surface reflection image for an object at infinite distance and a normal vector of the constraint line at the image position a. The intersection position of lines LS and LR represents the 3D position PO (ˆ pO ) of the object. In real situations, the object position PO (ˆ pO ) is determined using the following equation, which minimizes the distance between the two lines. 1 (4) p ˆO = [(qS + vS tS ) + (qR + vR tR )] 2    −1   tS qS · vS −qS · vR qS · qR − qS 2 = tR qR · vS −qR · vR qR 2 − qS · qR For the above, we use the following relations presented in Fig. 4. (s − p) · n ˆ v·n ˆ r(v, n ˆ) = v − 2ˆ nv · n ˆ

q(p, v, s, n ˆ) = p + v

(5)

1  2  2  v    v 2   −n ˆ n  −   v · n ˆ v · n ˆ

t(v, n ˆ, n) = 

3.2



v v · n ˆ

 −n ˆ +n ˆ

(6) (7)

Parameters for Reflection Stereo

Reflection stereo with a non-parallel planar plate requires not only a lens focal length f /δ as a parameter, but also the following parameters, which are equivalent to extrinsic parameters in stereo vision. P( )

Q( )

S( )

n ˆ

Fig. 4. A ray and a normal vector of the plate

180

– – – –

M. Shimizu and M. Okutomi

surface and rear-surface normal vectors nS and nR of the plate, a position mS on the surface, plate thickness d, and relative refraction index n.

With these extrinsic parameters, the object range is obtainable using Eq. (4) with a displacement in a complex image.

4 4.1

Surface Shape Estimation by Reflection Stereo Piecewise Non-parallel and Planar Model of the Plate

Real surfaces of acrylic or glass plates are not perfectly flat on some level. Consequently, the extrinsic parameters are position-variant in the complex image. The relative refraction index can differ among production lots and manufacturers; it might also be position-variant. The proposed method assumes that the plate is piecewise non-parallel and planar, as illustrated in Fig. 5, for the reflection stereo with a real transparent plate. In other words, the method assumes that the extrinsic parameters are constant in a small region to form a complex image for an interest position in a complex image. The object range p ˆ Oy (a, d|pe (a)) along the Y -axis is obtainable using Eq. (4) for the small region that includes the image position a, where pe (a) = {nS (a), nR (a), mS (a), d(a), n(a)} denotes the extrinsic parameter vector at position a. 4.2

Parameter Estimation Method

The extrinsic parameters can be estimated from M sets of the object distance Dy and its corresponding displacement d at position a in a complex image, as follows analogously with calibration for stereo vision [10]. p ˆe (a) = arg min E(a)

(8)

pe (a)

E(a) =

M

2

Dyi − p ˆ Oy (a, di |pe (a)) + α ˜ pe − pe (a)

2

i=1

Whole transparent planar plate Image plane

u O A(a)

Piecewise planar region p e (a) n S (a), n R (a), m S (a), d (a), n(a)

Fig. 5. Extrinsic parameters for a piecewise non-parallel and planar model

Microscopic Surface Shape Estimation of a Transparent Plate

181

The second term in the objective function E(a) is a stabilization term that prevents the estimated parameters from deviating markedly from design (catalogue) values p ˜e . The weight α is set empirically to the minimum value. We have used the conjugate gradient method to minimize the nonlinear objective function with initial parameters of the design values. The displacement d in a complex image can be detected as the second peak location of the autocorrelation function of the complex image without any knowledge about a constraint if the image contains a rich and dense texture. The seven components in the extrinsic parameters can be estimated from M ≥ 4 sets of observations1. 4.3

Parametric Expression for Range Estimation

The estimated parameters are considered as continuous and their changes are small with respect to the image position, as is clear from observations of real acrylic plates. The following parametric expression with a two-dimensional cubic function with respect to the image position u = (u, v) is used for range estimation: p ¯e (u) = Φ1 +Φ2 u+Φ3 v+Φ4 uv+Φ5 u2 +Φ6 v 2 +Φ7 u2 v+Φ8 uv 2 +Φ9 u3 +Φ10 v 3 , (9) where Φj (j = 1, 2, .., 10) denotes a coefficient vector corresponding to the seven components of the extrinsic parameters. These vectors are obtainable using leastsquares estimation, with estimated extrinsic parameters p ˆ e (al ) at image positions al , as follows. ¯ pe (al ) − p ˆe (al )2 (10) Φj (j = 1, 2, .., 10) = arg min Φ

l

The constraint line direction and the infinite object position with respect to the image position are also obtained in advance, and parameterized similarly as the extrinsic parameters for range estimation.

5

Experimental Results

5.1

System Setup

The experimental system was constructed as shown in Fig. 6. The plate is set with an angle π/4 for the optical axis. The system includes a black and white camera (60 FPS, IEEE1394, Flea; Point Gray Research Inc.). The design values of the plate are 10.0 [mm] thickness and the relative refraction index of 1.49. The measured distance from the lens optical center to the plate along the optical axis is Dco = 66.2 [mm]. We used a well-known camera calibration tool [2] for 1

The normal vector has two degrees-of-freedom (2-DOF), the position on the surface is 1-DOF, as determined by a distance along Z-axis if the surface normal is given. In total, the extrinsic parameters are 2 + 2 + 1 + 1 + 1 = 7 DOF.

182

M. Shimizu and M. Okutomi

Fig. 6. Configuration of the experimental system Textured plane

Camera

Transparent plate

Dc

D yi

Fig. 7. Setup of the calibration target

intrinsic parameter estimation. The calibrated parameters are the focal length f /δ = 1255.7 and the image center (cu , cv ) = (−31.845, −11.272); the image size is 640 × 480 [pixel]. Image distortion was alleviated using the estimated lens distortion parameters. 5.2

Calibration Targets

A random textured planar target at 13 different positions is used for extrinsic parameter estimation. As depicted in Fig. 7, the experimental system is mounted on a motorized long travel translation stage. Figure 8 shows a complex image for the plane distance of 500 [mm]. As described in 4.2, the displacement (shift value) along the constraint line in the complex image is Fig. 8. Complex image example (Do = obtainable as the second peak location 500 [mm]) of the autocorrelation function. The target positions are at Dyi = 350 + 50i + De [mm] (i = 1, 2, .., 13) from the optical center in Y -axis, where

Microscopic Surface Shape Estimation of a Transparent Plate

183

[mm]

120

0

-120

-160

-320

0

(a) Estimated thickness

160

320

[pixel]

120

0 -120 -320

-160

0

320

160

[pixel]

(b) Estimated refraction index

Fig. 9. Estimated plate thickness and refraction index w.r.t. the image positions

De is a unknown value. To estimate De , first the following sum of the residuals with three predetermined values (De = 0, ±25) is obtained; then the three sums of residuals are used to estimate the De for the minimum sum by a parabola fitting over the three sums. M l

(350 + 50i + De ) − p ˆOy (al , di |pe (al ))2

(11)

i=1

Figure 9 displays the estimated plate thickness d and relative refraction index n of transparent plate “A” (described in the next subsection), at 24 × 18 = 432 image positions. The corresponding size of the plate to the image positions is about 30 × 30 [mm]. The mean and standard deviation of the plate thickness are, respectively, 10.005712 and 5.351757 × 10−3 [mm], those of the refraction index are 1.489652 and 4.427796 × 10−4 , whereas the catalogue values are 10.0 [mm] and 1.49. 5.3

Microscopic Surface Shape Estimation of a Transparent Plate

Two types of commercially produced transparent acrylic plate are used to estimate the extrinsic parameters. Both plates have the same thickness of 10 [mm]; plate “A” was cut from a cast acrylic plate to 100 × 100 [mm], whereas plate “B” was a cast acrylic plate marketed at that size without after-sales cutting. Figure 10 displays the estimated surface and rear-surface positions of plate “A”, corresponding to 24 × 18 = 432 image positions. In this figure resolution, the microscopic surface shape difference between plate “A” and “B” is invisible. Figures 11 and 12 respectively display the estimated surface (a) and rearsurface (b) shapes of plates “A” and “B”. The surface shapes are displayed with respect to least-squares approximation surfaces for each plate. The two plates are measured using the same conditions. The vertical axes are extremely magnified. The standard deviation of the plate “A” surface is 0.013077, whereas that of plate “B” is 0.010649. More than the numerical difference, the shape difference between the two plates is readily apparent from the figures.

184

M. Shimizu and M. Okutomi 20 10 0 -10 -20 10 0 -10

[mm]

Fig. 10. Estimated surface and rear-surface positions of plate “A”

10

10 0

-10 -20 -20

-10

0

10

20

0 -10 -20 -20

[mm]

㩿㪸㪀㩷㪜㫊㫋㫀㫄㪸㫋㪼㪻㩷㫊㫌㫉㪽㪸㪺㪼㩷㫊㪿㪸㫇㪼

-10

0

10

20

[mm]

㩿㪹㪀㩷㪜㫊㫋㫀㫄㪸㫋㪼㪻㩷㫉㪼㪸㫉㪄㫊㫌㫉㪽㪸㪺㪼㩷㫊㪿㪸㫇㪼

Fig. 11. Estimated surface and rear-surface shapes of plate “A”

10

10 0

-10 -20 -20

-10

0

10

[mm]

㩿㪸㪀㩷㪜㫊㫋㫀㫄㪸㫋㪼㪻㩷㫊㫌㫉㪽㪸㪺㪼㩷㫊㪿㪸㫇㪼

20

0 -10 -20 -20

-10

0

10

[mm]

㩿㪹㪀㩷㪜㫊㫋㫀㫄㪸㫋㪼㪻㩷㫉㪼㪸㫉㪄㫊㫌㫉㪽㪸㪺㪼㩷㫊㪿㪸㫇㪼

Fig. 12. Estimated surface and rear-surface shapes of plate “B”

20

Microscopic Surface Shape Estimation of a Transparent Plate

6

185

Conclusions

In this paper, we have proposed a method to estimate the surface shape of a transparent plate using a complex image. The displacement in the complex image depends not only on the object range but also on the normal vectors of the plate surfaces, plate thickness, relative refraction index, and the plate position. These parameters can be estimated using multiple planar targets with a random texture at known distances. Experimental results show that the proposed method can detect microscopic surface shape differences between two different commercially available transparent acrylic plates. Future works will include an investigation of the shape estimation accuracy and resolution using an optical standard, and a study of the precision limits of the proposed method.

References 1. Ben-Ezra, M., Nayar, S.K.: What Does Motion Reveal about Transparency? Proc. on ICCV 2, 1025–1032 (2003) 2. Bouguet, J.-Y.: Camera Calibration Toolbox for Matlab (2007), http://www.vision.caltech.edu/bouguetj/calib doc/index.html 3. Hata, S., Saitoh, Y., Kumamura, S., Kaida, K.: Shape Extraction of Transparent Object using Genetic Algorithm. Proc. on ICPR 4, 684–688 (1996) 4. Manabe, Y., Tsujita, M., Chihara, K.: Measurement of Shape and Refractive Index of Transparent Object. Proc. on ICPR 2, 23–26 (2004) 5. Murase, H.: Surface Shape Reconstruction of a Nonrigid Transport Object using Refraction and Motion. IEEE Trans. on PAMI 14(10), 1045–1052 (1992) 6. Saito, M., Sato, Y., Ikeuchi, K., Kashiwagi, H.: Measurement of Surface Orientations of Transparent Objects using Polarization in Highlight. Proc. on ICPR 1, 381–386 (1999) 7. Shimizu, M., Okutomi, M.: Reflection Stereo – Novel Monocular Stereo using a Transparent Plate. In: Proc. on Third Canadian Conference on Computer and Robot Vision, pp. 14–14 (2006) (CD-ROM) 8. Shimizu, M., Okutomi, M.: Monocular Range Estimation through a Double-Sided Half-Mirror Plate. In: Proc. on Fourth Canadian Conference on Computer and Robot Vision, pp. 347–354 (2007) 9. Whitted, T.: An Improved Illumination Model for Shaded Display. Communications of the ACM 23(6), 343–349 (1980) 10. Zhang, Z.: Flexible Camera Calibration by Viewing a Plane from Unknown Orientations. Proc. on ICCV 1, 666–673 (1999)

Shape Recovery from Turntable Image Sequence H. Zhong, W.S. Lau, W.F. Sze, and Y.S. Hung Department of Electrical and Electronic Engineering, the University of Hong Kong Pokfulam Road, Hong Kong, China {hzhong, sunny, wfsze, yshung}@eee.hku.hk

Abstract. This paper makes use of both feature points and silhouettes to deliver fast 3D shape recovery from a turntable image sequence. The algorithm exploits object silhouettes in two views to establish a 3D rim curve, which is defined with respect to the two frontier points arising from two views. The images of this 3D rim curve in the two views are matched using cross correlation technique with silhouette constraint incorporated. A 3D planar rim curve is then reconstructed using point-based reconstruction method. A set of rims enclosing the object can be obtained from an image sequence captured under circular motion. The proposed method solves the problem of reconstruction of concave object surface, which is usually left unresolved in general silhouette-based reconstruction methods. In addition, the property of the organized reconstructed rim curves allows fast surface extraction. Experimental results with real data are presented. Keywords: silhouette; rim reconstruction; surface extraction; circular motion.

1 Introduction The 3D modelling of real world objects is an important problem in computer vision and has many practical applications, such as in virtual reality, video games, motion tracking systems, etc. The point-based approach is the oldest technique for 3D reconstruction. Once feature points across different views are matched, a cloud of feature points lying on the surface of object can be recovered by triangulation methods. However, these points are not ordered, which means that the topological relationship among them is unknown. As a result, they are not readily usable for reconstructing the surface of an object. Besides feature points, silhouettes are prominent features of an image that can be extracted reliably. They provide useful information about both the shape and motion of an object, and indeed are the only features that can be extracted from a textureless object. But they do not provide point correspondences between images due to the viewpoint dependency of silhouettes. There is a need for the development of methods combining both point-based and silhouettes approaches for modelling a 3D object. The contribution of this paper is that it takes advantage of both point-based and silhouettes approaches to provide fast shape recovery. The method reconstructs a set of 3D rim curves on the object surface from a calibrated circular motion image sequence, with concavities on the object surface retained. These 3D rim curves are reconstructed in an organized manner consistent with the circular motion image order, leading to low computation complexity in the subsequent surface triangulation. Y. Yagi et al. (Eds.): ACCV 2007, Part II, LNCS 4844, pp. 186–195, 2007. © Springer-Verlag Berlin Heidelberg 2007

Shape Recovery from Turntable Image Sequence

187

This paper is organized as follows. Section 2 briefly reviews the literature on model reconstruction. Section 3 introduces the theoretical principles which are used in this paper. Section 4 describes how the rim curves are computed. The extraction of surface from the reconstructed rim curves is given in section 5. Experimental results are given in section 6 and a summary follows in section 7.

2 Previous Works There are two main streams of methods for model reconstruction, namely point-based and silhouette-based methods. In a point-based method, a limited number of 3D points are initially reconstructed from matched feature points. A 3D model is then obtained by computing a dense depth map of the 3D object from the calibrated images using stereo matching techniques in order to recover the depths of all the object points, (see, for example, [1, 2]). It is further necessary to fuse all the dense depth map into a common 3D model [3] and extract coherently connected surface by interpolation [4]. Although a good and realistic 3D model may be obtained by carrying out these processes, both the estimation of dense depth maps and the fusion of them are of high computation complexity and reported to be formidably time consuming [5]. Another model reconstruction approach utilizes object silhouettes instead of feature points, see seminal works [6, 7]. There are two groups of methods in this approach, namely surface and volumetric methods. The surface method aims to determine points on the surface of the visual hull by either updating depth intervals along viewing lines [8] or using a dual-space approach via epipolar parameterization [9, 10]. On the other hand, volume segment representation, introduced in [11], was for constructing the bounding volume which approximated the actual three-dimensional structure of the rigid object generating the contours in multiple views. In this technique, octree representation is adopted to provide a volumetric description of an object in terms of regular grids or voxels [12-14]. Given the volumetric data in form of an octree, the marching cubes algorithm [15] can be used to extract a triangulated surface mesh for visualization. Since only silhouettes are used, no feature points are needed. However, the aforementioned methods produce only the visual hull of the object with respect to the set of viewpoints from which the image sequence is captured. The main drawback is hence that concave surfaces of the object cannot be reconstructed. More recently, many methods are proposed to combine the silhouette and the photo-consistency constraint to obtain higher quality results based on different formulations, cf. [16-20]. Broadly speaking, these methods all employ a two-step approach to surface reconstruction. In the first step, a surface mesh is initialized by computing a visual hull satisfying the silhouette consistency constraint. In the second step, the surface mesh is refined using the photo-consistency constraint. In this paper, we also consider combining the properties of silhouettes and feature points in shape recovery from a calibrated circular motion image sequence. However, unlike the methods mentioned above which aimed to reconstruct the whole photoconsistent surface based on a visual hull, the proposed method computes a set of discrete photo-consistent surface curves which are circumnavigating the object to represent the shape. The advantages of using discrete surface curves to represent

188

H. Zhong et al.

object shape and to form surface are two fold. First, no visual hull and time-consuming surface optimization are needed, and secondly, surface concavities can be recovered. This is particularly suitable for fast modelling with more accurate shape description than visual hull. Experimental results show that through combining the photo-consistency and silhouette constraints, the surface curves can be computed accurately.

3 Theoretical Principles 3.1 Geometry of the Surface Consider an object with a smooth surface viewed by a pin-hole camera. The contour generator on the object surface is the set of points at which the rays cast from the camera center is orthogonal to the surface normal [21]. The projection of a contour generator on the image plane forms an apparent contour. A silhouette is a subset of the apparent contour where the viewing rays of the contour generator touch the object [22]. Due to the viewpoint dependency of the contour generators, silhouettes from two distinct viewpoints will be the projections of two different contour generators. As a result, there will be no point correspondence in the two silhouettes except for the frontier point(s) [21]. A frontier point is the intersection of the two contour generators in space and is visible in both silhouettes. For two distinct views, there are in general two frontier points at which the two outer epipolar planes are tangent to the object surface. Hence, the projections of them in one view are two outer epipolar tangent points on the object silhouette. The outer epipolar tangent points in the two associated views are corresponding points. X1

A 3D rim curve

C1

C2

A straight line

X2

A 2D rim curve

Fig. 1. A 3D rim curve associated with frontier points for two views. The two cameras centers C1 and C2 define two epipolar planes tangent to the objects at two frontier points X1 and X2 whose images are outer epipolar tangent points. The plane containing C1, X1 and X2 cuts the object at a 3D planar rim curve. The curve segment visible to cameras C1 and C2 projects onto them as a straight line and a 2D rim curve respectively.

Shape Recovery from Turntable Image Sequence

189

3.2 Definition of a 3D Rim Curve A 3D rim curve on an object surface is defined with respect to two distinct views as follows. The two frontier points associated with the two views, together with one of the two camera centers, define a plane. This plane intersects the object surface in a 3D planar curve on the object surface. The two frontier points are on this curve and they cut the curve into segments. The one closer to the chosen camera center is defined a 3D rim curve. The projection of this rim curve in the view associated with the chosen camera is a straight line, and the image of it in the other view is generally a 2D curve. Fig. 1 illustrates the rim curve so defined. 3.3 Cross-Correlation Cross-correlation is a general technique for finding correspondences between points in two images. The matching technique is based on the similarity between two image patches, and cross-correlation gives a measure of similarity. Let Ik(uk, vk) be the intensity of image k at pixel xk=(uk, vk) and the correlation mask size be (2n+1)×(2m+1). When n=m, then n is called half mask size of n. The normalized correlation score between a pair of points x1 and x2 is given by: n

r ( x1 , x2 ) =

m

∑ ∑ [ I (u

i =− n j =− m

1

1

+ i, v1 + j ) − I1 (u1 , v1 )] ⋅ ⎡⎣ I 2 (u2 + i, v2 + j ) − I 2 (u2 , v2 ) ⎤⎦ (1)

[(2n + 1) × (2m + 1)] ⋅ δ ( I1 ) ⋅ δ ( I 2 )

where n

I k (uk , vk ) =

m

∑∑I

i =− n j =− m

k

(uk + i, vk + j ) (2)

(2n + 1) × (2m + 1)

is average intensity about point xk, and n

δ (Ik ) =

m

∑ ∑ ⎡⎣ I

i =− n j =− m

⎤ k (uk + i , vk + j ) − I k (uk , vk ) ⎦ (2n + 1) × (2m + 1)

2

(3)

is the standard deviation of intensity over the mask. The feature point x2 which maximizes the response of the normalized correlation score is deemed to be the best match to x1., and it is usual to define a threshold T to remove dissimilar matches (i.e. the correlation score between corresponding points have to be larger than T, otherwise it will be not considered as a match).

4 Fast Modeling by Rim Reconstruction Given a calibrated circular motion image sequence, silhouettes are first extracted reliably from the images. For two adjacent views, the images of the two frontier

190

H. Zhong et al.

points are estimated using the silhouettes. A 2D rim (the image of a 3D rim curve in one view) is then defined as a line joining the two projected frontier points in an image. The correspondences of this 2D rim in the second image are found using cross-correlation methods. A 3D rim curve is then reconstructed by the linear triangulation method [23]. Repeating this process for the image sequence on a sequential two-view basis, a set of rim curves can be reconstructed. Since the object undergoes circular motion, the rims are constructed in a known order. 4.1 Silhouette Extraction

In order to extract silhouettes, the cubic B-spline snakes [24] are adopted for achieving sub-pixel accuracy. It provides a concise representation of silhouettes. In addition, using a B-spline representation allows explicit computation of tangency at each point on the silhouette which simplifies the subsequent identification of outer epipolar tangents (to be described in section 4.2). We will assume that each silhouette consists of one closed curve. 4.2 Estimation of Outer Epipolar Tangents

An outer epipolar tangent is an epipolar line tangent to the silhouette passing through the epipole. Given the B-spline and the known epipolar geometry between two views, the outer epipolar tangent points can be calculated as follows. Let Ai (i=1,2) be the set of sample points on the B-spline and ei be the epipole in image i from camera center j (j=1,2, j≠i), then the two outer epipolar tangent points in image i are determined by ⎛ p ( y ) − ei ( y ) ⎞ ⎛ pi ( y ) − ei ( y ) ⎞ min ⎜⎜ i max and (4) ⎟ ⎜ ⎟. ⎟ pi ∈ Ai pi ∈ Ai ⎜ p ( x ) − e ( x ) ⎟ i ⎝ pi ( x ) − ei ( x ) ⎠ ⎝ i ⎠ Fig. 2 illustrates the two outer epipolar tangents determined in view one based on the epipolar geometry between view one and view two.

Outer epipolar tangent points

Silhouette curve

Fig. 2. Estimation of outer epipolar tangent

4.3 Matching of 2D Rims

Given two views, the outer epipolar tangents in two views can be computed based on the estimated epipoles and the silhouettes as described in the last subsection. Let us name the two views as the source view and the target view respectively. In the source view, the rim is a straight line connecting the two epipolar tangent points. We need to

Shape Recovery from Turntable Image Sequence

191

find the counterpart of this rim in the target view. First, points on the rim in the source view are sampled. As the silhouettes of all images are available, for each sample point sp in the rim in source view, we can determine the associated visual hull surface points corresponding to the ray back-projected from sp [8]. Suppose the pair of visual hull surface points on the ray associated with sp is V1 and V2. We can limit the search region for sp in the target view by making use of V1 and V2, i.e. the epipolar line of sp in the target view to be searched for matched point is bounded by the projections of V1 and V2. Furthermore, we can use a parameter λ to reduce the searchable depth range from V1 to (1-λ)V1+λV2 based on prior knowledge about the shape of the object. For example, taking a λ=0.5 assumes that the maximum concavity of the object at sp is not larger than half of the depth from V1 to V2. Thus the search region in the target view can be accordingly reduced. After the search region along the epipolar line is defined, each corresponding point of the rim in the target view is determined as the pixel within the delimited epipolar line which has the highest cross correlation score with the sampled point in the source view. An example of matched rims in two consecutive views is shown in Fig. 3, from which we see that the reduction of depth search range greatly increases the matching accuracy.

(a)

(b)

(c)

Fig. 3. Matching of a rim in two views. The rim in the source view (a) is a straight line (green). The search region in the target view (b) is bounded by the projections of visual hull surface points determined based on silhouette information. The rim in the target view (c) is found by searching matched points along the delimited scan lines (blue) joining the two bounding points.

4.4 Reconstruction and Insertion of 3D Rims

Once the 2D rim points are matched, the 3D rim structure can be easily computed using optimal triangulation method. Repeating this process for every two successive views in the image sequence, a set of 3D rims on the object surface can be obtained. After this reconstruction processes, the rims of the model are generated, as shown in Fig. 4(a). However, there may be big gaps between pairs of rims where the surface is concave, see Fig. 4(a). This is because frontier points never occur on the concave part of a surface where the outer epipolar tangents cannot reach. To fill in a gap between two rims, new rims can be inserted. We first compute a 2D interpolated line between two rims in an image for which it is required to insert a new rim. Then, the correspondences in the target image are determined by feature point matching process as described in section 4.2. As a result, a 3D interpolated rim can be generated on the concave surface. This insertion of new rims is carried out until not rims are needed to be inserted. Fig. 4(b) shows the newly added rims (red) and the original rim sets (blue).

192

H. Zhong et al.

(a)

(b)

Fig. 4. Insertion of new 3D rims on concave surface. In (a) there may be a gap between two rims; in (b) more rims (red color) are inserted to fill in gaps.

5 Surface Formation 5.1 Slicing the 3D Rims

The proposed surface extraction method is built on a slice based re-sampling method, and it produces fairly evenly distributed mesh grids. The rims are first re-sampled to give cross-sectional contours by parallel slicing planes. Each slicing plane contains the intersections of the rims with the plane. For a better visual effect, the normal of the slicing planes is chosen to be the direction parallel to the rotation axis of circular motion for imaging the object. Since the 3D rims are generated in the order in accordance with the circular motion image sequence, in general the sampled points on each slice can be re-grouped by linking these points according to the spatial order of the corresponding rims. However, if the frontier points (the top most and lowest points) in the rims are very close, the sample points on the slicing plane near those frontier points may not exactly be in the desired order, for example, points on the slicing plane near the top of the object. So we reorder the sample points on the slicing planes before forming the polygonal cross section. Nonetheless, due to the sequential reconstruction of the rims, only the points on the top two slicing planes are affected. In the matching process, through enforcing the silhouette constraint, the search region on the epipolar line for matched point is substantially reduced, giving a very high accuracy of matching and thus a good reconstruction of surface points. However, there may be still few points reconstructed to be inside the object surface because of wrong matching. This type of error cannot be detected by the silhouette constraint, but can be reduced by smoothing the polygon on the slicing plane and by spatial smoothing on the extracted surface. The result of slice-based model after smoothing is shown in Fig. 6(a). 5.2 Surface Triangulation

In order to allow the reconstructed 3D model to be displayed using conventional rendering algorithms (e.g. in VRML standard adopted here), a triangulated mesh is extracted from the polygons on each slicing plane using the surface triangulation method, and triangle patches are produced that best approximate the surface. Since we have the same number of sample points on each slicing plane and the sample points are on the organized rims, triangle patches can be formed by a simple

Shape Recovery from Turntable Image Sequence

193

strategy: starting from the first sample point, say, p1i on layer i, the closest point pki+1 on layer i+1 to p1i is located. Then the third vertex can be either p2i or pk+1i+1, and is determined by considering two factors, the angles formed at p1i and at pki+1 must not exceed a threshold and the sum of distances to p1i and pki+1. If both points meet the angle constraint or conversely both are violated, the point between p2i or pk+1i+1 giving the shortest distance is chosen; otherwise, the one satisfying the angle constraint is selected.

6 Experimental Results Our approach has been tested on real data. Due to space limit, an experimental result of one circular motion image sequence is shown in this section. The experiment is performed over a turntable sequence of 36 images of a “Clay man” toy with moderately complex surface structure (see Fig. 3). All the cameras have been calibrated, which means that the projection matrices for cameras are already known. Since the performance of the proposed method depends on the accuracy of matching, we evaluate the proposed method by comparing our matching results with that generated by the graph-cut based stereo vision algorithm [25].

(a)

(b)

(c)

(d)

Fig. 5. Comparison of matching results between the graph-cut based method (GCM) and the proposed method. (a) 2D rim points in the source view (white dots); (b) Matched 2D rim points in the target view computed by GCM (green triangles) and the proposed method (white dots). (c) and (d) show a zoom-in region of (a) and (b) respectively.

We calculate the 2D rim curves on a successive view pair basis on gray scale images. A half window size of 10 pixels is used in calculating the normalized cross correlation and the parameter λ is chosen to be 0.5 in our method. For the graph-cut based method (GCM), the default parameter values are used except for the disparity range. The disparity search range is from -1 pixel to +40 pixels. As the images are large, the images are first reduced to a region containing the object with small margins before submitting to GCM. Despite so, the running time of GCM for matching one pair of size-reduced images was more than 40 minutes. In contrast, our

194

H. Zhong et al.

method took less than half a minute to match the 2D rim curves in the original images. Fig. 5 shows the matching results of two views by our method and GCM. It can be seen from Fig. 5 (b) and (d) that our method finds all correct matches but GCM does not. In the experiment, the percentage of correctly determined correspondences is about 99% by our method through visual judgment whereas that of GCM is less than 25%. Similar results are obtained for other view pairs, which demonstrate the effectiveness and accuracy of the proposed method.

(a)

(b)

(c)

Fig. 6. (a) The model after slicing the reconstructed rims using parallel planes, (b) the triangulated mesh model, (c) the textured model in VRML from a new viewpoint

After reconstructing all 3D rim curves, the surface of the object can be formed as discussed in section 5.2. The final result of our method is shown in Fig. 6 where Fig. 6 (b) presents the model in terms of triangle patches computed and Fig. 6 (c) shows the final result after texture mapping from the original image in VRML format.

7 Discussion This paper presents a new model reconstruction method from a circular motion image sequence. It combines the merits of both silhouette-based and point-based methods. By making use of the outer epipolar tangent points, a set of structured 3D rims can be rapidly reconstructed. The proposed rim structure greatly reduces the computation complexity of surface extraction and provides a flexible reconstruction. The flexibility lies in different levels of quality of the reconstruction, depending on the number of rims and slicing planes used to approximate the surface.

References 1. Sun, J., Zheng, N.N., Shum, H.Y.: Stereo Matching Using Belief Propagation. IEEE Transation on Pattern Analysis and Machine Intelligence 25(7), 787–800 (2003) 2. Sun, C.: Fast Stereo Matching Using Rectangular Subregioning and 3D Maximum-Surface Techniques. International Journal of Computer Vision 47(1/2/3), 99–117 (2002) 3. Koch, R., Pollefeys, M., Van Gool, L.: Multi viewpoint stereo from uncalibrated video sequences. In: Burkhardt, H.-J., Neumann, B. (eds.) ECCV 1998. LNCS, vol. 1406, pp. 55–71. Springer, Heidelberg (1998)

Shape Recovery from Turntable Image Sequence

195

4. Pollefeys, M., Van Gool, L.: From Images to 3D Models, Ascona, pp. 403–410 (2001) 5. Tang, W.K.: A Factorization-Based Approach to 3D Reconstruction from Multiple Uncalibrated Images. In: Department of Electrical and Electronic Engineering, p. 245. The University of Hong Kong, Hong Kong (2004) 6. Baumgart, B.G.: Geometric Modelling for Computer Vision. Standford University (1974) 7. Laurentini, A.: The visual hull concept for silhouette-based image understanding. IEEE Transactions on Pattern Analysis and Machine Intelligence 16(2), 150–162 (1994) 8. Boyer, E., Franco, J.S.: A hybrid approach for computing visual hulls of complex objects. Computer Vision and Pattern Recognition, 695–701 (2003) 9. Brand, M., Kang, K., Cooper, B.: An algebraic solution to visual hull. Computer Vision and Pattern Recognition, 30–35 (2004) 10. Liang, C., Wong, K.Y.K.: Complex 3D Shape Recovery Using a Dual-Space Approach. In: IEEE International Conference on Computer Vision and Pattern Recognition, San Diego, CA, USA, pp. 878–884. IEEE Computer Society Press, Los Alamitos (2005) 11. Martin, W.N., Aggarwal, J.K.: Volumetric descriptions of objects from multiple views. IEEE Transactions on Pattern Analysis and Machine Intelligence 5(2), 150–158 (1983) 12. Chien, C.H., Aggarwal, J.K.: Volume/surface octrees for the representation of threedimensional objects. Computer Vision, Graphics and Image Processing 36(1), 100–113 (1986) 13. Hong, T.H., Shneier, M.O.: Describing a robot’s workspace using a sequence of views from a moving camera. IEEE Transactions on Pattern Analysis and Machine Intelligence 7(6), 721–726 (1985) 14. Szeliski, R.: Rapid octree construction from image sequences. Computer Vision, Graphics and Image Processing 58(1), 23–32 (1993) 15. Lorensen, W.E., Cline, H.E.: Marching cubes: a high resolution 3D surface construction algorithm. ACM Computer Graphics 21(4), 163–169 (1987) 16. Esteban, C.H., Schmitt, F.: Silhouette and Stereo fusion for 3D object modeling. Computer Vision and Image Understanding (96), 367–392 (2004) 17. Sinha, S., Pollefeys, M.: Multi-view reconstruction using Photo-consistency and Exact silhouette constraints: A Maximum-Flow Formulation. In: IEEE International Conference on Computer Vision, pp. 349–356. IEEE, Los Alamitos (2005) 18. Furukawa, Y., Ponce, J.: Carved Visual Hulls for Image-Based Modeling. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3951, Springer, Heidelberg (2006) 19. Kutulakos, K.N., Seitz, S.M.: A theory of shape by space carving. In: International Conference on Computer Vision, Kerkyra, Greece, pp. 307–314 (1999) 20. Isidoro, J., Sclaroff, S.: Stochastic Refinement of the Visual Hull to Satisfy Photometric and Silhouette Consistency Constraints. In: IEEE International Conference on Computer Vision, pp. 1335–1342. IEEE Computer Society Press, Los Alamitos (2003) 21. Cipolla, R., Giblin, P.J.: Visual Motion of Curves and Surfaces. Cambridge Univ. Press, Cambridge, U.K. (1999) 22. Wong, K.Y.K.: Structure and Motion from Silhouettes. In: Department of Engineering, p. 196. University of Cambridge, Cambridge (2001) 23. Hartley, R., Zisserman, A.: Multiple View Geometry in Computer Vision. Cambridge University Press, Cambridge (2000) 24. Cipolla, R., Blake, A.: The dynamic analysis of apparent contours. In: International Conference on Computer Vision, Osaka, Japan, pp. 616–623 (1990) 25. Kolmogorov, V., Zabih, R.: Multi-camera Scene Reconstruction via Graph Cuts. In: Heyden, A., Sparr, G., Nielsen, M., Johansen, P. (eds.) ECCV 2002. LNCS, vol. 2350, pp. 82–96. Springer, Heidelberg (2002) 26. Franco, J.S., Boyer, E.: Exact Polyhedral Visual Hulls. In: British Conference on Computer Vision, pp. 329–338 (2003)

Shape from Contour for the Digitization of Curved Documents Fr´ed´eric Courteille, Jean-Denis Durou, and Pierre Gurdjos IRIT, Toulouse, France {courteil,durou,gurdjos}@irit.fr Abstract. We are aiming at extending the basic digital camera functionalities to the ability to simulate the flattening of a document, by virtually acting like a flatbed scanner. Typically, the document is the warped page of an opened book. The problem is stated as a computer vision problem, whose resolution involves, in particular, a 3D reconstruction technique, namely shape from contour. Assuming that a photograph is taken by a camera in arbitrary position or orientation, and that the model of the document surface is a generalized cylinder, we show how the corrections of its geometric distortions, including perspective distortion, can be achieved from a single view of the document. The performances of the proposed technique are assessed and illustrated through experiments on real images.

1

Introduction

The digitization of documents currently knows an increasing popularity, because of the expansion of Internet browsing. The traditional process, which uses a flatbed scanner, is satisfactory for flat documents, but is unsuitable for curved documents like for example a thick book, since some defects will appear in the digitized image. Several specific systems have been designed, but such systems are sometimes intrusive with regard to the documents and, before all, they cannot be referred to as consumer equipments. An alternative consists in simulating the flattening of curved documents i.e., in correcting the defects of images provided by a flatbed scanner or a digital camera. In this paper, we describe a new method of simulation of document flattening which uses one image taken from an arbitrary angle of view, and not only in frontal view (as this is often the case in the literature). The obtained results are very encouraging. In Section 2, different techniques of simulation of document flattening are reviewed. In Section 3, a new 3D-reconstruction method based on the so-called shape-from-contour technique is discussed. In Section 4, this method is applied to the flattening simulation of curved documents. Finally, Section 5 concludes our study and states several perspectives.

2

Techniques of Simulation of Document Flattening

One can address a purely 2D-deformation of the image, in order to correct its defects according to an a priori modelling of the flattened document [1,2]. In [3,4,5], Y. Yagi et al. (Eds.): ACCV 2007, Part II, LNCS 4844, pp. 196–205, 2007. c Springer-Verlag Berlin Heidelberg 2007 

Shape from Contour for the Digitization of Curved Documents

197

the characters orientation is estimated, so as to straighten them out. In all these papers, the results are of poor quality because, if the lines of text are rather well uncurved, the narrowing of the characters near the binding is not well corrected. In [6], a judicious 2D-deformation is introduced, which considers that the contour of each page becomes rectangular after flattening: the results are nice, but a “paper checkerboard pattern” must be placed behind the document to force both them having the same 3D-shape, and this makes the process rather complicated. In order to successfully simulate the document flattening, it is necessary to compute its surface shape. Stereoscopy aims at reconstructing the shape of a document from several photographs taken from different angles of view. In [7], the CPU time is very high when dealing with two images of size 2048 × 1360. On the other hand, this technique works well only if the stereo ring has been intrinsically and extrinsically calibrated. Stereophotometry requires several photographs taken from the same angle of view, but under different lightings. A modelling linking the image greylevel to the orientation of the surface is then used. It has been implemented by Cho et al. [8]. The results are of mean quality since, for photographs taken at close range, perspective should be taken into account. Structuredlighting systems make also use of two photographs taken under two different lightings, knowing that, for one of the photographs, a pattern is projected onto the document [9,10]. The deformation of the pattern in this image gives some information on the surface shape. A second photograph is required, in order to avoid possible artefacts of the pattern in the flattening simulation. The best results using this technique are presented in [11], but they make use of a dedicated imaging system. Shape-from-texture has also been used. An a priori knowledge on the document assumes that the text is printed along parallel lines, which is the case for most documents. Hence, the shape of the document surface may be deduced from the deformation of the lines of text in the image. This technique, that has been implemented by Cao et al. [12] on photographs of cylindrical books taken in frontal view, works well and quickly (some seconds on an image of size 1200 × 1600). Its crucial step consists in extracting the lines of text. In [13], it is generalized to any angles of view. Nevertheless, the latter work assumes that the lines of text are also equally spaced. The oldest contribution to the simulation of document flattening uses the shape-from-shading technique. Wada et al. [14] take advantage of the greylevel gradation in the non-inked areas of a scanned image, in order to estimate the slope of the document surface. This idea has been resumed and improved by Tan et al. [15], whose results are of good quality, and also by Courteille et al. [16]. The latter paper provides two noticeable improvements: a digital camera replaces the flatbed scanner, so as to accelerate the digitization process; a new modelling of shape-from-shading is stated, that takes perspective into account. Finally, the shape-from-contour technique may be used i.e., the deformation of the contours in the image provides information on the surface shape. This technique has been implemented in [17,6,18] on photographs of cylindrical books taken in frontal view. In [19], it is generalized to any applicable surfaces: the results are of mean quality, but this

198

F. Courteille, J.-D. Durou, and P. Gurdjos

last contribution is worth of mention, since it reformulates the problem elegantly, as a set of differential equations. The method of simulation of document flattening that we discuss in this paper uses the shape-from-contour technique to compute the shape of the document from one photograph taken from an arbitrary angle of view.

3

3D-Reconstruction Using Shape-from-Contour

In the most general situation, shape-from-contour (SFC) is an ill-posed problem: the same contours may characterize a lot of different scenes. To make the problem well-posed, it is necessary to make some assumptions. In [20], the scene is supposed to be cylindrically-symmetrical. In the present work, we suppose that its surface is a generalized cylinder having straight meridians. 3.1

Notations and Choice of the Coordinate Systems

The photographic bench is represented in Fig. 1: f refers to the focal length and C to the optical center; the axis Cz coincides with the optical axis, so that the equation of the image plane Πi is z = f . The digital camera is supposed to lie in an arbitrary position with regard to the book, apart from the fact that the optical axis must be non-parallel to the binding. Hence, the vanishing point F of the binding direction can be located at infinity, but it is separate from the principal point O. Thus, we can define the axis Ox by the straight line F O, in such a way that F O > 0. Furthermore, we complete Ox by an axis Oy such that Oxy is an orthonormal coordinate system of the image plane Πi (cf. Fig. 1). It is convenient to define two 3D orthonormal coordinate systems: Ro = Cxyz and Rp = Cuvw, where Cu coincides with the straight line F C, which is parallel to the binding, and where Cv coincides with Cy. Since Cu intersects Πi at F = O, it follows that Cw intersects Πi at a point Ω which also lies on the axis Ox. We introduce the orthonormal coordinate system Ri = Ωxy of Πi . The angle between Cz and Cw is denoted α (cf. Fig. 1). The case α = 0 corresponds to the frontal orientation of the camera. Denoting c = cos α, s = sin α and t = tan α, it can easily be stated that OΩ = t f , F O = f /t and F Ω = f /(c s). Finally, we denote Πr the plane orthogonal to Cw and containing the binding, whose equation is w = δ. 3.2

Relations Between Object and Image

Let P be an object point, whose coordinates are (X, Y, Z) w.r.t. Ro and (U, V, W ) w.r.t. Rp . The transformation rules between these two sets of coordinates are: X = c U + s W, Y = V,

(1a) (1b)

Z = −s U + c W.

(1c)

Shape from Contour for the Digitization of Curved Documents

199

F Πi Πr

y

δ

C

y

O

z

α u

α y u f x

Ω x

w

Fig. 1. Representation of the photographic bench

Let Q be the image point conjugated to P , whose coordinates are (u, v) w.r.t. Ri . Using the perspective projection rules, we obtain: f X − t f, Z f v = Y. Z Denoting f  = f /c, the equations (1a), (1b), (1c), (2a) and (2b) give: u=

U , −s U + c W V v=f . −s U + c W

u = f

(2a) (2b)

(3a) (3b)

Let us define the “pseudo-image” Q of P as the image of the orthogonal projection P of P on Πr . As the coordinates of P are (U, V, δ) w.r.t. Rp , the coordinates (u, v) of Q w.r.t. Ri are: U , −s U + c δ V . v=f −s U + c δ

u = f

(4a) (4b)

Dividing (4a) by (3a) and (4b) by (3b), we find: u v = . u v

(5)

200

F. Courteille, J.-D. Durou, and P. Gurdjos

This equality means that the image points Q, Q and Ω are aligned i.e., Ω is the vanishing point of the direction orthogonal to Πr . In a general way, the knowledge of an image point Q does not allow us to compute its conjugated object point P . But, if we also know the pseudo-image Q associated to P , then we can compute the coordinates of P . Actually, it can be deduced from (3a), (3b) and (4a): cu , f + s u v u , V =δ u f + s u u f + s u W =δ . u f + s u U =δ

(6a) (6b) (6c)

Note that (6c) gives W = δ when u = u i.e., for image points such that Q = Q. For a given image point Q, if the location of the associated pseudo-image Q is known, the coordinates (U, V, W ) of the conjugated object point P can be computed using (6a), (6b) and (6c). Nevertheless, in a general way, the location of the pseudo-image Q on the straight line ΩQ is unknown. 3.3

Additional Assumptions

Within the framework of our application, the scene is a book. We make two additional assumptions: • A1 - The flattened pages are rectangular. • A2 - The pages of the book are curved in a such way that they form a generalized cylinder. As the surface of the book is a generalized cylinder, the lower and upper contours are located in two planes which are orthogonal to the binding. Thanks to this property, the SFC problem becomes well-posed. As the binding belongs to the plane Πr , its image and its pseudo-image coincide. Under the assumptions A1 and A2 , it is easy to predict that the pseudoimage of the upper and lower contours of the book (whose images are called Cu and Cl ) are the straight lines Lu and Ll , parallel to Ωy and passing through the ends Bu and Bl of the image B of the binding (cf. Fig. 2). Let Q be an image point of coordinates (u, v) w.r.t. Ri = Ωxy. We call θ the polar angle in the coordinate system F xy, and Qu and Ql the two image points located on Cu and Cl , which have the same polar angle θ as Q (cf. Fig. 2). Considering the assumptions A1 and A2 , and knowing that F is the vanishing point of the binding direction, the object point P conjugated to Q has the same coordinate W as the two object points conjugated to Qu and Ql . We denote uu (θ) and ul (θ) the abscissas of Qu and Ql w.r.t. Ri . Finally, we denote θB the polar angle of B. According to these notations, uu (θB ) and ul (θB ) are the abscissas of

Shape from Contour for the Digitization of Curved Documents

201

y

F

Cu

Qu

Bu

Qu

Lu

B Q Q

θB

θ

Cl

Ql Bl

Ql

Ll y

Ω x

Fig. 2. Geometric construction of the pseudo-image Q associated to an image point Q

the pseudo-images Qu and Ql associated to Qu and Ql , w.r.t. Ri . Hence, if we denote f  = f  /s, the equation (6c) gives, when applied to Qu and to Ql : uu (θB ) uu (θ) ul (θB ) W =δ ul (θ)

W =δ

f  + uu (θ) , f  + uu (θB ) f  + ul (θ) . f  + ul (θB )

(7a) (7b)

From one of both these expressions of W , we can deduce the other coordinates U and V of P , solving the system of two equations (3a) and (3b). Considering the expressions (7a) and (7b) of W and the equations (3a) and (3b), it appears that the computation of the shape of the document requires the knowledge of some parameters: the focal length f , the viewing angle α and the location of the principal point O. On the other hand, the parameter δ can be chosen arbitrarily, because the shape of the document can be computed only up to a scale factor.

4

Application to the Simulation of Document Flattening

Due to lack of space, we do not show any result on synthetic images, but only on real images. The left column of Fig. 3 shows three photographs of the same

202

F. Courteille, J.-D. Durou, and P. Gurdjos

(a)

(b)

(c)

(d)

(e)

(f)

Fig. 3. Three photographs of the same book taken from three different angles of view, and the flattening simulations obtained from each of them: (a-b) α = 1.5◦ ; (c-d) α = 20.38◦ ; (e-f) α = 40.54◦

book taken from three different angles of view. For each of them, the shape of the document is computed using the method described in Section 3. The flattening simulations are shown on the right column of Fig. 3, knowing that a generalized cylinder is particularly easy to flatten. The same six images are zoomed on the area near the binding which contains a picture representing a samurai (cf. Fig. 4). It appears that when the angle of view increases, the quality of the flattening

Shape from Contour for the Digitization of Curved Documents

(a)

(c)

(e)

(b)

(d)

(f)

203

Fig. 4. Zooms on a common area of the six images of Fig. 3

simulation decreases but, even for a strong angle of view (cf. Fig. 3-e), the result remains acceptable (cf. Fig. 3-f). A second result proves that our method works well even with the most general case of camera pose i.e., when the optical axis is also tilted in the direction perpendicular to the axis of the cylinder (cf. Fig. 5).

5

Conclusion and Perspectives

In this paper, we generalize the 3D-shape reconstruction of a document from its contours, as it had previously been stated in frontal view in [17,6,18], to the case of an arbitrary view. We validate this result by simulating the flattening of curved documents taken from different angles of view. Even when the angle of view noticeably increases, the quality of the result remains rather good, in

204

F. Courteille, J.-D. Durou, and P. Gurdjos

(a)

(b)

Fig. 5. Most general case of camera pose: (a) original and (b) flattened images

comparison with other results in the literature that are obtained under similar conditions. In the present state of our knowledge, the focal length and the location of the principal point have to be known. As a first perspective, we aim at generalizing the 3D-shape reconstruction to the case of an uncalibrated camera. In addition, when the angle of view is too large, then focusing blur occurs, which inevitably restricts the quality of the flattening simulation. Rather than enduring this defect, it could be interesting to correct it, knowing that the 3D-shape of the document could allow us to predict the focusing blur magnitude.

References 1. Tang, Y.Y., Suen, C.Y.: Image Transformation Approach to Nonlinear Shape Restoration. IEEE Trans. Syst. Man and Cybern. 23(1), 155–172 (1993) 2. Weng, Y., Zhu, Q.: Nonlinear Shape Restoration for Document Images. In: Proc. IEEE Conf. Comp. Vis. and Patt. Recog., San Francisco, California, USA, pp. 568–573. IEEE Computer Society Press, Los Alamitos (1996) 3. Zhang, Z., Tan, C.L.: Recovery of Distorted Document Images from Bound Volumes. In: Proc. 6th Int. Conf. Doc. Anal. and Recog., Seattle, Washington, USA, pp. 429–433 (2001) 4. Lavialle, O., Molines, X., Angella, F., Baylou, P.: Active Contours Network to Straighten Distorted Text Lines. In: Proc. IEEE Int. Conf. Im. Proc., Thessaloniki, Greece, vol. III, pp. 748–751. IEEE Computer Society Press, Los Alamitos (2001) 5. Wu, C.H., Agam, G.: Document Image De-Warping for Text/Graphics Recognition. In: Caelli, T.M., Amin, A., Duin, R.P.W., Kamel, M.S., de Ridder, D. (eds.) SPR 2002 and SSPR 2002. LNCS, vol. 2396, pp. 348–357. Springer, Heidelberg (2002) 6. Tsoi, Y.C., Brown, M.S.: Geometric and Shading Correction for Images of Printed Materials: A Unified Approach Using Boundary. In: Proc IEEE Conf. Comp. Vis. and Patt. Recog., vol. I, pp. 240–246. IEEE Computer Society Press, Washington, D.C., USA (2004)

Shape from Contour for the Digitization of Curved Documents

205

7. Yamashita, A., Kawarago, A., Kaneko, T., Miura, K.T.: Shape Reconstr. and Image Restoration for Non-Flat Surfaces of Documents with a Stereo Vision System. In: Proc. 17th Int. Conf. Patt. Recog., Cambridge, UK, vol. I, pp. 482–485 (2004) 8. Cho, S.I., Saito, H., Ozawa, S.: A Divide-and-conquer Strategy in Shape from Shading Problem. In: Proc.IEEE Conf. Comp. Vis. and Patt. Recog., San Juan, Puerto Rico, pp. 413–419. IEEE Computer Society Press, Los Alamitos (1997) 9. Doncescu, A., Bouju, A., Quillet, V.: Former books digital processing: image warping. In: Proc. IEEE Worksh. Doc. Im. Anal., San Juan, Puerto Rico, pp. 5–9. IEEE Computer Society Press, Los Alamitos (1997) 10. Brown, M.S., Seales, W.B.: Document Restoration Using 3D Shape: A General Deskewing Algorithm for Arbitrarily Warped Documents. In: Proc. 8th IEEE Int. Conf. Comp. Vis., Vancouver, Canada, vol. I, pp. 367–375. IEEE Computer Society Press, Los Alamitos (2001) 11. Sun, M., Yang, R., Yun, L., Landon, G., Seales, W.B., Brown, M.S.: Geometric and Photometric Restoration of Distorted Documents. In: Proc. 10th IEEE Int. Conf. Comp. Vis., Beijing, China, vol. II, pp. 1117–1123. IEEE Computer Society Press, Los Alamitos (2005) 12. Cao, H., Ding, X., Liu, C.: Rectifying the Bound Document Image Captured by the Camera: A model Based Approach. In: Proc. 7th Int. Conf. Doc. Anal. and Recog., Edinburgh, UK, pp. 71–75 (2003) 13. Liang, J., DeMenthon, D., Doermann, D.: Flattening Curved Documents in Images. In: Proc. IEEE Conf. Comp. Vis. and Patt. Recog., San Diego, California, USA, vol. II, pp. 338–345. IEEE Computer Society Press, Los Alamitos (2005) 14. Wada, T., Ukida, H., Matsuyama, T.: Shape from Shading with Interreflections Under a Proximal Light Source: Distortion-Free Copying of an Unfolded Book. Int. J. Comp. Vis. 24(2), 125–135 (1997) 15. Tan, C.L., Zhang, L., Zhang, Z., Xia, T.: Restoring Warped Document Images through 3D Shape Modeling. IEEE Trans. PAMI 28(2), 195–208 (2006) 16. Courteille, F., Crouzil, A., Durou, J.D., Gurdjos, P.: Shape from Shading for the Digitization of Curved Documents. Mach. Vis. and Appl. (to appear) 17. Kashimura, M., Nakajima, T., Onda, N., Saito, H., Ozawa, S.: Practical Introduction of Image Processing Technology to Digital Archiving of Rare Books. In: Proc. Int. Conf. Sign. Proc. Appl. and Techn., Toronto, Canada, pp. 1025–1029 (1998) 18. Courteille, F., Durou, J.D., Gurdjos, P.: Transform your Digital Camera into a Flatbed Scanner. In: Proc. 9th Eur. Conf. Comp. Vis., 2nd Works. Appl. Comp. Vis., Graz, Austria, pp. 40–48 (2006) 19. Gumerov, N., Zandifar, A., Duraiswami, R., Davis, L.S.: Structure of Applicable Surfaces from Single Views. In: Pajdla, T., Matas, J(G.) (eds.) ECCV 2004. LNCS, vol. 3022, pp. 482–496. Springer, Heidelberg (2004) 20. Colombo, C., Del Bimbo, A., Pernici, F.: Metric 3D Reconstruction and Texture Acquisition of Surfaces of Revolution from a Single Uncalibrated View. IEEE Trans. PAMI 27(1), 99–114 (2005)

Improved Space Carving Method for Merging and Interpolating Multiple Range Images Using Information of Light Sources of Active Stereo Ryo Furukawa1, Tomoya Itano1 , Akihiko Morisaka1 , and Hiroshi Kawasaki2 1

Faculty of Information Sciences, Hiroshima City University, 3-4-1, Ozuka-higashi, Asaminami-ku, Hiroshima, Japan [email protected], {t itano,a morisaka}@toc.cs.hiroshima-cu.ac.jp 2 Faculty of Engineering, Saitama University, 255, Shimo-okubo, Sakura-ku, Saitama, Japan [email protected]

Abstract. To merge multiple range data obtained by range scanners, filling holes caused by unmeasured regions, the space carving method is a simple and effective method. However, this method often fails if the number of the input range images is small, because unseen voxels that are not carved out remains in the volume area. In this paper, we propose an improved algorithm of the space carving method that produces stable results. In the proposed method, a discriminant function defined on volume space is used to estimate whether each voxel is inside or outside the objects. Also, in particular case that the range images are obtained by active stereo method, the information of the positions of the light sources can be used to improve the accuracy of the results.

1 Introduction Shape data obtained by 3D measurement systems are often represented as range images. In order to obtain the entire shape model from multiple scanning results, it is necessary to merge these range images. For this task, several types of approaches have been proposed. Among them, the methods using signed distance fields have been widely researched because they are capable of processing a large volume of input data efficiently. Signed distance fields have also been used as a shape representation in order to interpolate holes appearing in unmeasured parts of object surfaces. Curless and Levoy filled holes of a measured shape by classifying each voxel in volume space as either Unseen (unseen regions), Nearsurface (near the surfaces), or Empty (outside the object), and generating a mesh between Unseen and Empty regions [1]. This process is known as space carving (SC) method. The SC method is capable of effectively interpolating missing parts when there is plenty of observed data. However, when only a few range images are captured, the SC method often fails. This problem occurs because the target object and the ”remains of carving” in volume space become connected. One of the approaches to solve this problem would be classifying the Unseen voxels as either inside or outside of the object. In this paper, object merging and the interpolation method is proposed based on this approach. Since Unseen voxels include both Y. Yagi et al. (Eds.): ACCV 2007, Part II, LNCS 4844, pp. 206–216, 2007. c Springer-Verlag Berlin Heidelberg 2007 

Improved SC Method for Merging and Interpolating Multiple Range Images

207

unobserved voxels inside an object (due to occlusion or low reflection) and voxels outside the object, it is necessary to discriminate these cases. To classify voxels, we take the following two approaches: (1) defining discriminant function for classifying Unseen voxels, and (2) using the positions of light sources if the range images are obtained using active stereo methods. Under the proposed method, the large “remains of carving” that often occurs in SC method are not generated. Also, since all voxels are classified as inside, outside or near the surface, closed shapes are always obtained. A unique property of the proposed algorithm is that it can be implemented on GPU. Recently, many methods have been proposed for utilizing the computational performance of GPUs for general calculations besides graphics. Also, our algorithm can be executed efficiently on GPU.

2 Related Work To merge multiple range data into a single shape, several types of approaches have been proposed, such as, for example, generating a mesh from unorganized points [2], using deformable model represented as a level-set [3], stitching meshes at the overlapped surfaces (zippering)[4], and methods using signed distance fields [1,5,6]. In particular, methods using signed distance fields have been widely researched since they are capable of processing a large volume of input data. In order to express the distance from a voxel to the object’s surface, the Euclidean distance (the shortest distance between the center of the voxel and the object’s surface) is theoretically preferable[5,6]. However, since computational cost of calculating the Euclidean distance is large, simplified methods such as line-of-sight distance (the distance between the voxel center and the object’s surface measured along the line of sight from the viewpoint of the measurement) are sometimes used [1] instead. Regarding filling holes of the surface of the shape defined as isosurfaces of signed distance fields, a number of methods have already been proposed. Davis et al. [7] presented a method in which signed distance field volume data is diffused using a smoothing filter. As shown in the experiments later, this method sometimes propagates the isosurface in incorrect directions and yields undesirable results (Figure 4(b)). According to the SC method proposed by Curless et al. [1], all voxels are first initialized as Unseen. Then, all voxels between the viewpoints and the object surfaces are set to Empty (this method carves the voxels in front of the observed surfaces, in contrast to the SC method used for 3D reconstruction from 2D silhouettes in which the voxels in the surrounding unobserved regions are carved). This method only carves the volume space in front of the observed surfaces, so, in practice, the unobserved voxels outside of the object remain as Unseen, and excess meshes are generated on the borders between the Unseen and the Empty regions. When range images from a sufficient number of observation points are merged, this kind of excess meshes are not connected to the target object mesh, and can be simply pruned away. However, when the number of observation points is small, or when the object is not observed from certain directions, the excess meshes often become connected to the object (Figure 4(a)). Sagawa et al. succeeded in interpolating scenes with complex objects by taking the consensus between each voxel in a signed distance field with its surrounding voxels

208

R. Furukawa et al.

Unseen

observed surface

v5

Angle of view v1

v2

v3 sensor1

sensor2

v4

Angle of view

Fig. 1. (Left)Shape interpolation using space carving, and (right) classes of voxels used in the proposed method

[8]. Masuda proposed a method for filling unmeasured regions by fitting quadrics to the gaps in the signed distance field [9]. They use a Euclidean distance for their calculation to achieve high quality interpolation at the cost of high computational complexity.

3 Shape Merging and Interpolation Using Class Estimation of Unseen Voxels 3.1 Outline A signed distance field is a scalar field defined for each voxel in 3D space such that the absolute value of the scaler is the distance between the voxel and the object surface and the sign of the scaler indicates whether the voxel is inside or outside of the object (in this paper, voxels inside the object are negative, and those outside are positive). By describing a signed distance field as D(x), it is possible to define an object’s surface as the isosurface satisfying D(x) = 0. In order to express the distance from a voxel to the object’s surface, although there exist several problems for accuracy of hole-filling process, we adopt the line-of-sight distance (the distance from the voxel center to the object’s surface measured along the line of sight from the viewpoint of the measurement) instead of the Euclidean distance, since its computational cost is relatively small. The signed distance D(v) for a voxel v is calculated by accumulating the signed distances from each of the range images, d1 (v), d2 (v), · · · , dn (v), each multiplied by a weight wi (v). It is obtained with the following formula. D(v) =



di (v)wi (v)

(1)

i

The weights represent the accuracy of each distance value, and are often decided according to the angles between the directions of the line-of-sight from the camera and the directions of the normal vectors of the surface. In the constructed signed distance field D(x), the isosurface satisfying D(x) = 0 is the merged shape. In order to generate the mesh model of the merged result, the marching cubes method [10] is used.

Improved SC Method for Merging and Interpolating Multiple Range Images

209

The SC method of Curless et al. [1] divides all the voxels in volume space into one of the three types: Unseen (not observed), Empty (outside the object) and NearSurface (near the surface of the object). Shape interpolation is then performed by generating a mesh over the border between Unseen and Empty voxels (Figure 1(left)). A large problem with this method is that all the voxels in the following three cases are classified as Unseen: (1) voxels outside the object, that do not exist on any line-of-sight to an observed region in the range images, (2) voxels outside the object that exist BEHIND a surface of a range image (occluded voxels outside the object), and (3) voxels inside the object. In the proposed method, each Unseen voxel is classified as either outside or inside of objects using a discriminant function to solve this problem. We now describe the classification of voxels according to the proposed method. For a given voxel, the information obtained from a range image takes one of the following four types ((Figure 1(right)). – NearSurface (near the surface): The absolute value of the signed distance is below the assigned threshold, the voxel in question is classified as “near the surface”, and the signed distance is retained (case of v1 in Figure 1(right)). – Outside (outside the object): The absolute value of the signed distance is larger than the threshold and the sign is positive. The voxel in question exists between the object and the camera, so the voxel can be classified as “outside the object” (v2 ). – Occluded (occluded region): The absolute value of the signed distance is larger than the threshold and the sign is negative. It is not possible to assert unconditionally whether the voxel is inside or outside the object. In this case, the classification of the voxel in question is temporarily set to Occluded. Whether the voxel is inside or outside is estimated afterwards (v3 ). – NoData (deficient data): The signed distance value cannot be obtained due to missing data in the range images. It cannot be judged whether the voxel is inside or outside. In this case, the classification of the voxel is temporarily set to NoData. Whether the voxel is inside or outside is estimated afterwards (v5 ). The case of v4 in Figure 1(right), when the voxel in question is outside the angle of view of the range image, may be handled as either Outside or NoData according to the application. For many applications, the voxels outside of the view can be treated as Outside, but in cases such as zooming and measuring a large object, they should be treated as NoData. When merging multiple range images, the information obtained from each range image regarding voxels may differ. In such cases, priority is given to NearSurface over other classes. The second priority is given to the Outside class. When the information from range images is only Occluded or NoData, it is estimated whether the voxel is inside or outside the object according to the discriminant function defined in Section 3.2. The classified voxels are tagged as Inside or Outside. By performing the above process, all the voxels are classified into the three types: Inside (inside the object), NearSurface (near the surface), and Outside (outside the object). The usual signed distances are assigned to the NearSurface voxels, and a fixed negative and positive values are assigned Inside and Outside voxels, respectively. By generating the isosurface of the constructed signed distance field, a merged polygon mesh can be obtained.

210

R. Furukawa et al.

3.2 Voxel Class Estimation If the information of a voxel obtained from all the range images is only Occluded or NoData, whether the voxel is inside or outside the object should be estimated. For this purpose, discriminant functions based on Bayes estimation are defined. We consider scalar values that are positive outside the object and negative inside the object, and we estimate the probability distributions of these values based on Bayes estimation. The subjective probability distribution when there is no data is a uniform distribution. Based on the posterior probabilities obtained from the data calculated from each range image, the subjective probability distributions are updated according to Bayes theory. Using normal distributions N (μ, σ) for the posterior distributions, a heuristics for assigning the voxel to Inside or Outside is represented as a mean value μ, and the degree of confidence in the heuristics is represented as the standard deviation σ (the higher σ, the lower the degree of confidence). When the voxel in question is Occluded in a given range image, the voxel is behind the surface from the viewpoint. When the absolute value of the distance from the surface of the voxel (expressed as Dist) is relatively small, the confidence that the voxel is inside the object is high. On the other hand, when it is far from the surface (Dist is large), the confidence that it is inside the object is low. In this case, the degree of confidence that the voxel is inside is 1/Dist, so the corresponding posterior distribution is N (−1, Dist). When the voxel in question is NoData for a given range image, it may be either an outside voxel or an unobserved voxel inside the object. For actual cases, pixels of NoData in the range images often indicate outside regions. From this heuristics, a constant value is assigned to the degree of confidence that the voxel is outside, so the posterior distribution corresponding to NoData is represented as N (1, Const), where Const is a user-defined parameter. According to experiments, reasonable results can be obtained by setting Const to a value near the smallest thickness of the object. In Bayes estimation using a normal distribution, the prior distribution N (μ, σ) is updated using the posterior distribution N (μ , σ  ) as follows. μ←

1 1 1 σ  μ + σμ ← +  , σ + σ σ σ σ

(2)

After performing this for all range images, a voxel is classified as Outside if the sign of the mean value μ of the final probability distribution is positive, and Inside if it is negative. By defining the discriminant function C(v) for voxel v as  −1/|di (v)| if Occ(i, v) , (3) ci (v) = 1/Const if N od(i, v)  ci (v), (4) C(v) = i

the above judgment is equivalent to determining the inside or outside of the object according to the sign of C(v), where Occ(i, v) and N od(i, v) mean that the information for the voxel v obtained from the range image i is Occluded or NoData, and di (v) is the signed line-of-sight distance between the voxel v and the surface of the range image i.

Improved SC Method for Merging and Interpolating Multiple Range Images

211

3.3 Utilizing the Position of Light Sources in Active Stereo Method Active stereo methods, in which the light sources are used as well as the camera, are common techniques for obtaining dense 3D shapes. In these methods, the occlusion of light sources causes missing data, but when data is measured at a point, it proves that the voxels between the point and the light source are outside the object. In this research, we utilize the position of light sources in an active stereo system by using additional range images from virtual cameras located at the light sources. For each of the measurement, we consider using two range images, from both the camera and the light source position (described below as the camera range image, and light source range image). The light source range image can be generated by projecting the 3D positions of pixels of the camera range image onto the virtual camera at the light source position. In the case shown in Figure 2(a), the data ends up missing in the camera range image since the light from the light source is occluded, therefore, at the voxel shown in Figure 2(b), the range data from the camera range image is missing. However, by referring to the light source range image, the line-of sight distance from the light source position to the voxel may be obtained. There are two advantages in using this information. First, by using light source range images, the number of voxels that can be classified as NearSurface or Outside increases. In addition, even if a voxel is not classified to these classes, we can define an improved discriminant function that utilizes more information than the one described in Section 3.2. The voxel types are mainly the same as when using the camera range images alone, but the order of priority for the voxel classifications is set to NearSurface in the camera range image, NearSurface in the light source range image, Outside in the camera range image, followed by Outside in the light source image. The discriminant function C(v) for voxel classification that utilizes the positions of the light sources is as follows. ⎧ −1 + −1 if Occc (i, v) ∧ Occl (i, v) c l ⎪ ⎪ ⎨ |di (v)| c |di (v)| if Occc (i, v) ∧ N odl (i, v) , (5) ci (v) = −1/|dil (v)| ⎪ −1/|di (v)| if N odc (i, v) ∧ Occl (i, v) ⎪ ⎩ 1/Const if N odc (i, v) ∧ N odl (i, v)  C(v) = ci (v). (6) i

The superscript symbols c and l of Occ, N od, di (v) mean the camera range images and light source range images, respectively.

4 Implementation Using a GPU The merging and interpolation algorithms in this paper can be executed efficiently by utilizing a GPU. Since GPUs can only output 2D images, the volume of the signed distance field is calculated slice by slice. The signed distance value D(v) and discriminant function value C(v) are calculated by rendering each slice in a frame buffer. Rendering is performed by multi-pass rendering using programmable shaders. Each pass performs the rendering for one range image, and the results are added using blending functions. When performing the rendering for the ith camera range image

212

R. Furukawa et al.

Missing data

Camera range image

Light source (Projector)

(a)

No data

Camera range image

d

Light source range image

(b)

Fig. 2. Using light sources of active stereo methods: (a) Missing data caused by occlusion, and (b) using light source range images

(range image i), the camera range image and the corresponding light source range image are treated as floating point textures. Then, by issuing an instruction to draw one rectangle in the whole frame buffer, the pixel-shader process for each pixel is executed. The pixel shaders calculate di (v)wi (v) and ci (v) using the range image and voxel positions as input values, and add them to the frame buffer. This process is repeated for the number of measured shapes, while changing the textures of the camera range images and the light source range images. Finally, the frame buffer holding the weighted sum of signed distances D(v), and the values of C(v) is read back into the CPU. The flags for voxel classes are checked, the signed distance and the discriminant function are combined, and a slice of the signed distance field is calculated. Since only small parts in the entire volume space are related to the isosurface, the processing cost can be reduced by first computing the signed distance field in a coarse resolution, and performing the same computation in the high resolution for only the voxels determined to be near the isosurfaces in the coarse resolution. In the experiments described in section 5, the implementation of the proposed method uses this coarse-tofine method to reduce the computational cost and time.

5 Experiments In order to demonstrate the effectiveness of the proposed method, experiments were conducted using two types of data: synthetic data formed by seven range images generated from an existing shape model (mesh data), and an actual object (an ornament in the shape of a boar) measured from 8 viewpoints with an active stereo method. For synthetic data, the points on the surface where the light is self-occluded by the object were treated as missing data even if they were visible from the camera, as occurs in measurements by an active stereo method. Each data set was merged and a mesh model was generated using the SC method, the volumetric diffusion method [7] (VD method), the method proposed by Sagawa et al. [8] (Consensus method), and the proposed method (the information regarding the light source position for active stereo was used). In the SC method, the VD method, and the proposed method, the signed distance field is calculated using line-of-sight distance.

Improved SC Method for Merging and Interpolating Multiple Range Images

213

(a)Synthetic data (b)Real data Fig. 3. Results of merging(without interpolation)

(a)SC

(b)VD

(c)Consensus

(d)Proposed(Smoothing)

Fig. 4. Results of merging(synthetic data)

In the Consensus method, however, the signed distance field is calculated using the Euclidean distance. In the VD method, the diffusion of the volume was repeated 30 times. Also, the size of the volume was 512 × 512 × 512. For efficiency, in the proposed method, the procedure is first executed with resolution of 62 × 62 × 62 and rendering with the high resolution was only performed for the region where the surface exists. We performed experiments regarding the proposed method, applying a smoothing filter with a 3 × 3 × 3 size to the volume, and also without such a filter. The filter prevents aliasing on the interpolated surface (the smoothing was not performed in the SC method or the VM method). The SC method, the VD method and the proposed method were executed on PC with an Intel Xeon (2.8GHz) CPU, and an NVIDIA GeForce8800GTX GPU installed. The Consensus method was implemented on a PC constructed with 2 Opteron 275 (2.2GHz) CPUs (a total of 4 CPUs). The results of merging with no interpolation are shown in Figure 3, and the results of the interpolation process with each method (for the proposed method, the case when smoothing was applied) are shown in Figures 4 and 5. Also, Figures 6(a)-(f) show the signed depth fields for each experiments on the synthetic data sliced at a certain zcoordinate.

214

R. Furukawa et al.

(a)SC

(b)VD

(c)Consensus

(d)Proposed(Smoothing)

Fig. 5. Results of merging(real data)

(a)No interpolation

(e)Magnified image of (a)

(b)SC

(c)VD

(f)Magnified image of (c)

(d)Proposed

(g)Discriminant function

Fig. 6. (a)-(f):Slice images of the signed distance field: For “No interpolation”, “SC” and “VD,” green color represents Unseen, black represent Empty, and gray colors represents signed distance at NearSurface regions. For the proposed method, green represents Inside, black represents Outside. (g):The discriminant function. Gray intensities mean plus values and red color means region of minus value.

Figure 4(a) and Figure 5(a) show that excess meshes were produced surrounding the target object using the SC method, since Unseen (not observed) regions remained uncarved. Figure 4(b) shows that, using the VD method, the mesh surrounding the holes beneath the ears and on the chest spread in undesirable directions. These phenomena often occurred where the borders between the NearSurface and Unseen regions (the red line in Figure 6(e)) were not parallel with the normal directions of the surface of the object as shown in Figure 6(e). In such regions, the expansion of the isosurfaces due to the filter to occur in wrong directions (in Figure 6(f) the isosurface expands downwards and to the right). A similar phenomenon also occurred in the actual data (Figure 5(b)). The Consensus method produced interpolated meshes with the best quality. However, processing time was long since the Euclidean distance was required. The proposed method produced the results whose qualities are the best next to the Consensus method. Since our method does not use signed distance field, but only uses

Improved SC Method for Merging and Interpolating Multiple Range Images

215

Table 1. Execution time in seconds. The volume size of the Consensus method for synthesized data was 128 × 128 × 128. Methods

SC VD

Consensus

Proposed No smoothing Smoothing Synthetic data 55 168 15 min. for merging, 18 sec. for interpolation 25 36 Real data 38 120 7 hours for merging, 280 sec. for interpolation 21 28

a discriminant function for Unseen voxels to fill holes, smoothness or continuity of the shapes are not considered, thus there remains a good chance to improve the quality. Using both a signed distance field and a discrimination function may be a promising way to improve our algorithm in terms of qualities of results. Figure 6(g) shows the values of the discriminant function C(v). From the figure, we can see that regions where C(v) has plus values coarsely approximate the shapes of the target objects. Table 1 is the execution times of each method. It shows that the proposed method was executed much faster than the other methods.

6 Conclusion In this paper, the space carving method was improved, and an interpolation algorithm yielding stable results even when there are few range images was proposed. This method was realized by defining a discriminant function based on Bayes estimation in order to determine the inside and outside of an object in a signed distance field, even for unseen voxels. In addition, a method was proposed for improving the accuracy of this discriminant function by using range images obtained using an active stereo method. Furthermore, a technique for implementing the proposed method using a GPU was stated, which realized a reduction in computational time by a wide margin. Finally, experiments were conducted regarding the proposed method and existing methods, and the effectiveness of the proposed method was confirmed.

References 1. Curless, B., Levoy, M.: A volumetric method for building complex models from range images. Computer Graphics(Annual Conference Series) 30, 303–312 (1996) 2. Hoppe, H., DeRose, T., Duchamp, T., McDonald, J., Stuetzle, W.: Surface reconstruction from unorganized points. In: ACM SIGGRAPH, pp. 71–78. ACM Press, New York (1992) 3. Whitaker, R.T.: A level-set approach to 3d reconstruction from range data. IJCV 29(3), 203– 231 (1998) 4. Turk, G., Levoy, M.: Zippered polygon meshes from range images. In: SIGGRAPH 1994, pp. 311–318. ACM Press, New York (1994) 5. Masuda, T.: Registration and integration of multiple range images by matching signed distance fields for object shape modeling. CVIU 87(1-3), 51–65 (2002) 6. Sagawa, R., Nishino, K., Ikeuchi, K.: Adaptively merging large-scale range data with reflectance properties. IEEE Trans. on PAMI 27(3), 392–405 (2005)

216

R. Furukawa et al.

7. Davis, J., Marschner, S.R., Garr, M., Levoy, M.: Filling holes in complex surfaces using volumetric diffusion. In: 3DPVT 2002. Proc. of International Symposium on the 3D Data Processing, Visualization, and Transmission, pp. 428–438 (2002) 8. Sagawa, R., Ikeuchi, K.: Taking consensus of signed distance field for complementing unobservable surface. In: Proc. 3DIM 2003, pp. 410–417 (2003) 9. Masuda, T.: Filling the signed distance field by fitting local quadrics. In: 3DPVT 2004. Proc. of International Symposium on the 3D Data Processing, Visualization, and Transmission, pp. 1003–1010. IEEE Computer Society Press, Washington, DC, USA (2004) 10. Lorensen, W.E., Cline, H.E.: Marching cubes: A high resolution 3d surface construction algorithm. In: SIGGRAPH 1987, pp. 163–169. ACM Press, New York, NY, USA (1987)

Shape Representation and Classification Using Boundary Radius Function Hamidreza Zaboli and Mohammad Rahmati Department of Computer Engineering, Amirkabir University of Technology, Tehran, Iran [email protected], [email protected], [email protected]

Abstract. In this paper, a new method for the problem of shape representation and classification is proposed. In this method, we define a radius function on the contour of the shape which captures for each point of the boundary, attributes of its related internal part of the shape. We call these attributes as “depth” of the point. Depths of boundary points generate a descriptor sequence which represents the shape. Matching of sequences is performed using dynamic programming method and a distance measure is acquired. At last, different classes of shapes are classified using a hierarchical clustering method and the distance measure. The proposed method can analyze features of each part of the shape locally which this leads to the ability of part analysis and insensitivity to local deformations such as articulation, occlusion and missing parts. We show high efficiency of the proposed method by evaluating it for shape matching and classification of standard shape datasets. Keywords: Computer vision, shape matching, shape classification, boundary radius function.

1 Introduction In the recent years, shape recognition and retrieval has been an important research area in the computer vision field. Various approaches and methods are proposed to deal with this problem. In spite of propounding state of the art methods in this field, there are serious drawbacks to overcome the deformations which may appear in the real applications. These deformations may appear in the form of noises, occlusions, missing parts and articulations. Different methods such as contour-based methods, region-based methods, skeleton-based methods and statistical methods have tried to overcome them in different ways [1]. Contour-based methods generally use features of the contour (boundary) of the shape without considering internal parts of the shape. This may reduces amount of computations to a process on the contour of the shape. On the other hand, regionbased methods consider the information represented by all points of the shape. Skeleton-based methods extract a rich descriptor of the shape which is called “skeleton”. Skeletons are first introduced by Blum in [2]. While the skeletons are rich descriptors of shapes, they are sensitive structures to deformations of the shape, Y. Yagi et al. (Eds.): ACCV 2007, Part II, LNCS 4844, pp. 217–226, 2007. © Springer-Verlag Berlin Heidelberg 2007

218

H. Zaboli and M. Rahmati

especially for natural objects such as human and animal [1]. Moreover, skeleton by itself and the structures and features which are extracted from it, are complex structures. These complex structures make the matching process a hard and sometimes inefficient step. Siddiqi and Sebastian at al. in [3,4] propose a shock graph method which extracts a directed acyclic graph (DAG) from the skeleton of the shape and uses a sub-graph isomorphism method at the matching stage. While the region-based methods, especially skeleton-based methods represent perfect description of the shape, extracting a simple and efficient structure from the skeleton remains as a challenge. In this work, we look for a simple and efficient structure which completely represents interior of the shape. In the skeleton-based methods, there is a function, called "radius function" [1,3]. This function perfectly captures convexities and concavities of different parts of the shape, along the skeletal lines and curves. In this paper, we define a new radius function on the boundary of the shape instead of the skeleton. By traversing the boundary of the shape, the boundary radius function (BRF) models behavior of the shape in the different parts and segments. In this way, we have a simple sequential structure which describes convexities and concavities of the shape as well as skeletal radius function. Variations of the BRF are considered as variations of the different parts of the shape which this creates the convexities and concavities of the shape. We use the variations of the BRF to represent shapes of different objects and classify them. Our method performs part analysis of the shape and it is not sensitive to articulations. Furthermore, due to the part analysis and local description of the different parts, the method is robust under high amount of occlusion and missing parts. Experiments confirm this robustness. The rest of the paper is organized as follows: In section 2, we present the related works. Section 3 presents new definition of the boundary radius function (BRF) and how to extract a descriptor of the shape using the BRF. Section 4 gives method of matching the descriptors of shapes. In Section 5, experimental results are presented and Section 6 concludes the paper.

2 Related Work Employing the variations of the shape contour has been reported widely in the previous works. These variations have been measured based on different criteria. Bernier et al. in [5] used a polar transformation of the distances between contour points and the geometric center of the shape. Hoan et al. used a similar approach in [6]. Variations of contour of the shape are measured by Euclidean distances between the contour points and a fixed reference point. The drawback is sensitivity of the reference point. Location of the reference point has a high potential to vary in the presence of small deformations, articulations, occlusion and missing parts. In fact, dependency of the descriptor to just a single and unstable reference point, makes these methods inefficient. In another work, Belongie et al. in [7] proposed a shape descriptor called "shape context" which captures for each point on the boundary of the shape, Euclidean distances and orientations of other contour points, relative to the current point. While, the descriptor is rich, the Euclidean distance and orientation between the contour

Shape Representation and Classification Using Boundary Radius Function

219

points may vary under deformations and clutter, especially articulations. Thayananthan et al. in [8] used Chamfer matching to improve robustness of the method. Another shortcoming is that the descriptor of a point depends on location of all other point on the boundary of the shape. This makes the descriptor sensitive to occlusion and missing parts which occur in the segments far from the current point. Therefore, the methods can not perform part analysis. In contrast, Siddiqi in [3] proposed a skeleton-based method with a structural descriptor, which can performs part analysis of the shape. It represents shape by a hierarchical structure called "shock graph". The method suffers from the difficult matching process and sensitivity to noises, occlusions and other deformations which appear on the different parts of the shape. Sebastian et al. in [4] have improved the method by defining a graph edit distance algorithm. In this paper, we introduce a contour descriptor using boundary radius function (BRF) which captures efficiently the internal parts of the shape. We define an edit distance algorithm for matching the descriptors and we perform shape recognition and classification using the resulted distance.

3 Boundary Radius Function In the skeleton-based methods, radius function on the skeleton of the shape is defined as follows: Assume S denotes the skeleton of a shape and consider point p where p ∈ S . Radius function R(p) is the radius of the maximal inscribed disc touches the boundary of the shape. With respect to this definition, we define the boundary radius function(BRF) on the boundary of the shape and we show how to calculate values of this function on the boundary of a shape. We define BRF as follows: Definition 1: Assume B denotes the boundary of a shape and consider point p where p ∈ B . Boundary radius function BRF(p) for point p, is defined as radius of the minimal disk centered at p, which touches nearest point q on the boundary of the shape, such that the line segment which connects point p to q lies entirely within the shape and is perpendicular to a ε -neighborhood of q on the boundary. Fig. 1 shows this definition.

(a)

(b)

(c)

(d)

Fig. 1. (a): A horse and BRF(p) for different points on the boundary. (b): line segment pq is perpendicular to the boundary segment in the point q. (c): BRF for a non smooth shape. (d): new BRF (definition 2) for point p on the rear part of the horse shape.

220

H. Zaboli and M. Rahmati

BRF for three sample point is shown in Fig. 1(a). In Fig. 1(b) line segment pq which is radius of the circle, is perpendicular to the boundary at point q. This verticality can be proved for any smooth closed curve. Although boundary of a shape is a closed curve, it can be uneven with sharp corners. In Fig. 1(c). an example of an uneven boundary with sharp corners is shown. As seen in this figure, the boundary at point q has a corner and is not smooth. This causes that the boundary at point q not to be differentiable. Therefore, there is no tangent line on the boundary at point q. As a result in such cases, Definition 1 may not true because there is no point q on the boundary which its tangent is perpendicular to the radius line pq. In fact, Definition 1 is true for smooth closed curves not for any closed shape boundary. Thus we give a new definition of BRF which is less formal but more applicable. Definition 2: Assume B denotes the boundary of a shape and consider point p where p ∈ B . Moreover, assume minimal circle C centered at p which touches boundary B at two different points b1 , b2. By increasing radius of the circle C until it touches a third boundary point q, such that line segment pq lies entirely within the shape, then boundary radius function BRF(p) for point p, is radius of the circle C. Note that in the above definition, radius of circle C is equal to length of the line segment pq. In Fig. 1(c). an example of definition 2 is illustrated. In this figure, by increasing radius of the circle, it touches the boundary at third point q and line segment pq lies entirely within the shape. Fig 1(d). shows another example of BRF with definition 2. In this figure, BRF(p) is not the line segment pa because the line segment pa is not entirely within the shape. While pq is the right line segment for BRF(p). Definition 2 is a useful and applicable definition for BRF in any closed shape. Therefore, we use it for calculating BRF on the boundary of shapes. BRF can analyze and capture related internal part of the shape for any boundary point. This provides enough information for local analysis of parts which is useful for part analysis, articulations, occlusion and missing parts. Given a sample shape, it is straightforward to compute values of the BRF for boundary points of the shape. Fig. 2(a) shows two sample shapes and values of the BRF for their boundary points. These values are shown as a set of transformed points for each sample shape, such that each point s with loci s(x, y) on the boundary of the sample shape is transformed into point s′ with loci s ′( x, y + BRF ( s )) .

(a)

(b)

Fig. 2. (a),(b): Two sample shapes, their boundaries and their transformed point set at their rightwards. Note that the X and Y values increase rightwards and downwards, respectively.

Shape Representation and Classification Using Boundary Radius Function

221

As seen in Fig. 2. the parts with high “depth” in their fronts, have high values of related BRFs. This causes that these parts locate at down sections (i.e. high values of y in their loci (x,y)) of the transformed points. Up to this point, we have for any given shape a transformed point set which reflects for each boundary point, attribute and feature of the related internal parts. This feature is like depth of the shape for the point which its BRF is computed. Although in some cases, BRF may not gives the real and accurate related depth for each boundary point, but it can be considered as a feature similar to depth. Moreover, values of BRF for different points of the boundary accurately represent shapes and it is not necessary to have a real depth measure for describing the shape. Depth-like measure i.e. BRF, represents for each segment of the shape, a view of corresponding segment of the boundary in front of the segment which includes convex and concave parts of the shape. Note that different shapes are created by these convexities and concavities in the different parts of shapes.

4 Extracting Structure from the BRFs In this section we intend to extract a suitable structure from the boundary points such that the structure contains BRF measures. Since the boundary points of a shape generate contour of the shape and the contour has a sequential structure, we use a vector structure containing values of BRFs to represent a shape. Order of the boundary points is kept and the relative BRF value is assigned to each one. In this way, we have a directed attributed vector for any shape to represent it. We call this vector as boundary radius vector (BRV). Therefore, matching of shapes can be performed by matching their related BRVs. One thing to mention is about length of the vectors. It is obvious that bigger shapes have longer boundaries. As a result, length of their BRV will be longer than small shapes. This can cause some problems especially in the presence of some transformations like scale. To overcome this, for matching of any two given shapes, we consider n points on their boundaries for both of them, such that the n points divide boundary of each shape into n equal segments. Thus, we have for each shape, a BRV of length n. Note that n is fixed in a matching process. After computing BRV for a shape, variations of the BRF values in the BRV should be considered. As seen in Fig. 2(a)., in the boundary of the shape, what is creates two convex parts in the two ends of the shape, is variations of the BRF values. In this figure, from point a to b, fixed values of BRF create a flat part between these points. Also, from point b to c and then c to d, increase and decrease of BRF values, respectively, create a bowed curve. Thus, it is necessary to analyze variations of BRF values for a shape, instead of values themselves. Although BRF values are important, their variations are more important. To achieve this goal, we define a first order differential on the BRF values of BRV. This definition is given in below: Definition 3: Assuming BRV of a shape with length n, variations of BRF for ith value in BRV(ith point on contour) is denoted by VBRF(i) and computed as follows:

222

H. Zaboli and M. Rahmati i +α

VBRF (i) =

∑ (BRF (k ) − BRF (k − 1))

k = i −α +1

2α

, α +1 ≤ i ≤ n − α −1

(1)

where, α is a parameter of resolution. Bigger α lead to less accuracy for analyzing details of parts. It is obvious that very little values of α may not leads to superior description of the shape and may not leads to superior efficiency and recognition rate of final recognition system. Using definition 3, we can compute variations of BRF (VBRF) for any given shape. Therefore, we have a sequence of values of VBRFs for any given shape. Similarly, this sequence specifies a vector containing values of VBRF, calculated for a shape. We call this final vector as “descriptor vector”. Note that for each value of the descriptor vector(VBRF(i)), we take into account its related BRF value(BRF(i)) as a weight factor for it. In this way, huge parts of the shape which are deeper than other parts, are considered more important. This will raise recognition rate of our method, especially in the presence of occlusion and missing parts.

5 Matching Descriptor Vectors Up to this point, we have introduced a descriptor vector for any given shape. For matching, classifying and recognizing of shapes, it is sufficient to match the descriptor vectors and compute difference between them. The resulted difference between the shapes is calculated as a distance between the shapes and can be used as a distance measure. Due to the sequential structure of the descriptor vectors, we use a dynamic programming method for matching the vectors. This method is used widely for matching attributed and sequential data structures e.g. strings. Moreover, this method has a high efficiency to deal with deleted and inserted parts in the sequential structures and it can find optimal matching. This ability is very useful for dealing with some deformations which occur to shapes such as occlusion and missing parts. Dynamic programming compares and matches the vectors and generates a cost for the matching. This cost is cost of transforming one of the vectors into the other one and is known as “edit distance”. Resulted edit distance is considered as distance between the two shapes. To use dynamic programming method, we define below cost functions: Cost(i, j) = min(cost(i, j − 1),cost(i − 1, j),cost(i − 1, j − 1) + ⎧⎪ BRF ( i ) − BRF ( j ) ⎨ ⎪⎩ (BRF ( i ) × VBRF ( i )) − (BRF ( j ) × VBRF ( j ))

if VBRF ( i ) = VBRF ( j ) ⎫⎪ ⎬ if VBRF ( i ) ≠ VBRF ( j )⎪⎭

(2)

In relation 2, Cost(i,j) is distance between values 1..i of first vector and values 1..j of the second vector. For a complete matching, i must be equal to the number of values in the first vector and j must be equal to the number of values in the second vector. As seen in this relation, variations of boundary radius function(VBRF), are considered as a basic for matching any two values in the descriptor vectors. After that, values of the related BRFs are taken into account in a next level.

Shape Representation and Classification Using Boundary Radius Function

223

Using the cost function in relation 2, we may compute edit distance between any two descriptor vectors. As a result, it is straightforward to calculate distance between any two given shapes. Therefore, given any two shapes S1 , S2 the process of matching of S1 , S2 can be performed as follows: 1. 2. 3.

Traversing on the boundaries of S1 , S2 and computing BRF for boundary points of S1 , S2 and generating BRV1 , BRV2 , respectively. Computing variations of BRFs and transforming BRV1 , BRV2 into descriptor vectors. Matching descriptor vectors using dynamic programming.

Assuming BRF is computed for n boundary points, the first stage takes time complexity of O(nk2), where k2 is complexity of computing BRF and k is related radius of a boundary point. In the worst case, k is not greater than n/2. The worst case occurs when the shape is like an ellipse. Thus, the first stage takes O(n3/4)=O(n3). Time complexity of the second stage is O(n) and at the third stage, dynamic programming takes O(nm) which n and m are length of the descriptor vectors. Therefore, assuming n ≥ m , overall time complexity of the method in the worst case is O(n3+n+nm) which is still O(n3).

6 Experiments The distance measure computed in section 5 can be used for shape matching and retrieval. We evaluated our method using two standard shape datasets and many customized sample shapes. The datasets are kimia[11] and MPEG-7[12] binary image databases. Moreover we designed and applied a new set of customized and deformed shapes and we performed experiments on this shape dataset. Note that in our experiments, number of sample boundary points (n), discussed in Section 4, is equal to 100. Also, resolution parameter α is considered less than or equal to 3. First experiment is comparison and matching of different shape from different classes. In this experiment, a set of randomly selected shapes from the datasets is considered and distance between each pair of them is computed using the proposed method. Evaluated shapes contain various deformed shapes such as occluded shapes, shapes with articulation and missing parts. Table 1. shows this experiment for 13 sample shapes out of about 100 evaluated shapes. Each value in the table, indicates distance between the shapes at relative row and column. As seen in the table, distances between the shapes which belong to a similar class, are closer than other distances. For example, the horse at column 10 has distance equal to 0.581 with the other horse at row 9 and distance 0.789 with the dog at row 3. Also each shape in the table has distance equal to 0 with itself (e.g. row10, column 10). In the next experiment, we classify shapes of different classes from the datasets based on the distance between the shapes. To achieve this goal, we use a hierarchical clustering method (single linkage).

224

H. Zaboli and M. Rahmati Table 1. 13 out of 100 sample evaluated shapes and distance between each pair of them

0

0.757 1.105 1.37 1.486 0.879 1.740 1.301 1.436 0

1.232 1.419 1.670 0.909 1.787 1.306 1.341 0

1.582 1.838 1.117 1.664 1.784 0.709 0

0.301 1.207 1.366 1.341 1.589 0

1.717 1.587 1.505 1.724 0

1.785 1.377 1.360 0

0.591 1.596 0

1.403 1.466 1.117 1.125 1.366 1.477 1.147 1.198 0.789 1.695 1.183 1.030 1.649 1.445 1.262 1.342 1.785 1.584 1.307 1.328 1.321 1.370 1.224 1.203 1.633 1.973 0.936 0.929

1.669

1.754 1.875 0.936 0.866

0

0.581 1.709 1.168 1.009 0

1.591 1.230 1.083 0

0.627 0.724 0

0.431 0

This method categorizes similar shapes into a class based on the fact that shapes of a class have minimal distances between themselves and have more distances with other classes of shapes. Using this method, we classified 340 shapes of different classes and reached to great results. Fig. 3 shows the results of this classification. Our method measured distances between each pair of shapes of different classes and generated a distance table. Then, clustering method categorized the shapes based on the distance table. As seen in Fig. 3. our method correctly has classified the shapes and has linked similar classes first. In this classification, first classes of horse and deer are linked. Also classes of rabbit and mouse are linked and after that, classes of pliers and scissors and classes of dummies and planes are linked, respectively. Also the dog class is linked to classes of horse and deer which is a correct classification. The classification is continued and similar classes are linked until the last class which class of leaves. This class is linked to the other classes at the last level due to its huge difference with the other classes. In the last experiment, we evaluate our method against the methods proposed in [5,7]. The first method, proposed in [5] is a close method to ours. Both of the two methods capture shape by traversing on the contour and based on its interior.

Shape Representation and Classification Using Boundary Radius Function

225

Fig. 3. Classification of different classes of shapes

The difference between them is that the proposed method in [5] considers all parts of the shape relative to a reference point, while our method in most cases considers each part based on a local analysis. Next method which is called “shape context”, proposed in [7] is a contour-based method which saves a profile for each boundary point. Each profile contains distances and orientations of other boundary points relative to the current point. As mentioned earlier, the method suffers from high dependency of its descriptor to deformations and variations of other parts of the boundary. In this experiment, we evaluate our method for retrieving most similar and closest matches. We perform this retrieval test on 530 randomly selected shapes from the kimia and MPEG-7 datasets. A query shape is selected from the dataset and 10 closest matches are retrieved. Results of these retrievals are shown in Table 2 and are compared with results of [5] and [7]. As shown in the table, Our method performs superior and achieves great results. While polar signature[5] and shape context[7] almost lose efficient retrieval after 5th Table 2. Result of retrieving of 530 shapes by our method and [5,7] and percent of recognition rate of the three methods for top 10 matches

1st

2nd

3rd

4th

5th

6th

7th

8th

9th

10th

100

100

99

99

98

97

97

97

94

84

Polar signature[5]

100

100

97

96

91

82

78

72

57

39

Shape Context[7]

97

91

88

85

84

77

75

66

56

37

Method Boundary Radius Function (BRF)

226

H. Zaboli and M. Rahmati

closest match, our method has recognition rate higher that 90 % for top 9 closest matches.

7 Conclusions We proposed a new descriptor which captures interior of a shape by walking along its contour. Result of this stage is a vector which represents the shape based on the interior of the shape. Matching of the vectors is performed using dynamic programming method. Since the proposed method and its descriptor, analyze locally each part of the shape, the method is robust under complex and problematic deformations, especially articulation, occlusion, missing parts and clutters. Moreover, matching process of the method i.e. dynamic programming, is an efficient and simple matching method. However, the matching process can be done using other methods such as computing the area lying between VBRFs.

References 1. Zhang, D., Lu, G.: Review of shape representation and description techniques. Pattern Regocnition 37, 1–19 (2004) 2. Blum, H.: A Transformation for extracting new descriptors of Shape. In: Whaten-Dunn, W. (ed.) Models for the perception of Speetch and Visual Forms, pp. 362–380. MIT Press, Cambridge (1967) 3. Siddiqi, K., Shokoufandehs, A., Dickinsons, S.J., Zucker, S.W.: Shock Graphs and Shape Matching. International Journal of Computer Vision 35(1), 13–32 (1999) 4. Sebastian, T.B., Klein, P.N., Kimia, B.B.: Recognition of Shapes by Editing Shock Graphs. In: ICCV 2001, pp. 755–762 (2001) 5. Bernier, T., Landry, J.-A.: A New Method for Representing and Matching Shapes of Natural Objects. Pattern Recognition 36, 1711–1723 (2003) 6. Kang, S.K., Ahmad, M.B., Chun, J.H., Kim, P.K., Park, J.A.: Modified Radius-Vector Function for Shape Contour Description. In: Laganà, A., Gavrilova, M., Kumar, V., Mun, Y., Tan, C.J.K., Gervasi, O. (eds.) ICCSA 2004. LNCS, vol. 3046, pp. 940–947. Springer, Heidelberg (2004) 7. Belongie, S., Malik, J., Puzicha, J.: Shape Matching and Object Recognition using Shape Contexts. IEEE Transactions on Pattern Analysis and Machine Intelligence 24(24), 509– 522 (2002) 8. Thayananthan, A., Stenger, B., Torr, P.H.S., Cipolla, R.: Shape Context and Chamfer Matching in Cluttered Scenes. IEEE CVPR 1, 127–133 (2003) 9. Gorelick, L., Galun, M., Sharon, E., Basri, R., Brandt, A.: Shape Representation and Classification Using the Poisson Equation. IEEE Transaction on Pattern Recognition and Machine Intelligence 28(12), 1991–2005 (2005) 10. Navarro, G.: A guided tour to approximate string matching. ACM Computing Surveys (CSUR) 33(1), 31–88 (2000) 11. Kimia Image Database (May 2007), Available at http://www.lems.brown.edu/ dmc/main.html 12. MPEG7 CE Shape Database (May 2007), Available at http://www.imageprocessingplace.com/ DIP/dip_image_dabases/image_databases.htm

A Convex Programming Approach to the Trace Quotient Problem Chunhua Shen1,2,3 , Hongdong Li1,2 , and Michael J. Brooks3 1

2

NICTA Australian National University 3 University of Adelaide

Abstract. The trace quotient problem arises in many applications in pattern classification and computer vision, e.g., manifold learning, low-dimension embedding, etc. The task is to solve a optimization problem involving maximizing the ratio of two traces, i.e., maxW Tr(f (W ))/Tr(h(W )). This optimization problem itself is non-convex in general, hence it is hard to solve it directly. Conventionally, the trace quotient objective function is replaced by a much simpler
quotient trace formula, i.e., maxW Tr h(W )−1 f (W ) , which accommodates a much simpler solution. However, the result is no longer optimal for the original problem setting, and some desirable properties of the original problem are lost. In this paper we proposed a new formulation for solving the trace quotient problem directly. We reformulate the original non-convex problem such that it can be solved by efficiently solving a sequence of semidefinite feasibility problems. The solution is therefore globally optimal. Besides global optimality, our algorithm naturally generates orthonormal projection matrix. Moreover it relaxes the restriction of linear discriminant analysis that the projection matrix’s rank can only be at most c − 1, where c is the number of classes. Our approach is more flexible. Experiments show the advantages of the proposed algorithm.

1 Introduction The problem of dimensionality reduction—extracting low dimensional structure from high dimensional data—is extensively studied in pattern recognition, computer vision and machine learning. Many of the dimensionality reduction methods, such as linear discriminant analysis (LDA) and its kernel version, end up with solving a trace quotient problem Tr W Sα W , (1) W  = argmax  W W =Id Tr W Sβ W where Sα , Sβ are positive semidefinite (p.s.d.) matrices (Sα  0, Sβ  0), Id the d × d identity matrix1 and Tr(·) denoting the matrix trace. W ∈ RD×d is target projection matrix for dimensionality reduction (typically d  D). In the supervised learning framework, usually Sα represents the distance of different classes while Sβ is the distance between data points in the same class. For example, Sα is the “between classes scatter matrix” and Sβ is the “within classes scatter matrix” for LDA. By formulating 1

The dimension of I is omitted when it can be seen from the context.

Y. Yagi et al. (Eds.): ACCV 2007, Part II, LNCS 4844, pp. 227–235, 2007. c Springer-Verlag Berlin Heidelberg 2007 

228

C. Shen, H. Li, and M.J. Brooks

the problem of dimensionality reduction in a general setting and constructing Sα and Sβ in different ways, we can analyze many different types of data in the above underlying mathematical framework. Despite the importance of the trace quotient problem, it lacks a direct and globally optimal solution. Usually, as an approximation, the quotient trace −1

Tr (W Sα W )

W Sβ W ,

(instead of the trace quotient) is used. Such an approximation readily leads to a generalized eigen-decomposition (GEVD) solution, via which a close-form solution is readily available. It is easy to check that when rank(W ) = 1, i.e., W is a vector, then Equation (1) is actually a Rayleigh quotient problem, which can be solved by the GEVD. The eigenvector corresponding to the eigenvalue of largest magnitude gives the optimal W  . Unfortunately, when rank(W ) > 1, the problem becomes much more complicated. Heuristically, the dominant eigenvectors corresponding to the largest eigenvalues are used to form the W  . It is believed that the largest eigenvalues contains more useful information. However such a GEVD approach cannot produce the optimal solution to the original optimization problem (1) [1]. Furthermore, the GEVD does not yield an orthogonal projection matrix. Orthogonal LDA (OLDA) is proposed to compute a set of orthogonal discriminant vectors via the simultaneous diagonalisation of the scatter matrices [2]. In this paper, we proffer a novel semidefinite programming (SDP) based method to solve the trace quotient problem directly, which has the following properties: – It optimises the original problem (Equation (1)) directly; – The target low dimensionality is selected by the user and the algorithm guarantees an globally optimal solution since the optimisation is convex. In other words, it is local-optima-free; – The projection matrix is orthonormal naturally; – Unlike the GEVD approach to LDA, the data are not necessary to be projected to at most c − 1 dimensions with our algorithm solving LDA. c is the number of classes. To our knowledge, this is the first attempt which directly solves the trace quotient problem and at the same time, a global optimum is deterministically guaranteed.

2 SDP Approach to the Trace Quotient Problem In this section, we show how the trace quotient is reformulated into an SDP problem. 2.1 SDP Formulation By introducing an auxiliary variable δ, the problem (1) is equivalent to maximize subject to

(2a)

δ 



Tr W Sα W ≥ δ · Tr W Sβ W

(2b)

A Convex Programming Approach to the Trace Quotient Problem

229

W W = Id

(2c)

W ∈R

(2d)

D×d

The variables we want to optimise here are δ and W . But we are only interested in W with which the value of δ is maximised. This problem is clearly not convex because the constraint (2b) is not convex and (2d) is actually a non-convex rank constraint. Let us define a new variable Z ∈ RD×D , Z = W W , and now the constraint (2b) is converted to Tr(Sα − δSβ )Z ≥ 0 under the fact that Tr W SW = Tr SW W = Tr SZ. Because Z is a product of matrix W and its transpose, it must be p.s.d. Overton and Womersley [3] have shown that the set of Ω1 = {W W : W W = Id } is the set of extreme points of Ω2 = {Z : Z = Z , Tr Z = d, 0  Z  I}.2 That means, as constraints, Ω1 is more strict than Ω2 . Therefore constraints (2c) amd (2d) can be relaxed into Tr Z = d and 0  Z  I, which are both convex. When the cost function is linear and it is subject to Ω2 , the solution will be at one of the extreme points [4]. Consequently, for linear cost functions, the optimization problems subject to Ω1 and Ω2 are exactly equivalent. Moreover, the same nice property follow even when the objective function is a quotient (i.e. fractional programming), which is precisely the case we are dealing with here. With respect to Z and δ, (2b) is still non-convex: the problem may have locally optimal points. But still the global optimum can be efficiently computed via a sequence of convex feasibility problems. By observing that the constraint is linear if δ is known, we can convert the optimization problem into a set of convex feasibility problems. A bisection search strategy is adopted to find the optimal δ. This technique is widely used in fractional programming. Let δ  denote the unknown optimal value of the cost function. Given δ∗ ∈ R, if the convex feasibility problem3 find Z subject to Tr(Sα − δ∗ Sβ )Z ≥ 0

(3a) (3b)

Tr Z = d 0ZI

(3c) (3d)

is feasible, then we have δ  ≥ δ∗ . Otherwise, if the above problem is infeasible, then we can conclude δ  < δ∗ . This way we can check whether the optimal value δ  is smaller or larger than a given value δ∗ . This observation motivates a simple algorithm for solving the fractional optimisation problems using bisection search, which solves a convex feasibility problem at each step. Algorithm 1 shows how it works. At this point, a question remains to be answered: are constraints (3c) and (3d) equivalent to constraints (2c) and (2d) for the feasibility problem? Essentially the feasibility problem is equivalent to maximize 2 3

Tr(Sα − δ∗ Sβ )Z

(4a)

Our notation is used here. The feasibility problem has no cost function. The objective is to check whether the intersection of the convex constraints is empty.

230

C. Shen, H. Li, and M.J. Brooks

Algorithm 1. Bisection search Require: δL : Lower bounds of δ; δU : Upper bound of δ and the tolerance σ > 0. while δU − δL > σ do U . δ = δL +δ 2 Solve the convex feasibility problem described in (3a)–(3d). if feasible then δL = δ; else δU = δ. end if end while

subject to

Tr Z = d 0ZI

(4b) (4c)

If the maximum value of the cost function is non-negative, then the feasibility problem is feasible. Conversely, it is infeasible. Because this cost function is linear, we know that Ω1 can be replaced by Ω2 , i.e., constraints (3c) and (3d) are equivalent to (2c) and (2d). Note that constraint (3d) is not in the standard form of SDP. It can be rewritten into the standard form as   Z 0  0, (5a) 0Q Z + Q = I,

(5b)

where the matrix Q acts as a slack variable. Now the problem can be solved using standard SDP packages such as CSDP [5] and SeDuMi [6]. We use CSDP in all of our experiments. 2.2 Recovering W from Z From the convariance matrix Z learned by SDP, we can recover the output W by eigen-decomposition. Let Vi denote the ith eigenvector, with eigenvalue λi . Let λ1 ≥ λ2 ≥ √· · · ≥ the sorted eigenvalues. It is straightforward to see that W = √ λD be √ diag( λ1 , λ2 , · · · , λD )V  , where diag(·) is a square matrix with the input as its diagonal elements. To obtain a D × d projection matrix, the smallest D − d eigenvalues are simply truncated. This is the general treatment for recovering a low dimensional projection from a covariance matrix, e.g., principal component analysis (PCA). In our case, this procedure is precise, i.e., there is no information loss. This is obvious: λi , the eigenvalues of Z = W W , are the same as the eigenvalues of W W = Id . That means, λ1 = λ2 = · · · = λd = 1 and the left D − d eigenvalues are all zeros. Hence in our case we can simply stack the first d leading eigenvectors to obtain W .

A Convex Programming Approach to the Trace Quotient Problem

231

2.3 Estimating Bounds of δ The bisection search procedure requires a low bound and an upper bound of δ. The following theorem from [3] is useful for estimating the bounds. Theorem 1. Let S ∈ RD×D be a symmetric matrix, and μS1 ≥ μS2 ≥ · · · ≥ μSD be the sorted eigenvalues of S from largest to smallest, then max Tr W SW = W W =Id d S i=1 μi . Refer to [3] for the proof. This theorem can be extended to obtain the following corollary (following the proof for Theorem 1): S Corollary 1. Let S ∈ RD×D be a symmetric matrix, and ν1S ≤ ν2S ≤ · · · ≤ νD be its  d  S sorted eigenvalues from smallest to largest, then min Tr W SW = i=1 νi . W W =Id

Therefore, we estimate the upper bound of δ: d μSi α δU = i=1 . Sβ d i=1 νi

(6)

In the trace quotient problem, both Sα and Sβ are p.s.d. This is equivalent to say, all of their eigenvalues are non-negative. Recall that the denominator of (6) could be zeros and δU = +∞. This occurs when the d smallest eigenvalues of Sβ are all zeros. In this case, rank(Sβ ) ≤ D − d. For LDA, rank(Sβ ) = min(D, N ). Here N is the number of training data. When N ≤ D − d, which is termed the small sample problem, δU is invalid. A PCA data prep-processing can always be performed to remove the null space of the covariance matrix of the data, such that δU becomes valid. A lower bound of δ is then d νiSα δL = i=1 . (7) Sβ d i=1 μi Clearly δL ≥ 0.

3 Related Work The closest work to ours is [1] in the sense that it also proposes a method to solve the trace quotient directly. [1] finds the projection matrix W in the Grassmann manifold. Compared with optimization in the Euclidean space, the main advantage of optimization on the Grassman manifold is fewer variables. Thus the scale of the problem is smaller. There are major differences between [1] and our method: (i) [1] optimises Tr W Sα W − δ · Tr W Sβ W and they have no a principled way to determine the optimal value of δ. In contrast, we optimize the trace quotient function itself and a deterministic bisection search guarantees the optimal δ; (ii) The optimization in [1] is non-convex (difference of two quadratic functions). Therefore it might become trapped into a local maximum, while our method is globally optimal.

232

C. Shen, H. Li, and M.J. Brooks

Xing et al. [7] propose a convex programming approach to maximize the distances between classes and simultaneously to clip (but not to minimis) the distances within classes. Unlike our method, in [7] the rank constraint is not considered. Hence [7] is metric learning but not necessary a dimensionality reduction method. Furthermore, although the formulation of Xing et al. is convex, it is not SDP. It is more computationally expensive and general-purpose SDP solvers are not applicable. SDP (or other convex programming) is also used in [8,9] for learning a distance metric.

4 Experiments In this work, we consider optimizing the LDA criterion using the proposed SDP approach. Sα is the“between classes scatter matrix” and Sβ is the “within classes scatter matrix”. However, there are many different ways of constructing Sα and Sβ , e.g., the general methods considered in [7]. UCI data Firstly, we test whether the optimal value of the cost function, δ  obtained by our SDP bisection search, is indeed larger than the one obtained by GEVD (conventional LDA). In all the experiments, the tolerance σ = 0.1. In this experiment, two datasets (“iris” and “wine”) from UCI machine learning repository [10] are used. We randomly sample 70% of the data each time and run 100 tests for the two datasets. The target low dimension d is set to 2. Figure 1 plots the difference of δ  obtained by two approaches. Indeed, SDP consistently yields larger δ  than LDA does. To see the difference between these two approaches, we project the “wine” data into 2D space and plot the projected data in Figure 2. As can be seen, the SDP algorithm has successfully brought together the points in the same class, while keeping dissimilar ones apart (for the original data and the PCA projected data, different classes are entangled together.) While both of the two discriminant projection algorithms can separately the data successfully, SDP intentionally finds a projection of the data onto a straight line that maintains the separation of the clusters. Xing et al. [7] have reported very similar observations with their convex metric learning algorithm. To test the influence of the SDP algorithm for classification, again we randomly select 70% of the data for training and the left 30% for testing. Both of the two data are projected to 2D. Results with a kNN classifier (k = 1) are collected. We run 100 tests for each data set. For the “iris” data, SDP is slightly better (test error 3.71%(±2.74%)) than LDA (test error 4.16%(±2.56%)). While for the “wine” data, LDA is better than SDP (1.53%(±1.62%) against 8.45%(±4.30%)). It means that a larger LDA cost δ  does not necessarily produce better classification accuracy because: (i) The LDA criterion is not directly connected with the classifier’s performance. It is somewhat a heuristic criterion; (ii) As an example, Figure 2 indicates that SDP might over-fit the training data. When LDA already well separates the data, SDP aligns the data into a line for larger δ; (iii) With noisy training data, LDA can denoise by truncating the eigenvectors corresponding to the smaller eigenvalues, similar to what PCA does. During the learning of SDP, it takes noises into consideration as well, which appears to be harmful. We believe that some regularization to the LDA criterion would be beneficial. Also, other different criteria might perform differently in terms of over-fitting. These topics remain future research directions.

A Convex Programming Approach to the Trace Quotient Problem

233

650 600

δSDP − δLDA

550 500 450 400 350 300 0

20

40 60 Test # (UCI data: iris)

80

100

40 60 80 Test # (UCI data: wine)

100

250

LDA

100

δ

−δ

150

SDP

200

50

0 0

20

Fig. 1. The optimal value of δ obtained by our SDP approach (δSDP ) minus the value obtained by the conventional LDA (δLDA ). For all the runs, δSDP is larger than δLDA .

USPS handwritten digits data Experiments are also conducted on the full USPS data set. The US Postal (USPS) handwritten digit data-set is derived from a project on recognizing handwritten digits on envelopes. The digits were down sampled to 16 × 16 pixels. The training set has 7291 samples, and the test set has 2007 samples. The test set is rather difficult: the error rate achieved by human is 2.5% [11]. In the first experiment, we only use 7291 training digits. 70% are randomly selected for training and the other 30% for testing. The data are linearly mapped from 256D to 55D using PCA such that 90.14% of the energy is preserved. Then for LDA, we map them to 9D (because they are totally ten classes). SDP’s target low dimension is 50D. We run the experiments 20 times. The results are somewhat surprising. The 1NN classification (i.e, nearest neighbor) test error for LDA is 6.99%± 0.49%. SDP achieves much better performance: a 1NN test error 2.79%±0.27%. Note that if we set the target low dimension to 9D for SDP, SDP performs worse than LDA does. In the second experiment, we use 7291 training data for training and USPS’ 2007 test data for testing. Again at first they are mapped to 55D using PCA. LDA reduces

234

C. Shen, H. Li, and M.J. Brooks Original (wine)

PCA (wine)

6

40

5

20

4 0 3 −20 2 −40

1 0 11

12

13 LDA (wine)

−60 15 −1000

14

1.5

−0.4

1

−0.6

0.5

−0.8

0

−1

−0.5

−1.2

−1

−1.4

−1.5

−1.6

−2 −3

−2

−1

0

1

2

−1.8 −7

−500

−6

−5

0 SDP (wine)

−4

−3

500

−2

−1

Fig. 2. (1) Original data (first two dimensions are plotted); (2) Projected to 2D with PCA; (3) Projected to 2D with LDA; (4) Projected to 2D with SDP

the dimensionality to 9D and SDP to 54D. LDA has a 1NN test error 10.36% while our SDP achieves a 5.13% test error. Note that in these experiments we have not tuned all the parameters carefully.

5 Conclusion We have proposed a new formulation for directly solving the trace quotient problem. It is based on SDP, combining with a bisection search approach for solving a fractional programming, which allow us to derive a guaranteed globally optimal algorithm. Compared with LDA, the algorithm also relaxes the restriction of linear discriminant analysis that the projection matrix’s rank can only be at most c − 1. In the USPS classification experiment, it shows that this restriction might significantly affect the LDA’s performance. Our experiments have validated the advantages of the proposed algorithm.

Acknowledgements National ICT Australia (NICTA) is funded through the Australian Government’s Backing Australia’s Ability initiative, in part through the Australian Research Council.

A Convex Programming Approach to the Trace Quotient Problem

235

References 1. Yan, S., Tang, X.: Trace quotient problems revisited. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3951, pp. 232–244. Springer, Heidelberg (2006) 2. Ye, J., Xiong, T.: Null space versus orthogonal linear discriminant analysis. In: Proc. Int. Conf. Mach. Learn., Pittsburgh, Pennsylvania, pp. 1073–1080 (2006) 3. Overton, M.L., Womersley, R.S.: On the sum of the largest eigenvalues of a symmetric matrix. SIAM J. Matrix Anal. Appl. 13(1), 41–45 (1992) 4. Overton, M.L., Womersley, R.S.: Optimality conditions and duality theory for minimizing sums of the largest eigenvalues of symmetric matrices. Math. Program 62, 321–357 (1993) 5. Borchers, B.: CSDP, a C library for semidefinite programming. Optim. Methods and Software 11, 613–623 (1999) 6. Sturm, J.F.: Using SeDuMi 1.02, a matlab toolbox for optimization over symmetric cones (updated for version 1.05). Optim. Methods and Software 11-12, 625–653 (1999) 7. Xing, E., Ng, A., Jordan, M., Russell, S.: Distance metric learning, with application to clustering with side-information. In: Proc. Adv. Neural Inf. Process. Syst., MIT Press, Cambridge (2002) 8. Weinberger, K.Q., Blitzer, J., Saul, L.K.: Distance metric learning for large margin nearest neighbor classification. In: Proc. Adv. Neural Inf. Process. Syst. (2005) 9. Globerson, A., Roweis, S.: Metric learning by collapsing classes. In: Proc. Adv. Neural Inf. Process. Syst. (2005) 10. Newman, D., Hettich, S., Blake, C., Merz, C.: UCI repository of machine learning databases (1998) 11. Simard, P., LeCun, Y., Denker, J.S.: Efficient pattern recognition using a new transformation distance. In: Proc. Adv. Neural Inf. Process. Syst., pp. 50–58. MIT Press, Cambridge (1993)

Learning a Fast Emulator of a Binary Decision Process ˇ Jan Sochman and Jiˇr´ı Matas Center for Machine Perception, Dept. of Cybernetics, Faculty of Elec. Eng. Czech Technical University in Prague, Karlovo n´ am. 13, 121 35 Prague, Czech Rep. {sochmj1,matas}@cmp.felk.cvut.cz Abstract. Computation time is an important performance characteristic of computer vision algorithms. This paper shows how existing (slow) binary-valued decision algorithms can be approximated by a trained WaldBoost classifier, which minimises the decision time while guaranteeing predefined approximation precision. The core idea is to take an existing algorithm as a black box performing some useful binary decision task and to train the WaldBoost classifier as its emulator. Two interest point detectors, Hessian-Laplace and Kadir-Brady saliency detector, are emulated to demonstrate the approach. The experiments show similar repeatability and matching score of the original and emulated algorithms while achieving a 70-fold speed-up for KadirBrady detector.

1

Introduction

Computation time is an important performance characteristic of computer vision algorithms. We show how existing (slow) binary-valued classifiers (detectors) can be approximated by a trained WaldBoost detector [1], which minimises the decision time while guaranteeing predefined approximation precision. The main idea is to look at an existing algorithm as a black box performing some useful binary decision task and to train a sequential classifier to emulate its behaviour. We show how two interest point detectors, Hessian-Laplace [2] and KadirBrady [3] saliency detector, can be emulated by a sequential WaldBoost classifier [1]. However, the approach is very general and is applicable in other areas as well (e.g. texture analysis, edge detection). The main advantage of the approach is that instead of spending man-months on optimising and finding a fast and still precise enough approximation to the original algorithm (which can be sometimes very difficult for humans), the main effort is put into finding a suitable set of features which are then automatically combined into a WaldBoost ensemble. Another motivation could be an automatic speedup of a slow implementation of one’s own detector. A classical approach to optimisation of time-to-decision is to speed-up an already working approach. This includes heuristic code optimisations (e.g. FastSIFT [4] or SURF [5]) but also very profound change of architecture (e.g. classifier cascade [6]). A less common way is to formalise the problem and try to solve the error/time trade-off in a single optimisation task. Y. Yagi et al. (Eds.): ACCV 2007, Part II, LNCS 4844, pp. 236–245, 2007. c Springer-Verlag Berlin Heidelberg 2007 

Learning a Fast Emulator of a Binary Decision Process

"Black Box"

Vision Algorithm

(t)

Training Set samples binary output

WaldBoost Learning

training sample?

(t)

f (x), θA , θB

Emulator

data request

labels

237

Wald decision Bootstrap management

images

images Image pool

Fig. 1. The proposed learning scheme

Our contribution is a proposal of a general framework for speeding up existing algorithms by a sequential classifier learned by the WaldBoost algorithm. Two examples of interest point detectors were selected to demonstrate the approach. The experiments show a significant speed-up of the emulated algorithms while achieving comparable detection characteristics. There has been much work on the interest point detection problem [7] but to our knowledge, learning techniques has been applied only to subproblems but not to the interest point detection as a whole. Lepetit and Fua [8] treated matching of detected points of interest as a classification problem, learning the descriptor. Rosten and Drummond [9] used learning techniques to find parameters of a hand-designed tree-based Harris corner classifier. Their motivation was to speed-up the detection process, but the approach is limited to the Harris corner detection. Martin et al. [10] learned a classifier for edge detection, but without considering the decision time and with significant manual tuning. Nevertheless, they tested a number of classifier types and concluded that a boosted classifier was comparable in performance to these classifiers and was preferable for its low model complexity and low computational cost. The rest of the paper is structured as follows. The approximation of a blackbox binary valued decision algorithm by a WaldBoost classifier is discussed in §2. Application of the approach to interest point detectors is described in §3. Experiments are given in §4 and the paper is concluded in §5.

2

Emulating a Binary-Valued Black Box Algorithm with WaldBoost

The structure of the approach is shown in Figure 1. The black box algorithm is any binary-valued decision algorithm. Its positive and negative outputs form a labelled training set. The WaldBoost learning algorithm builds a classifier sequentially and when new training samples are needed, it bootstraps the training set by running the black box algorithm on new images. Only the samples not decided yet by the so far trained classifier are used for training. The result of the

238

ˇ J. Sochman and J. Matas

process is a WaldBoost sequential classifier which emulates the original black box algorithm. The bootstrapping loop uses the fact that the black box algorithm can provide practically unlimited number of training data. This is in contrast to commonly used human labelled data which are difficult to obtain. Next, a brief overview of the WaldBoost learning algorithm is presented. 2.1

WaldBoost

WaldBoost [1] is a greedy learning algorithm which finds a quasi-optimal sequential strategy for a given binary-valued decision problem. WaldBoost finds a sequential strategy S ∗ such that S ∗ = arg min T¯S S

subject to βS ≤ β,

αS ≤ α

(1)

for specified α and β. T¯S is average time-to-decision, αS is false negative and βS false positive rate of the sequential strategy S. A sequential strategy is any algorithm (in our case a classifier) which evaluates one measurement at a time. Based on the set of measurements obtained up to that time, it either decides for one of the classes or postpones the decision. In the latter case, the decision process continues by taking another measurement. To find the optimal sequential strategy S ∗ , the WaldBoost algorithm combines the AdaBoost algorithm [11] for feature (measurement) selection and Wald’s sequential probability ratio test (SPRT) [12] for finding the thresholds which are used for making the decisions. The input of the algorithm is a labelled training set of positive and negative samples, a set of features F - the building blocks of the classifier, and the bounds on the final false negative rate α and the false positive rate β. The output is an ordered set of weak classifiers h(t) , t ∈ {1, . . . , T } each one corresponding to one (t) (t) feature and a set of thresholds θA , θB on the response of the strong classifier for all lengths t. During the evaluation of the classifier on a new observation x, one weak classifier is evaluated at time t and its response is added to the response function t  h(q) (x). (2) ft (x) = q=1

The response function ft is then compared to the corresponding thresholds and the sample is either classified as positive or negative, or the next weak classifier is evaluated and the process continues ⎧ (t) ⎪ +1, ft (x) ≥ θB ⎨ (t) (3) Ht (x) = −1, ft (x) ≤ θA ⎪ ⎩ (t) (t) continue, θA < ft (x) < θB . If a sample x is not classified even after evaluation of the last weak classifier, a threshold γ is imposed on the real-valued response fT (x).

Learning a Fast Emulator of a Binary Decision Process

239

Early decisions made in classifier evaluation during training also affect the training set. Whenever a part of the training set is removed according to eq. 3, new training samples are collected (bootstrapped) from yet unseen images. In the experiments we use the same asymmetric version of WaldBoost as used in [1]. When setting the β parameter to zero, the strategy becomes  (t) −1, ft (x) ≤ θA (4) Ht (x) = (t) continue, θA < ft (x) and only decisions for the negative class are made during the sequential evaluation of the classifier. A (rare) positive decision can only be reached after evaluating all T classifiers in the ensemble. In the context of fast black box algorithm emulation, what distinguishes training for different algorithms is the feature set F . A suitable set has to be found for every algorithm. Hence, instead of optimising the algorithm itself, the main burden of development lies in finding a proper set F . The set F can be very large if one is not sure which features are the best. The WaldBoost algorithm selects a suitable subset together with optimising the time-to-decision.

3

Emulated Scale Invariant Interest Point Detectors

In order to demonstrate the approach, two similarity invariant interest point detectors have been chosen to be emulated: (i) Hessian-Laplace [2] detector, which is a state of the art similarity invariant detector, and (ii) Kadir-Brady [3] saliency detector, which has been found valuable for categorisation, but is about 100× slower. Binaries of both detectors are publicly available1 . We follow standard test protocols for evaluation as described in [7]. Both detectors are similarity invariant (not affine), which is easily implemented via a scanning window over positions and scales plus a sequential test. For both detectors, the set F contains the Haar-like features proposed by Viola and Jones [6], plus a centre-surround feature from [13], which has been shown to be useful for blob-like structure detectors [4]. Haar-like features were chosen for their high evaluation speed (due to integral image representation) and since they have a potential to emulate the Hessian-Laplace detections [4]. For the Kadir-Brady saliency detector emulation, however, the Haar-like features turned out not to be able to emulate the entropy based detections. To overcome this, and still keep the efficiency high, “energy” features based on the integral images of squared intensities were introduced. They represent intensity variance in a given rectangle. To collect positive and negative samples for training, a corresponding detector is run on a set of images of various sizes and content. The considered detectors assign a scale to each detected point. Square patches of the size twice the scale are used as positive samples. The negative samples representing the “background” 1

http://www.robots.ox.ac.uk/~vgg/research/affine/

240

ˇ J. Sochman and J. Matas o

1 dc R

r

r R

dc

R

r+R

d

Fig. 2. The non-maximum suppression algorithm scheme for two detections

class are collected from the same images at positions and scales not covered by positive samples. Setting α.There is no error-free classification, the positive and negative classes are highly overlapping in feature space. As a consequence, the WaldBoost classifier responses on many positions and scales – false positives. One way of removing less reliable detections is to threshold the final response function fT at some higher value γ. This would lead to less false positives, more false negatives and very slow classifier (whole classifier evaluated for most samples). A better option is to set α to a higher value and let the training to prune the negative class sequentially. Again, it results in less false positives and controllable amount of false negatives. Additionally, the classifier becomes much faster due to early decisions. An essential part of a detector is the non-maximum suppression algorithm. Here the output differs from that obtained from the original detectors. Instead of having a real-valued map over whole image, sparse responses are returned by the WaldBoost detector due to early decisions – value of ft , t < T available for early decisions is not comparable to fT of positive detections. Thus a typical cubic interpolation and a local maximum search cannot be applied. Instead, the following algorithm is used. Any two detections are grouped together if their overlap is higher than a given threshold (parameter of the application). Only the detection with maximal fT in each group is preserved. The overlap computation is schematically shown in Figure 2. Each detection is represented by a circle inscribed to the box (scanning window) reported as a detection (Figure 2, left). For two such circles, let us denote radius of the smaller circle as r and radius of the bigger one as R. The distance of circle centres will be denoted by d. The following approximation to the actual circles overlap is used to avoid computationally demanding goniometric functions. The measure has an easy interpretation in two cases. First, when the circle centres coincide, the overlap is approximated as r/R. It equals to one for two circles of the same radius and decreases as the radiuses become different. Second, when two circles have just one point in common (d = r + R), the overlap is zero. These two situations are marked in Figure 2, right by blue dots. Linear interpolation (blue solid line in Figure 2, right) is used to approximate the overlap between these two states. Given two radiuses r and R where r ≤ R and circle centres distance dc , the overlap o is computed as

Learning a Fast Emulator of a Binary Decision Process

r o= R

4

 1−

dc r+R

241

.

Experiments

This section describes experiments with two WaldBoost-emulated detectors Hessian-Laplace [2] and Kadir-Brady [3] saliency detector. The Hessian-Laplace detector is expected to be easily emulated due to its blob-like detections. This allows to keep the first experiment more transparent. The Kadir-Brady detector is more complex one due to its entropy based detections. Kadir-Brady detector shows rather poor results in classical repeatability tests [7] but has been successfully used in several recognition tasks [14]. However, its main weakness for practical applications is its very long computation time (in order of minutes per image!). 4.1

Hessian-Laplace Emulation

The training set for the WaldBoost emulation of Hessian-Laplace is created from 36 images of various sizes and content (nature, urban environment, hand drawn, etc.) as described in §3. The Hessian-Laplace detector is used with threshold 1000 to generate the training set. The same threshold is used throughout all the experiments for both learning and evaluation. Training has been run for T = 20 (training steps) with α = 0.2 and β = 0. The higher α allows fast pruning of less trustworthy detections during sequential evaluation of the detector. The detector has been assessed in standard tests proposed by Mikolajczyk et al. [7]. First, repeatability of the trained WaldBoost detector has been compared with the original Hessian-Laplace detector on several image sequences with variations in scale and rotation. The results on two selected sequences, boat and east south, from [15] are shown in Figure 3 (top row). The WaldBoost detector achieves similar repeatability as the original Hessian-Laplace detector. In order to test the trained detectors for their applicability, a matching application scenario is used. To that effect, a slightly different definition of matching score is used than that of Mikolajczyk [7]. Matching score as defined in [7] is computed as the number of correct matches divided by the smaller number of correspondences in common part of the two images. However, the matches are computed only pairwise for correspondences determined by the geometry ground truth. Here, the same definition of the matching score is used, but the definition of a correct match differs. First, tentative matches using the SIFT detector are computed and mutually nearest matches are found. These matches are then verified by the geometry ground truth and only the verified matches are called correct. Comparison of the trainer and the trainee outputs on two sequences is given in Figure 3 (bottom row). The WaldBoost detector achieves similar matching score on both sequences while producing consistently more detections and matches.

ˇ J. Sochman and J. Matas

20 0 1

1.5 2 scale change

3000 2000 1000 0 1

2.5

(a) 80 60 40 20 0 1

1.5 2 scale change

(e)

60 40 20 0 1

2.5

(b) WB HL

2.5

4000 #correct matches

matching score

100

1.5 2 scale change

80

2000 1000 0 1

1.5 2 scale change

(f)

1.5 2 scale change

3000 2000 1000 0 1

2.5

2.5

100

60 40 20 0 1

1.5 2 scale change

(g)

1.5 2 scale change

2.5

(d) WB HL

80

WB HL

4000

(c) WB HL

3000

WB HL

#correspondences

40

5000

100 repeatability %

60

WB HL

4000

matching score

repeatability %

WB HL

80

2.5

4000 #correct matches

5000

100

#correspondences

242

WB HL

3000 2000 1000 0 1

1.5 2 scale change

2.5

(h)

Fig. 3. Comparison of Hessian-Laplace detector and its WaldBoost emulation. Top row: Repeatability on boat (a) and east south (c) sequences and corresponding number of detections (b), (d). Bottom row: Matching score (e), (g) and corresponding number of correct matches (f), (h) on the same sequences.

Fig. 4. First centre-surround and energy feature found in WaldBoost Hessian-Laplace (left) and Kadir-Brady (right) emulated detector. The underlying image is generated as E(|xi − 127.5|) and E(xi ) respectively, where E() is the average operator and xi is the i-th positive training example.

The WaldBoost classifier evaluates on average 2.5 features per examined position and scale. This is much less than any reported speed for face detection [1]. The evaluation times are compared in Table 1. The WaldBoost emulation speed is comparable to manually tuned Hessian-Laplace detector. The Hessian-Laplace detector finds blob-like structures. The structure of the trained WaldBoost emulation should reflect this property. As shown in Figure 4, the first selected feature is of a centre-surround type which gives high responses to blob-like structures. The outputs of the trained WaldBoost emulation of Hessian-Laplace and the original algorithm are compared in Figure 5. To find the original Hessian-Laplace detection correctly found by the WaldBoost emulator, correspondences based on Mikolajczyk’s overlap criterion [7] have been found between the original and WaldBoost detections. The white circles show repeated correspondences. The black circles show the detections not found by the WaldBoost emulation. Note that most of the missed detections have a correct detection nearby, so the

Learning a Fast Emulator of a Binary Decision Process

(a)

243

(b)

Fig. 5. Comparison of the outputs of the original and WaldBoost-emulated (a) HessianLaplace and (b) Kadir-Brady saliency detectors. The white circles show repeated detection. The black circles highlight the original detections not found by the WaldBoost detector. Note that for most of missed detections there is a nearby detection on the same image structure. The accuracy of the emulation is 85 % for Hessian-Laplace and 96 % for Kadir-Brady saliency detector. Note that the publicly available Kadir-Brady algorithm does not detect points close to image edges.

corresponding image structure is actually found. The percentage of repeated detections of the original algorithm is 85 %. To conclude, the WaldBoost emulator of the Hessian-Laplace detector is able to detect points with similar repeatability and matching score while its speed is comparable to speed of the original algorithm. This indicates that the proposed approach is able to minimise the decision time down to a manually tuned algorithm speed. 4.2

Fast Saliency Detector

The emulation of the Kadir-Brady saliency detector [3] was trained on the same set of images as the WaldBoost Hessian-Laplace emulator. The saliency threshold of the original detector was set to 2 to limit the positive examples only to those with higher saliency. Note, that as opposed to the Hessian-Laplace emulation where rather low threshold was chosen, it is meaningful to use only the top most salient features from the Kadir-Brady detector. This is not true for HessianLaplace detector since its response does not correspond to the importance of the feature. The Haar-like feature set was extended by the “energy” feature described in §3. The training was run for T = 20 (training steps) with α = 0.2 and β = 0. The same experiments as for the Hessian-Laplace detector have been performed. The repeatability and the matching score of the Kadir-Brady detector and its WaldBoost emulation on boat and east south sequences are shown in Figure 6. The trained detector performs slightly better than the original one.

ˇ J. Sochman and J. Matas

20 0 1

1.5 2 scale change

2000 1000 0 1

2.5

(a) 80 60 40 20 0 1

1.5 2 scale change

(e)

60 40 20

1.5 2 scale change

0 1

2.5

1.5 2 scale change

(b) WB KB

2.5

2500 #correct matches

matching score

100

WB KB

80

1500 1000 500 0 1

1.5 2 scale change

2.5

100

2000 1500 1000 500 0 1

2.5

60 40 20 0 1

1.5 2 scale change

(f)

1.5 2 scale change

2.5

(d) WB KB

80

WB KB

2500

(c) WB KB

2000

#correspondences

40

3000

100

2000 #correct matches

60

WB KB

3000

repeatability %

repeatability %

WB KB

80

#correspondences

4000

100

matching score

244

WB KB

1500 1000 500

2.5

(g)

0 1

1.5 2 scale change

2.5

(h)

Fig. 6. Comparison of Kadir-Brady detector and its WaldBoost emulation. Top row: Repeatability on boat (a) and east south (c) sequences and corresponding number of detections (b), (d). Bottom row: Matching score (e), (g) and corresponding number of correct matches (f), (h) on the same sequences. Table 1. Speed comparison on the first image (850×680) from the boat sequence

Hessian-Laplace Kadir-Brady

original 1.3s 1m 44s

WaldBoost 1.3s 1.4s

The main advantage of the emulated saliency detector is its speed. The classifier evaluates on average 3.7 features per examined position and scale. Table 1 shows that the emulated detector is 70× faster than the original detector. Our early experiments showed that the Haar-like features are not suitable to emulate the entropy-based saliency detector. With the energy features, the training was able to converge to a reasonable classifier. In fact, the energy feature is chosen for the first weak classifier in the WaldBoost ensemble (see Figure 4). The outputs of the WaldBoost saliency detector and the original algorithm are compared in Figure 5. The coverage of the original detections is 96 %. To conclude, the Kadir-Brady emulation gives slightly better repeatability and matching score. But, most importantly, the decision times of the emulated detector are about 70× lower than that of the original algorithm. That opens new possibilities for using the Kadir-Brady detector in time sensitive applications.

5

Conclusions and Future Work

In this paper a general learning framework for speeding up existing binary-valued decision algorithms by a sequential classifier learned by WaldBoost algorithm has been proposed. Two interest point detectors, Hessian-Laplace and Kadir-Brady saliency detector, have been used as black box algorithms and emulated by the

Learning a Fast Emulator of a Binary Decision Process

245

WaldBoost algorithm. The experiments show similar repeatability and matching scores of the original and emulated algorithms. The speed of the Hessian-Laplace emulator is comparable to the original manually tuned algorithm, while the Kadir-Brady detector was speeded up seventy times. The proposed approach is general and can be applied to other algorithms as well. For future research, an interesting extension of the proposed approach would be to train an emulator with not only similar outputs to an existing algorithm but also with some additional quality like higher repeatability or specialisation to a given task.

Acknowledgement The authors were supported by Czech Science Foundation Project 102/07/1317 ˇ (JM) and by EC project FP6-IST-027113 eTRIMS (JS).

References ˇ 1. Sochman, J., Matas, J.: WaldBoost - learning for time constrained sequential detection. In: CVPR, Los Alamitos, USA, vol. 2, pp. 150–157 (2005) 2. Mikolajczyk, K., Schmid, C.: Scale and affine invariant interest point detectors. IJCV 60(1), 63–86 (2004) 3. Kadir, T., Brady, M.: Saliency, scale and image description. IJCV 45(2) (2001) 4. Grabner, M., Grabner, H., Bischof, H.: Fast approximated SIFT. In: ACCV, pp. I:918–927 (2006) 5. Bay, H., Tuytelaars, T., Van Gool, L.: SURF: Speeded up robust features. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3951, Springer, Heidelberg (2006) 6. Viola, P., Jones, M.: Robust real time object detection. In: SCTV, Vancouver, Canada (2001) 7. Mikolajczyk, K., Tuytelaars, T., Schmid, C., Zisserman, A., Matas, J., Schaffalitzky, F., Kadir, T., Van Gool, L.: A comparison of affine region detectors. In: IJCV (2005) 8. Lepetit, V., Lagger, P., Fua, P.: Randomized trees for real-time keypoint recognition. In: CVPR, vol. II, pp. 775–781 (2005) 9. Rosten, E., Drummond, T.: Machine learning for high-speed corner detection. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3951, pp. 430– 443. Springer, Heidelberg (2006) 10. Martin, D., Fowlkes, C., Malik, J.: Learning to detect natural image boundaries using local brightness, color, and texture cues. PAMI 26(5), 530–549 (2004) 11. Schapire, R.E., Singer, Y.: Improved boosting algorithms using confidence-rated predictions. Machine Learning 37(3), 297–336 (1999) 12. Wald, A.: Sequential analysis. Dover, New York (1947) 13. Lienhart, R., Maydt, J.: An extended set of haar-like features for rapid object detection. In: ICIP (2002) 14. Fergus, R., Perona, P., Zisserman, A.: A sparse object category model for efficient learning and exhaustive recognition. In: CVPR, vol. 1, pp. 380–387 (2005) 15. Mikolajczyk, K.: Detection of local features invariant to affines transformations. PhD thesis, INPG, Grenoble (2002)

Multiplexed Illumination for Measuring BRDF Using an Ellipsoidal Mirror and a Projector Yasuhiro Mukaigawa, Kohei Sumino, and Yasushi Yagi The Institute of Scientific and Industrial Research, Osaka University

Abstract. Measuring a bidirectional reflectance distribution function (BRDF) requires long time because a target object must be illuminated from all incident angles and the reflected light must be measured from all reflected angles. A high-speed method is presented to measure BRDFs using an ellipsoidal mirror and a projector. The method can change incident angles without a mechanical drive. Moreover, it is shown that the dynamic range of the measured BRDF can be significantly increased by multiplexed illumination based on the Hadamard matrix.

1

Introduction

In recent years, the measurement of geometric information (3D shapes) has become easier by using commercial range-finders. However, the measurement of photometric information (reflectance properties) is still difficult. Reflection properties depend on the microscopic shape of the surface, and they can be used for a variety of applications such as computer graphics and inspection of painted surfaces. However, there is no standard way for measuring reflection properties. The main reason for this is that the dense measurement of BRDFs requires huge amounts of time because a target object must be illuminated from every incident angle and the reflected lights must be measured from every reflected angle. Most existing methods use mechanical drives to rotate a light source, and as a result, the measuring time becomes very long. In this paper, we present a new method to measure BRDFs rapidly. Our system substitutes an ellipsoidal mirror for a mechanical drive, and a projector for a light source. Since our system completely excludes mechanical drive, high-speed measurement can be realized. Moreover, we present an algorithm that improves the dynamic range of measured BRDFs. The combination of a projector and an ellipsoidal mirror can produce any illumination. Hence, the dynamic range is significantly increased by multiplexing illumination based on the Hadamard matrix, while the capturing time remains the same as for normal illumination.

2

Related Work

If the reflectance is uniform over the surface, the measurement becomes easier by merging the BRDFs at every point. Matusik et al.[1] measured isotropic BRDFs by capturing a sphere. Anisotropic BRDFs can also be measured by Y. Yagi et al. (Eds.): ACCV 2007, Part II, LNCS 4844, pp. 246–257, 2007. c Springer-Verlag Berlin Heidelberg 2007 

Multiplexed Illumination for Measuring BRDF

247

Table 1. Comparison of major BRDF measuring devices Device Camera Light source Density of BRDF Li [5] mechanical rotation mechanical rotation dense Dana [10] fixed mechanical translation dense M¨ uller [6], Han [12] fixed fixed sparse Our system fixed fixed dense

capturing a plane [2] or a cylinder[3]. Marschner et al.[4] measured the BRDFs of general shapes using a 3D range sensor. However, these methods cannot measure specially varying BRDFs. The most straightforward way to measure BRDFs is to use a gonioreflectometer, which allows a light source and a sensor to rotate around the target material. Li et al.[5] have proposed a three-axis instrument. However, because the angle needs to be altered mechanically, the measurement of dense BRDFs takes a long time. To speed up the measurement, rotational mechanisms should be excluded. By placing many cameras and light sources around the target object, BRDFs can be measured for some of the angle combinations of the incident and reflective directions. M¨ uller et al.[6] have constructed a system including 151 cameras with a flash. However, dense measurement is physically difficult. In the optics field, some systems that utilize catadioptric devices have been proposed. Davis and Rawling[7] have patented a system using an ellipsoidal mirror to collect the reflected light. Mattison et al.[8] have developed a handheld instrument based on the patent. The patent focuses only on gathering reflected light, and does not mention the control of the incident direction. Although Ward[9] used a hemispherical half-mirror and a fish-eye lens, it requires a rotational mechanism for the light source. Although Dana[10] used a paraboloidal mirror, a translational mechanism for the light source remains necessary. To avoid using mechanical drive, some systems include catadioptric devices. Kuthirummal et al.[11] used a cylindrical mirror and Han et al.[12] combined a projector and several plane mirrors similar to those used in a kaleidoscope. However, these systems can measure only sparse BRDFs because measurable incident and reflective angles are quite discrete. We, on the other hand, propose a new system that combines an ellipsoidal mirror and a projector. Since our system completely excludes a mechanical device, high-speed measurement is realized. The system can measure dense BRDFs because both lighting direction and viewing direction are densely changed.

3 3.1

BRDF Isotropic and Anisotropic BRDFs

To represent reflection properties, a BRDF is used. The BRDF represents the ratio of outgoing radiance in the viewing direction (θr , φr ) to incident irradiance from a lighting direction (θi , φi ), as shown in Fig.2(a).

248

Y. Mukaigawa, K. Sumino, and Y. Yagi

When a camera and a light source are fixed, the rotation of an object around the surface normal changes the appearance of some materials. Such reflection is called anisotropic reflection, and typical materials of this type are brushed metals and cloth fabrics such as velvet and satin. To perfectly describe anisotropic reflection, the BRDF should be defined by four angle parameters. On the other hand, the appearance does not change according to the rotation around the surface normal for many materials. Such reflection is called isotropic reflection. If isotropic reflection can be assumed, the BRDF can be described using only three parameters, θi , θr , and φ (φ = φi − φr ). If the number of parameters can be reduced from four to three, then the measuring time and data size can be significantly reduced. 3.2

Problems with a 4-Parameter Description

There are two major problems associated with a 4-parameter description: data size and measuring time. First, let us consider the data size. If the angles θr , φr , θi , and φi are rotated at one degree intervals, and the reflected light is recorded as R, G, and B colors for each angle, then the required data size becomes 360×90×360×90×3 = 3, 149, 280, 000 bytes. The size of 3GB is not impractical for recent PCs. Moreover, BRDFs can be effectively compressed because they include much redundancy. Therefore, data size is not a serious problem. On the other hand, the problem of measuring time remains serious. Since the number of combinations of lighting and sensing angles becomes extremely large, a long measuring time is required. If the sampling interval is one degree, the total number of combinations becomes 360 × 90 × 360 × 90 = 1, 049, 760, 000. This means that it would require 33 years if it takes one second to measure one reflection color. Of course the time can be shortened by using a high-speed camera, but the total time required would still remain impractical. While the problem of data size is not serious, the problem of measuring time warrants consideration. In this paper, we straightforwardly tackle the problem of measuring time for 4-parameter BRDFs by devising a catadioptric system.

4 4.1

BRDF Measuring System Principle of Measurement

An ellipsoid has two focal points. All rays from one focal point reflect off the ellipsoidal mirror and reach the other focal point. This property is used for measuring BRDFs. The target object is placed at the focal point, and a camera is placed at the other focal point. Since rays reflected in all directions from the target object converge at a single point, all the rays can be captured at once. The most significant characteristic of the system is that an ellipsoidal mirror is combined with a projector. The projector serves as a substitute for the light source. The projection of a pattern in which only one pixel is white corresponds to the illumination by a point light source. Moreover, changing the location of the white pixel corresponds to rotating the incident angle. Since the change

Multiplexed Illumination for Measuring BRDF

249

Object Mirror

Black Plate

Ellipsoidal Mirror

N

θr

Image Plane

Ellipsoidal Mirror

Half Mirror φr

θi

φi

Beam Splitter

Projector

Camera

Image Plane Camera

Object

Projector

(a) Vertical setup

(b) Horizontal setup

y

y

x

z

z

(c) Mirror for vertical setup

x

(d) Mirror for horizontal setup

Fig. 1. The design of the BRDF measuring device

of the projection pattern is faster than mechanical rotation, rapid and dense measurement can be achieved. 4.2

Design of the Measuring System

Based on the principle described in the previous section, we developed two BRDF measuring devises that have differently shaped ellipsoidal mirrors. One is a vertical setup in which a target material is placed vertically to the long axis as shown in Fig.1(a). The shape of the mirror is an ellipsoid that is cut perpendicularly to the long axis as shown in Fig.1(c). The other is a horizontal setup in which a target material is placed horizontally to the long axis as shown in Fig.1(b). The shape of the mirror is an ellipsoid that is cut parallel and perpendicularly to the long axis as shown in Fig.1(d). The major optical devices are a projector, a digital camera, an ellipsoidal mirror, and a beam splitter. The illumination from the projector is reflected on the beam splitter and the ellipsoidal mirror, and finally illuminates a single point on the target object. The reflected light on the target object is again reflected by the ellipsoidal mirror, and is recorded as an image. The vertical setup has merit in that the density of the BRDF is uniform along φ because the long axis of the ellipsoid and the optical axes of the camera and projector are the same. Moreover, this kind of mirror is available commercially because it is often used as part of a usual illumination device. However, target materials must be cut into small facets to be placed at the focal point. On the other hand, the horizontal setup has merit in that the target materials do

250

Y. Mukaigawa, K. Sumino, and Y. Yagi φ=0

N incoming

outgoing

θr

θi

φ=60

φ=300 φ=270

φ=90 θ=30 θ=60

φ=240

φr

φ=30

φ=330

φi

(a) Four angle parameters

φ=210

φ=180

φ=150

φ=120

φ=240

φ=300 φ=0 φ=60 θ=90 φ=330 φ=30 φ=90 φ=120 θ=60

φ=270

θ=30

θ=90

(b) Vertical setup

(c) Horizontal setup

Fig. 2. Angular parameters of BRDF. (a) Four angle parameters. (b)(c) Relationship between the angles (θ, φ) and the image location.

not have to be cut. Hence, the BRDF of cultural heritages can be measured. However, the mirror must be specially made by the cutting operation. 4.3

Conversion Between Angle and Image Location

The lighting and viewing directions are specified as angles, while they are expressed as 2-D locations in the projection pattern or the captured image. The conversion between the angle and the image location is easy if geometric calibration is done for the camera and the projector. Figures 2 (b) and (c) illustrate the relationship between the angle and the image location for the vertical and horizontal setup, respectively.

5

Multiplexed Illumination

In this section, the problem of low dynamic range inherent in the projectorbased system is clarified, and this problem is shown to be solved by multiplexed illumination based on the Hadamard matrix. 5.1

Dynamic Range Problem

There are two main reasons for low dynamic range. One of these is the difference in intensities for specular and diffuse reflections. If a short shutter speed is used to capture the specular reflection without saturation, diffuse reflection tends to be extremely dark as shown in Fig.3(a). Conversely, a long shutter speed to capture bright diffuse reflection creates saturation of the specular reflection as shown in Fig.3(b). This problem is not peculiar to our system, but is common in general image measurement systems. The other reason is peculiar to our system that uses a projector for illumination. Generally, the intensity of the black pixel in the projection pattern is not perfectly zero. A projector emits a faint light when the projection pattern is black. Even if the intensity of each pixel is small, the sum of the intensities converging on one point cannot be ignored. For example, let us assume that the contrast ratio of the projector is 1000 : 1 and the size of the projection pattern

brightness

brightness

(a)

angle

(b)

angle

251

Specular Diffuse Sum of black projection

brightness

Multiplexed Illumination for Measuring BRDF

(c)

angle

Fig. 3. The problem of low dynamic range

is 1024 × 768. If 10 pixels in a projection pattern are white and the others are black, the intensity ratio of the white pixels to the black pixels is 10 × 1000 : (1024 × 768 − 10) × 1 = 1 : 79.

(1)

Thus the intensity of the black pixels is larger than one of the white pixels. This means that the measured data include a large amount of unnecessary information which should be ignored as shown in Fig. 3(c). By subtracting the image that is captured when a uniform black pattern is projected, this unnecessary information can be eliminated. As only a few bits are required to express the necessary information, a radical solution is required. 5.2

Multiplexed Illumination

Optical multiplexing techniques from the 1970s have been investigated in the spectrometry field[13]. If the spectrum of a light beam is measured separately for each wavelength, each spectral signal becomes noisy. Using the optical multiplexing technique, multiple spectral components are simultaneously measured to improve the quality. In the computer vision field, Schechner et al.[14] applied the multiplexing technique to capture images under varying illumination. In this method, instead of illuminating each light source independently, the multiplexed light sources are simultaneously illuminated. From the captured images, an image that is illuminated by a single light source is calculated. Wenger et al.[15] evaluated the effects of noise reduction using multiplexed illumination. We briefly describe the principle of multiplexed illumination. Let us assume that there are n light sources, and that s denotes the intensities of a point in the images when each light source turns on. The captured images are multiplexed by the weighting matrix W . The intensities m of the point by the multiplexed illumination are expressed by m = W s. (2) The intensities of the point under the single illumination can be estimated by s = W −1 m.

(3)

In our BRDF measuring system, a projector is used instead of an array of light sources. Hence the weighting matrix W can be arbitrarily controlled. It is

252

Y. Mukaigawa, K. Sumino, and Y. Yagi

known that if the component of the matrix W is −1 or 1, the Hadamard matrix is the best √ multiplexing weight[13]. In this case, the S/N ratio is increased by a factor of n. The n × n Hadamard matrix satisfies H Tn H n = nI n ,

(4)

where I n denotes an n × n identity matrix. In fact, as negative illumination cannot be given by a projector, the Hadamard matrix cannot be used directly. It is also known that if the component of the matrix W is 0 or 1, the S-matrix is the best√ multiplexing weight[13]. In this case, the S/N ratio is increased by a factor of n/2. The S-matrix can be easily generated from the Hadamard matrix. Hence, the projection pattern is multiplexed using the S-Matrix. Since the illumination can be controlled for each pixel using a projector, n becomes a large number and dramatic improvement can be achieved.

6 6.1

Experimental Results BRDF Measuring Systems

We constructed BRDF measuring systems named RCGs (Rapid Catadioptric Gonioreflectometers) as shown in Figs.4 (c) and (d). The RCG-1 includes a PointGrey Flea camera, an EPSON EMP-760 projector, and a Melles Griot ellipsoidal mirror as shown in Fig.4 (a). The RCG-2 includes a Lucam Lu-160C camera and a TOSHIBA TDP-FF1A projector. The ellipsoidal mirror for the RCG-2 is designed so that BRDFs can be measured for all angles of θ within 0 ≤ φ ≤ 240 as shown in Fig.4 (b).

Ellipsoidal mirror

Plate mirror

Object

(a)

(b)

Beam splitter

Projector Camera

(c) RCG-1 (vertical setup)

(d) RCG-2 (horizontal setup)

Fig. 4. The BRDF measuring systems

Multiplexed Illumination for Measuring BRDF

(a) velvet

(b) satin

(c) polyurethane

253

(d) copper

Fig. 5. Target materials

(a)

(b)

(c)

(d)

Fig. 6. BRDF of velvet and satin. (a)(b) examples of captured images, (c)(d) rendering result from measured BRDFs.

6.2

Measurement of Velvet and Satin

In this section, the results of measured BRDFs using the RCG-1 are shown. The target objects are velvet and satin, both of which have anisotropic reflections, as shown in Figs.5(a) and (b). First, the measuring time is evaluated. The sampling interval was set to one degree. The pattern corresponding to the lighting directions θi = 30 and φi = 250 was projected, and the reflected images were captured, as shown in Figs.6 (a) and (b) for velvet and satin, respectively. It is noted that some BRDFs could not be measured because the ellipsoidal mirror of the RCG-1 has a hole at the edge of the long axis. 360 × 90 = 32400 images were captured for each material. The measuring time was about 50 minutes. Figures 6 (c) and (d) are generated images of a corrugated plane that have the measured BRDFs of velvet and satin. The rendering process for this corrugated shape fortunately does not require the missing data. It can be seen that the characteristics of anisotropic reflection are reproduced. 6.3

Measurement of a Polyurethane Sphere

To evaluate the effectiveness of the multiplexed illumination, the isotropic BRDF of a polyurethane sphere was measured as shown in Fig.5(c). In this case, the lighting direction is varied by 1-DOF rotation because of the isotropic reflection. That is, the azimuth angle φi is fixed, and the elevation angle is varied 0 ≤ θi ≤ 180. Figure 7(a) shows an example of multiplexed illumination by a 191 × 191 S-matrix, and (b) shows the captured image after subtracting an image of projecting a black pattern. The captured images of the lighting direction θi = 10, φi = 270 were compared under several conditions. Figures 8(a) and (b) show the distribution of the reflected light without multiplexing. (a) is a single captured image, while (b) is

254

Y. Mukaigawa, K. Sumino, and Y. Yagi

(a)

(b)

Fig. 7. An example of the multiplexed illumination. (a) Projected pattern multiplexed by a 191 × 191 S-Matrix, and (b) the captured image.

(a) Single illumination without averaging

(b) Single illumination with averaging

(c) Multiplexed illumination (d) Multiplexed illumination without averaging with averaging Fig. 8. The reflected light of the lighting direction (θl = 10, φl = 270)

the average of ten captured images. The captured images are very noisy even with the averaging process. Figures 8(c) and (d) show the results with multiplexing. A sequence of multiplexed illumination patterns were projected, and the distribution of the reflected light corresponding to the same lighting direction is estimated. As before, (c) is the result without averaging, while (d) is the result with averaging ten images. Obviously, the noise is dramatically decreased in the multiplexed illumination. To find the spatial distribution of the reflected light, the changes in intensity of y = 60, x = 30 − 200 are plotted as shown in Fig.9. (a) shows the intensities without averaging, while (b) shows those with averaging ten images. In the graphs, blue and red lines represent the distribution with and without multiplexing, respectively. It is interesting that the result of multiplexing without averaging is more accurate than the result of the single illumination with averaging. While the time taken for capturing images is ten times greater for the averaging process, the multiplexed illumination can improve accuracy without increasing the capturing time. Figure 10 shows rendered images of a sphere and a corrugated surface using the BRDF measured by multiplexed illumination with averaging. Compared with the real sphere, the distribution of the specular reflection is slightly wide. One of the reasons is that the reflected light on the ellipsoidal mirror does not converge

Multiplexed Illumination for Measuring BRDF brightness 1.6

255

brightness 1.6 single at once multiplexed at once

1.4 1.2

single average multiplexed average

1.4 1.2

1

1

0.8

0.8

0.6

0.6

0.4

0.4

0.2

0.2

0

0

-0.2

-0.2

-0.4 30 40 50 60 70

80 90 100 110 120 130 140 150 160 170 180 190 200 pixel

(a) without averaging

-0.4 30 40 50 60 70

80 90 100 110 120 130 140 150 160 170 180 190 200 pixel

(b) with averaging ten images

Fig. 9. The distribution of the intensities

(a)

(b)

(c)

(d)

Fig. 10. Rendering results of the pink polyurethane sphere

perfectly on the target material because the alignment of the optical devices is not perfect. Since the target object is a sphere, the normal direction varies if the measuring point is different. As a result, the wide specular reflections may appear to be generated incorrectly. Unnatural reflections were observed in the upper area in Figures 8(c) and (d). This problem may be caused by the cutting operation error of the ellipsoidal mirror. Therefore, the BRDF of θ = 65 is substituted for the missing data of θ > 65. One of our future aims is to improve the accuracy of the optical devices. 6.4

Measurement of a Copper Plate

The isotropic BRDFs of a copper plate were measured as shown in Fig.5(d). Metal is the most difficult material for which to measure the BRDFs accurately, because the intensity levels of specular and diffuse reflections are vastly different. Figure 11 shows the rendering results of a corrugated surface using the measured BRDFs. (a) represents the results of a single illumination, while (b) represents those of multiplexed illumination. Since a fast shutter speed is used when measuring BRDFs to avoid saturation, the captured images are very dark. Hence, the rendering results are brightly represented in this figure. In the rendered images of (a) and (b), incorrect colors such as red or blue are observed. These incorrect colors seem to be the result of magnifying noise during the brightening process. On the other hand, noise can be drastically decreased in the rendered images of (c) and (d) using multiplexed illumination.

256

Y. Mukaigawa, K. Sumino, and Y. Yagi

(a)

(b)

(c)

(d)

Fig. 11. Comparison of the rendered results of the copper plate. (a) and (b) Single illumination. (c) and (d) Multiplexed illumination.

Although the dynamic range of the measured BRDFs is suitably widened, some noise is still observed in the rendered images. Ratner et al.[16] pointed out the fundamental limitation of Hadamard-based multiplexing. The dynamic range problem can also be reduced by incorporating the use of several images captured with varying shutter speeds[17], while the capturing time increases.

7

Conclusion

In this paper, we proposed a new high-speed BRDF measurement method that combines an ellipsoidal mirror with a projector, and solved the low dynamic range problem by applying multiplexed illumination to pattern projection. Two BRDF measuring systems were developed, which include differently shaped ellipsoidal mirrors. The proposed systems can measure complex reflection properties including anisotropic reflection. Moreover, the measuring time of BRDFs can be significantly shortened by the exclusion of a mechanical device. This paper focuses only on the BRDF measuring speed of the developed systems. The accuracy of the measured BRDFs needs to be evaluated. For the evaluation, we are attempting to compare the measured BRDFs and the ground truth using reflectance standards for which reflection properties are known. This research was partially supported by the Ministry of Education, Science, Sports and Culture, Grant-in-Aid for Young Scientists (A).

References 1. Matusik, W., Pfister, H., Brand, M., McMillan, L.: A Data-Driven Reflectance Model. In: Proc. SIGGRAPH 2003, pp. 759–769 (2003) 2. Karner, K.F., Mayer, H., Gervautz, M.: An image based measurement system for anisotropic reflection. Computer Graphics Forum (Eurographics 1996 Proceedings) 15(3), 119–128 (1996) 3. Lu, R., Koenderink, J.J., Kappers, A.M.L.: Optical Properties (Bidirectional Reflection Distribution Functions) of Velvet. Applied Optics 37(25), 5974–5984 (1998) 4. Marschner, S.R., Westin, S.H., Lafortune, E.P.F., Torrance, K.E., Greenberg, D.P.: Image-Based BRDF Measurement Including Human Skin. In: Proc. 10th Eurographics Workshop on Rendering, pp. 139–152 (1999) 5. Li, H., Foo, S.C., Torrance, K.E., Westin, S.H.: Automated three-axis gonioreflectometer for computer graphics applications. In: Proc. SPIE, vol. 5878, pp. 221–231 (2005)

Multiplexed Illumination for Measuring BRDF

257

6. M¨ uller, G., Bendels, G.H., Klein, R.: Rapid Synchronous Acquisition of Geometry and Appearance of Cultural Heritage Artefacts. In: VAST2005, pp. 13–20 (2005) 7. Davis, K.J., Rawlings, D.C.: Directional reflectometer for measuring optical bidirectional reflectance, United States Patent 5637873 (June 1997) 8. Mattison, P.R., Dombrowski, M.S., Lorenz, J.M., Davis, K.J., Mann, H.C., Johnson, P., Foos, B.: Handheld directional reflectometer: an angular imaging device to measure BRDF and HDR in real time. In: Proc. SPIE, vol. 3426, pp. 240–251 (1998) 9. Ward, G.J.: Measuring and Modeling anisotropic reflection. In: Proc. SIGGRAPH 1992, pp. 255–272 (1992) 10. Dana, K.J., Wang, J.: Device for convenient measurement of spatially varying bidirectional reflectance. J. Opt. Soc. Am. A 21(1), 1–12 (2004) 11. Kuthirummal, S., Nayar, S.K.: Multiview Radial Catadioptric Imaging for Scene Capture. In: Proc. SIGGRAPH2006, pp. 916–923 (2006) 12. Han, J.Y., Perlin, K.: Measuring Bidirectional Texture Reflectance with a Kaleidoscope. ACM Transactions on Graphics 22(3), 741–748 (2003) 13. Harwit, M., Sloane, N.J.A.: Hadamard Transform Optics. Academic Press, London (1973) 14. Schechner, Y.Y., Nayar, S.K., Belhumeur, P.N.: A Theory of Multiplexed Illumination. In: Proc. ICCV 2003, pp. 808–815 (2003) 15. Wenger, A., Gardner, A., Tchou, C., Unger, J., Hawkins, T., Debevec, P.: Performance Relighting and Reflectance Transformation with Time-Multiplexed Illumination. In: Proc. SIGGRAPH2005, pp. 756–764 (2005) 16. Ratner, N., Schechner, Y.Y.: Illumination Multiplexing within Fundamental Limits. In: Proc. CVPR2007 (2007) 17. Debevec, P., Malik, J.: Recovering High Dynamic Range Radiance Maps from Photographs. In: Proc. SIGGRAPH1997, pp. 369–378 (1997)

Analyzing the Influences of Camera Warm-Up Effects on Image Acquisition Holger Handel Institute for Computational Medicine (ICM), Univ. of Mannheim, B6, 27-29, 69131 Mannheim, Germany [email protected]

Abstract. This article presents an investigation of the impact of camera warmup on the image acquisition process and therefore on the accuracy of segmented image features. Based on an experimental study we show that the camera image is shifted to an extent of some tenth of a pixel after camera start-up. The drift correlates with the temperature of the sensor board and stops when the camera reaches its thermal equilibrium. A further study of the observed image flow shows that it originates from a slight displacement of the image sensor due to thermal expansion of the mechanical components of the camera. This sensor displacement can be modeled using standard methods of projective geometry in addition with bi-exponential decay terms to model the temporal dependence. The parameters of the proposed model can be calibrated and then used to compensate warmup effects. Further experimental studies show that our method is applicable to different types of cameras and that the warm-up behaviour is characteristic for a specific camera.

1 Introduction In the last couple of years much work has been done on camera modeling and calibration (see [1], [2], [3], [4] to mention a few). The predominant way to model the mapping from 3D world space to 2D image space is the well-known pinhole camera model. The ideal pinhole camera model has been extended with additional parameters to regard radial and decentering distortion ([5], [6], [7]) and even sensor unflatness [8]. These extensions have led to a more realistic and thus more accurate camera model (see Weng et al. [5] for an accuracy evaluation). Beside these purely geometrical aspects of the imaging process additional work has also been done on the electrical properties of the camera sensor and its influence on the image acquisition process. Some relevant variables are dark current, fixed pattern noise and line jitter ([9], [10], [11]). An aspect which has rarely been studied is the effect of camera warm-up on the imaging process. Beyer [12] reports a drift of measured image coordinates to an extent of some tenth of a pixel during the first hour after camera start-up. Wong et al. also report such an effect [13] as well as Robson et al. [14] . All of them only report drift distortions due to camera warm-up but do not give any further explanation of the origins of the observed image drift nor any way to model and compensate for these distortions. Today machine vision techniques have gained a great extension in many sensitive areas like industrial production and medical invention where errors of some tenth of a pixel in image feature Y. Yagi et al. (Eds.): ACCV 2007, Part II, LNCS 4844, pp. 258–268, 2007. c Springer-Verlag Berlin Heidelberg 2007 

Analyzing the Influences of Camera Warm-Up Effects

259

segmentation caused by sensor warm-up can result in significant reconstruction errors. In [15] measurement drifts of an optical tracking system up to 1 mm during the first 30 minutes after start-up are reported. In many computer assisted surgery applications such reconstruction errors are intolerable. Thus, a better understanding of the impact of camera warm-up on the image acquisition process is crucial. In this paper we investigate the influence of camera warm-up on the imaging process. We will show that the coordinates of segmented feature points are corrupted by a drift movement the image undergoes during camera warm-up. To our opinion this drift is caused by thermal expansion of a camera’s sensor board which results in a slight displacement of the sensor chip. We develop a model for the image plane movement which can be used to compensate distortions in image segmentation during a camera’s warmup period. Finally, we provide further experimental results approving the applicability of our method. The paper is organized as follows. Section 2 describes the experimental setup and the image segmentation methods from which we have observed the warm-up drift. Section 3 presents our model of warm-up drift and a way to calibrate the relevant parameters. Furthermore, a procedure is described to compensate for the image drift which fits easily in the distortion correction models widely used. Section 4 provides further experiments with different types of cameras.

2 Observing Warm-Up Drift To analyze the impact of temperature change after camera start-up a planar test field consisting of 48 white circular targets printed on a black metal plate is mounted in front of a camera (equipped with a 640 × 480 CMOS sensor). The test pattern is arranged to cover the entire field of view of the camera. The complete setup is rigidly fixed. The center points of the targets are initially segmented using a threshold technique. The coordinates of the target centers are refined using a method described in [16]. For each target the gray values along several rays beginning at the initial center are sampled until a gray value edge is detected. The position of the found edge is further refined to sub-pixel precision using moment preservation [17]. Next, a circle is fitted to the found sub-pixel edge points for each target using least squares optimization. The centers of the fitted circles are stored together with the current time elapsed since camera startup. The segmentation process is continuously repeated and stopped after approximately 45 minutes. At the same time the temperature on the sensor board is measured. The obtained data basis has a temporal resolution of approximately three seconds. Since the relative position between the test pattern and the camera is fixed the coordinates of the segmented target centers are not expected to vary systematically over time except noise. The results of the experiment are shown in figure 1 and figure 2.

3 Modeling Warm-Up Drift As one can see from figs. 1 and 2 the temperature increase of the camera sensor board induces an optical flow. To our hypothesis, this flow field results from a movement of the image plane due to thermal expansion of the sensor board. The CTE (Coefficient

260

H. Handel 500

sensor board temperature y-coordinate [pixel]

temperature [ C]

30 28.5 27 25.5 24 22.5

400 300 200 100

21 0 0

5

10 15 20 25 30 35 40 45

0

100 200 300 400 500 600

time [min]

x-coordinate [pixel]

(a)

(b) gray value shift of a falling edge 250

200

200 gray value

gray value

gray value shift of a rising edge 250

150 100

150 100

50

50

0

0 0

1

2

3

4

5

6

7

8

9 10

0

1

2

3

4

pixel

5

6

7

8

9 10

pixel

(c) Fig. 1. Warm-up drift. Fig. 1(a): Measured temperature on the sensor board. Fig. 1(b): Total displacement from camera start-up until thermal equilibrium, the lengths of the arrows are scaled by a factor of 150. Fig. 1(c): Gray value change for sampled line. The red curve shows the sampled gray values immediately after start-up and the blue curve after thermal stabilization.

x-coordinate drift

y-coordinate drift

point drift

39.25 39.2 39.15 39.1

42 y-coordinate [pixel]

41.95 y-coordinate [pixel]

x-coordinate [pixel]

39.3

41.9 41.85 41.8 41.75

5 10 15 20 25 30 35 40 45

41.95 41.9 41.85 41.8 41.75

5 10 15 20 25 30 35 40 45

time [min]

39.1

time [min]

39.2

39.3

x-coordinate [pixel]

(a) y-coordinate drift

5 10 15 20 25 30 35 40 45 time [min]

439.14

point drift y-coordinate [pixel]

596.15 596.1 596.05 596 595.95 595.9 595.85 595.8 595.75

y-coordinate [pixel]

x-coordinate [pixel]

x-coordinate drift

439.12 439.1 439.08 439.06 5 10 15 20 25 30 35 40 45 time [min]

439.14 439.12 439.1 439.08 439.06 595.8

595.9

596

596.1

x-coordinate [pixel]

(b) Fig. 2. Coordinate displacement. Coordinate changes of the top left target (fig. 2(a)) and the bottom rigth one (fig. 2(b)).

Analyzing the Influences of Camera Warm-Up Effects

261

of Thermal Expansion) of FR-4, the standard material which is used for printed circuit boards, can take values up to 150 − 200 ppm/K. Assuming a temperature increase to an extent of 10 to 20 K in the immediate vicinity of the sensor chip one would expect a thermal expansion and thus a displacement of the camera sensor to an extent of some microns. Taking the widely used pinhole camera model to describe the imaging process we can principally distinguish two cases: – The thermal expansion of the sensor board affects only the image plane. The center of projection remains fixed. – Both the image plane and the center of projection are displaced due to thermal expansion. Both cases can be found in real cameras. In the first case, the objective is fixed at the camera housing and is thus not affected by the local temperature increase of the sensor board since the distance to the board is relatively high. This configuration is typical for cameras equipped with C-mount objectives. In the second case, the lens holder of the objective is directly mounted on the circuit board. Thus, an expansion of the board displaces the lens and for this reason the center of projection. This configuration can be found at miniature camera devices used e.g. in mobile phones. Mathematically, the two cases have to be treated separately. In the remaining sections we use the following notation for the mapping from 3D world space to 2D image space: x = K [R|t] X

(1)

where x = (x, y, 1)T denotes the homogeneous image coordinates of world point X also described by homogeneous coordinates. The camera is described by its internal parameters K with ⎛ ⎞ f x 0 cx K = ⎝ 0 f y cy ⎠ 0 0 1 The exterior orientation of the camera is given by the rotation R and the translation t (see [4] for details). 3.1 Fixed Center of Projection If the center of projection remains fixed the observed optical flow will result from a movement of the image plane alone. In this case, the coordinate displacement can be described by a homography [4]. Let x(t0 ) and x(t1 ) denote the coordinates of the same target feature for the time t0 , i.e. immediately after camera start-up, and an arbitrary time t1 . Then, x(t0 ) = K(t0 )[I|0]X x(t1 ) = K(t1 )[R(t1 )|0]X = K(t1 )R(t1 )K−1 (t0 )(K(t0 )[I|0]X) = K(t1 )R(t1 )K−1 (t0 )x(t0 )

262

H. Handel

so that x(t1 ) = H(t1 )x(t0 ) with the time dependent homography H(t1 ): H(t1 ) = K(t1 )R(t1 )K−1 (t0 )

(2)

˜ 1 ) = K(t1 )R(t1 ). Since H(t) ˜ is invertible we Setting x ˜ = K−1 (t0 )x(t0 ) we get H(t can write ˜ −1 (t) = (K(t)R(t))−1 = R−1 (t)K−1 (t) H = RT (t)K−1 (t)

(3)

Since RT is orthogonal and K−1 is an upper diagonal matrix we can use QR decom˜ −1 is given [18]. For a rotation by a small angle position to obtain RT and K−1 once H ΔΩ around axis l we can further use the following approximation [19] R(t) = I + W(t)ΔΩ(t) + O(ΔΩ 2 ),

(4)

where the matrix W(t) is given by ⎞ 0 −l3 (t) l2 (t) 0 −l1 (t) ⎠ W(t) = ⎝ l3 (t) 0 −l2 (t) l1 (t) ⎛

(5)

The vector l is a unit vector and thus has two degrees of freedom. We can identify the rotation by the three component vector ˜l = ΔΩl. From ˜l we get ΔΩ = ˜l  and ˜ ˜ becomes l = ˜ll . The homography H(t) ˜ H(t) = (K(t0 ) + ΔK(t))R(t)

(6)

where ΔK(t) denotes the time dependent offset to the original camera parameters and is given by ⎞ ⎛ 0 Δcx (t) Δfx (t) Δfy (t) Δcy (t) ⎠ ΔK(t) = ⎝ 0 (7) 0 0 1 ˜ is determined by seven time dependent parameters, namely Δfx (t), Δfy (t), Thus, H(t) Δcx (t), Δcy (t), the changes of the internal camera parameters and l˜1 (t), l˜2 (t), l˜3 (t), the external orientation. Motivated by the results of our empirical studies (see section 4 for further details) we choose bi-exponential functions for the time dependent parameters: f (t) = a0 + a1 e−k1 t − (a0 + a1 )e−k2 t

(8)

The parameterization of f (t) is chosen in such a way that f (0) = 0 and thus H(0) = I. Since f (t) is determined by the four parameters a0 , a1 , k1 , k2 and we have seven time ˜ dependent parameters for H(t) the complete warm-up model comprises 28 parameters. In section 4 it is shown that the total number of parameters can be reduced in praxis.

Analyzing the Influences of Camera Warm-Up Effects

263

3.2 Moving Center of Projection In this case we use the simplifying assumption that the center of projection and the image plane are equally translated, i.e. the internal parameters of the imaging device remain constant during the warm-up period. This assumption will later be justified empirically. Then, we get the following relations x(t0 ) = K [I|0] X x(t1 ) = K [R(t1 )|t(t1 )] X Since the observed targets lie on a plane the image coordinate changes can again be described by a homography (see [3] for a strict treatment). x(t1 ) = K [r1 (t1 ) r2 (t1 ) t(t1 )] x(t0 )

(9)

where ri (t) denotes the i-th column of R(t). Thus, we get H(t) = K [R(t)|t(t)]. Given the homography H(t) the external parameters can be computed as follows [3] r1 = λK−1 h1 r2 = λK−1 h2 r3 = r1 × r2 t = λK−1 h3 with λ = 1/  K−1 h1 . Using the axis angle notation for the rotation R(t) we get six temporal dependent parameters, namely the three rotation parameters ˜l1 (t), ˜l2 (t), ˜l3 (t) as well as the three translational parameters t1 (t), t2 (t), t3 (t). Again, we use biexponential terms to describe the temporal behaviour of the parameter values. Thus, we have 24 parameters. 3.3 Warm-Up Model Calibration In the previous sections we have shown how to model the coordinate displacement of segmented image features during camera warm-up. We now outline an algorithm to calibrate the parameters of the models: 1. Determine the internal camera parameters using a method described in [3] or [2] based on a few images taken immediately after camera start-up. The obtained values are used for K(0) or K respectively. 2. Collect image coordinates by continuously segmenting target center points. 3. For each segmented image determine the homography H(t) (see [4], [19]) 4. Use a factorization method described in section 3.1 or 3.2 depending on the type of camera to obtain values for the internal/external parameters. 5. Fit a bi-exponential function to the values of each camera parameter. 6. Perform a non-linear least squares optimization over all 28(24) parameters minimizing the following expression M  N 

 xj (ti ) − H(ti ; β)xj (0) 2

(10)

j=1 i=1

where M denotes the number of feature points and β the current parameter vector.

264

H. Handel Focal length shift [pixel] 0.3

Principal point shift [pixel] 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0

delta f

0.25 0.2 0.15 0.1 0.05

delta cx delta cy

0 10

20

30 40 time [min]

50

60

70

10

20

30 40 time [min]

50

60

70

(a) Rotation 0.002

x-axis y-axis z-axis

0.0015 0.001 0.0005 0 -0.0005 -0.001 10

20

30

40

50

60

70

time [min]

(b) Fig. 3. Estimated internal and external parameters of the SonyFCB-EX780BP over time x-coordinate drift

x-coordinate drift

point drift

56.6 56.4 56.2 56 55.8 55.6

52.4

y-coordinate [pixel]

x-coordinate [pixel]

x-coordinate [pixel]

57 56.8

52.2 52 51.8 51.6

0

10

20

30

40

50

60

0

10

20

time [min]

30

40

50

52.4 52.2 52 51.8 51.6 55.6 55.8 56 56.2 56.4 56.6 56.8 57

60

time [min]

x-coordinate [pixel]

(a) x-coordinate drift

y-coordinate drift

point drift

56.6 56.4 56.2 56 55.8 55.6

52.4

y-coordinate [pixel]

y-coordinate [pixel]

x-coordinate [pixel]

57 56.8

52.2 52 51.8 51.6

0

10

20

30

40

50

60

70

0

time [min]

10

20

30

40

time [min]

50

60

70

52.4 52.2 52 51.8 51.6 55.6 55.8 56 56.2 56.4 56.6 56.8 57 x-coordinate [pixel]

(b) Fig. 4. Application of warm-up calibration to the SonyFCB-EX780BP. Fig. 4(a) shows the results of the drift calibration. The red curve depicts the segmented image coordinates and the blue one the ideal trajectory according to the calibrated drift-model. Fig. 4(b) shows the results of the drift compensation.

3.4 Warm-Up Drift Compensation With a calibrated warm-up model we can regard the influences of the sensor warmup on the imaging process. For cameras whose center of projection remains fixed an

Analyzing the Influences of Camera Warm-Up Effects

Translation [mm] 0.07

Rotation 0.0004

x-axis y-axis z-axis

0.05 0.03

265

x-axis y-axis z-axis

0.0003 0.0002

0.01 0.0001

-0.01

0

-0.03 -0.05 5

10

15

20

25

30

35

5

10

time [min]

15

20

25

30

35

time [min]

Fig. 5. Estimated external camera parameters of the VRmagic-C3 over time

x-coordinate drift

y-coordinate drift

point drift

39.25 39.2 39.15 39.1

42 y-coordinate [pixel]

41.95 y-coordinate [pixel]

x-coordinate [pixel]

39.3

41.9 41.85 41.8 41.75

5 10 15 20 25 30 35 40 45

41.95 41.9 41.85 41.8 41.75

5 10 15 20 25 30 35 40 45

time [min]

39.1

time [min]

39.2

39.3

x-coordinate [pixel]

(a) y-coordinate drift

point drift 439.15 y-coordinate [pixel]

439.15 y-coordinate [pixel]

x-coordinate [pixel]

x-coordinate drift 596.15 596.1 596.05 596 595.95 595.9 595.85 595.8 595.75

439.1

439.05 5 10 15 20 25 30 35 40 45 time [min]

439.1

439.05 5 10 15 20 25 30 35 40 45 time [min]

595.8

595.9

596

596.1

x-coordinate [pixel]

(b) Fig. 6. Fig. 6(a) and 6(b) show the predicted trajectories of the point coordinates (blue) compared to the observed ones (red)

image coordinate correction is straight forward. Given observed image coordinates xo at time t after camera start-up the undistorted image coordinates xu can be computed by multiplying with the inverse of H(t) xu = H−1 (t)xo

(11)

This correction is independent from the structure of the scene, i.e. the distance of the observed world point from the camera. Fig. 4(b) shows the results of this drift correction. In the second case, where the center of projection is not fixed, a direct correction of the image coordinates is not possible since the image displacement of an observed feature point depends on its position in the scene. In this case, the drift model can only be applied in reconstruction algorithms where the position of the camera is corrected accordingly.

266

H. Handel

Table 1. Camera motion parameters and duration until thermal equilibrium for a single camera (VRmagic-C3, CMOS)

1 2 3 4 5

Translation tx ty 0.024286 -0.004156 0.022227 -0.004706 0.023420 -0.004448 0.018973 -0.004655 0.022384 -0.004380

Rotation lx 0.412771 0.445256 0.377107 0.328780 0.460083

tz -0.037418 -0.039590 -0.034522 -0.033378 -0.040338

0 -0.01 z-axis -0.02 -0.03 -0.04 0.5

1

1.5 2 x-axis

2.5

3

3.5

0

0.5

1

1.5

2

2.5 y-axis

ΔΩ 0.011553 0.012630 0.010667 0.009569 0.013023

Time T99 19.62 18.08 19.37 17.16 18.72

Residuals σ2 0.000108 0.000465 0.000114 0.000112 0.000100

0 -0.005 -0.01 -0.015 -0.02 -0.025 -0.03 -0.035 -0.04

-0.04 -0.050

0.5

1

1.5 2 x-axis

2.5

3

3.5

0

0.5

1

1.5

2

2.5 y-axis

0 -0.005 0.01 -0.01 -0.015 0 -0.02 -0.025 -0.01 -0.03 z-axis -0.035 -0.02 -0.04 -0.03

0.01 0 -0.01 z-axis -0.02 -0.03 -0.04 -0.050

lz 0.783463 0.748210 0.810767 0.786722 0.745038

0 -0.005 0.01 -0.01 -0.015 0 -0.02 -0.025 -0.03 z-axis-0.01 -0.035 -0.02 -0.04 -0.03

0.01

-0.050

ly 0.458647 0.487992 0.426833 0.352122 0.482951

0.5

1

1.5 2 x-axis

2.5

3

3.5

0

0.5

1

1.5

2

2.5 y-axis

0 -0.005 -0.01 -0.015 -0.02 -0.025 -0.03 -0.035 -0.04

-0.04 -0.050

0.5

1

1.5 2 x-axis

2.5

3

3.5

0

0.5

1

1.5

2

2.5 y-axis

Fig. 7. Reconstruction of the position of the image plane for the VRmagic-C3 after 0s, 100s, 600s and 1000s since camera start-up. The axis units are mm. The depicted range in x- and y-direction corresponds roughly to the dimensions of the active sensor chip area.

4 Experimental Results This section presents experimental studies which justify the applicability of the proposed warm-up model. The experiment described in section 2 is conducted for two different types of cameras. The one is a VRmagic-C3, a miniature sized camera whose lens is directly mounted on the circuit board. The camera is equipped with a CMOS based active pixel sensor. The other camera is a SonyFCB-EX780BP, a CCD-based camera whose objective is not directly connected to the sensor circuit board. The initially estimated motion parameters are shown in figures 3(a)-3(b) and 5 respectively. As the figures show, our choice for a bi-exponential function describing the temporal dependence of the camera parameters seems resonable. Furthermore, one can see that for some camera parameters a simple exponential term is sufficient reducing the total number of parameters. Figure 4(a) and 6(a) show the applicability of the chosen models

Analyzing the Influences of Camera Warm-Up Effects

267

to explain the observed image displacement. Figs. 4(b) and 6(b) show the results of the drift correction described in section 3.4. In a second experiment we examine the repeatability of the calibration. The drift model is repeatedly calibrated for one camera. The data has been collected over several weeks. Table (1) shows the resulting parameters. The table contains the values of the motion parameters when the camera comes to a thermal equilibrium. The column T99 denotes the time in minutes until the drift displacement reaches 99% of its final amount. The results show that the warm-up behaviour is characteristic for a specific device. Finally, fig. 7 shows a reconstruction of the image plane during the camera warm-up period.

5 Conclusion We have presented a study of the impact of camera warm-up on the coordinates of segmented image features. Based on experimental observations we have developed a model for image drift and a way to compensate for it. Once the warm-up model is calibrated for a specific camera we can use the parameters for drift compensation. The formulation of our displacement correction fits well in the widely used projective framework used in the computer vision community. Thus, the standard camera models used in computer vision can easily be extended to regard for warm-up effects. Further experimental evaluations have shown that our warm-up model is principally applicable for all kinds of digital cameras and additionally that the warm-up behaviour is characteristic for a specific camera. In the future we plan to use cameras with an on-board temperature sensor to get direct access to the camera’s temperature. The formulation of our model presented here is based on the time elapsed since camera start-up assuming that the temperature always develops similarly. A direct measurement of the temperature instead of time will propably increase accuracy further.

References 1. Brown, D.C.: Close-range camera calibration. Photogrammetric Engineering 37(8), 855–866 (1971) 2. Tsai, R.Y.: A versatile camera calibration technique for 3d machine vision. IEEE Journal for Robotics & Automation RA-3(4), 323–344 (1987) 3. Zhang, Z.: A flexible new technique for camera calibration. IEEE Transactions on Pattern Analysis and Machine Intelligence 22(11), 1330–1334 (2000) 4. Hartley, R.I., Zisserman, A.: Multiple View Geometry in Computer Vision. Cambridge University Press, Cambridge (2000) 5. Weng, J., Cohen, P., Herniou, M.: Camera calibration with distortion models and accuracy evaluation. IEEE Transactions on Pattern Analysis and Machine Intelligence 14(10), 965– 980 (1992) 6. El-Melegy, M., Farag, A.: Nonmetric lens distortion calibration: Closed-form solutions, robust estimation and model selection. In: Proceedings ICCV, pp. 554–559 (2003) 7. Devernay, F., Faugeras, O.: Straight lines have to be straight. MVA 13(1), 14–24 (2001) 8. Fraser, C.S., Shortis, M.R., Ganci, G.: Multi-sensor system self-calibration. In: SPIE Proceedings, vol. 2598, pp. 2–15 (1995)

268

H. Handel

9. Healey, G., Kondepudy, R.: Radiometric ccd camera calibration and noise estimation. PAMI 16(3), 267–276 (1994) 10. Clarke, T.A.: A frame grabber related error in subpixel target location. The Photogrammetric Record 15(86), 315–322 (1995) 11. Ortiz, A., Oliver, G.: Radiometric calibration of ccd sensors: dark current and fixed pattern noise estimation. IEEE International Conference on Robotics and Automation 5, 4730–4735 (2004) 12. Beyer, H.A.: Geometric and radiometric analysis of a ccd-camera based photogrammetric close-range system. Mitteilungen Nr. 51 (1992) 13. Wong, K.W., Lew, M., Ke, Y.: Experience with two vision systems. Close Range Photogrammetry meets machine vision 1395, 3–7 (1990) 14. Robson, S., Clarke, T.A., Chen, J.: Suitability of the pulnix tm6cn ccd camera for photogrammetric measurement. SPIE Proceedings, Videometrics II 2067, 66–77 (1993) 15. Seto, E., Sela, G., McIlroy, W.E., Black, S.E., Staines, W.R., Bronskill, M.J., McIntosh, A.R., Graham, S.J.: Quantifying head motion associated with motor tasks used in fmri. NeuroImage 14, 284–297 (2001) 16. F¨orstner, W., G¨ulch, E.: A fast operator for detection and precise location of distinct points, corners and centers of circular features. ISPRS Intercommission Workshop on Fast Processing of Photogrammetric Data (1987) 17. Tabatabai, A.J., Mitchell, O.R.: Edge location to subpixel values in digital imagery. IEEE transactions on pattern analysis and machine intelligence 6(2), 188–201 (1984) 18. Golub, G.H., VanLoan, C.F.: Matrix computations. Johns Hopkins University Press, Baltimore, MD (1997) 19. Kanatani, K.: Geometric Computation for Machine Vision. Oxford University Press, Oxford, UK (1993)

Simultaneous Plane Extraction and 2D Homography Estimation Using Local Feature Transformations Ouk Choi, Hyeongwoo Kim, and In So Kweon Korea Advanced Institute of Science and Technology

Abstract. In this paper, we use local feature transformations estimated in the matching process as initial seeds for 2D homography estimation. The number of testing hypotheses is equal to the number of matches, naturally enabling a full search over the hypothesis space. Using this property, we develop an iterative algorithm that clusters the matches under the common 2D homography into one group, i.e., features on a common plane. Our clustering algorithm is less affected by the proportion of inliers and as few as two features on the common plane can be clustered together; thus, the algorithm robustly detects multiple dominant scene planes. The knowledge of the dominant planes is used for robust fundamental matrix computation in the presence of quasi-degenerate data.

1

Introduction

Recent advances in local feature detection have achieved affine/scale covariance of the detected region according to the varying viewpoint [11][12][13][8]. In the description phase of the detected region, geometric invariance to affinity or similarity is achieved by explicitly or implicitly transforming the detected region to the standard normalized region. (See Fig. 1.) Statistics robust to varying illumination or small positional perturbation are extracted from the normalized region and are used in the matching phase. After the tentative matching of the normalized regions, not only the matching feature coordinates but also the feature transformations from image to image are given. In this paper, we are interested in further exploiting the local feature transformations for simultaneously extracting scene planes and estimating the induced 2D homographies. Local feature matches have been treated as point-to-point correspondences, not as region matches with feature transformations, in the literature on 2D homography or fundamental matrix computation [1][2][6][5][3][4]. The main reason is that non-covariant local features (e.g., single-scale Harris corners) are less elaborately described (e.g., the template itself) so that the affine/similar transformation is not uniquely determined. Even the approaches that use affine/scale covariant local features [12][7] do not utilize the feature transformations thoroughly. Some approaches propagate local feature matches into the neighborhood regions for simultaneous object recognition and segmentation [9][10]. These approaches use the feature transformation elaborately and show that a few matches Y. Yagi et al. (Eds.): ACCV 2007, Part II, LNCS 4844, pp. 269–278, 2007. c Springer-Verlag Berlin Heidelberg 2007 

270

O. Choi, H. Kim, and I.S. Kweon

T1i

xi

T2i T1j

xj

T2j

x2j

x2i x2k

xk

T1k Image 1

T2k Normalized regions

Image 2

Fig. 1. Affine covariant regions are detected in each image independently and each region is transformed to the normalized region. The normalized regions are matched using statistics robust to varying illumination or small positional perturbation. Cyan-colored rectangles represent the features that are tentatively matched. Each tentative match provides not only the matching feature coordinates but also the feature transformations from image to image.

[9], or even only one [10], can be grown over a large portion of the object. Inspired by these approaches, we propagate the feature transformation of one match to the other matches and update the feature transformation iteratively so that the 2D homography of a plane can be estimated; as well, the local features on the common plane can be grouped together. The main difference between our approach and those mentioned is that we are more concerned about sparse scene geometry rather than the verification of a single or a few matches by gathering more evidence in a dense neighborhood. Our approach, however, can also be interpreted as a match verification process, because we group the local feature matches using 2D homography, as Lowe does in [12]. The difference between our algorithm and Lowe’s is that we use the feature transformation directly rather than the clustering with the Hough transform. The main advantage is that our algorithm does not require the labor of determining the resolution of the Hough bins. In Section 2, we develop a simple algorithm that simultaneously groups the coplanar features and estimates the 2D homography. In Section 3, the knowledge of the detected dominant planes is used to develop an importance sampling procedure for robust fundamental matrix computation in the presence of quasidegenerate data [4]. We show some experimental results for both plane extraction and fundamental matrix computation in Section 4. Finally, some advantages and limitations of our approach and future work are discussed in Section 5.

2

Simultaneous Plane Extraction and 2D Homography Estimation

We characterize a feature match as a set of the center coordinates x1i , x2i of matching regions, the feature transformations Hi , H−1 and a membership i variable hi (see Fig. 1 for details).

Simultaneous Plane Extraction and 2D Homography Estimation

where

271

mi = {x1i , x2i , Hi , H−1 i , hi },

(1)

−1 = T2i T−1 Hi = T1i T−1 2i , Hi 1i ,

(2)

T1i and T2i are estimated as described by Mikolajczyk et al. in [8]. In this section, we develop an algorithm that determines hi and updates Hi , H−1 for i each match mi . Feature transformations differ in quality. Under some transformations, many features are transformed with small residual errors, while under other transformations, few features are transformed. Our algorithm described in Fig. 2 selects good transformations with high dominance scores defined as (3); therefore, the algorithm works in the presence of a number of erroneous initial feature transformations on the assumption that there exists at least one good initial seed for each dominant plane. It is important to note that a match already contains a transformation, in other words a homography, so that the algorithm does not require three or four true matches for constructing a hypothesis. This fact enables as few as two matches to be clustered together once one match is dominated by the other. We define –dominance to assess the quality of transformations in a systematic way. Definition 1. mi dominates mj if x2j −Hi (x1j ) <  and x1j −H−1 i (x2j ) < , where H(x) is the transformed Euclidean coordinates of x by H. The predicate in Def. 1 is identical to the inlier test in RANSAC (RANdom SAmple Consensus) approaches [1][5] with the maximum allowable residual error . Our algorithm can be considered as a deterministic version of the RANSAC algorithm, where random sampling is replaced by a full search over the hypothesis space. Nothing prevents us from modifying our algorithm to be random; however, the small number of hypotheses, equal to the number of matches, attracts us to choose the full search. The dominance score ni is defined as ni =

N 

dij ,

(3)

j=1

where dij =1 when mi dominates mj , otherwise 0. The dominance score ni is the number of features that are transformed by Hi with a residual error less than . Once the ni and dij values are calculated, the membership hi is determined in such a way that the mk with the largest dominance score collects its inliers first and the collected inliers are omitted from the set of matches for the detection of the next dominant plane. We do not cover the model selection issue [6] thoroughly in this paper. The in Fig. 2 is simply decided by the number of inliers and update of Hk , H−1 k the residual error. When the number of inliers is less than three, we do not update the feature transformation and the match is discarded from gathering

272

O. Choi, H. Kim, and I.S. Kweon

Input: M0 = {mi |i = 1...N }. Output: updated hi , Hi , H−1 i , and ni for each mi . Variables: - it: an iteration number. - ti : an auxiliary variable that temporarily stores hi for each iteration. - M : a set of matches whose membership is yet to be determined. - The other variables: explained in the text. 1. Initialize it, hi . - it ← 0. - For i = 1 . . . N , {hi ← i.} 2. Initialize M , ti , ni , dij . - M ← M0 . - For i = 1 . . . N , {ti ← 0.} - For i = 1 . . . N , {for j = 1 . . . N , {calculate dij .}} - For i = 1 . . . N , {calculate ni .} 3. k = argmaxi:mi ∈M ni . 4. If nk = 0, {go to 10.} 5. For i : mi ∈ M , {if dki = 1 and ti = 0, {ti ← hi ← k.}} 6. Update Hk , H−1 k . 7. M ← M − {mi |ti = 0}. 8. Update ni , dij . - For i : mi ∈ M , {for j : mj ∈ M , {calculate dij .}} - For i : mi ∈ M , {calculate ni .} 9. nk ← 0. Go to 3. 10. For i = 1 . . . N , {if hi has changed, {it ← it + 1, go to 2.}} 11. Finalize. = H−1 - For i = 1 . . . N , {Hi = Hhi , H−1 i hi , ni = j:hj =i 1.}



Fig. 2. Algorithm sketch of the proposed simultaneous plane detection and 2D homography estimation technique. The symbol ‘←’ was used to denote replacement.

more inliers. An affine transformation is calculated when the number of inliers is more than two, and a projective transformation is also calculated when the number of inliers is more than three. Three transformations compete with one another for each mk : the original transformation, and the newly calculated affine and projective transformations. The transformation with the minimum cost is finally selected at each iteration. The cost is defined as f (Hk ) =



2 x2i − Hk (x1i )2 + x1i − H−1 k (x2i ) .

(4)

i:hi =k

The cost is minimized using the Levenberg–Marquardt algorithm [14][15] using the original Hk as the initial solution. Refer to [16] for more statistically sound homography estimation.

Simultaneous Plane Extraction and 2D Homography Estimation

(a)  = 3.0, it = 0.

273

(b)  = 3.0, it = 2.

Fig. 3. (a) shows the plane detection result at it = 0. The same colored features were detected to be on the common plane. For visibility, planes with less than three features are not displayed. The dominance score decreases in the order of red, green, blue, yellow, purple. The algorithm converges very quickly at it = 2 (see (b)). The main reason is that the scene is fully affine so that our initial seeds can collect many inliers in the 0th iteration.

Figure 3 shows the clustered features using our algorithm. One obvious plane on top of the ice cream box was not detected for lack of matching features because of severe distortion (see also Fig. 1). The front part of the detergent box (Calgon) could not be clustered because of the slight slant of the top. The total number of dominance calculations is O(N 2 × K × it) and Hk , H−1 k update is O(K × it), where N is the total number of matches, K is the average number of planes at each iteration and it is the total number of iterations until convergence. The former is the main sources of computational complexity, which is equivalent to testing N × it hypotheses for each plane extraction in conventional RANSAC procedures. More experiments using various scenes are described in Section 4.

3

Importance-Driven RANSAC for Robust Fundamental Matrix Computation

The fundamental matrix can be calculated using the normalized eight-point algorithm. This algorithm requires eight perfect matches. This number is larger than the minimum number required for the fundamental matrix computation. However, we use the eight-point algorithm because it produces a unique solution for general configurations of the eight matches. The main degeneracy occurs when there is a dominant plane and more than five points are sampled from it [5]. The knowledge of the detected planes can be used to develop an importance sampling procedure that avoids the obvious degenerate conditions, i.e., more than five points on the detected common plane. Any feature match that is not grouped with other matches can be considered either a mismatch or a valuable true match off the planes, and a plane that includes many feature matches is highly likely to be a true scene plane, for which the included matches are also likely to be true. To balance between avoiding

274

O. Choi, H. Kim, and I.S. Kweon

mismatches and degeneracy, we propose an importance sampling method that first decides a plane according to its importance Ik = min(nk , n),

(5)

where Ik is the sampling importance of the k’th plane, nk is the dominance score, and n is a user-defined threshold value. Once the plane is determined, a match on the plane is sampled according to the uniform importance. This sampling procedure becomes equivalent to the standard RANSAC algorithm [1][5] when n = ∞, i.e., Ik = nk . The sampling importance decreases for dominant planes that have more than n inliers. n is typically set to five in the experiments in Section 4. The final sample set with eight matches is discarded if any six or more matches are sampled from a common plane. The following cost is minimized for each eight-match sample set. The F with the largest number of inliers is chosen as the most probable relation.  xˆ2i T Fxˆ1i 2 . (6) g(F) = mi ∈S8

The inlier test is based on the following predicate, ˆ 2i = 0) < , ˆ T Fˆ ˆ T FT x mi ∈ Sin if d⊥ (x2i , x x1i = 0) <  and d⊥ (x1i , x

(7)

ˆ is used to denote the homogeneous coordinates where Sin is the set of inliers, x and d⊥ is the distance from a point to a line in the line normal direction. Refer to [17] for more statistically sound fundamental matrix estimation.

4

Experiments

In this section, we show some experimental results for various images, including repeating patterns and dominant planes. The images in Figures 1, 3 and 4 were adapted from [3]. The castle images that contain repeating patterns (Fig. 5) were downloaded from http://www.cs.unc.edu/˜marc/. We extracted two kinds of features in the feature extraction stage. Maximally stable extremal regions (MSER) [3] and generalized robust invariant features (GRIF) [13] were used. MSERs are described using SIFT neighborhoods [12] and GRIFs are described using both SIFT and a hue histogram. The feature pairs were classified as tentative matches if the distance between the description vectors was smaller than 0.4 and the normalized cross correlation between the normalized regions was larger than 0.6 in all the experiments in this paper.  is the only free parameter in our plane extraction algorithm; it is the maximum allowable error in model fitting. Large  tends to produce a smaller number of planes with a large number of inliers. Dominant planes invade nearby or even distant planes when  is large. The homography is rarely updated with small  because the initial feature transformation fits the inliers very tightly.  trades off between the accuracy of the homography and the robustness to the perturbed feature position. Figure 4 shows this trade-off. Good results were obtained in the

Simultaneous Plane Extraction and 2D Homography Estimation

(a)  = 1

(b)  = 5

(c)  = 10

(d)  = 20

275

Fig. 4. Plane detection results with varying . See the text for details.

range of 2 <  < 10.  = 5 produced the best results in Fig. 4. We used  = 5 for all the experiments in this paper, unless otherwise mentioned. Figure 5 shows the plane extraction and epipolar geometry estimation results on the image pairs with varying viewpoint. It is hard to find the true correspondences in these image pairs because many features are detected on the repeating pattern, e.g., the windows on the wall. The number of planes that humans can manually detect is six in this scene and our algorithm finds only one or two planes that can be regarded as correct. However, it is important to note that the most dominant plane was always correctly detected, because true matches on the distinctive pattern have grown their evidence over the matches on the repeating pattern. Moreover, the number of tentative matches is not enough to detect other planes, i.e., our algorithm does not miss a plane once a true feature correspondence lies on the plane. Figure 6 shows the results for the images with a dominant plane. Among the total 756 tentative matches, 717 matches were grouped to belong to the redcolored dominant plane. For the standard RANSAC approach, the probability that the eight points contain more than two points off the plane is: P =

8 m  C8−m 717 C39 = 0.0059 ≈ 0.6%. 8 C756 m=3

(8)

We ran the standard random sampling procedure [1][5] 1000 times to count the number of occasions more than five matches were sampled from the detected dominant plane. The number of occasions was, unsurprisingly, 995, i.e.,

276

O. Choi, H. Kim, and I.S. Kweon

(a) Images with small viewpoint change

(b) Images with moderate viewpoint change

(c) Images with severe viewpoint change Fig. 5. Experiments on varying the viewpoint and repeating patterns. The number of tentative matches decreases with increasing viewpoint change. Many tentative matches are mismatches because of the repeating patterns (windows). The most dominant two planes were correctly detected in (a) and (b) (red–green and red–yellow, respectively). Only one plane was correctly detected in (c) for lack of feature matches (red). The epipolar geometry was correctly estimated in (a) and (b) with d⊥ 0.97 and 0.99 pixels respectively. The epipolar geometry could not be correctly estimated for lack of offplane correct matches in (c).

99.5% of cases were degenerate and only 0.5% were not degenerate cases, on the assumption that our dominant plane is correct, which is very close to the theoretical value. Our algorithm does not suffer from the degeneracy problem on the assumption that the detected planes are correct. Figure 6 shows the two representative solutions that were most frequently achieved during the proposed sampling procedure. It is clear that no more than five matches are sampled from the dominant plane.

Simultaneous Plane Extraction and 2D Homography Estimation

277

(a) Detected dominantplanes

(b) 751/756 inliers, d⊥ = 0.52 (c) 751/756 inliers, d⊥ = 0.58 Fig. 6. Epipolar geometry estimation in the presence of quasi-degenerate data. The two solutions (b) and (c) that are most frequently achieved show the effectiveness of our algorithm.

5

Conclusion

We developed a simultaneous plane extraction and homography estimation algorithm using local feature transformations that are already estimated in the matching process of the affine/scale covariant features. Our algorithm is deterministic and parallel in the sense that all the feature transformations compete with one another at each iteration. This property naturally enables the detection of multiple planes without the independent RANSAC procedure for each plane or the labor in determining the resolution of the Hough bins. Our algorithm always produces consistent results and even two or three matches can be grouped together if one match dominates the others. The knowledge of the detected dominant planes is used to safely discard the degenerate samples so that the fundamental matrix can be robustly computed in the presence of quasidegenerate data. Our plane detection algorithm does not depend on the number of inliers, but depends on the number of supports that enable the original transformation to be updated. In further work, we hope to adapt more features (e.g., single-scale, affine/scale covariant Harris corner [8]) into our framework so that both the number of supports and the number of competing hypotheses can be increased. It is clear that our algorithm can be applied to sparse or dense motion segmentation [6][7]. Dense motion segmentation is another area of future research.

278

O. Choi, H. Kim, and I.S. Kweon

Acknowledgement This work was supported by the IT R&D program of MIC/IITA [2006-S-02801, Development of Cooperative Network-based Humanoids Technology] and the Korean MOST for NRL program [Grant number M1-0302-00-0064].

References 1. Fishler, M.A., Bolles, R.C.: Random sampling Consensus: A paradigm for model fitting with application to image analysis and automated cartography. Communications of the ACM 24(6), 381–395 (1981) 2. Torr, P.H., Zisserman, A.: Robust Computation and Parameterization of Multiple View Relations. In: ICCV (1998) 3. Chum, O., Werner, T., Matas, J.: Two-view Geometry Estimation Unaffected by a Dominant Plane. In: CVPR (2005) 4. Frahm, J., Pollefeys, M.: RANSAC for (Quasi-)Degenerate data (QDEGSAC). In: CVPR (2006) 5. Hartley, R., Zisserman, A.: Multiple View Geometry in Computer Vision. Cambridge University Press, Cambridge (2000) 6. Torr, P.H.S.: Geometric Motion Segmentation and Model Selection. Philosophical Transactions of the Royal Society, pp. 1321–1340 (1998) 7. Bhat, P., Zheng, K.C., Snavely, N., Agarwala, A., Agrawala, M., Cohen, M.F., Curless, B.: Piecewise Image Registration in the Presence of Multiple Large Motions. In: CVPR (2006) 8. Mikolajczyk, K., Tuytelaars, T., Schmid, C., Zisserman, A., Matas, J., Schaffalitzky, F., Kadir, T., Van Gool, L.: A Comparison of Affine Region Detectors. IJCV 65(1-2), 43–72 (2005) 9. Ferrari, V., Tuytelaars, T., Van Gool, L.: Simultaneous Object Recognition and Segmentation from Single or Multiple Model Views. IJCV 67(2), 159–188 (2006) 10. Vedaldi, A., Soatto, S.: Local Features, All Grown Up. In: CVPR (2006) 11. Matas, J., Chum, O., Urban, M., Pajdla, T.: Robust Wide Baseline Stereo from Maximally Stable Extremal Regions. In: BMVC, pp. 384–393 (2002) 12. Lowe, D.G.: Distinctive Image Features from Scale-Invariant Keypoints. IJCV 60(2), 91–110 (2004) 13. Kim, S.H., Yoon, K.J., Kweon, I.S.: Object Recognition using Generalized Robust Invariant Feature and Gestalt Law of Proximity and Similarity. In: CVPR 2006. 5th IEEE Workshop on Perceptual Organization in Computer Vision (2006) 14. Levenberg, K.: A Method for the Solution of Certain Problems in Least Squares. Quart. Appl. Math. 2, 164–168 (1944) 15. Marquardt, D.: An Algorithm for Least-Squares Estimation of Nonlinear Parameters. SIAM J. Appl. Math. 11, 431–441 (1963) 16. Kanatani, K., Ohta, N., Kanazawa, Y.: Optimal Homography Computation with a Reliability Measure. IEICE Trans. Information Systems E83-D(7), 1369–1374 (2000) 17. Kanatani, K., Sugaya, Y.: High Accuracy Fundamental Matrix Computation and its Performance Evaluation. In: BMVC (2006)

A Fast Optimal Algorithm for L2 Triangulation Fangfang Lu and Richard Hartley Australian National University

Abstract. This paper presents a practical method for obtaining the global minimum to the least-squares (L2 ) triangulation problem. Although optimal algorithms for the triangulation problem under L∞ -norm have been given, finding an optimal solution to the L2 triangulation problem is difficult. This is because the cost function under L2 -norm is not convex. Since there are no ideal techniques for initialization, traditional iterative methods that are sensitive to initialization may be trapped in local minima. A branch-and-bound algorithm was introduced in [1] for finding the optimal solution and it theoretically guarantees the global optimality within a chosen tolerance. However, this algorithm is complicated and too slow for large-scale use. In this paper, we propose a simpler branch-and-bound algorithm to approach the global estimate. Linear programming algorithms plus iterative techniques are all we need in implementing our method. Experiments on a large data set of 277,887 points show that it only takes on average 0.02s for each triangulation problem.

1

Introduction

The triangulation problem is one of the simplest geometric reconstruction problems. The task is to infer the 3D point, given a set of camera matrices and the corresponding image points. In the presence of noise, the correct procedure is to find the solution that reproduces the image points as closely as possible. In other words, we want to minimize the residual errors between the reprojected and measured image points. Notice that residual errors measured under different norms lead to different optimization problems. It is shown in [2] that a quasi-convex cost function arises and thus a single local minimum exists if we choose to calculate the residual errors under L∞ -norm. However, the problem with the residual errors measured under L2 -norm still remain of primary interest. Suppose we want to recover a 3D point X = (x, y, z) . Let Pi (i = 1, 2, . . . , n) denote a set of n camera matrices, ui the corresponding image coordinates and ˆ = (X; 1) the homogenous coordinates of X. Under the L2 -norm, we are led to X solve the following optimization problem: 

The second author is also affilated with NICTA, a research instutute funded by the Australian Government’s Backing Australia’s Ability initiative, in part through the Australian Research Council.

Y. Yagi et al. (Eds.): ACCV 2007, Part II, LNCS 4844, pp. 279–288, 2007. c Springer-Verlag Berlin Heidelberg 2007 

280

F. Lu and R. Hartley

minimize C2 (X) =

n 

ˆ )2 d(ui , Pi X

subject to λi (X) > 0

(1)

i=1

where d(·, ·) is the Euclidean metric and λi (X) is the depth of the point relative to image i. The optimal solution to (1) is not easy, since the cost function C2 (X) is nonconvex. In the worst case, multiple local minima may occur. Iterative methods, so-called bundle adjustment methods [3], usually work well, though they are dependent on initialization. Both the L∞ solution [4] and the linear solution [5] are useful for initialization, but neither initialization can theoretically guarantee global optimality. A branch-and-bound algorithm was given in [1] and it provably finds the global optimum within any tolerance . However, the way of bounding the cost-function is complicated. The computational cost is high so it may not be used to deal with large-scale data sets. A simpler way of obtaining the global estimates by a branch-and-bound process is presented in this paper. The main feature is to get the lower bound for the cost-function in a bounding box. We then use a much simpler convex lower bound for the cost than is used in [1]. Instead of using Second Order Cone Programming as in [1], we need only Linear Programming and simple iterative convex optimization (such as a Newton or Gauss-Newton method). This makes the implementation very easy. Experimental results show that the method can theoretically guarantee the global optimum within small tolerance, and also it works fast, taking on average 0.02s for each triangulation problem. The only other known methods for finding a guaranteed optimal solution to the L2 triangulation problem are those given for the two-view ([6]) and the three-view problem ([7]), which involve finding the roots of of a polynomial. For the two-view triangulation problem this involves the solution of a degree-6 polynomial, whereas for 3-view problem, the polynomial is of degree 43. This approach does not generalize in any useful manner to larger numbers of views.

2

Other Triangulation Methods

Various other methods for triangulation have been proposed, based on simple algebraic or geometric considerations. Two of the most successful are briefly discussed here. 2.1

Linear Triangulation Methods

ˆ = (X; 1) is given, let p1i  , ˆ where X For every image i the measurement ui = Pi X p2i  , p3i  be the rows of Pi and xi , yi be the coordinates of ui , then two linear equations are given as follows.     3 ˆ = 0, ˆ = 0. xi p3i  − p1i  X yi pi − p2i  X In all, a set of 2n linear equations are composed for computing X in the n-view triangulation problem. The linear least-squares solution to this set of equations

A Fast Optimal Algorithm for L2 Triangulation

281

provides the linear solution to the triangulation problem. The minimum is denoted by Xl here. To be a little more precise, there are two slightly different methods that may ˆ to 1, as suggested be considered here. One may either set the last coordinate of X above, and solving using a linear least-squares solution method. Alternatively the ˆ may be treated as a variable. The optimal solution may then last coordinate of X be solved using Singular Value Decomposition (see [5], DLT method). 2.2

L∞ Framework

The L∞ formulation leads to the following problem: ˆ ) subject to λi (X) > 0. minimize C∞ (X) = max d(ui , Pi X i

(2)

This is a quasi-convex optimization problem. A global solution is obtained using a bisection algorithm in which a convex feasibility problem is solved in each step. Please refer to [4] for details. We denote the minimum by X∞ in this paper. Other methods are mentioned in [6]. The most commonly used technique is to use some non-optimal solution such as the algebraic or (as more recently suggested) the optimal L∞ solution to initialize an iterative optimization routine such as Levenberg-Marquardt to find the L2 minimum. This method can not be guaranteed to find the optimal solution, however. The purpose of this paper is to give a method guaranteed to find the optimal solution.

3 3.1

Strategy Branch and Bound Theory

Branch and Bound algorithms are classic methods for finding the global solution to non-convex optimization problems. By providing a provable lower or upper bound, which is usually a convex function to the objective cost-function, and a dividing scheme, the algorithms tend to achieve the global estimates within an arbitrary small tolerance. 3.2

Strategy

In this paper, our strategy is to find the L2 optimal point Xopt by a process of branch-and-bound on a bounding box B0 . We start with the bounding box and a best point Xmin found so far (X∞ plus local refinement will do in this paper). At each step of the branch and bound process, we consider a bounding box Bi obtained as part of a subdivision of B0 . A lower bound for the minimum of the cost function is estimated on Bi and compared with the best cost found so far, namely C2 (Xmin ). If the minimum cost on Bi exceeds C2 (Xmin ), then we abandon Bi and go on to the next box to be considered. If, on the other hand, the minimum cost on Bi is smaller than C2 (Xmin ), then we do two things: we

282

F. Lu and R. Hartley

evaluate the cost function at some point inside Bi to see if there is a better value for Xmin , and we subdivide the box Bi and repeat the above process with the subdivided boxes.

4

Process

In this paper, the Branch and Bound process is shown considering three aspects. Firstly an initial bounding box is computed. Secondly the bounding method is presented which means the provably lower or upper bound of the objective costfunction is calculated. Then the branching part, that is, a subdivision scheme is given. Since Branch and Bound algorithms can be slow, a branching strategy should be devised to save the computation cost. 4.1

Obtaining the Initial Bounding Box B0

We start by an initial estimate Xinit for the optimum point. If Xopt is the true L2 minimum, it follows that C2 (Xopt ) ≤ C2 (Xinit ). This may be written as: C2 (X) =

n 

ˆ opt )2 ≤ C2 (Xinit ) d(ui , Pi X

(3)

i=1

ˆ opt )2 is ˆ opt = (Xopt ; 1). We can see that the sum of the values d(ui , Pi X with X ˆ opt )2 is less than this value. bounded by C2 (Xinit ), and  this means each d(ui , Pi X ˆ opt ) ≤ δ for all i. This In particular, defining δ = C2 (Xinit ) we have d(ui , Pi X equation can also be written as  



 1X 2X ˆ opt 2 ˆ opt 2 p p  i i ˆ opt ) = + yi − 3  ≤δ d(ui , Pi X xi − 3  ˆ opt ˆ opt pi X pi X which is satisfied for each i. This means the following two constraints are satisfied for each i.     1 ˆ 2 ˆ     yi − pi Xopt  ≤ δ xi − pi Xopt  ≤ δ ,    ˆ ˆ 3 3 pi Xopt pi Xopt  Notice for n-view triangulation, we have a total number of 4n linear constraints ˆ , formulated by multiplying both sides of the above constraint on the position X ˆ opt . equations with the depth term p3i  X We wish to obtain bounds for Xopt = (xopt , yopt , zopt ) . That is to find xmin , xmax , ymin , ymax , zmin , zmax such that xmin ≤ xopt ≤ xmax , ymax ≤ yopt ≤ ymax , zmin ≤ zopt ≤ zmax . For each of them, we can formulate a linear programming (LP) problem by linearizing the constraints in the above equations. For instance, xmin is the smallest value of the x-coordinate of Xopt with respect to the following linear constraints. ˆ opt ≤ 0 , (xi p3i  − p1i  − δp3i  )X 3 2 3 ˆ (yi pi − pi − δpi )Xopt ≤ 0 ,

ˆ opt ≤ 0 (−xi p3i  + p1i  − δp3i  )X 3 2 3 ˆ (−yi pi + pi − δpi )Xopt ≤ 0 .

A Fast Optimal Algorithm for L2 Triangulation

283

The other bound values are found in the similar way. This process then provides an initial bounding box B0 for the optimal point Xopt . Note: In this paper we got the initial optimal point Xinit by local refinement from X∞ , It should be mentioned that any point X may be used instead of this Xinit for initialization. However, we would like to choose a relatively tight initial bounding box since it will reduce the computation complexity. Using the local minimum from X∞ would be a good scheme because it is likely to produce a relatively small value of the cost-function C2 , and hence a reasonably tight bound. 4.2

Bounding

Now we consider the problem of finding a minimum value for C2 (X) on a box B. We rewrite the L2 cost-function as follows:  ˆ )2 d(ui , Pi X C2 (X) = i





 ˆ 2 ˆ 2 p1i  X p2i  X = + yi − 3  xi − 3  ˆ ˆ pi X pi X i  f 2 (X) + g 2 (X) i i = 2 (X) λ i i Here fi , gi , λi are linear functions in the coordinates of X. And the depths λi (X) can be assumed to be positive, by cheirality(see [5]). Note that for this we must choose the right sign for each Pi , namely that it is of the form Pi = [Mi |mi ] where det Mi > 0. It is observed in [4] that each of the functions fi2 (X) + gi2 (X) such that λi (X) > 0 λi (X) is convex. Now we define wi = maxX∈B λi (X), where B is the current bounding box. The value of each wi can be easily found using Linear Programming(LP). Then we may reason as follows, for any point X ∈ B, wi 1/wi fi2 (X) + gi2 (X) wi λi (X)  f 2 (X) + g 2 (X) i i w λ ( X) i i i

≥ λi (X) ≤ 1/λi (X) f 2 (X) + gi2 (X) ≤ i λ2i (X)  f 2 (X) + g 2 (X) i i ≤ = C2 (X) 2 (X) λ i i

(4)

However, the left-hand side of this expression is a sum of convex functions, and hence is convex. It is simple to find the minimum of this function on the box

284

F. Lu and R. Hartley

B, hence we obtain a lower bound L(X) = C2 (X).

i

fi2 (X)+gi2 (X) wi λi (X)

for the cost function

BFGS algorithm: In this paper, we adopted the BFGS algorithm([8]) to find the minimum of the convex function L(X) within the bounding box B. The BFGS algorithm is one of the main Quasi-Newton methods in convex optimization problems ([9]). It inherits good properties of the Newton Method such as fast convergence rate while avoiding the complexity of computing the Hessian. This significantly improves the computation speed. 4.3

Branching

Given a box Bi , first we evaluate the lower bound of the cost. If the lower bound exceeds the best cost C2 (Xmin ) we have got so far, we abandon the box and go on to the next box. If on the other hand the lower bound is less than C2 (Xmin ), we evaluate the cost of some point in the current box. If the minimum value is less than C2 (Xmin ), we change the C2 (Xmin ) to the current minimum value and subdivide the block into two along its largest dimension. We repeat the steps until the dimension of the box approaches zero within a given tolerance. Note: How exactly the lower bound approximates the cost depends essentially on how closely maxX∈B λi (X) approximates the value of λi (X) for arbitrary points X in B. It is best if λi (X) does not vary much in B. Note that λi (X) is the depth of the point X with respect to the i-th camera. Thus it seems advantageous to choose the boxes to be shallow with respect to their depth from the cameras. This suggests that a more sophisticated scheme for subdividing boxes may be preferabel to the simple scheme we use of subdividing in the largest dimension.

5

Proof of Optimality

The complete algorithm is given in Fig 1. We have claimed that the method will find the optimal solution. That will be proved in this section. It will be assumed that the bounding box B0 is finite, as in the description of the algorithm. Because we use a FIFO structure to hold the boxes, as the algorithm progresses, the size of boxes B decreases towards zero. Note that at any time, the best result so-far found gives an upper bound for the cost of the optimal solution. We will also define a lower bound as follows. At time j, just before removing the j-th box from the queue, define lj = min min L(X) . B∈Q X∈B

It is clear that lj gives a lower bound for the optimal solution C2 (Xopt ), since L(X) < C2 (X) for all X. We will show two things.

A Fast Optimal Algorithm for L2 Triangulation

285

Algorithm. Branch and Bound Given an initial bounding box B0 , initial optimal point X0 with value f0 = C2 (X0 ), and tolerance  > 0. 1. 2. 3. 4. 5.

6. 7. 8. 9. 10.

Initialize Q, a FIFO queue of bounding boxes, with Q = {B0 }. If Q is empty, terminate with globally optimal point X0 and optimal value f0 . Take the next box B from Q. Compute the largest dimension lmax of box B. If lmax < , – Set fC = C2 (XC ) where XC is the centroid of the box B. – If fC < f0 , set f0 = fC and X0 = XC . – Goto 2. Find the minimum of L(X) in B denoted by XL . Set fL = L(XL ). If fL > f0 , goto 2. Find the local minimum of C2 (X) denoted by XC with XL as the initial point. Set fC = C2 (XC ). If fC < f0 , set f0 = fC and X0 = XC . Subdivide B into two boxes B1 and B2 along the largest dimension, and insert B1 and B2 into Q. Goto 2.

Fig. 1. Branch and bound algorithm for optimal L2 triangulation. Alternative stopping conditions are possible, as discussed in the text.

1. The sequence of values for f0 set at line 8 of algorithm 1 is a decreasing sequence converging to C2 (Xopt ). 2. The sequence of values lj is an increasing sequence converging to C2 (Xopt ). Thus, the optimal value C2 (Xopt ) is sandwiched between two bounds, which can always be tested as the algorithm proceeds. The algorithm can be terminated when the difference between the bounds is small enough. We now proceed with the proof. Let δ > 0 be chosen. We will show that some value of f0 will be smaller than C2 (Xopt ) + δ, so the values taken by f0 converge to C2 (Xopt ). First, note that f0 will never be less than C2 (Xopt ) because of the way it is assigned at step 5 or 8 of the algorithm. The cost function C2 (X) is continuous on the box B0 , so the set {X ∈ B0 | C2 (X) < C2 (Xopt ) + δ} is an open set. Thus, there exists a ball S containing Xopt on which C2 takes values within δ of the optimal. Consider the sequence of boxes which would be generated in this algorithm if no boxes were eliminated at step 7. Since these boxes are decreasing in size, one of them B  must lie inside the ball S. Thus C2 (X) < C2 (Xopt ) + δ on B  . Note that no box that contains the point Xopt can be eliminated during the course of the branch-and-bound algorithm, so the box B  must be one of the boxes Bj that will eventually be evaluated. Since box B  can not be eliminated at step 7 of the algorithm, step 8 will be executed. The value fC found by optimizing starting in box Bj will result in a value less than C2 (Xopt ) + δ, since all points in Bj satisfy this condition. Thus, f0 will be assigned a value less than C2 (Xopt ) + δ, if it does not already satisfy this condition. This completes the proof that f0 converges to C2 (Xopt ).

286

F. Lu and R. Hartley

Next, we prove that lj converges to C2 (Xopt ). As before, define w ¯i = maxX∈B λi (X). Also define wi = minX∈B λi (X). As the size of the boxes diminishes towards zero, the ratio w ¯i /wi decreases towards 1. We denote by Bj the j-th box taken from the queue during the course of the algorithm 1. Then, for any  > 0 there exists an integer value N such that w ¯i < (1 + )wi for all i and all j > N. Now, using the same reasoning as in (4) with the directions of the inequalities reversed, we see that  f 2 (X) + g 2 (X) i

i

i

w ¯i λi (X)

≤ C2 (X) ≤

 f 2 (X) + g 2 (X) i

i

i

wi λi (X)

.

So, if j > N , then for any point X ∈ Bj , L(X) ≤ C2 (X) ≤ (1 + )L(X) .

(5)

We deduce from this, and the definition of lj that lj < C2 (Xopt ) ≤ (1 + )lj if j > N . Thus, lj converges to C2 (Xopt ).

6

Experiments

We tested our experiments with a set of trials involving all the points in the “Notre Dame” data set([10]). This data set contains 277,887 points and 595 cameras, and involves triangulation problems involving from 2 up to over 100 images. We compared our optimal triangulation method with both L∞ and Linear methods of triangulation, as well as with iterative methods using the BFGS algorithm and Levenberg-Marquardt, initialized by the Linear and L∞ algorithms. The results of these experiments are shown in Figures 2 and 3. Although the experiments were run on all 277,887 points, only 30,000 points (randomly chosen) are shown in the following graphs, because of difficulties plotting 270,000 points. Synthetic Data. Suspecting that previous methods will have difficulty with points near infinity, we devised an experiment that tested this. Results are a little preliminary for this, but they appear to show that the “homogeneous” Linear method (see section 2.1) still works well, but the inhomogeneous SVD method fails a few percent of the time, and iteration will not recover the failure. 6.1

Timing

Our algorithm is coded in C++ and takes about 0.02 seconds per point to run on a standard desktop computer in C++. We were unable to evaluate the algorithm of [1] directly, and its speed is a little uncertain. The authors have claimed 6-seconds (unpublished work) for average timing, but we can not verify this. Their algorithm is in Matlab, but the main numerical processing is done in Sedumi (which is coded in C). The algorithm is substantially more complex, and it is unlikely that the difference between Matlab and C++ would account for the 300-times speed-up of our algorithm compared with theirs.

A Fast Optimal Algorithm for L2 Triangulation

287

Fig. 2. Plot of Linear (left) and L∞ (right) triangulation versus the optimal algorithm. The graph shows the results for 30,000 points randomly chosen from the full Notre Dame set. The top row shows plots of the difference in residual (in pixels) between the two methods, plotted agains the point number. The points sorted according to the difference in residual so as to give a smooth curve, from which one sees the quantitative difference between the optimal and Linear or L∞ algorithms more easily. The plot for the Linear data is truncated on the Y -axis; the maximum difference in residual in reality reaches 12 pixels. The second row presents the same results in a different way. It shows a scatter plot of the optimal versus the Linear or L∞ residuals. Note that the optimal algorithm always gives better results than the Linear or L∞ algorithms. (Specifically, the top row shows that the difference of residuals is always positive.) This is expected and only serves to verify the optimality of the algorithm. In addition, L∞ is seen to outperform the Linear method.

7

Conclusion

The key feature of the proposed method is that it guarantees global optimality with a reasonable computation cost. It can be applied to large-scale triangulation problems. Although the given experiments do show that traditional local methods also work very well on most occasions, the problem of depending on the initialization will always be a disadvantage. Our method may be still a little slow for some large-scale applications, but it does provide an essential benchmark against which other algorithms may be tested, to verify whether they are obtaining optimal results. On real examples where the triangulated points were at a great distance from the cameras, the Linear algorithm gave such poor results that iteration was unable to find the optimal solution in many cases. On the other hand conventional

288

F. Lu and R. Hartley

Fig. 3. Plot of Levenberg-Marquardt (LM) refined results versus the optimal algorithm. The stopping criterion for the LM algorithm was our default (relatively quick) stopping criterion, resulting in about three iterations. The optimal algorithm still does best, but only on a few examples. Note that for visibility, we only show the first 2000 (out of 30,000) sorted points. When the LM algorithm and BFGS algorithms were run for more iterations (from either Linear or L∞ initialization), the results were largely indistinguishable on this data set from the optimal results. Also shown is a snapshot of the dataset that we used.

iterative methods worked well on the Notre Dame data set, because most of the points were relatively close or triangulated from a wide baseline.

References 1. Agarval, S., Chandraker, M., Kahl, F., Belongie, S., Kriegman, D.: Practical global optimization for multiview geometry. In: Proc. European Conference on Computer Vision (2005) 2. Hartley, R., Schaffalitzky, F.: L∞ minimization in geometric reconstruction problems. In: Proc. IEEE Conference on Computer Vision and Pattern Recognition, Washington DC, pp. I–504–509 (2004) 3. Triggs, W., McLauchlan, P.F., Hartley, R.I., Fitzgibbon, A.: Bundle adjustment for structure from motion. In: Triggs, B., Zisserman, A., Szeliski, R. (eds.) Vision Algorithms: Theory and Practice. LNCS, vol. 1883, pp. 298–372. Springer, Heidelberg (2000) 4. Kahl, F.: Multiple view geometry and the L∞ -norm. In: Proc. International Conference on Computer Vision, pp. 1002–1009 (2005) 5. Hartley, R.I., Zisserman, A.: Multiple View Geometry in Computer Vision, 2nd edn. Cambridge University Press, Cambridge (2003) 6. Hartley, R.I., Sturm, P.: Triangulation. Computer Vision and Image Understanding 68(2), 146–157 (1997) 7. Stewenius, H., Schaffalitzky, F., Nister, D.: How hard is 3-view triangulation really. In: Proc. International Conference on Computer Vision, pp. 686–693 (2005) 8. Nocedal, J., Wright, S.J.: Numerical Optimization, 2nd edn. Oxford University Press, Oxford (2006) 9. Boyd, S., Vanderberghe, L.: Convex Optimization. Cambridge University Press, Cambridge (2004) 10. Snavely, N., Seitz, S., Szeliski, R.: Photo tourism: Exploring photo collections in 3d. ACM Trans on Graphics 25(3), 835–846 (2006)

Adaptively Determining Degrees of Implicit Polynomial Curves and Surfaces Bo Zheng, Jun Takamatsu, and Katsushi Ikeuchi Institute of Industrial Science, The University of Tokyo, Komaba 4-6-1, Meguro-ku, Tokyo, 153-8505 Japan [email protected]

Abstract. Fitting an implicit polynomial (IP) to a data set usually suffers from the difficulty of determining a moderate polynomial degree. An over-low degree leads to inaccuracy than one expects, whereas an overhigh degree leads to global instability. We propose a method based on automatically determining the moderate degree in an incremental fitting process through using QR decomposition. This incremental process is computationally efficient, since by reusing the calculation result from the previous step, the burden of calculation is dramatically reduced at the next step. Simultaneously, the fitting instabilities can be easily checked out by judging the eigenvalues of an upper triangular matrix from QR decomposition, since its diagonal elements are equal to the eigenvalues. Based on this beneficial property and combining it with Tasdizen’s ridge regression method, a new technique is also proposed for improving fitting stability.

1

Introduction

Recently representing 2D and 3D data sets with implicit polynomials (IPs) has been attractive for vision applications such as fast shape registration, pose estimation [1,2,3,4], recognition [5], smoothing and denoising, image compression [6], etc. In contrast to other function-based representations such as B-spline, Non-Uniform Rational B-Splines (NURBS), and radial basis function (RBF) [7], IPs are superior in the areas of fast fitting, few parameters, algebraic/geometric invariants, robustness against noise and occlusion, etc. A 3D IP function of degree n is defined as:  aijk xi y j z k = (1 . . . z n)(a000 a100 . . . a00n )T , (1) fn (x) =  x     0≤i,j,k;i+j+k≤n m(x)T a where x = (x y z) is a data point. fn (x)’s zero set {x|fn (x) = 0} is used to represent the given data set. The estimation of IP coefficients belongs to the conventional fitting problem, and various methods have been proposed [1,2,3,4,8]. These fitting methods cannot adaptively control the IP degrees for different object shapes; it is well known that simple shapes correspond to low-degree IPs whereas complicated shapes correspond to the higher ones. Fig.1 shows that Y. Yagi et al. (Eds.): ACCV 2007, Part II, LNCS 4844, pp. 289–300, 2007. c Springer-Verlag Berlin Heidelberg 2007 

290

B. Zheng, J. Takamatsu, and K. Ikeuchi

→

(a)

(b)

(c)

→

(c’)

(d)

(d’)

Fig. 1. IP fitting results: (a) Original data set; (b) 4-degree IP surface; (c) 8-degree IP surface; (c’) Stability-improved 8-degree IP surface; (d) 10-degree IP surface; (d’) Stability-improved 10-degree IP surface

when fitting an object like Fig.1(a), an over-low degree leads to the inaccuracy (Fig.1(b)), whereas an over-high degree leads to an unstable result: too many unexpected zero sets appear (see Fig.1(d)). This paper provides a solution to automatically find the moderate degree IP (such as Fig.1(c)). Another issue of IP fitting is that there may be collinearity in the covariant matrix derived from the least squares method, making them prone to instability [4], e.g., the fitting results shown in Fig.1(c) and (d). In order to address this issue we propose a method for automatically checking out this collinearity and improving it. And we also combine the Ridge Regression (RR) technique recently introduced by [4,9]. Fig.1(c’) and (d’) show the improved results of Fig.1(c) and (d) respectively, where the redundant zero sets are retired. Note although Fig.1 (d) is globally improved by our method to Fig.1 (d’), since there are too many redundant zero sets that need to be eliminated, the local accuracy is also affected very much. Therefore, we first aim at adaptively finding a moderate degree, and then applying our stability-improving method to obtain a moderate result (accurate both locally and globally). This paper is organized as follows: Section 2 gives a review of IP fitting methods; Section 3 and 4 provide an incremental method for fitting IP with moderate degrees; Section 5 discusses on how to improve the global stability based on our algorithm; Section 6 presents experimental results followed by discussion and conclusions in section 7 and 8.

2

Implicit Polynomial Fitting

In order to estimate the coefficient vector a in (1), a simple method is to solve a linear system M a = b,

(2)

where M is the matrix of monomials, and the ith row of M is m(xi ) (see (1)) and b is a zero vector. But generally M is not a square matrix, and the linear system is usually solved by the least squares method. Solutions to this least squares problem are classified into nonlinear methods [1,2,10] and linear methods [3,4,8,9]. Because the linear methods are

Adaptively Determining Degrees of Implicit Polynomial Curves and Surfaces

291

simpler and much faster than the nonlinear ones, they are preferable and can be formulated as: M T M a = M T b.

(3)

Note this formula is just transformed from the least squares result, a = M † b, where M † = (M T M )−1 M T called a pseudo-inverse matrix. Direct solution of the linear system is numerically unstable, since M T M is nearly singular and b = 0. Thus a is determined from the kernel basis. Fortunately, the methods for improving the classical linear fitting (avoiding the ill-condition matrix M T M in (3)) have already been proposed by adding some constraints, such as the 3L method [3], the Gradient-One method [4] and the Min-Max method [8]. The singularity of M is improved and b is also derived as a nonzero vector. In the prior methods, the symmetric linear system (3) was solved by the classical solvers such as the Cholesky decomposition method, the conjugate gradient method, singular value decomposition (SVD), and their variations. But none of them allow changing the degree during the fitting procedure. This is the main reason why these prior methods require a fixed degree.

3

Incremental Fitting

This section shows computational efficiency of the proposed incremental fitting method. Although an IP degree is gradually increased until obtaining a moderate fitting result, the computational cost is saved because each step can completely reuse the calculation results of the previous step. In this section, first we describe the method for fitting an IP with the QR decomposition method. Next, we show the incrementability of Gram-Schmidt QR decomposition. After that, we clarify the amount of calculation in order to increase the IP degree. 3.1

Fitting Using QR Decomposition

Without solving the linear system (3) directly, our method first carries out QR decomposition on matrix M as M = QN ×m Rm×m , where Q satisfies the QT Q = I (I is the identity matrix), and R is an invertible upper triangular matrix. Then substituting M = QR into (2), we obtain: QRa = b → QT QRa = QT b → Ra = QT b → Ra =  b.

(4)

Since R is an upper triangular matrix, a can be quickly solved (in O(m2 )). 3.2

Gram-Schmidt QR Decomposition

Let us assume that matrix M consisting of columns {c1 , c2 , · · · , cm } is known. We show the method of Gram-Schmidt orthogonalization, that is, how to orthogonalize the columns of M into the orthonormal vectors {q1 , q 2 , · · · , q m } which

292

B. Zheng, J. Takamatsu, and K. Ikeuchi Ri +1

~ bi

ai

Ri

~ bi

ri , j

=

~ ai +1 bi +1 ~ bi

ri , j

G

=

0

0 0

Fig. 2. The triangular linear system grows from the ith step to the (i + 1)th, and only the calculation shown in light-gray is required

are columns of matrix Q, and simultaneously calculate the corresponding upper triangular matrix R consisting of elements ri,j . The algorithm is as follows: Initially let q 1 = c1 /  c1  and r1,1 = c1 . If {q1 , q 2 , · · · , q i } have been computed, then the (i+1)th step is: rj,i+1 = q Tj ci+1 , for j ≤ i, q i+1 = ci+1 −

i 

rj,i+1 q j ,

j=1

ri+1,i+1 = q i+1 , q i+1 = q i+1 /  q i+1  .

(5)

From this algorithm, we can see that Gram-Schmidt orthogonalization can be carried out in an incremental manner, which orthogonalizes the columns of M one by one. 3.3

Additional Calculation for Increasing an IP Degree

The incrementability in the QR decomposition with Gram-Schmidt orthogonalization makes our incremental method computationally efficient. Fig.2 illustrates this efficiency by the calculation from the ith step to the (i + 1)th step in our incremental process. It is only necessary to calculate the parts shown in light-gray. For constructing these two upper triangular linear systems from the ith step to the (i + 1)th step, we only need two types of calculation: 1) for growing the upper triangular matrix from Ri to Ri+1 , calculate the rightmost column and add it into the Ri to construct Ri+1 , and 2) for growing the right-hand vector from  bi to  bi+1 , calculate the bottom element and add it into the  bi to construct  bi+1 . For the first calculation, it can be simply obtained from Gram-Schmidt orthogonalization in (5). For the second calculation, assuming bi+1 is the bottom element of vector  bi+1 , the calculation of bi+1 can follow the (i + 1)th step of Gram-Schmidt orthogonalization in (5), and can be calculated as bi+1 = q Ti+1 b. In order to clarify the computational efficiency, let us assume a comparison between our method and an incremental method that iteratively calls a linear

Adaptively Determining Degrees of Implicit Polynomial Curves and Surfaces

293

method such as the 3L method at each step. It is obvious that, for solving coefficient a at the ith step, our method needs i inner-product operations for constructing the upper triangular linear system, and O(i2 ) for solving this linear system; whereas the latter method needs i × i inner-product operations for constructing linear system (3), and O(i3 ) for solving (3). Let us define a function G to denote the above calculation. Then if we repeat calling function G, we can obtain the incremental (dimension-growing) upper triangular linear systems, and the corresponding coefficient vectors with different degrees can be solved from them.

4

Finding the Moderate Degree

Now to construct an algorithm for finding the moderate degree, we are facing two problems: 1) What is the order for choosing cα ? 2) When should the incremental procedure be stopped? Note: as a matter of convenience, hereafter we use notation α to denote the column index of M instead of i. 4.1

Incremental Order of Monomials

Feeding the columns cα from M into the function G in a different order may lead to the different result. Therefore it is important to decide a suitable order. A reasonable way is to choose cα in the degree-growing order described by Tab.1. The reason is as follows: when we fit a two-degree IP to the data on a unit circle, a unique solution −1 + x2 + y 2 = 0 can be obtained, while if we choose a three-degree IP to fit, solutions such as x(−1 + x2 + y 2 ) = 0, are obtained. There exist some redundant zero set groups, such as x = 0. 4.2

Stopping Criterion

For the second problem, we have to define a stopping criterion based on our defined similarity measure between IP and data set. Once this stopping criterion is satisfied, we consider the desired accuracy is reached and the procedure is terminated. First let us introduce a set of similarity functions measuring the similarity between IP and data set, as follows: Ddist =

N 1  ei , N i=1

Dsmooth =

N 1  (N i · ni ), N i=1

(6)

|f (xi )| where N is the number of points, ei = f (xi ) , N i is the normal vector at a point obtained from the relations of the neighbor normals (here we refer to f (xi ) Sahin’s method [9]), and ni = f (xi ) is the normalized gradient vector of f at xi . ei has proved useful for approximating the Euclidean distance from xi to the IP zero set [2].

294

B. Zheng, J. Takamatsu, and K. Ikeuchi

Table 1. Index List: i, j and k are the powers of x, y and z respectively. α is the index of column of M . And the relations between α and (i, j, k) can be formulated as: α = j + (i + j + 1)(i + j)/2 + 1 (for 2D) and α = k + (j + k + 1)(j + k)/2 + (i + j + k + 2)(i + j + k + 1)(i + j + k)/6 + 1 (for 3D). α 1 2 3 4 5 6 7 8 9 10

[i j] [0 0] [1 0] [0 1] [2 0] [1 1] [0 2] [3 0] [2 1] [1 2] [0 3]

Form L0 (i + j = 0) L1 (i + j = 1)

L2 (i + j = 2)

L3 (i + j = 3) .. . m [0 n] Ln (i + j = n) (a) Index list for 2D

[i j k] Form [0 0 0] L0 (i + j + k [1 0 0] [0 1 0] [0 0 1] L1 (i + j + k [2 0 0] [1 1 0] [0 2 0] [1 0 1] [0 1 1] [0 0 2] L2 (i + j + k .. . m [0 n] Ln (i + j + k (b) Index list for 3D α 1 2 3 4 5 6 7 8 9 10

= 0)

= 1)

= 2) = n)

Ddist and Dsmooth can be considered as two measurements on distance and smoothness between data set and IP zero set. And we define our stopping criterion as: (Ddist < T1 ) ∧ (Dsmooth > T2 ). 4.3

(7)

Algorithm for Finding the Moderate IPs

Having the above conditions, our algorithm is simply described as follows: 1) Calling the function G to construct the upper triangular linear system; 2) Solving this linear system to obtain coefficient vector a; 3) Measuring the similarity for the obtained IP; 4) Stopping the algorithm if the stopping criterion (7) is satisfied; otherwise going to 1) for growing up the dimension.

5

Improving Global Stability

Linear fitting methods in general suffer from not achieving global stability, which is well discussed in [4,9]. Since our fitting method belongs to these linear methods, we face the same problem. We propose a method for solving this by controlling the condition number of matrix M . 5.1

Stability and Condition Number of M

An important reason for global instability is the collinearity of M , which causes the matrix M T M to be nearly singular with some eigenvalues negligible

Adaptively Determining Degrees of Implicit Polynomial Curves and Surfaces

295

compared to the others [4]. Such eigenvalues are degenerated to contribute very little to the overall shape of the fit. Fortunately, since M = QR, we can obtain M T M = RT R, and thus the condition number of M T M can be improved by controlling the eigenvalues of R. Note here we just consider the condition number as λmax /λmin , where λmax and λmin are the maximum and minimum eigenvalues respectively. And from the good properties that upper triangular matrix R’s eigenvalues lay on its main diagonal, we can easily evaluate the singularity of R by only observing the diagonal values. To improve the condition number of M T M , this paper gives a solution from two aspects: eliminating collinear columns of M and using the Ridge Regression method. 5.2

Eliminating Collinear Columns of M

The first simple idea is to check the eigenvalue ri,i to see whether it is too small (nearly zero), at each step. If ri,i is too small, that means the current column ci of M is nearly collinear to the space of {c1 , c2 , · · · , ci−1 }. Thus to find a viable value for R, it should be abandoned, and the subsequent columns should be tried. 5.3

Ridge Regression Method

Ridge regression (RR) regularization is an effective method that improves the condition number of M T M by adding a small moderate value to each diagonal element, e.g., adding a term κD to M T M [4,9]. Accordingly equation (3) can be modified as (M T M + κD)a = M T b and equation (4) can be modified as (R + κR−T D)a =  b

(8)

where κ is a small positive value called the RR parameter and D is a diagonal matrix. D will be chosen to maintain Euclidean invariance, and the simplest choice is to let D be an identity matrix. A cleverer choice has been proposed by T.Tasdizen et al. [4] for 2D and T.Sahin et al. [9] for 3D. In fact, their strategies are to add the constraints that keep the leading forms of this polynomial always strictly positive, which proves that the zero set of polynomials with an even degree are always bounded (see the proof in [4]). We give details of this derivation in the appendix.

6

Experimental Results

The setting for our experiments involve some pre-conditions. 1) As a matter of convenience, we employ the constraints of the 3L method [3]; 2) All the data sets are regularized by centering the data-set center of mass at the origin of the coordinate system and scaling it by dividing each point by the average length from points to origin; 3) We choose T1 in (7) with about 20 percent of the layer distance c of the 3L method as done in [3].

296

B. Zheng, J. Takamatsu, and K. Ikeuchi









Ddist Dsmooth

(b)

Ddist

(a)









 

(c)



Dsmooth











α









 

(d) (e)

Fig. 3. IP fitting results: (a) Original image. (b) α = 28 (six-degree). (c) α = 54 (ninedegree). (d) α = 92 (thirteen-degree). (e) Convergence of Ddist and Dsmooth . Note “o” symbols represent the boundary points extracted from the image and real lines represent the IP zero set in (b)-(d).

6.1

A Numerical Example

In this experiment, we fit an IP to the boundary of a cell shown in Fig.3 (a). The stopping criterion is set as T1 = 0.01, T 2 = 0.95, and the layer distance of the 3L method is c = 0.05. The moderate IP is found out automatically (see Fig.3(d)). To give a comparison, we also show some fits before the desired accuracy is reached (see Fig.3(b) and (c)). And these results are improved by our method mentioned in section 5. We also track the convergence of Ddist and Dsmooth shown in Fig.3(e). Although there are some small fluctuations on the graph, Ddist and Dsmooth are convergent to 0 and 1 respectively, which also proves the stopping criterion in (7) can effectively measure the similarity between the IP and the data set. 6.2

2D and 3D Examples

Some 2D and 3D experiments are shown in Fig.4 where the fitting results are obtained with the same parameters as those in the first example. As a conclusion here, objects with different shapes may obtain fitting results of different degrees, since these results always respect the complexity of shapes. 6.3

Degree-Fixed Fitting Compared with Adaptive Fitting

Fig.5 shows some comparisons between degree-fixed fitting methods and our adaptive fitting method. Compared to degree-fixed methods such as [3,4,8], the results of our method show that there is neither over-fitting nor the insufficient

Adaptively Determining Degrees of Implicit Polynomial Curves and Surfaces

11-degree

12-degree

12-degree

6-degree

8-degree

297

12-degree

Fig. 4. IP fitting results. First row: Original objects; Second row: IP fits.

fitting, and we also attain global stability. It shows that our method is more meaningful than the degree-fixed methods, since it fulfills the requirement that the degrees should be subject to the complexities of object shapes.

7 7.1

Discussion QR Decomposition Methods

Other famous algorithms of QR decomposition are Householder and Givens [11]. In the field of numerical computation, Householder and Givens have proved more stable than conventional Gram-Schmidt method. But in this paper, since our discussion is based on the good condition of a regularized data set, we ignore the small effect of rounding errors, which causes instability. Here we just would like to take advantage of the properties of QR decomposition that orthogonalize vectors one by one, to demonstrate the possibility of constructing the moderatedegree fitting algorithm described above. 7.2

IP vs Other Functions

In contrast to other function based representations such as B-spline, NURBS, and radial basis function, IP representation cannot give a relatively accurate model. But this representation is more attractive for applications that require fast registration and fast recognition (see the works [4,5,12,13,14]), because of its algebraic/geometric invariants [15]. Also, Sahin also showed some experiments for missing data in [9]. Further accurate representation of a complex object may require segmenting the object shapes and representing each segmented patch with an IP. We will consider this possibility in our future works.

298

B. Zheng, J. Takamatsu, and K. Ikeuchi

Original Objects:

Degree-fixed fitting in 2-degree: 2-degree *

2-degree †

2-degree †

4-degree ‡

4-degree †

4-degree †

2-degree *

6-degree *

12-degree *

Degree-fixed fitting in 4-degree:

Our method:

Fig. 5. Comparison between degree-fixed fitting and adaptive fitting. First row: Original objects. Second and third row: IP fits resulting from degree-fixed fitting with 2-degree and 4-degree fitting respectively. Fourth row: Adaptive fitting. Mark *: moderate fitting. †: insufficient fitting (losing accuracy). ‡: over-fitting.

7.3

Setting the Parameters T1 and T2

Since our stopping criterion can be approximately seen as a kind of Euclidean metric, it is intuitive to control moderate fitting accuracy by setting the appropriate values to T1 and T2 . Basically, these parameters should be decided based on object scale or statistics about data noise. Further discussion is beyond the scope of this paper. Fortunately it is intuitive to decide if the parameters are appropriate for your demand, since the 2D/3D Euclidean distance can be easily observed. In this paper, we practically let T1 and T2 be close to zero and one respectively for a smooth model and more tolerant values for a coarse one.

8

Conclusions

This paper provided an incremental method for fitting shape-representing IPs. By our stopping criterion, an IP of moderate degree can be adaptively found in one fitting process, and global fitting stability is successfully improved. Our results support the argument that IP degrees being adaptively determined by shapes is better than being fixed, because this not only saves much time for users, but also it is suited to the future applications involving automatic recognition systems.

Adaptively Determining Degrees of Implicit Polynomial Curves and Surfaces

299

Acknowledgements Our work was supported by the Ministry of Education, Culture, Sports, Science and Technology under the Leading Project: Development of High Fidelity Digitization Software for Large-Scale and Intangible Cultural Assets.

References 1. Keren, D., Cooper, D.: Describing Complicated Objects by Implicit Polynomials. IEEE Trans. on Patt. Anal. Mach. Intell (PAMI) 16(1), 38–53 (1994) 2. Taubin, G.: Estimation of Planar Curves, Surfaces and Nonplanar Space Curves Defined by Implicit Equations with Applications to Edge and Range Image Segmentation. IEEE Trans. on Patt. Anal. Mach. Intell (PAMI) 13(11), 1115–1138 (1991) 3. Blane, M., Lei, Z.B., Cooper, D.B.: The 3L Algorithm for Fitting Implicit Polynomial Curves and Surfaces to Data. IEEE Trans. on Patt. Anal. Mach. Intell (PAMI) 22(3), 298–313 (2000) 4. Tasdizen, T., Tarel, J.P., Cooper, D.B.: Improving the Stability of Algebraic Curves for Applications. IEEE Trans. on Imag. Proc. 9(3), 405–416 (2000) 5. Tarel, J.-P., Cooper, D.B.: The Complex Representation of Algebraic Curves and Its Simple Exploitation for Pose Estimation and Invariant Recognition. PAMI 22(7), 663–674 (2000) 6. Helzer, A., Bar-Zohar, M., Malah, D.: Using Implicit Polynomials for Image Compression. In: Proc. 21st IEEE Convention of the Electrical and Electronic Eng., pp. 384–388. IEEE Computer Society Press, Los Alamitos (2000) 7. Turk, G., OfBrieni, J.F.: Variational Implicit Surfaces. Technical Report GITGVU-99-15, Graphics, Visualization, and Useability Center (1999) 8. Helzer, A., Barzohar, M., Malah, D.: Stable Fitting of 2D Curves and 3D Surfaces by Implicit Polynomials. IEEE Trans. on Patt. Anal. Mach. Intell (PAMI) 26(10), 1283–1294 (2004) 9. Sahin, T., Unel, M.: Fitting Globally Stabilized Algebraic Surfaces to Range Data. In: Proc. 10th IEEE Int. Conf. on Compter Vision (ICCV), vol. 2, pp. 1083–1088 (2005) 10. Knanatani, K.: Renormalization for Computer Vision. The Institute of Elec., Info. and Comm. eng (IEICE) Transaction 35(2), 201–209 (1994) 11. Horn, R.A., Johnson, C.R.: Matrix Analysis: Section 2.8. Cambridge University Press, Cambridge (1985) 12. Tarel, J.P., Civi, H., Cooper, D.B.: Form 3d objects without point matching using algebraic surface models. In: Proceedings of IEEE Workshop Model Based 3D Image Analysis, pp. 13–21. IEEE Computer Society Press, Los Alamitos (1998) 13. Khan, N.: Silhouette-Based 2D-3D Pose Estimation Using Implicit Algebraic Surfaces. Master Thesis in Computer Science, Saarland University (2007) 14. Unsalan, C.: A Model Based Approach for Pose Estimation and Rotation Invariant Object Matching. Pattern Recogn. Lett. 28(1), 49–57 (2007) 15. Taubin, G., Cooper, D.: Symbolic and Numerical Computation for Artificial Intelligence. In: Computational Mathematics and Applications. ch. 6, Academic Press, London (1992)

300

B. Zheng, J. Takamatsu, and K. Ikeuchi

Appendix A. Choosing Diagonal Matrix D for RR Method A choice of diagonal matrix D for RR method was derived by T.Tasdizen et al. [4] and T.Sahin et al. [9] as: dαα = γ tˆ, where dαα is the αth diagonal element of D, and calculating dαα is respected to i, j, k. The relationship between index α and i, j, k are shown in Tab.1. γ is a free parameter for the (i + j)th form (2D) or (i + j + k)th form (3D) decided from the data set as follows: γi+j =

N 

(x2t + yt2 )i+j

(2D), γi+j+k =

t=1

N 

(x2t + yt2 + zt2 )i+j+k

(3D), (9)

t=1

where (xt yt ) and (xt yt zt ) are the data point. tˆ is a variable respected to i, j, k and for maintaining Euclidean invariance it can be derived as: tˆ =

i!j! (i + j)!

(2D) and tˆ =

i!j!k! (i + j + k)!

(3D).

(10)

Determining Relative Geometry of Cameras from Normal Flows Ding Yuan and Ronald Chung Department of Mechanical & Automation Engineering The Chinese University of Hong Kong Shatin, Hong Kong, China {dyuan,rchung}@mae.cuhk.edu.hk

Abstract. Determining the relative geometry of cameras is important in active binocular head or multi-camera system. Most of the existing works rely upon the establishment of either motion correspondences or binocular correspondences. This paper presents a first solution method that requires no recovery of full optical flow in either camera, nor overlap in the cameras’ visual fields and in turn the presence of binocular correspondences. The method is based upon observations that are directly available in the respective image stream – the monocular normal flow. Experimental results on synthetic data and real image data are shown to illustrate the potential of the method. Keywords: Camera calibration, Extrinsic camera parameters, Active Vision.

1 Introduction Active vision systems allow the position and orientation of each camera be arbitrarily controlled according to need and have many advantages. They however require the relative geometry of the cameras be determined from time to time for fusion of the visual information. Determination of the relative geometry of cameras is a wellstudied problem. What makes the problem unique and particularly challenging in active vision is that there is no guarantee on how much overlap there exists between the visual fields of the cameras. In the extreme case, there could be no overlap. There have been many methods in the literature proposed for binocular geometry determination. Some methods require certain specific objects appearing in the scene, such as planar surfaces [7] and cubic objects [9]. Such methods constitute simpler solution mechanisms, but they are restricted to certain applications or scenes. Other methods [3] [11] require the accessibility of either cross-camera feature correspondences [1] [11] or motion correspondences [2]. While cross-camera correspondences require the cameras to have much in common in what they picture, establishing motion correspondences is an ill-posed problem and the result is not always reliable when the scene contains much occlusion. The objective of this work is to develop a solution method that does not assume presence of calibration objects or specific shapes or features in the imaged scene, nor impose restriction on the viewing directions of the cameras, thus allowing the visual fields of the cameras to have little or zero overlap. Y. Yagi et al. (Eds.): ACCV 2007, Part II, LNCS 4844, pp. 301–310, 2007. © Springer-Verlag Berlin Heidelberg 2007

302

D. Yuan and R. Chung

In an inspiring work, Fermüller and Aloimonos [4][5] described a method (hereafter referred to as the FA-camera motion determination method) of determining the ego-motion of a camera directly from normal flow. Normal flow is the apparent motion of image position in the image stream picked up by the camera, which must be in the direction of the intensity gradient, and can be detected directly by applying some simple derivative filters to the image data. Fermüller and Aloimonos first define, for any arbitrarily chosen 3D axis p that contains the optical center, a vector field over the entire image space. Patterns with “+” and “-” candidates are generated according to the vector field and the normal flows. The camera motion parameters can be determined from those patterns. The FA-camera motion determination method forms the basis of a method we proposed earlier [10] for determining the relative geometry of two cameras. However, a key issue has not been addressed in [10]. Any p-axis allows not all but only a small subset of the data points – those whose normal flow is parallel or anti-parallel to the field vector there with respect to p-axis – to be usable. Different choices of the p-axis would allow different subsets of the data points to be usable, each with a different density of the data points in the space. The choices of the p-axis are thus crucial; they determine the density of the data points and the accuracy in determining the camera motions and in turn the binocular geometry. However, in both the FA-camera motion determination method [4][5] and in our earlier work on binocular geometry determination [10], no particular mechanism was devised in choosing the p-axes. This paper presents how the p-axes can be chosen optimally for best use of the data points. We assume that the intrinsic parameters of the cameras have been or are to be estimated by camera self-calibration methods like [8] [11]. The focus of this work is the estimation of the camera-to-camera geometry.

2 Copoint Vector Field with Respect to Arbitrary 3D Direction Fermüller and Aloimonos [4][5] proposed vector filed which was then applied in their camera motion determination method. Our binocular geometry determination method also makes use of this vector field. So here we give a brief review of it. Suppose the image space is represented by an image plane positioned perpendicular to the optical axis and at 1 unit away from the optical center. Pick any arbitrary axis p=[A B C] in space that contains the optical center; the axis hits the image plane at the point P=[A/C B/C]. The family of projection planes that contains axis p define the family of lines that contain point P on the image plane. The copoint vector field for the image space with respect to the p-axis is defined as the field with vectors perpendicular to the above family of lines about point P, as shown in Fig. 1 (a). In the figure each arrow represents the vector assigned to each image point, and it is [M x

My] =

[− y + B / C

x − A / C]

(2.1)

( x − A / C ) + (− y + B / C ) 2 2

Suppose the camera undergoes a pure rotation, which is represented in the rotationaxis form by a vector ω=[α β γ]. For any particular choice of the p-axis, we have a

Determining Relative Geometry of Cameras from Normal Flows

303

p-copoint field direction (Equation 2.1) at each image position. At any image position, the dot product between the p-axis induced field vector and the optical flow there allows the image position to be labeled as either: “+” if the dot product is positive, or “-” if the dot product is negative. By the distributions of the “+” and “-” labels, the image plane is divided into two regions: positive and negative, with the boundary being a 2nd order curve called the zero-boundary, as illustrated by Fig.1 (b). The zero-boundary is only determined by the ratios α/γ and β/γ. Fig.1 (c) illustrates the positive-negative pattern generated the same way as described above, when the camera takes a pure translation. Different from the pattern Fig.1 (b), the zeroboundary is a straight line (Fig.1 (c)), which is a function of the focus of expansion (FOE) of the optical flow and precisely describes the translational direction of the camera. If the motion of the camera includes both rotation and translation, the positivenegative pattern will include the positive region, the negative region, and the “Don’t know” region (depends on the structure), as shown in Fig. 1 (d). P=[A/C B/C ]

(a)

Positive

Negative

(b)

(c)

Don’t know

(d)

Fig. 1. p-copoint vector field and its positive-negative patterns in the image space. (a) pcopoint vector field; positive-negative labeled patterns when (b) camera undergoes pure rotation; (c) camera undergoes pure translation; (d) camera takes arbitrary motion (with both rotation and translation components).

The above labeling mechanism allows constraint for the camera motion to be constructed from optical flow. However, due to the aperture problem the full flow is generally not directly observable from the image data; only the normal flow, i.e., the component of the full flow projected to the direction of the local intensity gradient, is. The above labeling mechanism therefore has to be adjusted, and the positive-negative pattern still can be generated [4] [5]. The main difference is, while with the full flows all the data points can be labeled, with the normal flows only a handful of the data points can be labeled, and the localization of the zero-boundary from the sparsely labeled regions is much more challenging.

3 Binocular Geometry Determination Suppose the binocular geometry of two cameras at a particular configuration of the camera system is to be determined. Our procedure consists of the following. The binocular configuration is first frozen, and then moved rigidly in space while image streams are collected from the respective cameras to serve as data for the

304

D. Yuan and R. Chung

determination task. If the binocular geometry is expressed by a 4×4 matrix X, and the two camera motions by A and B, because of the rigidity of the motion of the camera pair we have the relationship AX=XB, which can be decomposed into the following two equations: R AR x = R x R B

(3.1)

(or ω A = R xω B in vector form) and (R A − I )t x = R x t B − tA

(3.2)

where Rx, RA, RB are the 3×3 orthonormal matrices representing the rotational components of X, A, B respectively, tx, tA, tB are the vectors representing the translational components, and ωA, ωB are the rotations RA, RB expressed in the axisangle form. The plan is, if the camera motions A and B can be determined from normal flows by the use of say the FA- camera motion determination method, Equations (3.1) and (3.2) would provide constraints for the determination of the binocular geometry parameters X. However, there are two complications. First, if general motion of any camera (i.e., motion that involves both translation and rotation) is involved, the image space associated with the camera under the FA- camera motion determination method contains two “Don’t know” regions as depicted by Fig. 1(d). The presence of the “Don’t know” regions would add much challenge to the localization of the zeroboundaries. Second, with only the normal flows in the respective image streams of the two cameras, only a small percentage of the data points can be made usable in the image space under a random choice of the p-axis. This complication is the most troubling, as the localization of the zero boundary from very sparsely labeled data points would be an extremely difficult task. On the first complication, we adopt specific rigid motions of the camera pair to avoid as much as possible the presence of general motion of any camera. On the second complication, we propose a scheme that allows the p-axes to be chosen not randomly as in previous work [4][5][10], but according to the how many data points they can make useful in the copoint vector field based method. The scheme allows each data point (an image position with detectable normal flow) to propose a family of p-axes that could make that data point useful, and vote for them in the space of all possible p-axes. The p-axes in the p-axis space that have the highest number of votes from all the data points are then the p-axes we use in the process. 3.1 Determination of Rx We first determine the rotation component Rx of the binocular geometry. We let the camera pair undergo a specific motion – pure translation – so as to reduce the complexity in locating the zero-boundary of the positive-negative labeled patterns in the image space. When the camera pair exercise a rigid-body pure translation, the motion of each camera is also a pure translation. From Equation (3.2) we have: ~ ~ (3.3) tA = R x tB

Determining Relative Geometry of Cameras from Normal Flows

305

where ~tA and ~tB are both unit vectors corresponding to the focus of expansions (FOEs) of the two cameras. Previously we proved at least two translational motions in different directions are required to achieve a unique solution of Rx [10]. Solution of Rx is presented in our previous work[10]. On determining ~tA and ~tB in each rigid-body translation of the camera pair, we adopt the p-copoint vector field model to generate patterns from the normal flows in the respective image streams. Both cameras will exhibit pattern like that shown in Fig. 1 (c), containing only the positive and the negative regions separated by a straight line (the zero boundary), without the “Don’t know” regions. 3.2 Determination of tx Up to Arbitrary Scale To determine the baseline tx of the binocular geometry, we let the camera pair undergo rigid-body pure rotations while the cameras capture the image stream data. However, tx can only be determined up to arbitrary scale unless certain metric measurement about the 3D world is available. Suppose the camera pair has a pure rotation about an axis passing through the optical center of one camera, say the optical center of camera A. Then camera A only undergoes a pure rotation, while camera B’s motion consists of a rotation about an axis containing the optical center of camera A, and a translation orthogonal to the baseline. In this case Equation (3.2) can be rewritten as: (3.4) (R A − I)t x = R xt B We then rewrite Equation (3.4) to a homogeneous system: )~ A tx = 0

(3.5)

) where tx is the normalized vector representing the direction of the baseline, and A is a )

2×3 matrix calculated from Rx, RA, tB with Rank( A ) = 1. At least two rotations are needed to determine tx uniquely [10]. Camera A, which has only pure rotations in the process, has the positive-negative labeled patterns in the image space just like the one shown in Fig.1 (b), in which a 2nd order curve (the zero-boundary) separates the ‘+’ and ‘-’ labeled regions. As for camera B, the positive-negative labeled patterns in the image space take the form of Fig.1 (d), and are more challenging because of the existence of two “Don’t know” regions. There are two zero boundaries to be determined: one a 2nd order curve, and the other a straight line. The strategy on calculating tx (up to arbitrary scale) by analyzing the positive-negative labeled patterns is presented in our earlier work [10].

4 Optimal Selection of p-Axes Under different choices of the p-axis, different subsets of data points would be made usable to generate the positive-negative labeled patterns in the image space. Obviously a higher density of the labeled patterns is desired, as it would make the localization of the zero-boundary easier. In this section we propose a scheme for that. In the following discussion, for simplicity we only describe the scheme under the case that the camera motion is a pure translation. The scheme for the pure rotation case is actually similar.

306

D. Yuan and R. Chung

4.1 From a Data Point of Normal Flow to a Locus of p-Axes For any given p-axis, only the data points with the normal flows exactly parallel or anti-parallel with the p-copoint field vectors there are usable data for participating in the positive-negative patterns in the image space. Viewing the whole process in the opposite angle, a data point (xi, yi) with normal flow (uin, vin) is usable only under a paxis whose equivalent image position P =[px , py] is located on the line li passing through the data point (xi, yi) and orthogonal to the normal flow (uin, vin), as illustrated by Fig. 2 . We call the line li the P-line of the data point (xi, yi), and it can be expressed as:

u i n p x + v i n p y − (u i n x i + v i n y i ) = 0

(4.1)

Thus, to find the p-axis which can make the maximum number of data points useful, a simple scheme is to let each data point vote for the members of its P-line in the space of all possible p-axes (which is only a two-dimensional space, as each paxes contains only two degrees of freedom). The p-axes that collect a large number of votes are then the good choices of p-axes we should use in the copoint vector field based method. li lj

(ujn, vjn)

(xi, yi)

(xj, yj)

(uin, vin) P =(px,py)

Fig. 2. The P-lines of data points (xi, yi) (with normal flow (uin, vin)) and (xj, yj) (with normal flow (ujn, vjn))

4.2 Optimal Determination of p-Axes

Obviously we could obtain a linear system of equations for the optimal p-axis (point P) from say n data points using Equation (4.1), and solve for the optimal p-axis. However, the normal flows’ orientations are extracted not without error, so each data point should vote not for a P-line, but a narrow cone centered at the data point and swung from the P-line. The size of the cone is a threshold that depends upon what estimated error we have in the extraction of the normal flows’ orientations. We thus adopt a voting scheme similar to the Hough Transform. We use an accumulator array to represent the entire space of all p-axes, and to collect votes from each data point. The accumulator is a two-dimensional array whose axes correspond to the quantized values of px and py. For each data point (an image point with detectable normal flow), we determine its P-line, look for bins in the accumulator array that the line falls into, and put one vote in each of those bins. After we finish this with all the data points, we identify the bins of the highest count of votes in the accumulator array. An example of an accumulator array is shown in Fig. 3(a). To

Determining Relative Geometry of Cameras from Normal Flows

(a )

(b)

(c)

307

(d)

Fig. 3. (a) Two-dimensional accumulation array that corresponds to various values of px and py. The P-line associated with each data point is determined, the array bins corresponding to the line are identified, and each of such bins has the vote count increased by one. The bin with the highest vote count is identified (and marked as a red circle in this figure), which corresponds to the optimal p-axis. (b) (c) (d): The development of the voting process under the coarse-to-fine strategy.

increase computational efficiency we use a coarse-to-fine strategy in the voting process, as illustrated by Fig. 3(b-d). Since the copoint vector field based method demands the use of not one but a few p-axes, in our case we use not only the optimal p-axis but a few p-axes of the highest number of vote counts. While in synthetic data experiments the scene texture (and thus the orientation of the normal flow) is often made random, making all p-axes having similar density of usable data points, in real image data the scene texture is often oriented to a few directions (and so is the normal flow), and densities of the usable data points could be drastically different under different choices of the p-axes. Our experience show that, especially in the real image data cases, the adoption of the optimal p-axes makes drastic improvement to the solution quality over those under random selection of the p-axes. More specifically, our experiments on real image data show that the pattern generated by the best p-axes often has 60% more data points than those under the average p-axes.

5 Experimental Results The whole method consists of two steps. First, the binocular cameras undergo pure translations twice as a whole, and each time they move in different directions. The rotational component Rx is computed in this first step. After that, we rotate the camera pair twice around two different axes passing through the optical center of one of the cameras. In this step tx is determined up to scale. 5.1 Experimental Results on Synthetic Data

The experiments on synthetic data aim at investigating the accuracy and precision of the method. Normal flows are the only input, same as in the case of real image experiments. We used image resolution 101×101 in the synthetic data.

308

D. Yuan and R. Chung

Estimation of Rx. The normal flows were generated by assigning to each image point a random intensity gradient direction. Dot product between the gradient direction and the optical flow incurred from the assumed camera motion determined the normal flow precisely. We selected the optimal set of p-axes first. With the first optimal p-axis we got the first positive-negative labeled pattern in the image space. After determining the pseudo FOEs at an accuracy of 0.25×0.25 pixel, a number of lines, determined from different pseudo FOEs, could well divide the pattern into two regions. Then we applied a second optimal p-axis to examine if those pseudo FOEs that had good performance in the first pattern still performed well in this new pattern. We kept those that still had good performance in the next round under the new p-axis. We repeated this process, until all possible FOEs were located within a small enough area. Then the center of these possible FOEs was considered as the input for computing Rx. We estimated the FOEs by locating the zero-boundaries for both camera A and B first, and the rotational component ωx of the binocular geometry was then estimated. The calculation result is shown in the Tab.1. The error was 0.7964o in direction, 1.2621% in length. Estimation of tx up to Arbitrary Scale. We assumed that the camera pair rotated about an axis passing the optical center of camera A at two different given velocities. As above, the normal flows were generated to be the inputs. We located the zero boundaries on the positive-negative labeled patterns to estimate rotations ωA of camera A, using the algorithm named “detranslation” [4][5]. FOE tB of camera B was obtained readily from the patterns. Finally we obtained tx up to arbitrary scale using Equation (3.5). The result, shown in Tab.1, is a unit vector describing the direction of the baseline. The angle between the ground truth and the result is 2.0907o. Table 1. Estimation of ω x and t x up to scale

ωx tx

Ground Truth [0.100 0.100 -0.200]T [-700 20 80]T

Experiment [ 0.097 0.204 -0.203] T [-0.988 0.043 0.147] T

In this experiment, synthetic normal flows, computed from full optical flows by allocating to each pixel a random gradient direction, are oriented more evenly to all directions than in real image data, because in real image data the scene texture is often oriented as discussed above. However, the accuracy of our method can be better explored in the synthetic data experiments. 5.2 Experimental Results on Real Image Data Here we only show results on the recovery of Rx (ωx) due to limitation of page space. We moved the camera pair on a translational platform. The image sequences were captured by Dragonfly CCD cameras of resolution 640×480. The first experiment is to investigate the accuracy of the algorithm. We used the algorithm described in [6] to estimate the intrinsic parameters of the two cameras.

Determining Relative Geometry of Cameras from Normal Flows

309

Input images were first smoothed by using Gaussian Filter with n=5 and σ=1.4 to eliminate the Gaussian noise. We examined pseudo FOEs pixel by pixel in the image frames. 377 p-axes were enough to pinpoint the locations of possible FOEs. The zeroboundaries determined by the estimated FOEs are shown in Fig 4.

(a)

(b)

(c)

(d)

Fig 4. The zero-boundaries (blue lines) determined by estimated FOEs. Green dots represent negative candidates; red dots represent positive candidates. (a) Camera A, Motion 1; (b) Camera B, Motion 1; (c) Camera A, Motion 2; (d) Camera B, Motion 2.

We then calibrated the binocular cameras using the traditional stereo calibration method [6], in which the inputs are the manually picked corner pairs of an imaged chess-board pattern in the stereo images. We compared in Tab. 2 the results from our method and from the traditional stereo calibration method. Table 2. Estimation of ω x Experiment 1--- by using our method; Experiment 2--- by using traditional stereo calibration method [6]

Experiment 1 [0.0129 -0.7896 0.5222]T

ωx

Experiment 2 [0.0270 -0.4109 -0.0100] T

Although there is still some error compared with the result by using tradition calibration method, our result is acceptable on the condition that we neither require any chess-board pattern appearing in the scene, nor need any manually intervention on selecting point-to-point correspondences across the image pairs. The second experiment is about the case where almost no overlap is there in the two cameras’ fields of view, as shown in Fig. 5. Estimating the binocular geometry of the cameras viewing such imaged scene could be a difficult task for the correspondence-based methods. However, our method is still effective.

(a)

(b)

(c)

(d)

Fig 5. The zero-boundaries (blue lines) determined by the estimated FOEs. Green dots represent negative candidates; red dots represent positive candidates. (a) Camera A, Motion 1, (b) Camera B, Motion 1, (c) Camera A, Motion 2, d) Camera B, Motion 2.

310

D. Yuan and R. Chung

Result of ωx in the experiment is shown in Table 3. Table 3. Estimation of the rotational component ω x of the binocular geometry

ωx

[0.1634 -0.0801 -2.1097] T

ωx

6 Conclusion and Future Work We have addressed in this work how the determination of inter-camera geometry from normal flows can be much improved by the use of better chosen p-axes, and how these better p-axes can be chosen. Our future work is to relax the requirement of the specific rigid-body motions required in the method. Acknowledgments. The work described in this paper was partially supported by a grant from the Research Grants Council of the Hong Kong Special Administrative Region, China (Project No. CUHK4195/04E), and is affiliated with the MicrosoftCUHK Joint Laboratory for Human-centric Computing and Interface Technologies.

References 1. Bjorkman, M., Eklundh, J.O.: Real-time epipolar geometry estimation of binocular stereo heads. IEEE Trans. on Pattern Analysis and Machine Intelligence 24(3) (March 2002) 2. Dornaika, F., Chung, R.: Stereo geometry from 3D ego-motion streams. IEEE Trans. On Systems, Man, and Cybernetics: Part B, Cybernetics 33(2) (April 2003) 3. Faugeras, O., Luong, T., Maybank, S.: Camera self-calibration: theory and experiments. In: Proc. 3rd European Conf. Computer Vision, Stockholm, Sweeden, pp. 471–478 (1994) 4. Fermüller, C., Aloimonos, Y.: Direct perception of 3D motion from patterns of visual motion. Science 270, 1973–1976 (1995) 5. Fermüller, C., Aloimonos, Y.: Qualitative egomotion. Int’ Journal of Computer Vision 15, 7–29 (1995) 6. Heikkil, J.: Geometric camera calibration using circular control points. IEEE Trans. Pattern Analysis and Machine Intelligence 22(10), 1066–1077 (2000) 7. Knight, J., Reid, I.: Self-calibration of a stereo rig in a planar scene by data. combination. In: Proc. of the International Conference on Pattern Recognition, pp. 1411–1414 (September 2000) 8. Maybank, S.J., Faugeras, O.: A Theory of self-calibration of a moving camera. Int’ Journal of Computer Vision 8(2), 123–152 (1992) 9. Takahashi, H., Tomita, F.: Self-calibration Of Stereo Cameras. In: Proc. 2nd Int’l Conference on Computer Vision, pp. 123–128 (1988) 10. Yuan, D., Chung, R.: Direct Estimation of the Stereo Geometry from Monocular Normal Flows. In: International Symposium on Visual Computing (1), pp. 303–312 (2006) 11. Zhang, Z., Luong, Q.-T., Faugeras, O.: Motion of an uncalibrated stereo rig: Selfcalibration and metric reconstruction. IEEE Trans. on Robotics and Automation 12(1), 103–113 (1996)

Highest Accuracy Fundamental Matrix Computation Yasuyuki Sugaya1 and Kenichi Kanatani2 Department of Information and Computer Sciences, Toyohashi University of Technology, Toyohashi, Aichi 441-8580 Japan [email protected] Department of Computer Science, Okayama University, Okayama 700-8530, Japan [email protected] 1

2

Abstract. We compare algorithms for fundamental matrix computation, which we classify into “a posteriori correction”, “internal access”, and “external access”. Doing experimental comparison, we show that the 7-parameter Levenberg-Marquardt (LM) search and the extended FNS (EFNS) exhibit the best performance and that additional bundle adjustment does not increase the accuracy to any noticeable degree.

1

Introduction

Computing the fundamental matrix from point correspondences is the first step of many vision applications including camera calibration, image rectification, structure from motion, and new view generation [6]. To compute the fundamental matrix accurately from noisy data, we need to solve optimization subject to the constraint that it has rank 2, for which typical approaches are: A posteriori correction. We first compute the fundamental matrix without considering the rank constraint and then modify the solution so that it is satisfied (Fig. 1(a)). Internal access. We minimally parameterize the fundamental matrix so that the rank constraint is always satisfied and do optimization in the reduced (“internal”) parameter space (Fig. 1(b)). External access. We do iterations in the redundant (“external”) parameter space in such a way that an optimal solution that satisfies the rank constraint automatically results (Fig. 1(c)). The aim of this paper is to find the best method by thorough performance comparison.

2

Mathematical Fundamentals

Fundamental matrix. Given two images of the same scene, a point (x, y) in the first image and the corresponding point (x , y  ) in the second satisfy the epipolar equation [6] Y. Yagi et al. (Eds.): ACCV 2007, Part II, LNCS 4844, pp. 311–321, 2007. c Springer-Verlag Berlin Heidelberg 2007 

312

Y. Sugaya and K. Kanatani

⎞⎛  ⎞ ⎛ ⎞ x /f0 x/f0 F11 F12 F13 (⎝ y/f0 ⎠ , ⎝ F21 F22 F23 ⎠ ⎝ y  /f0 ⎠) = 0, 1 F31 F32 F33 1 ⎛

(1)

where f0 is a scaling constant for stabilizing numerical computation [5] (In our experiments, we set f0 = 600 pixels). Throughout this paper, we denote the inner product of vectors a and b by (a, b). The matrix F = (Fij ) in Eq. (1) is of rank 2 and called the fundamental matrix . If we define u = (F11 , F12 , F13 , F21 , F22 , F23 , F31 , F32 , F33 ) , ξ = (xx , xy  , xf0 , yx , yy  , yf0 , f0 x , f0 y  , f02 ) ,

(2) (3)

Equation (1) can be rewritten as (u, ξ) = 0.

(4)

The magnitude of u is indeterminate, so we normalize it to u = 1, which is equivalent to scaling F so that F  = 1. With a slight abuse of symbolism, we hereafter denote by det u the determinant of the matrix F defined by u. Covariance matrices. Given N observed noisy correspondence pairs, we represent them as 9-D vectors {ξ α } in the form of Eq. (3) and write ξα = ξ¯α + Δξα , where ξ¯α is the true value and Δξ α the noise term. The covariance matrix of ξα is defined by (5) V [ξα ] = E[Δξ α Δξ  α ], where E[ · ] denotes expectation over the noise distribution. If the noise in the xand y-coordinates is independent and of mean 0 and standard deviation σ, the covariance matrix of ξ α has the form V [ξ α ] = σ 2 V0 [ξ α ] up to O(σ 4 ), where

      V0 [ξ α ] =       

x ¯2α + x ¯2 x ¯α y¯α f0 x ¯α x ¯α y¯α 0 0 f0 x ¯α 0 α   2 2 ¯α + y¯α f0 y¯α 0 x ¯α y¯α 0 0 f0 x ¯α x ¯α y¯α x ¯α f0 y¯α f02 0 0 0 0 0 f0 x 0 0 y¯α2 + x ¯2 x ¯α y¯α f0 x ¯α f0 y¯α 0 x ¯α y¯α α 0 x ¯α y¯α y¯α2 + y¯α2 f0 y¯α 0 f0 y¯α 0 x ¯α y¯α ¯α f0 y¯α f02 0 0 0 0 0 f0 x ¯α 0 0 f0 y¯α 0 0 f02 0 f0 x ¯α 0 0 f0 y¯α 0 0 f02 0 f0 x 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0

        .      

(6)

xα , y¯α ) are replaced by In actual computations, the true positions (¯ xα , y¯α ) and (¯   their data (xα , yα ) and (xα , yα ), respectively. ˆ of u by We define the covariance matrix V [ˆ u] of the resulting estimate u ˆ )(P U u ˆ ) ], V [ˆ u] = E[(P U u

(7)

where P U is the linear operator projecting R9 onto the domain U of u defined by ˆ by projecting the constraints u = 1 and det u = 0; we evaluate the error of u it onto the tangent space Tu (U) to U at u.

Highest Accuracy Fundamental Matrix Computation

optimal correction

SVD correction det F = 0

(a)

det F = 0

(b)

313

det F = 0

(c)

Fig. 1. (a) A posteriori correction. (b) Internal access. (c) External access.

Geometry of the constraint. The unit normal to the hypersurface defined by det u = 0 is (8) u† = N [∇u det u], where N [ · ] denotes normalization into unit norm. It is easily shown that the constraint det u = 0 is equivalently written as (u† , u) = 0.

(9)

Since the domain U is included in the unit sphere S 8 ⊂ R9 , the vector u is everywhere orthogonal to U. Hence, {u, u† } is an orthonormal basis of the orthogonal complement of the tangent space Tu (U). It follows that the projection operator P U in Eq. (7) has the following matrix representation: P U = I − uu − u† u† .

(10)

KCR lower bound. If the noise in {ξα } is independent and Gaussian with mean 0 and covariance matrix σ 2 V0 [ξ α ], the following inequality holds for an ˆ of u [7]: arbitrary unbiased estimator u V [ˆ u]  σ 2

N  (P U ξ¯α )(P U ξ¯α ) − . (u, V0 [ξ α ]u) 8 α=1

(11)

Here,  means that the left-hand side minus the right is positive semidefinite, and ( · )− r denotes the pseudoinverse of rank r. Chernov and Lesort [2] called the right-hand side of Eq. (11) the KCR (Kanatani-Cramer-Rao) lower bound and ˆ is not unbiased; it is sufficient showed that Eq. (11) holds up to O(σ 4 ) even if u ˆ → u as σ → 0. that u Maximum likelihood. If the noise in {ξα } is independent and Gaussian with mean 0 and covariance matrix σ 2 V0 [ξ α ], maximum likelihood (ML) estimation of u is to minimize the sum of square Mahalanobis distances J=

N 

¯ (ξ α − ξ¯α , V0 [ξ α ]− 2 (ξ α − ξ α )),

α=1

(12)

314

Y. Sugaya and K. Kanatani

subject to (u, ξ¯α ) = 0, α = 1, ..., N . Eliminating the constraint by using Lagrange multipliers, we obtain [7] J=

N 

(u, ξ α )2 . (u, V0 [ξ α ]u) α=1

(13)

ˆ minimizes this subject to u = 1 and (u† , u) = 0. The ML estimator u

3

A Posteriori Correction

The a posteriori correction approach first minimizes Eq. (13) without considering ˜ so as to satisfy it the rank constraint and then modifies the resulting solution u (Fig. 1(a)). A popular method is to compute the singular value decomposition (SVD) of the computed fundamental matrix and replace the smallest singular value by 0, resulting in a matrix of rank 2 “closest” to the original one in norm [5]. We call this SVD correction. A more sophisticated method is the optimal correction [7,11]. According to the statistical optimization theory [7], the covariance matrix V [˜ u] of the rank ˜ can be evaluated, so u ˜ is moved in the direction of the unconstrained solution u mostly likely fluctuation implied by V [˜ u] until it satisfies the rank constraint (Fig. 1(a)). The procedure goes as follows [7]: 1. Compute the 9 × 9 matrices ˜ = M

N 

ξα ξ α , (˜ u , V [ξ u) 0 α ]˜ α=1

(14)

˜− and V0 [˜ u] = M 8. ˜ as follows (˜ ˜ ): 2. Update the solution u u† is defined by Eq. (8) for u ˜ ← N [˜ u u−

˜ † )V0 [˜ 1 (˜ u, u u]˜ u† ]. 3 (˜ u† , V0 [˜ u]˜ u† )

(15)

˜ † ) ≈ 0, return u ˜ and stop. Else, update the matrix V0 [˜ 3. If (˜ u, u u] in the form ˜u ˜ , P u˜ = I − u

V0 [˜ u] ← P u˜ V0 [˜ u]P u˜ ,

(16)

and go back to Step 2. Before doing this, we need to solve unconstrained minimization of Eq. (13), for which many method exist: the FNS (Fundamental Numerical Scheme) of Chojnacki et al. [3], the HEIV (Heteroscedastic Errors-in-Variable) of Leedan and Meer [10], and the projective Gauss-Newton iterations of Kanatani and Sugaya [8]. Their convergence properties were studies in [8].

Highest Accuracy Fundamental Matrix Computation

4

315

Internal Access

The fundamental matrix F has nine elements, on which the normalization F  = 1 and the rank constraint det F = 0 are imposed. Hence, it has seven degrees of freedom. The internal access minimizes Eq. (13) by searching the reduced 7-D parameter space (Fig. 1(b)). Many types of 7-degree parameterizations have been proposed in the past [12,14], but the resulting expressions are often complicated, and the geometric meaning of the individual unknowns are not clear. This was overcome by Bartoli and Sturm [1], who regarded the SVD of F as its parameterization. Their expression is compact, and each parameter has its geometric meaning. They did tentative 3-D reconstruction using the assumed F and adjusted the reconstructed shape, the camera positions, and their intrinsic parameters so that the reprojection error is minimized; such an approach is known as bundle adjustment. Sugaya and Kanatani [13] simplified this: adopting the parameterization of Bartoli and Sturm [1], they directly minimized Eq. (13) by the LevenbergMarquardt (LM) method. Their 7-parameter LM search goes as follows: 1. Initialize F in such a way that det F = 0 and F  = 1, and express it as F = U diag(cos θ, sin θ, 0)V  . 2. Compute J in Eq. (13), and let c = 0.0001. 3. Compute the matrices F U and F V and the vector uθ as follows: 0 0 0

   −F FU =   −F  −F  F F

31 32 33

21 22

F23

F31 −F21 F32 −F22 F33 −F23 0 F11 0 F12 0 F13 −F11 0 −F12 0 −F13 0

   uθ =    

    ,   

FV

      =      

0

F13 −F12 −F13 0 F11 F12 −F11 0 0 F23 −F22 −F23 0 F21 F22 −F21 0 0 F33 −F32 −F33 0 F31 F32 −F31 0

U12 V12 cos θ − U11 V11 sin θ U12 V22 cos θ − U11 V21 sin θ U12 V32 cos θ − U11 V31 sin θ U22 V12 cos θ − U21 V11 sin θ U22 V22 cos θ − U21 V21 sin θ U22 V32 cos θ − U21 V31 sin θ U32 V12 cos θ − U31 V11 sin θ U32 V22 cos θ − U31 V21 sin θ U32 V32 cos θ − U31 V31 sin θ

        .      

        ,      

(17)

(18)

4. Compute the following matrix X: X=

N 

 (u, ξ )2 V0 [ξ ] ξα ξ α α α − . 2 (u, V [ξ ]u) (u, V [ξ ]u) 0 0 α α α=1 α=1 N

(19)

316

Y. Sugaya and K. Kanatani

5. Compute the first and (Gauss-Newton approximated) second derivatives of J as follows: ∇ω J = F  U Xu, ∇2ω J = F  UMF U,

∇ω J = F  V Xu, ∇2ω J = F  V MF V ,

∂∇ω J ∂J 2 = F = (uθ , M uθ ), U M uθ , 2 ∂θ ∂θ 6. Compute the following matrix H: ⎛ ∇2ω J ∇ωω J ∇2ω J H = ⎝ (∇ωω J)  (∂∇ω J/∂θ) (∂∇ω J/∂θ)

∂J = (uθ , Xu), ∂θ ∇ωω J = F  UMF V , ∂∇ω J = F V M uθ . ∂θ ⎞ ∂∇ω J/∂θ ∂∇ω J/∂θ ⎠ . ∂J 2 /∂θ2

7. Solve the simultaneous linear equations ⎞ ⎞ ⎛ ⎛ ω ∇ω J (H + cD[H]) ⎝ ω  ⎠ = − ⎝ ∇ω J ⎠ , ∂J/∂θ Δθ

8. 9. 10. 11. 12.

5

(20)

(21)

(22)

(23)

for ω, ω  , and Δθ, where D[ · ] denotes the diagonal matrix obtained by taking out only the diagonal elements. Update U , V , and θ in the form U  = R(ω)U , V  = R(ω  )V , and θ = θ + Δθ, where R(ω) denotes rotation around N [ω] by angle ω. Update F to F  = U  diag(cos θ , sin θ , 0)V  . Let J  be the value of Eq. (13) for F  . Unless J  < J or J  ≈ J, let c ← 10c, and go back to Step 7. If F  ≈ F , return F  and stop. Else, let F ← F  , U ← U  , V ← V  , θ ← θ , and c ← c/10, and go back to Step 3.

External Access

The external access approach does iterations in the 9-D u-space in such a way that an optimal solution satisfying the rank constraint automatically results (Fig. 1(c)). The concept dates back to such heuristics as introducing penalties to the violation of the constraints or projecting the solution onto the surface of the constraints in the course of iterations, but it is Chojnacki et al. [4] that first presented a systematic scheme, which they called CFNS (Constrained FNS ). Kanatani and Sugaya [9] pointed out, however, that CFNS does not necessarily converge to a correct solution and presented in a more general framework a new scheme, called EFNS (Extended FNS ), which is shown to converge to an optimal value. For fundamental matrix computation, it reduces to the following form: 1. Initialize u. 2. Compute the matrix X in Eq. (19).

Highest Accuracy Fundamental Matrix Computation

317

3. Computer the projection matrix P u† = I −u† u† (u† is defined by Eq. (8)). 4. Compute Y = P u† XP u† . 5. Solve the eigenvalue problem Y v = λv, and compute the two unit eigenvectors v 1 and v 2 for the smallest eigenvalues in absolute terms. ˆ = (u, v 1 )v 1 + (u, v 2 )v 2 . 6. Compute u ˆ ]. 7. Compute u = N [P u† u 8. If u ≈ u, return u and stop. Else, let u ← N [u + u ] and go back to Step 2.

6

Bundle Adjustment

The transition from Eq. (12) to Eq. (13) is exact ; no approximation is involved. Strictly speaking, however, the minimization of the (squared) Mahalanobis distance in the ξ-space (Eq. (13)) can be ML only when the noise in the ξ-space is Gaussian, because then and only then is the likelihood proportional to e−J/constant . If the noise in the image plane is Gaussian, on the other hand, the transformed noise in the ξ-space is no longer Gaussian, so minimizing Eq. (13) is not strictly ML in the image plane. In order to test how much difference is incurred, we also implemented bundle adjustment, minimizing the reprojection error (we omit the details).

7

Experiments

Figure 2 shows simulated images of two planar grid surfaces viewed from different angles. The image size is 600 × 600 pixels with 1200 pixel focal length. We added random Gaussian noise of mean 0 and standard deviation σ to the x- and y-coordinates of each grid point independently and from them computed the fundamental matrix by 1) SVD-corrected LS, 2) SVD-corrected ML, 3) CFNS, 4) optimally corrected ML, 5) 7-parameter LM, and 6) EFNS. “LS” means least squares (also called “8-point algorithm” [5]) minimizing N 2 α=1 (u, ξ α ) , which reduces to simple eigenvalue computation [8]. For brevity, we use the shorthand “ML” for unconstrained minimization of Eq. (13), for 0.2 1 3

5

2

4

6

0.1

0

0

1

2

3

σ

4

Fig. 2. Simulated images of planar grid surfaces and the RMS error vs. noise level. 1) SVD-corrected LS. 2) SVD-corrected ML. 3) CFNS. 4) Optimally corrected ML. 5) 7-parameter LM. 6) EFNS. The dotted line indicates the KCR lower bound.

318

Y. Sugaya and K. Kanatani 12

1.08 1

1.06

10 2 8

1.04

6 1.02 2

3

5

4

4

4

1

5 1

2

0.98

0

0.96

-2

0

1

2

(a)

3

σ

4

0

1

2

3

3

σ

4

(b)

Fig. 3. (a) The RMS error relative to the KCR lower bound. (b) Average residual minus minus (N − 7)σ 2 . 1) Optimally corrected ML. 2) 7-parameter LM started from LS. 3) 7-parameter LM started from optimally corrected ML. 4) EFNS. 5) Bundle adjustment.

which we used the FNS of Chojnacki et al. [3]. The 7-parameter LM and CFNS are initialized by LS. All iterations are stopped when the update of F is less than 10−6 in norm. On the right of Fig. 2 is plotted for σ on the horizontal axis the following rootmean-square (RMS) error D corresponding to Eq. (7) over 10000 independent trials:

  1 10000  ˆ (a) 2 . P U u (24) D= 10000 a=1 ˆ (a) is the ath value, and P U is the projection matrix in Eq. (10). The Here, u dotted line is the bound implied by the KCR lower bound (the trace of the right-hand side of Eq. (11)). Preliminary observations. We can see that SVD-corrected LS (Hartley’s 8point algorithm) performs very poorly. We can also see that SVD-corrected ML is inferior to optimally corrected ML, whose accuracy is close to the KCR lower bound. The accuracy of the 7-parameter LM is nearly the same as optimally corrected ML when the noise is small but gradually outperforms it as the noise increases. Best performing is EFNS, exhibiting nearly the same accuracy as the KCR lower bound. In contrast, CFNS performs as poorly as SVD-corrected ML. The reason for this is fully investigated by Kanatani and Sugaya [9]. Doing many experiments (not all shown here), we have observed that i) EFNS stably achieves the highest accuracy over a wide range of the noise level, ii) optimally corrected ML is fairly accurate and very robust to noise but gradually deteriorates as noise grows, and iii) 7-parameter LM achieves very high accuracy when started from a good initial value but is likely to fall into local minima if poorly initialized. The robustness of EFNS and optimally corrected ML is due to the fact that the computation is done in the redundant (“external”) u-space, where J has a simple form of Eq. (13). In fact, we have never experienced local minima in our

Highest Accuracy Fundamental Matrix Computation

Fig. 4. Left: Real images and 100 corresponding points. Right: Residuals and execution times (sec) for 1) SVDcorrected LS, 2) SVD-corrected ML, 3) CFNS, 4) optimally corrected ML, 5) direct search from LS, 6) direct search from optimally corrected ML, 7) EFNS, 8) bundle adjustment.

1 2 3 4 5 6 7 8

residual 45.550 45.556 45.556 45.378 45.378 45.378 45.379 45.379

319

time . 000524 . 00652 . 01300 . 00764 . 01136 . 01748 . 01916 . 02580

experiments. The deterioration optimally corrected ML in the presence of large noise is because linear approximation is involved in Eq. (15). The fragility of 7-parameter LM is attributed to the complexity of the function J when expressed in seven parameters, resulting in many local minima in the reduced (“internal”) parameter space, as pointed out in [12]. Thus, the optimal correction of ML and the 7-parameter ML have complementary characteristics, which suggests that the 7-parameter ML initialized by optimally corrected ML may exhibit comparable accuracy to EFNS. We now confirm this. Detailed observations. Figure 3(a) compares 1) optimally corrected ML, 2) 7-parameter LM started from LS, 3) 7-parameter LM started from optimally corrected ML, 4) EFNS, and 5) bundle adjustment. For visual ease, we plot the ratio D/DKCR of D in Eq. (24) to the corresponding KCR lower bound. Figure 3(b) plots the corresponding average residual J (minimum of Eq. (13). Since direct plots of J nearly overlap, we plot its difference from (N − 7)σ 2 , where N is the number of corresponding pairs. This is motivated by the fact ˆ 2 is subject to a χ2 distribution with N − 7 that to a first approximation J/σ degrees of freedom [7], so the expectation of Jˆ is approximately (N − 7)σ 2 . We observe from Fig. 3 that i) the RMS error of optimally corrected ML increases as noise increases, yet the corresponding residual remains low, ii) the 7-parameter LM started from LS appears to have high accuracy for noise levels for which the corresponding residual high, iii) the accuracy of the 7-parameter LM improves if started from optimally corrected ML, resulting in the accuracy is comparable to EFNS, and iv) additional bundle adjustment does not increase the accuracy to any noticeable degree. The seeming contradiction that solutions that are closer to the true value (measured in RMS) have higher residuals Jˆ implies that the 7-parameter LM failed to reach the true minimum of the function J, indicating existence of local minima located close to the true value. When initialized by the optimally corrected ML, the 7-parameter LM successfully reaches the true minimum of J, resulting in the smaller Jˆ but larger RMS errors. Real image example. We manually selected 100 pairs of corresponding points in the two images in Fig. 4 and computed the fundamental matrix from them.

320

Y. Sugaya and K. Kanatani

The final residual J and the execution time (sec) are listed there. We used Core2Duo E6700 2.66GHz for the CPU with 4GB main memory and Linux for the OS. We can see that for this example optimally corrected ML, 7-parameter LM started from either LS or optimally corrected ML, EFNS, and bundle adjustment all converged to the same solution, indicating that all are optimal. On the other hand, SVD-corrected LS (Hartley’s 8-point method) and SVD-corrected ML have higher residual than the optimal solution and that CFNS has as high a residual as SVD-corrected ML.

8

Conclusions

We compared algorithms for fundamental matrix computation (the source code is available from the authors’ Web page1 ), which we classified into “a posteriori correction”, “internal access”, and “external access”. We observed that the popular SVD-corrected LS (Hartley’s 8-point algorithm) has poor performance and that the CFNS of Chojnacki et al. [4], a pioneering external access method, does not necessarily converge to a correct solution, while the EFNS always yields an optimal value. After many experiments (not all shown here), we concluded that EFNS and the 7-parameter LM started from optimally corrected ML exhibited the best performance. We also observed that additional bundle adjustment does not increase the accuracy to any noticeable degree. Acknowledgments. This work was done in part in collaboration with Mitsubishi Precision, Co. Ldt., Japan. The authors thank Mike Brooks, Wojciech Chojnacki, and Anton van den Hengel of the University Adelaide, Australia, for providing software and helpful discussions. They also thank Nikolai Chernov of the University of Alabama at Birmingham, U.S.A. for helpful discussions.

References 1. Bartoli, A., Sturm, P.: Nonlinear estimation of fundamental matrix with minimal parameters. IEEE Trans. Patt. Anal. Mach. Intell. 26(3), 426–432 (2004) 2. Chernov, N., Lesort, C.: Statistical efficiency of curve fitting algorithms. Comput. Stat. Data Anal. 47(4), 713–728 (2004) 3. Chojnacki, W., Brooks, M.J., van den Hengel, A., Gawley, D.: On the fitting of surfaces to data with covariances. IEEE Trans. Patt. Anal. Mach. Intell. 22(11), 1294–1303 (2000) 4. Chojnacki, W., Brooks, M.J., van den Hengel, A., Gawley, D.: A new constrained parameter estimator for computer vision applications. Image Vis. Comput. 22(2), 85–91 (2004) 5. Hartley, R.I.: In defense of the eight-point algorithm. IEEE Trans. Patt. Anal. Mach. Intell. 19(6), 580–593 (1997) 1

http://www.iim.ics.tut.ac.jp/˜sugaya/public-e.html

Highest Accuracy Fundamental Matrix Computation

321

6. Hartley, R., Zisserman, A.: Multiple View Geometry in Computer Vision. Cambridge University Press, Cambridge, UK (2000) 7. Kanatani, K.: Statistical Optimization for Geometric Computation: Theory and Practice. Elsevier Science, Amsterdam, The Netherlands 1996, Dover, New York (2005) 8. Kanatani, K., Sugaya, Y.: High accuracy fundamental matrix computation and its performance evaluation. In: Proc. 17th British Machine Vision Conf., Edinburgh, UK, September 2006, vol. 1, pp. 217–226 (2006) 9. Kanatani, K., Sugaya, Y.: Extended FNS for constrained parameter estimation. In: Proc. 10th Meeting Image Recog. Understand, Hiroshima, Japan, July 2007, pp. 219–226 (2007) 10. Leedan, Y., Meer, P.: Heteroscedastic regression in computer vision: Problems with bilinear constraint. Int. J. Comput. Vision 37(2), 127–150 (2000) 11. Matei, J., Meer, P.: Estimation of nonlinear errors-in-variables models for computer vision applications. IEEE Trans. Patt. Anal. Mach. Intell. 28(10), 1537–1552 (2006) 12. Migita, T., Shakunaga, T.: One-dimensional search for reliable epipole estimation. In: Proc. IEEE Pacific Rim Symp. Image and Video Technology, Hsinchu, Taiwan, December 2006, pp. 1215–1224 (2006) 13. Sugaya, Y., Kanatani, K.: High accuracy computation of rank-constrained fundamental matrix. In: Proc. 18th British Machine Vision Conf., Coventry, UK (September 2007) 14. Zhang, Z., Loop, C.: Estimating the fundamental matrix by transforming image points in projective space. Comput. Vis. Image Understand 82(2), 174–180 (2001)

Sequential L∞ Norm Minimization for Triangulation Yongduek Seo1, and Richard Hartley2,  1 2

Department of Media Technology, Sogang University, Korea Australian National University and NICTA, Canberra, Australia

Abstract. It has been shown that various geometric vision problems such as triangulation and pose estimation can be solved optimally by minimizing L∞ error norm. This paper proposes a novel algorithm for sequential estimation. When a measurement is given at a time instance, applying the original batch bi-section algorithm is very much inefficient because the number of seocnd order constraints increases as time goes on and hence the computational cost increases accordingly. This paper shows that, the upper and lower bounds, which are two input parameters of the bi-section method, can be updated through the time sequence so that the gap between the two bounds is kept as small as possible. Furthermore, we may use only a subset of all the given measurements for the L∞ estimation. This reduces the number of constraints drastically. Finally, we do not have to reestimate the parameter when the reprojection error of the measurement is smaller than the estimation error. These three provide a very fast L∞ estimation through the sequence; our method is suitable for real-time or on-line sequential processing under L∞ optimality. This paper particularly focuses on the triangulation problem, but the algorithm is general enough to be applied to any L∞ problems.

1 Introduction Recently, convex programming technique has been introduced and widely studied in the area of geometric computer vision. By switching from an L2 sum-of-squared error function to an L∞ , we are now able to find the global optimum of the error function since the image re-projection error is of quasi-convex type that can be efficiently minimized by the bi-section method [1]. This L∞ norm miniminzation is advantageous because we do not need to build a linearized formulation to find an initial solution for iterative optimization like Levenverg-Marquadt, but also it provides the global optimum of the error function, which is geometrically meaningful, with the well-developed minimization algorithm. Applying an idea of L∞ optimization was presented by Hartley and Schaffalitzky in [2], where it was observed that many geometric vision problems have a single global 

This work was supported by the Korea Science and Engineering Foundation(KOSEF) grant funded by the Korean government(MOST) (No. R01-2006-000-11374-0). This research is accomplished as the result of the research project for culture contents technology development supported by KOCCA.  NICTA is a research centre funded by the Australian Government’s Department of Communications, Information Technology and the Arts and the Australian Research Council, through Backing Australia’s Ability and the ICT Research Centre of Excellence programs. Y. Yagi et al. (Eds.): ACCV 2007, Part II, LNCS 4844, pp. 322–331, 2007. c Springer-Verlag Berlin Heidelberg 2007 

Sequential L∞ Norm Minimization for Triangulation

323

minimum under L∞ error norm. Kahl, and Ke & Kanade, respectively, showed that the error functions are of quasi-convex and can be solved by Second Order Cone Programming (SOCP) [1,3]. Vision problems that can be solved under L∞ formulation include triangulation [2], homography estimation and camera resectioning [1,3], multiview reconstruction knowing rotations or homographies induced by a plane [1,3], camera motion recovery [4], and outlier removal [5]. Vision problems like building a 3D model or computing the camera motion of a video clip require a batch process. A fast algorithm for such batch computations is presented in [6]. However, some applications need a mechanism of sequential update such as navigation or augmented reality, e.g, [7,8,9]. This paper is about sequentially minimizing the L∞ error norm, which has not been considered yet in vision literature. Our research is aimed at on-line or real-time vision applications. The most important constraint in this case is that the optimization should be done within a given compuation time. Therefore, we need to develop a bi-section algorithm for this purpose. This paper first introduces the triangulation problem in Section 2 and analyzes the bi-section algorithm and suggests three methods to reduce the computation time without sacrificing any accuracy. Section 3 presents our novel bisection algorithm suitable for time sequence applications. Experimental results are given in Section 4 and concluding remarks in Section 5. We focus on triangulation problem in this paper. Triangulation alone may look very much restricted but note that motion estimation knowing rotation is equivalent to triangulation as can be found in [4]. In addition, if a branch-and-bound algorithm is adopted for rotation estimation, then a fast triangulation computation becomes also very much important for global optimization for pose estimation or multi-view motion computation.

2 Triangulation with L∞ Norm Triangulation is to find a 3D space point X when we are given two or more pairs of camera matrix Pt of dimension 3×4 and its image point ut = [u1t , u2t ] at time t. These quantities are related by the projection equation: uit =

pit X p3t X

for

i = 1, 2,

(1)

where pit denotes the i-th 4D row vector of Pt and X is a 4D vector represented by homogeneous coordinates (that is, the 4-th coordinate is one). Re-projection discrepancy dt of X for the measurement ut contaminated by noise is given by   p1t X p2t X 1 2  dt = ut − 3 , ut − 3 (2) pt X pt X and the quality of the error is given by the error function et (X) = dt (X). When we use L2 norm,  2  2  12 et (X) = dt 2 = d1t (X) + d2t (X) . (3) Any function f is called quasi-convex if its domain is convex and all its sub-level sets {x ∈ domain f | f (x) ≤ α} for α ∈ R are convex [10]. The error function in

324

Y. Seo and R. Hartley

Algorithm 1. Bisection method: L∞ norm minimization Input: initial upper(U )/lower(L) bounds, tolerance  > 0. 1: repeat 2: γ := (L + U )/2 3: Solve the feasibility problem (7) 4: if feasible then U := γ else L := γ 5: until U − L ≤ 

Equation (3) is of convex-over-concave and can be shown to be a quasi-convex function in the convex domain D = {X|p3t X ≥ 0}, which means that the scene is in front of the camera [11,1]. Given a bound γ, the inequality et ≤ γ defines a set Ct of X Ct = {X|et (X) ≤ γ} .

(4)

Note that the feasible set Ct is due to t-th measurement ut ; Ct is called a second order cone due to Equation (3). The bound γ is called the radius of the cone in this paper. Note that Equation (4) is given by the circular disk dt 2 ≤ γ in the image plane. The set Ct defines the cone whose apex is at the location of camera center and whose cutting shape with the image plane is the circular disk dt 2 ≤ γ. The vector e = [e1 , e2 , ...eT ] represents the error vector of T measurements. A feasible set Fγ given a constant γ is defined as the intersection of all the cones Fγ =

T 

{X|et ≤ γ}

(5)

t=1

= {X|e1 ≤ γ} ∩ . . . ∩ {X|eT ≤ γ}

(6)

We now have its feasibility problem for a given constant γ: find X subject to

et (X) ≤ γ,

t = 1, ...., T

(7)

The feasible set Fγ is convex because it is the intersection of (ice-cream shape) convex cones. Indeed, the feasibility problem (7) has already been investigated and well-known in the area of convex optimization; the solution X can be obtained by an SOCP solver. The L∞ norm of e is defined to be the maximum of et ’s, and the L∞ triangulation problem is to find the smallest γ that yields non-empty feasible set Fγ and its corresponding X . This can also be written as a min-max optimization problem min max {e1 , e2 , ..., eT } , X

(8)

and the global optimum can be found by the bi-section method presented in Algorithm 1. It consists of repeatedly solving the feasibility problem, adjusting the bound γ. Lemma 1. Since we minimize the maximum error, we don’t have to use all the measurements. There exist subsets of measurements that result in the same estimation. Reduced number of measurements will decrease the computation time, which is necessary for sequential applications. What we have to do is to choose some among those sequential measurements. Our approach will be provided in the next section together with our sequential bi-section algorithm.

Sequential L∞ Norm Minimization for Triangulation

325

3 Bisectioning for Sequential Update Problem 1. (Original Batch Problem) Given a set of image matches {ui , i = 1...T }, find their 3D point X that is optimal under L∞ error norm. As we mentioned in Section 1, the solution of this problem can be obtained by the bi-section method shown in Algorithm 1. From now on, the optimal solution with T ∞ measurements is represented by X∞ T and the corresponding minimum error by eT . Now let us cast our sequential problem. Problem 2. (Sequential Problem) The L∞ estimate X∞ T has been computed given image matches ui , i = 1, . . . , T . Now a new measurement uT +1 is arrived. Find the optimal estimate X∞ T +1 . Obviously, we might apply Algorithm 1 using all the T + 1 measurements again from scratch. However, we want to do it more efficiently in this paper; our first goal is to reduce the number of SOCP repetitions during the bisection algorithm. ∞ Lemma 2. If the re-projection error eT +1 (X∞ T ) for uT +1 is smaller than eT , that is, ∞ ∞ eT +1 ≤ e∞ , then no further minimization is necessary, and we can set e T T +1 = eT and ∞ = X . X∞ T +1 T ∞ If eT +1 ≤ e∞ T , the feasible cone CT +1 = {X|eT +1 (X) ≤ eT } for uT +1 is already a ∞ ∞ subset of FeT (i.e., γ = eT ). Therefore, we don’t have to run bisectioning to update the estimate. The only computation necessary is to evaluate the re-projection error eT +1 . This is because the bisection method is independent of the order of the input measurements {u1 , . . . , uT +1 }. The estimate X∞ T is already optimal and running the bisection ∞ algorithm with T + 1 measurements will result in the same output: X∞ T +1 = XT . Note that due to Lemma 2 the computational cost for evaluating eT +1 is so much less than the cost for running the bisection method.

Lemma 3. Otherwise (i.e., e∞ T < eT +1 ), we run the bisection algorithm but with different initial upper and lower bounds: U := eT +1 , L := e∞ T . In this case, e∞ T < eT +1 , we have ⊂ CT +1 = {X|eT +1 ≤ γ, where γ = eT +1 (X∞ Fe∞ T )} T

(9)

due to the fact that e∞ T < eT +1 ; therefore, the upper bound for the feasibility of X given T + 1 measurements can be set to U := eT +1 . In other words, the intersection of the T + 1 cones is non-empty when the cones are of radius eT +1 . It is natural that the initial lower bound be set to zero L0 := 0 to run the bisection algorithm. However, a lower bound greater than zero may reduce the number of iterations during the bi-sectioning. The feasible set FγT +1 with bound γ up to time T + 1 can be written as FγT +1 =

T +1

{X|et ≤ γ}

(10)

t=1

= [C1 ∩ . . . ∩ CT ] ∩ CT +1 =

FγT

∩ CT +1 .

(11) (12)

326

Y. Seo and R. Hartley

Algorithm 2. Sequential bi-section method with measurement selection. Input: Measurement set M, selected measurements S ⊂ M. 1: eT +1 := ReprojectionError (uT +1 , X∞ T ) 2: if eT +1 > e∞ T then 3: bool flag=FALSE 4: M := M ∪ {uT +1 }, U := eT +1 , L := e∞ T 5: repeat ∞ 6: (e∞ T +1 , XT +1 ) := BisectionAlgorithm(S, U , L) 7: (emax , tmax ) := ReprojectionErrors (M \ S, X∞ T +1 ) 8: if emax < e∞ T +1 then 9: flag=TRUE; 10: else 11: S := S ∪ {utmax } 12: L := e∞ T +1 13: end if 14: until flag=TRUE 15: end if

If the bound γ is smaller than e∞ T , the intersection of the first T cones results in an empty set because γ = e∞ is the smallest bound for FγT found from the bi-section algorithm T using T measurements. Consequently, it will make the total feasible set FγT +1 be empty or non-feasible. Therefore, we see that L0 = e∞ T is the greatest lower bound up to time T + 1 to execute the bi-section algorithm. Due to Lemma 3, we now have a much reduced gap of initial upper and lower bound; this decreases the number of itreations during the execution of bi-section method.

Fig. 1. A synthetic data sequence. Initial image location was at (0, 0), and each point was generated by Brownian camera motion. In total, a hundred data points were generated as an input sequence.

3.1 Measurement Selection From Lemma 1, we know that there is a possibility to reduce the number of measurements or constraints in SOCP without an accuracy loss. Here we explain our algorithm

Sequential L∞ Norm Minimization for Triangulation

327

Fig. 2. Evolution of L∞ estimation error through time. The initial error was from the first two measurements. The red line shows changes of L∞ error e∞ ; the green line (impulse style) at each time t corresponds to the re-projection error et . When e∞ < et (when the green line goes above the red line), our bisectioning re-computed the estimate Xt∞ . Otherwise, no more computation was necessary. The blue line denotes the evolution of RMS error for Xt∞ .

of selecting measurements presented in Algorithm 2. If we need to solve the feasibility problem due to the condition in Lemma 3 (e∞ T < eT +1 ), then we include uT +1 into the set S of selected measurements and run the bi-section algorithm to get the estima∞ tion results (e∞ T +1 , XT +1 ) (6th line). Using this estimate, we evaluate the reprojection errors for those un-selected measurements (7th line). If the estimation error is greater than the maximum error from the un-selected, emax < e∞ T +1 , then we are done (8th and 9th lines). Otherwise, we include the measurement utmax into the measurement set M (11th line) and repeat the operation. The lower bound is then set to the new value (12th line).

4 Experiments We implemented the algorithm using C/C++ language and tested for synthetic and real data. First, experiments with synthetic data set were done. A data set S was generated as follows: The center of the first camera was located at C1 = [0, 0, −1]T, and moved randomly with standard deviation σC of zero mean Gaussian. That is, Ct = Ct−1 + σC [N , N , N ]T

(13)

where N represents a random value from the standard Gaussian distribution. Then the space point X0 = [0, 0, 0]T was projected to the image plane ut = [u1t , u2t ]T with focal length f = 1000. Gaussian noise of level σ in the image space was then injected: ut = ut + σ[N , N ]. Figure 1 shows the trajectory of a random set S when σ = 1.5. All the camera parameters were assumed to be known in this paper. Figure 2 shows a sequential evolution of L∞ estimation error e∞ t through time t = 2, ..., 100, using the image data plotted in Figure 1. The initial error was from the first

328

Y. Seo and R. Hartley

Fig. 3. Evolution of the number of selected measurements. Initially two are necessary for triangulation. Only 16 among 100 measurements were selected. The increments were exactly at the time instance when we had e∞ t < et .

two measurements. The red line shows e∞ t ; the green line (impulse style) at each time t ∞ corresponds to the re-projection error et (Xt−1 ∞ ) defined in Equation 3. When et < et (that is, when the green line goes above the red line in this graph), our bisectioning was applied to compute the estimate X∞ t together with the scheme of measurement selection. Otherwise, no update was necessary. The blue line shows the evolution of RMS error for a comparison: RMSt =

1

di (Xt∞ )22 t i=1 t

 12 (14)

.

Accumulated Computation Time 70

60

50

Clocks

40

30

20

10

0 0

10

20

30

40

50 60 Sequence Index

70

80

90

100

Fig. 4. Accumulated computation time. Computation was necessary when a new measurement < et . The total compuation took 3,414 was included into the measurement set M when e∞ t clocks (time units) when we simplied adopted the batch algorithm using every measurement every time step without any speed improving method; with adjustment of upper and lower bounds it took 100 clocks, which then reduced to 63 clocks with measurement selection.

Sequential L∞ Norm Minimization for Triangulation

329

Fig. 5. The ratio of the accumulated time to the batch computation time for the case of 100 data sets. The average ratio was almost 1.0, which meant that the speed of our sequential update algorithm was almost the same as that of one batch computation on the average.

Figure 3 shows the evolution of the number of selected measurements. In this experiment, only 16 measurements among 100 were selected during the sequential update. Note that the number of measurements increases when e∞ t < et . Figure 4 shows the accumulated computation time. Main computation was done only when the feasibility computation was necessary as can be seen in the graph. The total compuation took 3,414 clocks (time units) when we simplied adopted the batch algorithm using every measurement every time step without any speed improving method; with adjustment of upper and lower bounds it took 100 clocks, which then reduced to 63 clocks with measurement selection. The batch computation took 157 clocks.

Fig. 6. Results of a real experiment. Evolution of L∞ estimation error through time. The initial error was from the first two measurements. The red line shows changes of L∞ error e∞ ; the green line (impulse style) at each time t corresponds to the re-projection error et . When e∞ < et (when the green line goes above the red line), our bisectioning re-computed the estimate Xt∞ . Otherwise, no more computation was necessary. The blue line denotes the evolution of RMS (L2 ) error for Xt∞ ..

330

Y. Seo and R. Hartley

Fig. 7. Evolution of L∞ estimation error through time for the 25th 3D point from Corridor data set

Fig. 8. L∞ estimation error plot for Corridor sequence

Figure 5 shows that such a speed upgrade was attained on the average. We did the same experiments using different data sets. A hundred repetition showed that the average of the ratio of computation time was 1.0; this experimentally implies that the speed of our sequential algorithm is almost the same as running the batch algorithm once with total measurements. We then generated 1000 data sets {Sk , k = 1, ..., 1000} and repeated the same experiment for each. The difference of re-projection errors diff errk = |e∞ (Xbat ) − e∞ (Xseq )| were computed to make sure that our algorithm results in the same estimation, where e∞ is the L∞ norm (maximum) of all the errors of 100 data in the set Sk . The average of the differences was approximately 7 × 10−9 , which meant that the two estimates were the same numerically. Figure 6 shows the same illustration as Figure 2 for a real experiment of sequence length 162. It took 305 time units (batch algorith took 719 time units). Notice that the error converges early in the sequence and there is almost no time update. This also shows that our algorithm is very much suitable when we need to update a large number of triangulation problems at the same time.

Sequential L∞ Norm Minimization for Triangulation

331

Finally, experiments with real data were done with Corridor sequence1 . Among the tracks of corners, those which had more than three matches were chosen. Figure 7 shows an exemplary L∞ evolution graph as Figure 2 did. Figure 8 shows the plot of L∞ errors for all the data sequence whose length was longer than three.

5 Conclusion This paper considered how to apply the bisection method for L∞ norm minimization to a sequential situation. The computation with bisection method did not have to be executed during the sequence when the re-projection of Xt−1 ∞ to the t-th camera yielded a smaller error than e∞ ; otherwise, bisectioning was necessary but with lower and upper bounds whose gap was narrower, resulting in faster computation. Measurement selection scheme was also provided to reduce the computational cost by decreasing the number of measurement cones (constraints). Our mathematical reasoning were provided and showed the performance of the sequential algorithm via synthetic and real experiments; our method is suitable for real-time or on-line applications.

References 1. Kahl, F.: Multiple view geometry and the L∞ -norm. In: Proc. Int. Conf. on Computer Vision, Beijing, China, pp. 1002–1009 (2005) 2. Hartley, R., Schaffalitzky, F.: L∞ minimization in geometric reconstruction problems. In: Proc. IEEE Conf. Computer Vision and Pattern Recognition (2004) 3. Ke, Q., Kanade, T.: Quasiconvex optimization for robust geometric reconstruction. In: Proc. Int. Conf. on Computer Vision, Beijing, China (2005) 4. Sim, K., Hartley, R.: Recovering camera motion using L∞ minimization. In: Proc. IEEE Conf. Computer Vision and Pattern Recognition (2006) 5. Sim, K., Hartley, R.: Removing outliers using the L∞ norm. In: Proc. IEEE Conf. Computer Vision and Pattern Recognition (2006) 6. Seo, Y., Hartley, R.: A fast method to minimize L∞ error norm for geometric vision problems. In: Proc. Int. Conf. on Computer Vision (2007) 7. Robert, L., Buffa, M., H´ebert, M.: Weakly-calibrated stereo perception for rover navigation. In: Fifth Inter. Conf. Comp. Vision (1995) 8. Lamb, P.: Artoolkit (2007), http://www.hitl.washington.edu/artoolkit 9. Bauer, M., Schlegel, M., Pustka, D., Navab, N., Klinker, G.: Predicting and estimating the accuracy of optical tracking system. In: IEEE and ACM International Symposium on Mixed and Augmented Reality, pp. 43–51 (2007) 10. Boyd, S., Vandenberghe, L.: Convex Optimization. Cambridge Press, Cambridge (2004) 11. Hartley, R.I.: The chirality. Int. Journal of Computer Vision 26, 41–61 (1998)

1

http://www.robots.ox.ac.uk/ vgg/data.html

Initial Pose Estimation for 3D Model Tracking Using Learned Objective Functions Matthias Wimmer1, and Bernd Radig2 1

Faculty of Science and Engineering, Waseda University, Tokyo, Japan Institut f¨ ur Informatik, Technische Universit¨ at M¨ unchen, Germany

2

Abstract. Tracking 3D models in image sequences essentially requires determining their initial position and orientation. Our previous work [14] identifies the objective function as a crucial component for fitting 2D models to images. We state preferable properties of these functions and we propose to learn such a function from annotated example images. This paper extends this approach by making it appropriate to also fit 3D models to images. The correctly fitted model represents the initial pose for model tracking. However, this extension induces nontrivial challenges such as out-of-plane rotations and self occlusion, which cause large variation to the model’s surface visible in the image. We solve this issue by connecting the input features of the objective function directly to the model. Furthermore, sequentially executing objective functions specifically learned for different displacements from the correct positions yields highly accurate objective values.

1

Introduction

Model-based image interpretation is appropriate to extract high-level information from single images and from image sequences. Models induce a priori knowledge about the object of interest and thereby reduce the large amount of image data to a small number of model parameters. However, the great challenge is to determine the model parameters that best match a given image. For interpreting image sequences, model tracking algorithms fit the model to the individual images of the sequence. Each fitting step benefits from the pose estimate derived from the previous image of the sequence. However, determining the pose estimate for the first image of the sequence has not been sufficiently solved yet. The challenge of this so-called initial pose estimation is identical to the challenge of fitting models to single images. Our previous work identifies the objective function as an essential component fitting models to single images [14]. This function evaluates how well a particular model fits to an image. Without losing generality, we consider lower values to represent a better model fitness. Therefore, algorithms search for the model parameters that minimize the objective function. Since the described methods 

This research is partly funded by a JSPS Postdoctoral Fellowship for North American and European Researchers (FY2007).

Y. Yagi et al. (Eds.): ACCV 2007, Part II, LNCS 4844, pp. 332–341, 2007. c Springer-Verlag Berlin Heidelberg 2007 

Initial Pose Estimation for 3D Model Tracking

333

are independent of the used fitting algorithm, we do not elaborate on them, but refer to Hanek et al. [5] for a recent overview and categorization of fitting algorithms. As our approach has only been specified for 2D models so far, this paper extends it to be capable for handling 3D models while considering a rigid model of a human face. In contrast to artificial objects, such as cars, faces highly vary in shape and texture and therefore the described face model fitting task represents a particular difficulty. Many researchers engage in fitting 3D models. Lepetit et al. [7] treat this issue as a classification problem and use decision trees for solution. As an example implementation, the ICP algorithm [2,12] minimizes the square error of the distance between the visible object and the model projected to the image. Problem Statement. Although the accuracy of model fitting heavily depends on the objective function, it is often designed by hand using the designer’s intuition about a reasonable measure of fitness. Afterwards, its appropriateness is subjectively determined by inspecting the objective function on example images and example model parameters. If the result is not satisfactory the function is tuned or redesigned from scratch [11,4], see Figure 1 (left). Therefore, building the objective function is very time consuming and the function does not guarantee to yield accurate results.

Fig. 1. The procedures for designing (left) and learning (right) objective functions

Solution Outline. Our novel approach focuses on the root problem of model fitting: We improve the objective function rather than the fitting algorithm. As a solution to this challenge, we propose to conduct a five-step methodology that learns robust local objective functions from annotated example images. We investigated this approach for 2D models so far [14]. This paper extends our methodology in order to generate objective functions that are capable of handling 3D models as well. The obtained functions consider specific aspects of 3D models, such as out-of-plane rotations and self-occlusion. We compute the features not in the 2D image plane but in the space of the 3D model. This requires connecting the individual features directly to the model. Contributions. The resulting objective functions work very accurately in realworld scenarios and they are able to solve the challenge of initial pose estimation that is required by model tracking. This easy-to-use approach is applicable to various image interpretation scenarios and requires the designer just to annotate example images with the correct model parameters. Since no further

334

M. Wimmer and B. Radig

computer vision expertise is necessary, this approach has great potential for commercialization. The paper proceeds as follows. In Section 2, we sketch the challenge of modelbased image interpretation. In Section 3, we propose our methodology to learn accurate local objective functions from annotated training images with particular focus on 3D models. Section 4 conducts experimental evaluations that verify this approach. Section 5 summarizes our approach and shows future work.

2

Model-Based Image Interpretation

Rigid 3D models represent the geometric properties of real-world objects. A six-dimensional parameter vector p=(tx , ty , tz , α, β, γ)T describes the position and orientation. The model consists of 1≤n≤N three-dimensional model points specified by cn (p). Figure 2 depicts our face model with N =214 model points. Fitting 3D models to images requires two essential components: The fitting algorithm searches for the model parameters that match the content of the image best. For this task, it searches for the minimum of the objective function f (I, p), which determines how well a model p matches an image I. As in Equation 1, this function is often subdivided into N local components fn (I, x), one for each model point [7,1,9]. These local functions determine how well the nth model point fits to the image. The advantage of this partitioning is that designing the local functions is more straightforward than designing the global function, because only the image content in the vicinity of one projected model point needs to be taken into consideration. The disadvantage is that dependencies and interactions between local errors cannot be combined. f (I, p) =

N 

fn (I, cn (p))

(1)

n=1

2.1

Characteristic Search Directions of Local Objective Functions

Fitting algorithms for 2D contour models usually search the minimum of the objective function along the perpendicular to the contour [3]. The objective function

Fig. 2. Our 3D model of a human face correctly fitted to images

Initial Pose Estimation for 3D Model Tracking

335

Fig. 3. Due to affine transformations of the face model the characteristic search directions will not be parallel to the image plane, in general. These three images show how the transformations affect one of these directions.

computes its value from image features in the vicinity of the perpendicular. We stick to this procedure, and therefore we create local objective functions that are specific to a search direction. These so-called characteristic directions represent three-dimensional lines and we connect them tightly to the model’s geometric structure, i.e. to the individual model points cn (p). Transforming the model’s pose transforms these directions equivalently, see Figure 3. The objective function computes its value from three-dimensional features, however their value is calculated projecting them to the image plane. An image is most descriptive for a characteristic direction if they are parallel. Unfortunately, transforming the model will usually yield characteristic directions that are not parallel to the image plane. Therefore, we consider not only one but 1≤l≤L characteristic directions per model point, which are differently oriented. These directions may be arbitrary, but we prefer them to be pairwise orthogonal. This yields L objective function fn,l (I, x) for each model point. In order not to increase computation time, we consider the characteristic direction, which is most parallel to the image, only. fn (I, x) = fn,gn (p) (I, x)

(2)

The model point’s local objective function fn is computed as in Equation 2. The indicator gn (p) computes the index of the characteristic direction that is most significant for the current pose p of the model, i.e. that is most parallel to the image plane.

3

Learning Objective Functions from Image Annotations

Ideally, local objective functions have two specific properties. First, they should have a global minimum that corresponds to the best model fit. Otherwise, we cannot be certain that determining the true minimum of the local objective function indicates the intended result. Second, they should have no other local minima. This implies that any minimum found corresponds to the global minimum, which facilitates search. A concrete example of an ideal local objective

336

M. Wimmer and B. Radig

function, that has both properties, is shown in Equation 3. pI denotes the model parameters with the best model fit for a certain image I. fn (I, x) = |x − cn (pI )|

(3)

Unfortunately, fn cannot be applied to unseen images, for which the best model parameters pI are not known. Nevertheless, we apply this ideal objective function to annotated training images and obtain ideal training data for  for the model point n and its characterlearning a local objective function fn,l istic direction l. The key idea behind our approach is that since the training data is generated by an ideal objective function, the learned function will also be approximately ideal. This has already been shown in [14]. Figure 1 (right) illustrates the proposed five-step procedure. Step 1: Annotating Images with Ideal Model Parameters. As in Figure 2, a database of 1≤k≤K images Ik is manually annotated with the ideal model parameters pIk , which are necessary to compute the ideal objective functions fn . This is the only laborious step of the entire procedure.

Fig. 4. Further annotations are generated by moving along the line that is longest when projected in the image. That line is colored white here. The directions illustrated in black are not used. Annotations on one of the directions that are not used are also shown to demonstrate that this direction is too short to be used.

Initial Pose Estimation for 3D Model Tracking

337

Fig. 5. This comprehensive set of image features is provided for learning local objective functions. In our experiments, we use a total number of A=6·3·5·5=450 features.

Step 2: Generating Further Image Annotations. The ideal objective function returns the minimum fn (I, x)=0 for all manual annotations x=cn (pIk ). These annotations are not sufficient to learn the characteristics of fn . Therefore, we will generate annotations x=cn (pIk ), for which fn (I, x)=0. In general, any 3D position x may represent one of these annotations, however, we sample −D≤d≤D positions along the characteristic direction with a maximum displacement Δ (learning radius), see Figure 4. This procedure learns the calculation rules of the objective function more accurately. In this paper, we use L=3 characteristic directions, because the model points vary within the 3D space. Note that the gn (pIk ) selects the most significant direction for the nth model point. Step 3: Specifying Image Features. Our approach learns the calculation  rules of a mapping fn,l from an image Ik and a location xk,n,d,l to the value   has no knowledge of pI , it must compute its result of fn (Ik , xk,n,d,l ). Since fn,l from the image content. Instead of learning a direct mapping from the pixel values in the vicinity of x to fn , we compute image features, first. Note that x does not denote a position in I but in 3D space. However, the corresponding pixel position is obtained via perspective projection. Our idea is to provide a multitude of 1≤a≤A features, and let the training  . Each algorithm choose which of them are relevant to the calculation rules of fn,l feature ha (I, x) is computed from an image I and a position x and delivers a scalar value. Our approach currently relies on Haar-like features [13,8] of different styles and sizes. Furthermore, the features are not only computed at the location of the model point itself, but also at positions on a grid within its vicinity, see

Fig. 6. The grid of image features moves along with the displacement

338

M. Wimmer and B. Radig

Figure 5. This variety of styles, sizes, and locations yields a set of A=450 image features as we use it in our experiments in Section 4. This multitude of features enables the learned objective function to exploit the texture of the image at the model point and in its surrounding area. When moving the model point, the image features move along with it, leading their values to change, see Figure 6. Step 4: Generating Training Data. The result of the manual annotation step (Step 1) and the automated annotation step (Step 2) is a list of correspondences between positions x and the corresponding value of fn . Since K images, N model points, and 2D+1 displacements are landmarked these correspondences amount to K·N ·(2D+1). Equation 4 illustrates the list of these correspondences. [

Ik ,

xk,n,d,l ,

[ h1 (Ik , xk,n,d,l ), . . . , hA (Ik , xk,n,d,l ),

fn (Ik , xk,n,d,l ) ]

(4)

fn (Ik ,

(5)

xk,n,d,l ) ]

with 1≤k≤K, 1≤n≤N, −D≤d≤D, l=gn (pIk )

Applying the list of image features to the list of correspondences yields the training data in Equation 5. This step simplifies matters greatly. Since each feature returns a single value, we hereby reduce the problem of mapping the vast amount of image data and the related pixel locations to the corresponding target value, to mapping a list of feature values to the target value. Step 5: Learning the Calculation Rules. Given the training data from  (I, x) that approximates Equation 5, the goal is to now learn the function fn,l  is not provided knowledge of pI . Therefore, it fn (I, x). The challenge is that fn,l can be applied to previously unseen images. We obtain this function by training a model tree [10,15] with the comprehensive training data from Equation 5. Note  have to be learned individually. However, that the N ·L objective functions fn,l  learning fn,l only requires the records of the training data (Equation 5) where n and l match. Model trees are a generalization of regression trees and, in turn, of decision trees. Whereas decision trees have nominal values at their leaf nodes, model trees have line segments, allowing them to also map features to a continuous value, such as the value returned by the ideal objective function. One of the reasons for deciding for model trees is that they tend to select only features that are relevant to predict the target value. Therefore, they pick a small number of Mn Haar-like features from the provided set of A  Mn features.  for After executing these five steps, we obtain a local objective function fn,l each model point n and each direction l. It can now be called with an arbitrary location x of an arbitrary image I. The learned model tree calculates the values of the necessary features from the image content and yields the result value.

4

Experimental Evaluation

In this section, two experiments show the capability of fitting algorithms equipped with a learned objective function to fit a face model to previously unseen images.

Initial Pose Estimation for 3D Model Tracking

339

Fig. 7. If the initial distance to the ideal model points is smaller than the learning radius this distance is further reduced with every iteration. Otherwise, the result of the fitting step is unpredictable and the model points are spread further.

The experiments are performed on 240 training images and 80 test images with a non-overlapping set of individuals. Furthermore, the images differ in face pose, illumination, and background. Our evaluations randomly displace the face models from the manually specified pose. The fitting process determines the position of every model point by exhaustively searching along the most significant characteristic direction for the global minimum of the local objective function. Afterwards, the model parameters p are approximated. The subsequent figures illustrate the average point-to-point error of the model points between the obtained model p and the manually specified model pI . Our first evaluation investigates the impact of executing the fitting process with a different number of iterations. Figure 7 illustrates that each iteration improves the model parameters. However, there is a lower bound to the quality of the obtained model fit. Obviously, more than 10 iterations do not improve the fraction of well-fitted models significantly. Note that the objective function’s value is arbitrary for high distances from the correct position. These models are further distributed with every iteration. Our second experiment conducts model fitting by subsequently applying the fitting process with two different objective functions f A and f B learned with decreasing learning radii Δ. f A with a large Δ is able to handle large initial displacements in translation and rotation. However, the obtained fitting result gets less accurate, see Figure 8. The opposite holds true for f B . The idea is to apply a local objective function learned with large Δ first and then gradually apply objective functions learned with smaller values for Δ. As opposed to the

340

M. Wimmer and B. Radig

Fig. 8. By combining fitting algorithms using objective functions with different learning radius we obtain result that show the strengths of both objective functions. The sequential approach shows the tolerance to errors of f A and the accuracy of f B .

previous experiment, where we iteratively executed the same objective function, this iteration scheme executes different objective functions, which compensates the weakness of one function by the strength of another. The advantage of concatenating algorithms with objective functions with decreasing learning radii compared to iterating one algorithm several times is illustrated by Figure 8. Sequentially applying f A and f B is significantly better than both of the other algorithms. Note that we execute each experiment with ten iterations, because we don’t expect any improvement in quality with a higher number of iterations, see our first experiment. Therefore, the obtained accuracy from the sequential execution does not base of the fact that some additional iterations are applied.

5

Summary and Conclusion

In this paper, we extended our five-step methodology for learning local objective functions that we introduced for 2D models so far [14]. 3D models can now be considered as well. This approach automates many critical decisions and the remaining manual steps require little domain-dependent knowledge. Furthermore, its process does not contain any time-consuming loops. These features enable non-expert users to customize the fitting application to their specific domain. The resulting objective function is not only able to process objects that look similarly, such as in [7,6] but objects that differ significantly in shape and texture, such as human faces. Being trained with a limited number of annotated images as

Initial Pose Estimation for 3D Model Tracking

341

described in Section 3 the resulting objective function is able to fit faces that are not part of the training data as well. However, the database of annotated faces must be representative enough. If there were no bearded men in the training data the algorithm would have problem in fitting the model to an image of such a man. The disadvantage of our approach is the laborious annotation step. Gathering and annotating hundreds of images requires several weeks.

References 1. Allezard, N., Dhome, M., Jurie, F.: Recognition of 3D textured objects by mixing view-based and model-based representations. ICPR, 960–963 (September 2000) 2. Besl, P.J., McKay, N.D.: A method for registration of 3-D shapes. IEEE Transactions on Pattern Analysis and Machine Intelligence 14(2), 239–256 (1992) 3. Cootes, T.F., Taylor, C.J., Lanitis, A., Cooper, D.H., Graham, J.: Building and using flexible models incorporating grey-level information. In: ICCV, pp. 242–246 (1993) 4. Cristinacce, D., Cootes, T.F.: Facial feature detection and tracking with automatic template selection. In: 7th IEEE International Conference on Automatic Face and Gesture Recognition, April 2006, pp. 429–434. IEEE Computer Society Press, Los Alamitos (2006) 5. Hanek, R.: Fitting Parametric Curve Models to Images Using Local Self-adapting Seperation Criteria. PhD thesis, Technische Universit¨ at M¨ unchen (2004) 6. Lepetit, V., Lagger, P., Fua, P.: Randomized trees for real-time keypoint recognition. In: CVPR 2005, Switzerland, pp. 775–781 (2005) 7. Lepetit, V., Pilet, J., Fua, P.: Point matching as a classification problem for fast and robust object pose estimation. In: CVPR 2004, June 2004, vol. 2, pp. 244–250 (2004) 8. Lienhart, R., Maydt, J.: An extended set of haar-like features for rapid object detection. In: IEEE ICIP, pp. 900–903. IEEE Computer Society Press, Los Alamitos (2002) 9. Marchand, E., Bouthemy, P., Chaumette, F., Moreau, V.: Robust real-time visual tracking using a 2D-3D model-based approach. In: ICCV, pp. 262–268 (September 1999) 10. Quinlan, R.: Learning with continuous classes. In: Proceedings of the 5th Australian Joint Conference on Artificial Intelligence, pp. 343–348 (1992) 11. Romdhani, S.: Face Image Analysis using a Multiple Feature Fitting Strategy. PhD thesis, University of Basel, Computer Science Department, Basel, CH (January 2005) 12. Simon, D., Hebert, M., Kanade, T.: Real-time 3-D pose estimation using a highspeed range sensor. In: ICRA 1994. Proceedings of IEEE International Conference on Robotics and Automation, vol. 3, pp. 2235–2241 (May 1994) 13. Viola, P., Jones, M.: Rapid object detection using a boosted cascade of simple features. Computer Vision and Pattern Recognition (CVPR) (2001) 14. Wimmer, M., Pietzsch, S., Stulp, F., Radig, B.: Learning robust objective functions with application to face model fitting. In: Proceedings of the 29th DAGM Symposium, Heidelberg, Germany, September 2007 (to appear) 15. Witten, I.H., Frank, E.: Data Mining: Practical machine learning tools and techniques, 2nd edn. Morgan Kaufmann, San Francisco (2005)

Multiple View Geometry for Non-rigid Motions Viewed from Translational Cameras Cheng Wan, Kazuki Kozuka, and Jun Sato Department of Computer Science and Engineering, Nagoya Institute of Technology, Nagoya 466–8555, Japan

Abstract. This paper introduces multiple view geometry under projective projections from four-dimensional space to two-dimensional space which can represent multiple view geometry under the projection of space with time. We show the multifocal tensors defined under space-time projective projections can be derived from non-rigid object motions viewed from multiple cameras with arbitrary translational motions, and they are practical for generating images of non-rigid object motions viewed from cameras with arbitrary translational motions. The method is tested in real image sequences.

1

Introduction

The multiple view geometry is very important for describing the relationship between images taken from multiple cameras and for recovering 3D geometry from images [1,2,3,4,6,7]. In the traditional multiple view geometry, the projection from the 3D space to 2D images has been assumed [3]. However, the traditional multiple view geometry is limited for describing the case where enough corresponding points are visible from a static configuration of multiple cameras. Recently, some efforts for extending the multiple view geometry for more general point-camera configurations have been made [5,8,10,11,12]. Wolf et al. [8] studied the multiple view geometry on the projections from N dimensional space to 2D images and showed that it can be used for describing the relationship of multiple views obtained from moving cameras and points which move on straight lines with constant speed. Thus the motions of objects are limited. Hayakawa et al. [9] proposed the multiple view geometry in the space-time which makes it possible to describe the relationship of multiple views derived from translational cameras and non-rigid arbitrary motions. However their multiple view geometry assumes affine projection, which is an ideal model and can not be applied if we have strong perspective distortions in images. In this paper we introduce the multiple view geometry under the projective projection from 4D space to 2D space and show that such a universal model can represent multiple view geometry in the case where non-rigid arbitrary motions are viewed from multiple translational projective cameras. We first analyze multiple view geometry under the projection from 4D to 2D, and show that we have multilinear relationships for up to 5 views unlike the traditional multilinear Y. Yagi et al. (Eds.): ACCV 2007, Part II, LNCS 4844, pp. 342–352, 2007. c Springer-Verlag Berlin Heidelberg 2007 

Multiple View Geometry for Non-rigid Motions

343

relationships. The three view, four view and five view geometries are studied extensively and new trilinear, quadrilinear and quintilinear relationships under the projective projection from 4D space to 2D space are presented. We next show that the newly defined multiple view geometry can be used for describing the relationship between images taken from non-rigid motions viewed from multiple translational cameras. We also show that it is very useful for generating images of non-rigid object motions viewed from arbitrary translational cameras.

2

Projective Projections from 4D to 2D

We first consider projective projections from 4D space to 2D space. This projection is used to describe the relationship between the real space-time and 2D images, and for analyzing the multiple view geometry under space-time projections. Let X = [X 1 , X 2 , X 3 , X 4 , X 5 ] be the homogeneous coordinates of a 4D space point projected to a point in the 2D space, whose homogeneous coordinates are represented by x = [x1 , x2 , x3 ] . Then, the extended projective projection from X to x can be described as follows: x ∼ PX

(1)

where (∼) denotes equality up to a scale, and P denotes the following 3 × 5 matrix: ⎤ ⎡ m11 m12 m13 m14 m15 P = ⎣m21 m22 m23 m24 m25 ⎦ (2) m31 m32 m33 m34 m35 From (1), we find that the extended projective camera, P, has 14 DOF. In the next section, we consider the multiple view geometry of the extended projective cameras.

3

Projective Multiple View Geometry from 4D to 2D

From (1), we have the following equation for N extended projective cameras: ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ X 0 P x 0 0 ··· 0 ⎢ ⎥ ⎢ P 0 x 0 · · · 0⎥ ⎢ λ ⎥ ⎢0⎥ ⎢  ⎥⎢ ⎥ ⎢ ⎥ (3) ⎢P 0 0 x · · · 0⎥ ⎢ λ ⎥ = ⎢0⎥ ⎣ ⎦⎢ ⎥ ⎣ ⎦ .. .. ⎣λ ⎦ .. .. . . . . where, the leftmost matrix, M, in (3) is 3N × (5 + N ), and the (5 + N ) × (5 + N ) minors Q of M constitute multilinear relationships under the extended projection projection as: det Q = 0. We can choose any 5 + N rows from M to constitute Q, but we have to take at least 2 rows from each camera for deriving meaningful N view relationships (note, each camera has 3 rows in M).

344

C. Wan, K. Kozuka, and J. Sato

Table 1. The number of corresponding points required for computing multifocal tensors in three, four and five views with nonlinear method and linear method views nonlinear mothod linear mothod three 9 13 four 8 10 five 8 9

Table 2. Trilinear relations between point and line coordinates in three views. The final column denotes the number of linearly independent equations. correspondence relation # of equations three points xi xj xk krv Tijr = 0v 2 1 two points, one line xi xj lr Tijr = 0 2 one point, two lines xi lq lr qjt Tijr = 0t 4 three lines lp lq lr pis qjt Tijr = 0st

Thus, 5 + N ≥ 2N must hold for defining multilinear relationships for N view geometry in the 4D space. Thus, we find that, unlike the traditional multiple view geometry, the multilinear relationship for 5 views is the maximal linear relationship in the 4D space. We next consider the minimum number of points required for computing the multifocal tensors. The geometric DOF of N extended projective cameras is 14N − 24, since each extended projective camera has 14 DOF and these N cameras are in a single 4D projective space whose DOF is 24. Meanwhile, if we are given M points in the 4D space, and let them be projected to N projective cameras defined in (1). Then, we derive 2M N measurements from images, while we have to compute 14N − 24 + 4M components for fixing all the geometry in the 4D space. Thus, this condition must hold for computing the multifocal tensors from images: 2M N ≥ 14N − 24 + 4M . We find that minimum of 9, 8, 8 points are required to compute multifocal tensors in three, four and five views (see Table 1). 3.1

Three View Geometry

We next introduce the multiple view geometry of three extended projective cameras. For three views, the sub square matrix Q is 8 × 8. From det Q = 0, we have the following trilinear relationship under extended projective camera projections: xi xj xk krv Tijr = 0v

(4)

where ijk denotes a tensor, which represents a sign based on permutation from {i,j,k} to {1,2,3}. Tijr is the trifocal tensor for the extended cameras and has the following form:

Multiple View Geometry for Non-rigid Motions

345

Table 3. Quadrilinear relations between point and line coordinates in four views correspondence relation # of equations four points xi xj xk xs jlu kmv snw Qlmn = 0 8 uvw i three points, one line xi xj xk ln jlu kmv Qlmn = 0uv 4 i   two points, two lines xi xj lm ln jlu Qlmn = 0u 2 i   lmn one point, three lines xi ll lm ln Q i =0 1   kiw lmn four lines lk ll lm ln  Q i = 0w 2

Table 4. Quintilinear relations between point and line coordinates in five views correspondence relation # five points xixjxkxsxt ila jmb knc sf d tge Rlmnf g = 0abcde four points, one line xi xj xk xs lg ila jmb knc sf d Rlmnf g = 0abcd three points, two lines xi xj xk lf lg ila jmb knc Rlmnf g = 0abc two points, three lines xi xj ln lf lg ila jmb Rlmnf g = 0ab one point, four lines xi l mln lf lg ila Rlmnf g = 0a five lines ll l mln lf lg Rlmnf g = 0

⎤ al ⎢ am ⎥ ⎢ q⎥ r ⎥ Tij = ilm jqu det ⎢ ⎢ bu ⎥ ⎣b ⎦ cr

of eq. 32 16 8 4 2 1

⎡

(5)

where ai denotes the ith row of P, bi denotes the ith row of P and ci denotes the ith row of P respectively. The trifocal tensor Tijr is 3 × 3 × 3 and has 27 entries. If the extended cameras are projective as shown in (1), we have only 26 free parameters in Tijr except a scale ambiguity. On the other hand, (4) provides us 3 linear equations on Tijr , but only 2 of them are linearly independent. Thus, at least 13 corresponding points are required to compute Tijr from images linearly. A complete set of the trilinear equations involving the trifocal tensor are given in Table 2. All of these equations are linear in the entries of the trifocal tensor Tijr . 3.2

Four View and Five View Geometry

Similarly, the four view and the five view geometry can also be derived for the extended projective cameras. The quadrilinear relationship under extended porjective projection is: = 0uvw xi xj xk xs jlu kmv snw Qlmn i

(6)

346

C. Wan, K. Kozuka, and J. Sato

Qlmn is the quadrifocal tensor whose form is described as: i ⎡ p⎤ a ⎢ aq ⎥ ⎢ l⎥ ⎥ = ipq det⎢ Qlmn i ⎢ bm ⎥ ⎣c ⎦ dn

(7)

where di denotes the ith row of P .The quadrifocal tensor Qlmn has 81 entries. i Excluding a scale ambiguity, it has 80 free parameters. And 27 linear equations are given from (6) but only 8 of them are linearly independent. Therefore, from imminimum of 10 corresponding points are required to compute Qlmn i ages linearly. The quadrilinear relationships involving the quadrifocal tensor are summerized in Table 3. We next introduce the multiple view geometry of five extended projective cameras. The quintilinear constraint is expressed as follows: xixjxkxsxt ila jmb knc sf d tge Rlmnf g = 0abcde

(8)

where Rlmnf g is the quintifocal tensor (five view tensor) whose form is represented as follows: ⎡ l ⎤ a ⎢ bm ⎥ ⎢ n⎥ ⎥ (9) Rlmnf g = det⎢ ⎢ cf ⎥ ⎣d ⎦ eg where ei denotes the ith row of P . The quintifocal tensor Rlmnf g has 243 entries. If the extended cameras are projective as shown in (1), we have only 242 free parameters in Rlmnf g except a scale. On the other hand, (8) provides us 243 linear equations on Rlmnf g , but only 32 of them are linearly independent. If we have N correponding points, 32N −N C2 independent constraints can be derived. Thus, at least 9 corresponding points are required to compute Rlmnf g from images linearly. The number of corresponding points required for computing multifocal tensors is summerized in Table 1. The quintilinear relationships are given in Table 4.

4

Multiple View Geometry for Multiple Moving Cameras

Let us consider a single moving point in the 3D space. If the multiple cameras are stationary, we can compute the traditional multifocal tensors [3] from the image motion of this point, and they can be used for constraining image points in arbitrary views and for reconstructing 3D points from images. However, if these cameras are moving independently, the traditional multifocal tensors cannot be computed from the image motion of a single point. Nonetheless, we in this section

Multiple View Geometry for Non-rigid Motions

camera motion

C3 (T + 2)

C1 (T + 2)

trajectory of point motion

C1 (T + 1)

347

camera motion

C3 (T + 1)

X(T + 2) X(T + 1)

C1 (T )

C3 (T )

X(T )

C2 (T + 2) C2 (T + 1) camera motion

C2 (T )

Fig. 1. A moving point in 3D space and its projections in three translational projective cameras. The multifocal tensor defined under space-time projections can describe the relationship between these image projections.

show that if the camera motions are translational as shown in Fig. 1, the multiple view geometry under extended projective projections can be computed from the image motion of a single point, and they can be used for, for example, generating image motions viewed from arbitrary translational cameras. We first show that the extended projective cameras shown in (1) can be used for describing non-rigid object motions viewed from stationary multiple cameras.We next show that this camera model can also be used for describing non-rigid object motions viewed from multiple cameras with translational motions of constant speed.  = [X, Y, Z] , in the real space can be considered The motions of a point, W = [X, Y, Z, T ] , in a 4D space-time where T denotes time as a set of points, X and ( ) denotes inhomogeneous coordinates. The motions in the real space are = [x, y] . Thus, if projected to images, and can be observed as a set of points, x we assume projective projections in the space axes, the space-time projections can be described by the extended projective cameras shown in (1). We next show that the multiple view geometry described in section 3 can also be applied for multiple moving cameras. Let us consider a usual projective camera which projects points in 3D to 2D images. If the translational motions of the projective camera are constant, non-rigid motions are projected to images as: ⎡ ⎤ ⎤ X(T ) − T ΔX ⎡ ⎤ ⎡ x(T ) a11 a12 a13 a14 ⎢ Y (T ) − T ΔY ⎥ ⎥ λ ⎣ y(T )⎦ = ⎣a21 a22 a23 a24 ⎦ ⎢ (10) ⎣ Z(T ) − T ΔZ ⎦ a31 a32 a33 a34 1 1 ⎡ ⎤ ⎤ X(T ) ⎡ ⎥ a11 a12 a13 −a11 ΔX − a12 ΔY − a13 ΔZ a14 ⎢ ⎢ Y (T ) ⎥ ⎥ (11) Z(T ) = ⎣a21 a22 a23 −a21 ΔX − a22 ΔY − a23 ΔZ a24 ⎦ ⎢ ⎢ ⎥ a31 a32 a33 −a31 ΔX − a32 ΔY − a33 ΔZ a34 ⎣ T ⎦ 1

348

C. Wan, K. Kozuka, and J. Sato

(a) Camera 1

(b) Camera 2

(c) Camera 3

Fig. 2. Single point motion experiment. (a), (b) and (c) show image motions of a single point viewed from camera 1, 2 and 3. The 13 green points in each image are corresponding points used for computing the trifocal tensor. Note that these 3 cameras are translating with different speed and direction.

y

y

200

200

150

150

100

100

50

50

50

100

150

(a)

200

250

300

x

50

100

150

200

250

300

x

(b)

Fig. 3. The white curve in (a) shows image motions recovered from the extended trifocal tensor, and the black curve shows real image motions observed in camera 3. (b) shows those recovered from the traditional trifocal tensor. The 13 black points in (a) and 7 black points in (b) show points used for computing the trifocal tensors.

where x(T ) and y(T ) denote image coordinates at time T , X(T ), Y (T ) and Z(T ) denote coordinates of a 3D point at time T , and ΔX, ΔY and ΔZ denote camera motions in X, Y and Z axes.Since the translational motion is constant in each camera, ΔX, ΔY and ΔZ are fixed in each camera. Then, we find, from (11), that the projections of non-rigid motions to multiple cameras with translational motions can also be described by the extended projective cameras shown in (1).Thus the multiple view geometry described in section 3 can also be applied to multiple projective cameras with constant translational motions. Note that if we have enough moving points in the scene, we can also compute the traditional multiple view geometry on the multiple moving cameras at each instant.

5

Experiments

We next show the results of experiments. We first show the results from real images that the trifocal tensor for extended projective cameras can be computed from image motions viewed from arbitrary translational cameras, and can be used for generating the third view from the first and the second view of moving cameras. We next evaluate the stability of extracted trifocal tensors for extended projective cameras.

Multiple View Geometry for Non-rigid Motions

(a1) Camera 1

(b1) Camera 2

(c1) Camera 3

(d1)

(a2) Camera 1

(b2) Camera 2

(c2) Camera 3

(d2)

349

Fig. 4. Other single point motion experiments. (ai), (bi) and (ci) show three views of the ith motion. The 13 green points in each image and black points in (di) are corresponding points. The white curve in (di) shows image motions recovered from the extended trifocal tensor, and the black curve shows real image motions observed in camera 3.

5.1

Real Image Experiment

In this section, we show the results from single point motion and multiple point motion experiments. In the first experiment, we used 3 cameras which are translating with different constant speed and different direction, and computed trifocal tensors between these 3 cameras by using a single moving point in the 3D space. Since multiple cameras are dynamic, we cannot compute the traditional trifocal tensor of these cameras from a moving point. Nonetheless we can compute the extended trifocal tensor and can generate image motions in one of the 3 views from the others. Fig. 2 (a), (b) and (c) show image motions of a single moving point in translational camera 1, 2 and 3 respectively. The trifocal tensor is computed from 13 points on the image motions in three views. They are shown by green points in (a), (b) and (c). The extracted trifocal tensor is used for generating image motions in camera 3 from image motions in camera 1 and 2. The white curve in Fig. 3 (a) shows image motions in camera 3 generated from the extended trifocal tensor, and the black curve shows the real image motions viewed from camera 3. As shown in Fig. 3 (a), the generated image motions almost recovered the original image motions even if these 3 cameras have unknown translational motions. To show the advantage of the extended trifocal tensor, we also show image motions generated from the traditional trifocal tensor, that is, trifocal tensor defined for projections from 3D space to 2D space. 7 points taken from the former 13 points are used as corresponding points in three views for computing the traditional projective trifocal tensor. The image motion in camera 3 generated from the image motions in camera 1 and 2 by using the extracted traditional trifocal tensor is shown by white curve in Fig. 3 (b). As shown in Fig. 3 (b), the generated image motion is very different from the real image motion shown by black curve as we

350

C. Wan, K. Kozuka, and J. Sato

(a1) Camera 1

(b1) Camera 2

(c1) Camera 3

(d1)

(a2) Camera 1

(b2) Camera 2

(c2) Camera 3

(d2)

(a3) Camera 1

(b3) Camera 2

(c3) Camera 3

(d3)

Fig. 5. Multiple point motion experiments. (ai), (bi) and (ci) show three views of the ith motion. The green curve and the red curve represent two different image motion. The 7 green points on the green curve and the 6 red points on the red curve in each image are corresponding points used for computing the trifocal tensor. Note that these 3 cameras are translating with different speed and direction. The white curve in (di) shows image motions recovered from the extended trifocal tensor, and the black curve shows real image motions observed in camera 3.

expected, and thus we find that the traditional multiple view geometry cannot describe such general situations, while the proposed multiple view geometry can as shown in Fig. 3 (a). The results from other single point motions are also given. In Fig. 4, (ai), (bi) and (ci) show three views of the ith motion. The 13 green points in each image are corresponding points used for computing the trifocal tensor. Note that these 3 cameras are translating with different speed and different direction. The white curve in (di) shows image motions recovered from the extended trifocal tensor in camera 3, and the black curve shows real image motions observed in camera 3. The 13 black points in (di) show points used for computing the trifocal tensor. As we can see, the trifocal tensor defined under space-time projective projections can be derived from arbitrary single point motions viewed from the 3 cameras with arbitrary translational motions, and they are practical for generating images of single point motions viewed from translational camera. Next we show the results from multiple point motions. In Fig. 5, (ai), (bi) and (ci) show three views of the ith motion. The green curve and the red curve represent two different image motion. The 7 green points on the green curve and the 6 red points on the red curve in each image are corresponding points used for computing the trifocal tensor. Note that these 3 cameras are

Multiple View Geometry for Non-rigid Motions

351

40

C1 C2

2 Z

0

C3

-2

5 2.5 0

0 x

-2.5

5

30 25 20 15 10 5 0 13

-5 10

(a)

y

reprojection error

35

15

17

19

21

number of points

          23   

          25   

(b)

Fig. 6. Stability evaluation.(a) shows 3 translating cameras and a moving point in the 3D space. The black points show the viewpoints of the cameras before translational motions, and the white points show those after the motions.(b) shows the relationship between the number of corresponding points and the reprojection errors.

translating with different speed and different direction. The white curve in (di) shows image motions recovered from the extended trifocal tensor in camera 3, and the black curve shows real image motions observed in camera 3. The 13 black points in (di) show points used for computing the trifocal tensor. According to these experiments, we found that the extended multifocal tensors can be derived from non-rigid object motions viewed from multiple cameras with arbitrary translational motions, and they are useful for generating images of non-rigid object motions viewed from cameras with arbitrary translational motions. 5.2

Stability Evaluation

We next show the stability of extracted trifocal tensors under space-time projections. Fig. 6 (a) shows a 3D configuration of 3 moving cameras and a moving point. The black points show the viewpoints of three cameras, C1 , C2 and C3 , before translational motions, and the white points show their viewpoints after the translational motions. The translational motions of these three cameras are different and unknown. The black curve shows a locus of a freely moving point. For evaluating the extracted trifocal tensors, we computed reprojection errors derived from the trifocal tensors. The reprojection error is defined as: 1 N ˆ i )2 , where d(mi , m ˆ i ) denotes a distance between a true point i=1 d(mi , m N mi and a point m ˆ i recovered from the trifocal tensor. We increased the number of corresponding points used for computing trifocal tensors in three views from 13 to 25, and evaluated the reprojection errors. The Gaussian noise with the standard deviation of 1 pixel is added to every point on the locus. Fig. 6 (b) shows the relationship between the number of corresponding points and the reprojection errors. As we can see, the stability is obviously improved by using a few more points than the minimum number of corresponding points.

352

6

C. Wan, K. Kozuka, and J. Sato

Conclusion

In this paper, we analyzed multiple view geometry under projective projections from 4D space to 2D space, and showed that it can represent multiple view geometry under space-time projections. In particular, we showed that multifocal tensors defined under space-time projective projections can be computed from non-rigid object motions viewed from multiple cameras with arbitrary translational motions. We also showed that they are very useful for generating images of non-rigid motions viewed from projective cameras with arbitrary translational motions. The method was implemented and tested by using real image sequences. The stability of extracted trifocal tensors was also evaluated.

References 1. Faugeras, O.D., Luong, Q.T.: The Geometry of Multiple Images. MIT Press, Cambridge (2001) 2. Faugeras, O.D., Mourrain, B.: On the geometry and algebra of the point and line correspondences beterrn N images. In: Proc. 5th International Conference on Computer Vision, pp. 951–956 (1995) 3. Hartley, R.I., Zisserman, A.: Multiple View Geometry. Cambridge University Press, Cambridge (2000) 4. Hartley, R.I.: Multilinear relationship between coordinates of corresponding image points and lines. In: Proc. International Workshop on Computer Vision and Applied Geometry (1995) 5. Heyden, A.: A common framework for multiple view tensors. In: Burkhardt, H.-J., Neumann, B. (eds.) ECCV 1998. LNCS, vol. 1406, pp. 3–19. Springer, Heidelberg (1998) 6. Heyden, A.: Tensorial properties of multiple view constraints. Mathematical Methods in the Applied Sciences 23, 169–202 (2000) 7. Shashua, A., Wolf, L.: Homography tensors: On algebraic entities that represent three views of static or moving planar points. In: Vernon, D. (ed.) ECCV 2000. LNCS, vol. 1842, Springer, Heidelberg (2000) 8. Wolf, L., Shashua, A.: On projection matrices P k → P 2 , k = 3, · · ·, 6, and their applications in computer vision. In: Proc. 8th International Conference on Computer Vision, vol. 1, pp. 412–419 (2001) 9. Hayakawa, K., Sato, J.: Multiple View Geometry in the Space-Time. In: Proc. Asian Conference on Computer Vision, pp. 437–446 (2006) 10. Wexler, Y., Shashua, A.: On the synthesis of dynamic scenes from reference views. In: Proc. Conference on Computer Vision and Pattern Recognition, pp. 576–581 (2000) 11. Hartley, R.I., Schaffalitzky, F.: Reconstruction from Projections using Grassman Tensors. In: Pajdla, T., Matas, J(G.) (eds.) ECCV 2004. LNCS, vol. 3021, pp. 363–375. Springer, Heidelberg (2004) 12. Sturm, P.: Multi-View Geometry for General Camera Models. In: Proc. Conference on Computer Vision and Pattern Recognition, pp. 206–212 (2005)

Visual Odometry for Non-overlapping Views Using Second-Order Cone Programming Jae-Hak Kim1 , Richard Hartley1 , Jan-Michael Frahm2 and Marc Pollefeys2 1

Research School of Information Sciences and Engineering The Australian National University National ICT Australia, NICTA 2 Department of Computer Science University of North Carolina at Chapel Hill

Abstract. We present a solution for motion estimation for a set of cameras which are firmly mounted on a head unit and do not have overlapping views in each image. This problem relates to ego-motion estimation of multiple cameras, or visual odometry. We reduce motion estimation to solving a triangulation problem, which finds a point in space from multiple views. The optimal solution of the triangulation problem in Linfinity norm is found using SOCP (Second-Order Cone Programming) Consequently, with the help of the optimal solution for the triangulation, we can solve visual odometry by using SOCP as well.

1

Introduction

Motion estimation of cameras or pose estimation, mostly in the case of having overlapping points or tracks between views, has been studied in computer vision research for many years [1]. However, non-overlapping or slightly overlapping camera systems have not been studied so much, particulary the motion estimation problem. The non-overlapping views mean that all images captured with cameras do not have any, or at most have only a few common points. There are potential applications for this camera system. For instance, we construct a cluster of multiple cameras which are firmly installed on a base unit such as a vehicle, and the cameras are positioned to look at different view directions. A panoramic or omnidirectional image can be obtained from images captured with a set of cameras with small overlap. Another example is a vehicle with cameras mounted on it to provide driving assistance such as side/rear view cameras. An important problem is visual odometry – how can we estimate the tracks of a vehicle and use this data to determine where the vehicle is placed. There has been prior research considering a set of many cameras moving together as one camera. In [2] an algebraic solution to the multiple camera motion problem is presented. Similar research on planetary rover operations has been conducted to estimate the motion of a rover on Mars and to keep track of the rover [3]. Other 

NICTA is funded by the Australian Government’s Backing Australia’s Ability initiative, in part through the Australian Research Council.

Y. Yagi et al. (Eds.): ACCV 2007, Part II, LNCS 4844, pp. 353–362, 2007. c Springer-Verlag Berlin Heidelberg 2007 

354

J.-H. Kim et al.

research on visual odometry has been performed to estimate the motion of a stereo rig or single camera [4]. Prior work on non-overlapping cameras includes most notably the paper [5]. This differs from our work in aligning independently computed tracks of the different cameras, whereas we compute a motion estimate using all the cameras at once. Finally, an earlier solution to the problem was proposed in unpublished work of [6], which may appear elsewhere. In this paper, we propose a solution to estimate the six degrees of freedom (DOFs) of the motion, three rotation parameters and three translation parameters (including scale), for a set of multiple cameras with non-overlapping views, based on L∞ triangulation. A main contribution of this paper is that we provided a well-founded geometric solution to the motion estimation in non-overlapping multiple cameras.

2

Problem Formulation

Consider a set of n calibrated cameras with non-overlapping fields of view. Since the cameras are calibrated, we may assume that they are all oriented in the same way just to simplify the mathematics. This is easily done by multiplying an inverse of the rotation matrix to the original image coordinates. This being the case, we can also assume that they all have camera matrices originally equal to Pi = [I| − ci ]. We assume that all ci are known. The cameras then undergo a common motion, described by a Euclidean matrix   R −R t M= (1) 0 1 where R is a rotation, and t is a translation of a set of cameras. Then, the i-th camera matrix changes to    R t Pi = Pi M−1 = [I | − ci ]  = [R | t − ci ] 0 1 which is located at R(ci − t). Suppose that we compute all the essential matrices of the cameras independently, then decompose them into rotation and translation. We observe that the rotations computed from all the essential matrices are the same. This is true only because all the cameras have the same orientation. We can average them to get an overall estimate of rotation. Then, we would like to compute the translation. As we will demonstrate, this is a triangulation problem. Geometric concept. First, let us look at a geometric idea derived from this problem. An illustration of a motion of a set of cameras is shown in Figure 1. A bundle of cameras is moved by a rotation R and translation t. All cameras at ci are moved to ci . The first camera at position c1 is a sum of vectors ci , ci − ci and c1 − ci where i = 1...3. Observing that the vector vi in Figure 1 is the same as the vector ci − ci and the vector c1 − ci is obtained by rotating

Visual Odometry for Non-overlapping Views

R, t

c3

c2

c3 c1

v1 c1

355

v2

c2

c2 + R(c1 − c2)

v3 c3 + R(c1 − c3)

Fig. 1. A set of cameras is moved by Euclidean motion of rotation R and translation t. The centre of the first camera c1 is moved to c1 by the motion. The centre c1 is a common point where all translation direction vectors meet. The translation direction vectors are indicated as red, green and blue solid arrows which are v1 , v2 and v3 , respectively. Consequently, this is a triangulation problem.

the vector c1 − ci , the first camera at position c1 can be rewritten as a sum of three vectors ci , R(c1 − ci ) and vi . Therefore, the three vectors vi , colored solid arrows in Figure 1 meet in one common point c1 , the position of the centre of the first camera after the motion. It means that finding the motion of the set of cameras is the same as solving a triangulation problem for translation direction vectors derived from each view. Secondly, let us derive detailed equations on this problem from the geometric concept we have described above. Let Ei be the essential matrix for the i-th camera. From E1 , we can compute the translation vector of the first camera, P1 , in the usual way. This is a vector passing through the original position of the first camera. The final position of this camera must lie along this vector. Next, we use Ei , for i > 1 to estimate a vector along which the final position of the first camera can be found. Thus, for instance, we use E2 to find the final position of P1 . This works as follows. The i-th essential matrix Ei decomposes into Ri = R and a translation vector vi . In other words, Ei = R[vi ]× . This means that the i-th camera moves to a point ci + λi vi , the value of λi being unknown. This point is the final position of each camera ci in Figure 1. We transfer this motion to determine the motion of the first camera. We consider the motion as taking place in two stages, first rotation, then translation. First the camera centre c1 is rotated by R about point ci to point ci + R(c1 − ci ). Then it is translated in the direction vi to the point c1 = ci + R(c1 − ci ) + λi vi . Thus, we see that c1 lies on the line with direction vector vi , based at point ci + R(c1 + ci ).

356

J.-H. Kim et al.

In short, each essential matrix Ei constrains the final position of the first camera to lie along a line. These lines are not all the same, in fact unless R = I, they are all different. The problem now comes down to finding the values of λi and c1 such that for all i: c1 = ci + R(c1 − ci ) + λi vi

for i = 1, ..., n .

(2)

Having found c1 , we can get t from the equation c1 = R(c1 − t). Averaging Rotations. From the several cameras and their essential matrices Ei , we have several estimates Ri = R for the rotation of the camera rig. We determine the best estimate of R by averaging these rotations. This is done by representing each rotation Ri as a unit quaternion, computing the average of the quaternions and renormalizing to unit norm. Since a quaternion and its negative both represent the same rotation, it is important to choose consistently signed quaternions to represent the separate rotations Ri . Algebraic derivations. Alternatively, it is possible to show an algebraic derivation of the equations as follows. Given Pi = [I| − ci ] and Pi = [R | t − ci ] ( See (2)), an essential matrix is written as Ei = R [ci + R(t − ci )]× I = [R ci + (t − ci )]× R .

(3)

Considering that the decomposition of the essential matrix Ei is Ei = Ri [vi ]× = [Ri vi ]× Ri , we may get the rotation and translation from (3), namely Ri = R and λi Ri vi = R ci + (t − ci ). As a result, t = λi R vi + ci − R ci which is the same equation derived from the geometric concept. A Triangulation Problem. Equation (2) gives us independent measurements of the position of point c1 . Denoting ci + R(c1 − ci ) by Ci , the point c1 must lie at the intersection of the lines Ci + λi vi . In the presence of noise, these lines will not meet, so we need find a good approximation to c1 . Note that the points Ci and vectors vi are known, having been computed from the known calibration of the camera geometry, and the computed essential matrices Ei . The problem of estimating the best c1 is identical with the triangulation problem studied (among many places) in [7,8]. We adopt the approach of [7] of solving this under L∞ norm. The derived solution is the point c1 that minimizes the maximum difference between c1 − Ci and the direction vector vi , for all i. In the presence of noise, the point c1 will lie in the intersection of cones based at the vertex Ci , and with axis defined by the direction vectors vi . To formulate the triangulation problem, instead of c1 , we write X as the final position of the first camera where all translations derived from each essential matrix meet together. As we have explained in the previous section, in the presence of noise we have n cones, each one aligned with one of the translation directions. The desired point X lies in the overlap of all these cones, and, finding this overlap region gives the solution we need in order to get the motion

Visual Odometry for Non-overlapping Views

357

of cameras. Then, our original motion estimation problem is formulated as the following minimization problem: min max X

i

||(X − Ci ) × vi || . (X − Ci ) vi

(4)

Note that the quotient is equal to tan2 (θi ) where θi is the angle between vi and (X − Ci ). This problem can be solved as a Second-Order Cone Programming (SOCP) using a bisection algorithm [9].

3

Algorithm

The algorithm to estimate motion of cameras having non-overlapping views is as follows: Given 1. A set of cameras described in initial position by their known calibrated camera matrices Pi = Ri [I| − ci ]. The cameras then move to a second (unknown) position, described by camera matrices Pi . 2. For each camera pair Pi , Pi , several point correspondences xij ↔ xij (expressed in calibrated coordinates as homogeneous 3-vectors). Objective: Find the motion matrix of the form (1) that determines the common motion of the cameras, such that Pi = Pi M−1 . Algorithm 1. Normalize the image coordinates to calibrated image coordinates by setting  ˆ ij = R−1 ˆ ij = R−1 x i xij and x i xij ,

2. 3. 4. 5.

ˆ ij /ˆ ˆ ij ← x ˆ ij /ˆ ˆ ij ← x xij  and x xij . then adjust to unit length by setting x  ˆ ij ↔ x ˆ ij for Compute each essential matrix Ei in terms of correspondences x the i-th camera. Decompose each Ei as Ei = Ri [vi ]× and find the rotation R as the average of the rotations Ri . Set Ci = ci + R(c1 − ci ). Solve the triangulation problem by finding the point X = c1 that (approximately because of noise) satisfies the condition X = Ci + λi vi for all i. Compute t from t = c1 − R c1 .

In our current implementation, we have used the L∞ norm to solve the triangulation problem. Other methods of solving the triangulation problem may be used, for instance the optimal L2 triangulation method given in [8]. Critical Motion. The algorithm has a critical condition when the rotation is zero. If this is so then, in the triangulation problem solved in this algorithm all the basepoints Ci involved are the same. Thus, we encounter a triangulation problem with a zero baseline. In this case, the magnitude of the translation can not be accurately determined.

358

4

J.-H. Kim et al.

Experiments

We have used SeDuMi and Yalmip toolbox for optimization of SOCP problems [10,11]. We used a five point solver to estimate the essential matrices [12,13]. We select the best five points from images using RANSAC to obtain an essential matrix, and then we improve the essential matrix by non-linear optimization. An alternative method for computing the essential matrix based on [14] was tried. This method gives the optimal essential matrix in L∞ norm. A comparison of the results for these two methods for computing Ei is given in Fig 6. Real data. We used Point Grey’s LadybugTM camera to generate some test data for our problem . This camera unit consists of six 1024×768 CCD color sensors with small overlap of their field of view. The six cameras, 6 sensors with 2.5 mm lenses, are closely packed on a head unit. Five CCDs are positioned in a horizontal ring around the head unit to capture side-view images, and one is located on the top of the head unit to take top-view images. Calibration information provided by Point Grey [15] is used to get intrinsic and relative extrinsic parameters of all six cameras. A piece of paper is positioned on the ground, and the camera is placed on the paper. Some books and objects are randomly located around the camera. The camera is moved manually while the positions of the camera at some points are marked on the paper as edges of the camera head unit. These marked edges on the paper are used to get the ground truth of relative motion of the camera for this experiment. The experimental setup is shown in Figure 2. A panoramic image stitched in our experimental setup is shown in Figure 3.

Fig. 2. An experimental setup of the LadybugTM camera on an A3 size paper surrounded by books. The camera is moved on the paper by hand, and the position of the camera at certain key frames is marked on the paper to provide the ground truth for the experiments.

Visual Odometry for Non-overlapping Views

359

Table 1. Experimental results of rotations at key frames 0, 30, 57 and 80, which correspond to the position number 0–3, respectively. For instance, a pair of rotation (R0 , R1 ) corresponds to a pair of rotations at key frame 0 and 30. Angles of each rotation are represented by the axis-angle rotation representation. Rotation pair (R0 , R1 ) (R0 , R2 ) (R0 , R3 )

True rotation Estimated rotation Axis Angle Axis [0 0 -1] 85.5◦ [-0.008647 -0.015547 0.999842] [0 0 -1] 157.0◦ [-0.022212 -0.008558 0.999717] [0 0 -1] 134.0◦ [ 0.024939 -0.005637 -0.999673]

Angle 85.15◦ 156.18◦ 134.95◦

Fig. 3. A panoramic image is obtained by stitching together all six images from the LadybugTM camera. This image is created by LadybugPro, the software provided by Point Grey Research Inc.

In the experiment, 139 frames of image are captured by each camera. Feature tracking is performed on the image sequence by the KLT (Kanade-Lucas-Tomasi) tracker [16]. Since there is lens distortion in the captured image, we correct the image coordinates of the feature tracks using lens distortion parameters provided by the Ladybug SDK library. The corrected image coordinates are used in all the equations we have derived. After that, we remove outliers from the feature tracks by the RANSAC (Random Sample Consensus) algorithm with a model of epipolar geometry in two view and trifocal tensors in three view [17]. There are key frames where we marked the positions of the camera. They are frames 0, 30, 57, 80, 110 and 138 in this experiment. The estimated path of the cameras over the frames is shown in Figure 4. After frame 80, the essential matrix result was badly estimated and subsequent estimation results were erroneous. A summary of the experimental results is shown in Tables 1 and 2. As can be seen, we have acquired a good estimation of rotations from frame 0 up to frame 80, within about one degree of accuracy. Adequate estimation of translations is reached up to frame 57 within less than 0.5 degrees. We have successfully tracked the motion of the camera through 57 frames. Somewhere between frame 57 and frame 80 an error occurred that invalidated the computation of the position of frame 80. Analysis indicates that this was due to a critical motion (near-zero rotation of the camera fixture) that made the translation estimation

360

J.-H. Kim et al.

0.1

0.08

0.06

0.04

0.03 0.025

0.02

0.02 0.015

0

0.01 0.005

−0.02

0 0.02

0

−0.02

−0.04

−0.06

−0.08

−0.1

−0.12

−0.14

−0.005 −0.14

(a) Top view

−0.12

−0.1

−0.08

−0.06

−0.04

−0.02

0

0.02

(b) Front view

Fig. 4. Estimated path of the LadybugTM camera viewed from (a) top and (b) front. The cameras numbered 0, 1, 2, 3, 4 and 5 are indicated as red, green, blue, cyan, magenta and black paths respectively.

Table 2. Experimental results of translation between two key frames are shown in scale ratio of two translation vectors and in angles of that at the two key frames. The translation direction vector t0i is a vector from the centre of the camera at the starting position, frame number 0, to the centre of the camera at the position number i. For example, t01 is a vector from the centre of the camera at frame 0 to the centre of the camera at frame 30. Translation Scale ratio Angles pair True value Estimated value True value Estimated value (t01 , t02 ) 0.6757 0.7424 28.5◦ 28.04◦ ◦ (t01 , t03 ) 0.4386 1.3406 42.5 84.01◦

inaccurate. Therefore, we have shown the frame-to-frame rotations, over frames in Figure 5-(a). As can be seen, there are frames for which the camera motion was less than 5 degrees. This occurred for frames 57 to 62, 67 to 72 and 72 to 77. In Figure 5-(c), we have shown the difference between the ground truth and estimated position of the cameras in this experiment. As can be seen, the position of the cameras is accurately estimated up to 57 frames. However, the track went off at frame 80. A beneficial feature of our method is that we can avoid such bad condition for the estimation by looking at the angles between frames and residual errors on the SOCP, and then we try to use other frames for the estimation. Using the L∞ optimal E-matrix. The results so-far were achieved using the 5-point algorithm (with iterative refinement) for calculating the essential matrix. We also tried using the method given in [14]. Since this method is quite new, we did not have time to obtain complete results. However, Fig 6 compares the angular error in the direction of the translation direction for the two methods. As may be seen, the L∞ -optimal method seems to work substantially better.

Visual Odometry for Non-overlapping Views

361

80 70 60

frame 30 frame 30

frame 0

50 60

frame 57 frame 57

40 420 −2 −4 −6 −8

30

40 20

frame 80

0

0

20 50

−20

10

frame 80

100

0 0

50

100

150

150

−40 −60

Fig. 5. The angles between pairs of frames used to estimate the motion are shown in (a). Note that a zero or near-zero rotation means a critical condition for estimating the motion of the cameras from the given frames. (b) Ground truth of positions (indicated as red lines) of the cameras with orientations at key frames 0, 30, 57 and 80, and estimated positions (indicated as black lines) of the cameras with their orientations at the same key frames. Orientations of the cameras are marked as blue arrows. Green lines are the estimated path through all 80 frames. Error of the estimation

Error of the estimation 30

Difference between the estimated translation direction and the translation from an essential matrix in degree

Difference between the estimated translation direction and the translation from an essential matrix in degree

30

25

20

15

10

5

0

0

20

40

60 Frames

80

(a) optimal

100

120

25

20

15

10

5

0

0

20

40

60

80

100

120

140

Frames

(b) 5-point

Fig. 6. Comparison of the angular error in the translation direction for two different methods of computing the essential matrix

5

Discussion

We have presented a solution to find the motion of cameras that are rigidly mounted and have minimally overlapping fields of view. This method works equally well for any number of cameras, not just two, and will therefore provide more accurate estimates than methods involving only pairs of cameras. The method requires a non-zero frame-to-frame rotation. Probably because of this, the estimation of motion through a long image sequence significantly went off track. The method geometrically showed good estimation results in experiments with real world data. However, the accumulated errors in processing long sequences

362

J.-H. Kim et al.

of images made the system produce bad estimations over long tracks. In the real experiments, we have found that a robust and accurate essential matrix estimation is a critical requirement to obtain correct motion estimation in this problem.

References 1. Hartley, R.I., Zisserman, A.: Multiple View Geometry in Computer Vision, 2nd edn. Cambridge University Press, Cambridge (2004) 2. Pless, R.: Using many cameras as one. In: CVPR 2003, vol. II, pp. 587–593 (2003) 3. Cheng, Y., Maimone, M., Matthies, L.: Visual odometry on the mars exploration rovers. In: Systems, Man and Cybernetics, 2005 IEEE International Conference (2005) 4. Nister, D., Naroditsky, O., Bergen, J.: Visual odometry. In: Conf. Computer Vision and Pattern Recognition, pp. 652–659 (2004) 5. Caspi, Y., Irani, M.: Aligning non-overlapping sequences. International Journal of Computer Vision 48(1), 39–51 (2002) 6. Clipp, B., Kim, J.H., Frahm, J.M., Pollefeys, M., Hartley, R.: Robust 6dof motion estimation for non-overlapping, multi-camera systems. Technical Report TR07-006 (Department of Computer Science, The University of North Carolina at Chapel Hill) 7. Hartley, R., Schaffalitzky, F.: L∞ minimization in geometric reconstruction problems. In: Conf. Computer Vision and Pattern Recognition, Washington, DC, USA, vol. I, pp. 504–509 (2004) 8. Lu, F., Hartley, R.: A fast optimal algorithm for L2 triangulation. In: Asian Conf. Computer Vision (2007) 9. Kahl, F.: Multiple view geometry and the L∞ -norm. In: Int. Conf. Computer Vision, Beijing, China, pp. 1002–1009 (2005) 10. Sturm, J.: Using SeDuMi 1.02, a MATLAB toolbox for optimization over symmetric cones. Optimization Methods and Software, Special issue on Interior Point Methods (CD supplement with software) 11–12, 625–653 (1999) 11. L¨ oberg, J.: Yalmip: A toolbox for modeling and optimization in MATLAB. In: Proceedings of the CACSD Conference, Taipei, Taiwan (2004) 12. Stew´enius, H., Engels, C., Nist´er, D.: Recent developments on direct relative orientation. ISPRS Journal of Photogrammetry and Remote Sensing 60, 284–294 (2006) 13. Li, H., Hartley, R.: Five-point motion estimation made easy. In: ICPR (1), pp. 630–633. IEEE Computer Society Press, Los Alamitos (2006) 14. Hartley, R., Kahl, F.: Global optimization through searching rotation space and optimal estimation of the essential matrix. In: Int. Conf. Computer Vision (2007) 15. Point Grey Research Incorporated: LadybugTM 2 camera (2006), http://www.ptgrey.com 16. Lucas, B., Kanade, T.: An iterative image registration technique with an application to stereo vision. In: IJCAI 1981, pp. 674–679 (1981) 17. Fischler, M.A., Bolles, R.C.: Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM 24(6), 381–395 (1981)

Pose Estimation from Circle or Parallel Lines in a Single Image Guanghui Wang1,2 , Q.M. Jonathan Wu1 , and Zhengqiao Ji1 1

Department of Electrical and Computer Engineering, The University of Windsor, 401 Sunset, Windsor, Ontario, Canada N9B 3P4 2 National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing, 100080, P.R. China [email protected], [email protected]

Abstract. The paper is focused on the problem of pose estimation from a single view in minimum conditions that can be obtained from images. Under the assumption of known intrinsic parameters, we propose and prove that the pose of the camera can be recovered uniquely in three situations: (a) the image of one circle with discriminable center; (b) the image of one circle with preassigned world frame; (c) the image of any two pairs of parallel lines. Compared with previous techniques, the proposed method does not need any 3D measurement of the circle or lines, thus the required conditions are easily satisfied in many scenarios. Extensive experiments are carried out to validate the proposed method.

1

Introduction

Determining the position and orientation of a camera from a single image with respect to a reference frame is a basic and important problem in robot vision field. There are many potential applications such as visual navigation, robot localization, object recognition, photogrammetry, visual surveillance and so on. During the past two decades, the problem was widely studied and many approaches have been proposed. One well known pose estimation problem is the perspective-n-point (PnP) problem, which was first proposed by Fishler and Bolles [5]. The problem is to find the pose of an object from the image of n points at known location on it. Following this idea, the problem was further studied by many researchers [6,8,9,15,14]. One of the major concerns of the PnP problem is the multi-solution phenomenon, all PnP problems for n ≤ 5 have multiple solutions. Thus we need further information to determine the correct solution [6]. Another kind of localization algorithm is based on line correspondences. Dhome et al. [4] proposed to compute the attitude of object from three line correspondences. Liu et al. [12] discussed some methods to recover the camera pose linearly or nonlinearly by using different combination of line and point features. Ansar and Daniilidis [1] presented a general framework which allows for a novel set of linear solutions to the pose estimation problem for both n points and n lines. Chen [2] proposed a polynomial approach to find close form solution for Y. Yagi et al. (Eds.): ACCV 2007, Part II, LNCS 4844, pp. 363–372, 2007. c Springer-Verlag Berlin Heidelberg 2007 

364

G. Wang, Q.M.J. Wu, and Z. Ji

pose determination from line-to-plane correspondences. The line based methods also suffer from the problem of multiple solutions. The above methods assume that the camera is calibrated and the positions of the points and lines are known. In practice, it may be hard to obtain the accurate measurements of these features in space. However, some geometrical constraints, such as coplanarity, parallelity and orthogonality, are abundant in many indoor and outdoor structured scenarios. Some researchers proposed to recover the camera pose from the image of a rectangle, two orthogonal parallel lines and some other scene constraints [7,18]. Circle is another very common pattern in man-made objects and scenes, many studies on camera calibration were based on the image of circles [10,11,13]. In this paper, we try to compute the camera’s pose from a single image based on geometrical configurations in the scene. Different from previous methods, we propose to use the image of only one circle, or the image of any two pairs of parallel lines that may not be coplanar or orthogonal. The proposed method is widely applicable since the conditions are easily satisfied in many scenarios.

2 2.1

Perspective Geometry and Pose Estimation Camera Projection and Pose Estimation

Under perspective projection, a 3D point x ∈ R3 in space is projected to an image point m ∈ R2 via a rank-3 projection matrix P ∈ R3×4 as ˜ = P˜ x sm x = K[R, t]˜ x = K[r1 , r2 , r3 , t]˜

(1)

˜ = [xT , w]T and m ˜ = [mT , w]T are the homogeneous forms of points where, x x and m respectively, R and t are the rotation matrix and translation vector from the world system to the camera system, s is a non-zero scalar, K is the camera calibration matrix. In this paper, we assume the camera is calibrated, thus we may set K = I3 = diag(1, 1, 1), which is equivalent to normalize the image coordinates by applying transformation K−1 . In this case, the projection matrix is simplified to P = [R, t] = [r1 , r2 , r3 , t]. When all space points are coplanar, the mapping between the space points and their images can be modeled by a plane homography H which is a nonsingular 3 × 3 homogeneous matrix. Without loss of generality, we may assume the coordinates of the space plane as [0, 0, 1, 0]T for a specified world frame, then we have H = [r1 , r2 , t]. Obviously, the rotation matrix R and translation vector t can be factorized directly from the homography. Proposition 1. When the camera is calibrated, the pose of the camera can be recovered from two orthogonal vanishing points in a single view. Proof. Without loss of generality, let us set the X and Y axes of the world system in line with the two orthogonal directions. In the normalized world coordinate ˜w = [0, 1, 0, 0]T system, the direction of X and Y axes are x ˜w = [1, 0, 0, 0]T and y

Pose Estimation from Circle or Parallel Lines in a Single Image

365

respectively, and the homogeneous vector of the world origin is ˜ ow = [0, 0, 0, 1]T . Under perspective projection, we have: ˜x = P x ˜w = [r1 , r2 , r3 , t][1, 0, 0, 0]T = r1 sx v ˜y = P y ˜w = [r1 , r2 , r3 , t][0, 1, 0, 0]T = r2 sy v

(2) (3)

so v ˜o = P ˜ ow = [r1 , r2 , r3 , t][0, 0, 0, 1]T = t

(4)

Thus the rotation matrix can be computed from r1 = ±

v ˜x v ˜y , r2 = ± , r3 = r1 × r2 ˜ vx  ˜ vy 

(5)

where the rotation matrix R = [r1 , r2 , r3 ] may have four solutions if righthanded coordinate system is adopted. While only two of them can ensure that the reconstructed objects lie in front of the camera, which may be seen by the camera. In practice, if the world coordinate frame is preassigned, the rotation matrix may be uniquely determined [19]. Since we have no metric information of the given scene, the translation vector can only be defined up to scale as t ≈ vo . This is to say that we can only recover the direction of the translation vector. In practice, the orthonormal constraint should be enforced during the computation since r1 and r2 in (5) may not be orthogonal due to image noise. Suppose the SVD decomposition of R12 = [r1 , r2 ] is UΣ VT , where Σ is a 3 × 2 matrix made of the two singular values of R12 . Thus we may obtain the best approxi 10 mation to the rotation matrix in the least square sense from R12 = U 0 1 VT , 00 since a rotation matrix should have unit singular values.   2.2

The Circular Points and Pose Estimation

The absolute conic (AC) is a conic on the ideal plane, which can be expressed in matrix form as Ω∞ = diag(1, 1, 1). Obviously, Ω∞ is composed of purely imaginary points on the infinite plane. Under perspective projection, we can obtain the image of the absolute conic (IAC) as ωa = (KKT )−1 , which depends only on the camera calibration matrix K. The IAC is an invisible imaginary point conic in an image. It is easy to verify that the absolute conic intersects the ideal line at two ideal complex conjugate points, which are called the circular points. The circular points can be expressed in canonical form as I = [1, i, 0, 0]T, J = [1, −i, 0, 0]T. Under perspective projection, their images can be expressed as: si m ˜ i = P I = [r1 , r2 , r3 , t][1, i, 0, 0]T = r1 + i r2 

sj m ˜ j = P J = [r1 , r2 , r3 , t][1, −i, 0, 0] = r1 − i r2 T

(6) (7)

Thus the imaged circular points (ICPs) are a pair of complex conjugate points, whose real and imaginary parts are defined by the first two columns of the rotation matrix. However, the rotation matrix can not be determined uniquely from the ICPs since (6) and (7) are defined up to scales.

366

G. Wang, Q.M.J. Wu, and Z. Ji

Proposition 2. Suppose mi and mj are the ICPs of a space plane, the world system is set on the plane. Then the pose of the camera can be uniquely determined from mi and mj if one direction of the world frame is preassigned. Proof. It is easy to verify that the line passing through the two imaged circular points is real, which is the vanishing line of the plane and can be computed from l∞ = mi × mj . Suppose ox is the image of one axis of the preassigned world frame, its vanishing point vx can be computed from the intersection of line ox with l∞ . If the vanishing point vy of Y direction is recovered, the camera pose can be determined accordingly from Proposition 1. Since the vanishing points of two orthogonal directions conjugate with  are vxT ωvy = 0 respect to the IAC, thus vy can be easily computed from . On the lT ∞ vy = 0 other hand, since two orthogonal vanishing points are harmonic with respect to the ICPs, their cross ratio Cross(vx vy ; mi mj ) = −1. Thus vy can also be computed from the cross ratio.  

3 3.1

Methods for Pose Estimation Pose Estimation from the Image of a Circle

Lemma 1. Any circle Ωc in a space plane π intersects the absolute conic Ω∞ at exactly two points, which are the circular points of the plane. Without loss of generality, let us set the XOY world frame on the supporting plane. Then any circle on the plane can be modelled in homogeneous form as (x − wx0 )2 + (y − wy0 )2 − w2 r2 = 0. The plane π intersects the ideal plane π∞ at the vanishing line L∞ . In the extended plane of the complex domain, L∞ has at most two intersections with Ωc . It is easy to verify that the circular points are the intersections. Lemma 2. The image of the circle Ωc intersects the IAC at four complex points, which can be divided into two pairs of complex conjugate points. Under perspective projection, any circle Ωc on space plane is imaged as a conic ωc = H−T Ωc H−1 , which is an ellipse in nondegenerate case. The absolute conic is projected to the IAC. Both the IAC and ωc are conics of second order that can be written in homogeneous form as xT ωc x = 0. According to B´ezout’s theorem, the two conics have four imaginary intersection points since the absolute conic and the circle have no real intersections in space. Suppose the complex point [a + bi] is one intersection, it is easy to verify that the conjugate point [a − bi] is also a solution. Thus the four intersections can be divided into two complex conjugate pairs. It is obvious that one pair of them is the ICPs, but the ambiguity can not be solved in the image with one circle. If there are two or more circles on the same or parallel space plane, the ICPs can be uniquely determined since the imaged circular points are the common intersections of each circle with the IAC in the image. However, we may have only one circle in many situations, then how to determine the ICPs in this case?

Pose Estimation from Circle or Parallel Lines in a Single Image

367

Proposition 3. The imaged circular points can be uniquely determined from the image of one circle if the center of the circle can be detected in the image. Proof. As shown in Fig.1, the image of the circle ωc intersects the IAC at two pairs of complex conjugate points mi , mj and mi , mj . Let us define two lines as l∞ = mi × mj , l∞ = mi × mj

(8)

then one of the lines must be the vanishing line and the two supporting points must be the ICPs. Suppose oc is the image of the circle center and l∞ is the vanishing line, then there is a pole-polar relationship between the center image oc and the vanishing line with respect to the conic. λl∞ = ωc oc

(9)

where λ is a scalar. Thus the true vanishing line and imaged circular points can be determined from (9).   Under perspective projection, a circle is transformed into a conic. However, the center of the circle in space usually does not project to the center of the corresponding conic in the image, since perspective projection (1) is not a linear mapping from the space to the image. Thus the imaged center of the circle can not be determined only from the contour of the imaged conic. There are several possible ways to recover the projected center of the circle by virtue of more geometrical information, such as by two or more lines passing through the center [13] or by two concentric circles [10,11]. vy Space plane

Y

Oc

Image plane

v′y

mj m′j

Ωc

y

ωc

m′i

oc

O

X

(a)

o

ωa

mi

vx

x

l∞′

l∞

v ′x

(b)

Fig. 1. Determining the ICPs from the image of one circle. (a) a circle and preassigned world frame in space; (b) the imaged conic of the circle.

Proposition 4. The imaged circular points can be recovered from the image of one circle with preassigned world coordinate system. Proof. As shown in Fig.1, suppose line x and y are the imaged two axes of the preassigned world frame, the two lines intersect l∞ and l∞ at four points. Since the two ICPs and the two orthogonal vanishing points form a harmonic relation. Thus the true ICPs can be determined by verifying the cross ratio of the

368

G. Wang, Q.M.J. Wu, and Z. Ji

two pairs of quadruple collinear points {mi , mj , vx , vy } and {mi , mj , vx , vy }. Then the camera pose can be computed according to Proposition 2.   3.2

Pose Estimation from Two Pairs of Parallel Lines

Proposition 5. The pose of the camera can be recovered from the image of any two general pairs of parallel lines in the space. Proof. As shown in Fig.2, suppose L11 , L12 and L21 , L22 are two pairs of parallel lines in the space, they may not be coplanar or orthogonal. Their images l11 , l12 and l21 , l22 intersect at v1 and v2 respectively, then v1 and v2 must be the vanishing points of the two directions, and the line connecting the two points must be the vanishing line l∞ . Thus mi and mj can be computed from the intersections of l∞ with the IAC. Suppose v1 is one direction of the world ⊥ frame, o2 is the image of the world origin. Then the vanishing point  v1 of the direction that is orthogonal to v1 can be easily computed from Cross(v1 v1⊥ ; mi mj )

v1T ωv1⊥ =0 ⊥ lT ∞ v1 =0

or

= −1, and the pose of the camera can be recovered from Proposition 1. Specifically, the angle α between the two pairs of parallel lines in v T ωa v 2 the space can be recovered from cos α = √ T 1 √ . If the two pairs of T v 1 ωa v 1

v 2 ωa v 2

lines are orthogonal with each other, then we have v1⊥ = v2 .

v1⊥ v 2

 

ωa mj

L22

l 22

L21 L11 L12

mi

l 21

l∞

v1

l 11

o2 α

o1

l 12

Fig. 2. Pose estimation from two pairs of parallel lines. Left: two pairs of parallel lines in the space; Right: the image of the parallel lines.

3.3

Projection Matrix and 3D Reconstruction

After retrieving the pose of the camera, the projection matrix with respect to the world frame can be computed from (1). With the projection matrix, any geometry primitive in the image can be back-projected into the space. For example, a point in the image is back-projected to a line, a line is back-projected to a plane and a conic is back-projected to a cone. Based on the scene constraints, many geometrical entities, such as the length ratios, angles, 3D information of some planar surfaces, can be recovered via the technique of single view metrology [3,17,18]. Therefore the 3D structure of some simple objects and scenarios can be reconstructed only from a single image.

Pose Estimation from Circle or Parallel Lines in a Single Image

4

369

Experiments with Simulated Data

During simulations, we generated a circle and two orthogonal pairs of parallel lines in the space, whose size and position in the world system are shown in Fig.3. Each line is composed of 50 evenly distributed points, the circle is composed of 100 evenly distributed points. The camera parameters were set as follows: focal length fu = fv = 1800, skew s = 0, principal point u0 = v0 = 0, rotation axis r = [0.717, −0.359, −0.598], rotation angle α = 0.84, translation vector t = [2, −2, 100]. The image resolution was set to 600 × 600 and Gaussian image noise was added on each imaged point. The generated image with 1-pixel Gaussian noise is shown in Fig.3. 15

L22 L21

10

200

Ωc

100

5

l 22

ωc

l 21

l 11

0

l 12

0 -100

-5

L11

-10

L12

-200

-10

0

10

o2 -200

0

200

Fig. 3. The synthetic scenario and image for simulation

In the experiments, the image lines and the imaged conic were fitted via least squares. We set L11 and L21 as the X and Y axes of the world frame, and recover the ICPs and camera pose according to the proposed methods. Here we only give the result of the recovered rotation matrix. For the convenience of comparison, we decomposed the rotation matrix into the rotation axis and rotation angle, we define the error of the axis as the angle between the recovered axis and the ground truth and define the error of the rotation angle as the absolute error of the recovered angle with the ground truth. We varied the noise level from 0 to 3 pixels with a step of 0.5 during the test, and took 200 independent tests at each noise level so as to obtain more statistically meaningful results. The mean and 0.1

0.05

Mean of rotation axis error

0.1

Mean of rotation angle error

0.05

STD of rotation axis error

STD of rotation angle error

0.08

Alg.1

0.04

Alg.1

0.08

Alg.1

0.04

Alg.1

0.06

Alg.2

0.03

Alg.2

0.06

Alg.2

0.03

Alg.2

0.04

0.02

0.04

0.02

0.02

0.01

0.02

0.01

0 0

1

2 Noise level

3

0 0

1

2 Noise level

3

0 0

1 2 Noise level

3

0 0

1

2

3

Noise level

Fig. 4. The mean and standard deviation of the errors of the rotation axis and rotation angle with respect to the noise levels

370

G. Wang, Q.M.J. Wu, and Z. Ji

standard deviation of the two methods are shown in Fig.4. It is clear that the accuracy of the two methods are comparable at small noise level (< 1.5 pixels), while the vanishing points based method (Alg.2) is superior to the circle based one (Alg.1) at large noise level.

5

Tests with Real Images

All images in the tests were captured by Canon Powershort G3 with a resolution of 1024 × 768. The camera was pre-calibrated via Zhang’s method [20]. Test on the tea box image: For this test, the selected world frame, two pairs of parallel lines and the two detected conics by the Hough transform are shown in Fig.5. The line segments were detected and fitted via orthogonal regression algorithm [16]. We recovered the rotation axis, rotation angle (unit: rad) and translation vector by the two methods as shown in Table 1, where the translation vector is normalized by t = 1. The results are reasonable with the imaging conditions, though we do not have the ground truth.

ω c2

ω c1

y x

Fig. 5. Test results of the tea box image. Upper: the image and the detected conics and parallel lines and world frame for pose estimation; Lower: the reconstructed tea box model at different viewpoints with texture mapping.

In order to give further evaluation of the recovered parameters, we reconstructed the 3D structure of the scene from the recovered projection matrix via the method in [17]. The result is shown from different viewpoints in Fig.5. We manually took the measurements of the tea box and the grid in the background and registered the reconstruction to the ground truth. Then we computed the relative error E1 of the side length of the grid, the relative errors E2 , E3 of the diameter and height of the circle. As listed in Table 1, we can see that the reconstruction error is very small. The results verifies the accuracy of the recovered parameters in return. Test on the book image: The image with detected conic and preassigned world frame and two pairs of parallel lines are shown in Fig.6. We recovered the

Pose Estimation from Circle or Parallel Lines in a Single Image

371

Table 1. Test results and performance evaluations for real images Images Method Alg.1 Box Alg.2 Alg.1 Book Alg.2

Raxis [-0.9746,0.1867,-0.1238] [-0.9748,0.1864,-0.1228] [-0.9173,0.3452,-0.1984] [-0.9188,0.3460,-0.1899]

Rangle 2.4385 2.4354 2.2811 2.3163

t [-0.08,0.13,0.98] [-0.08,0.13,0.98] [-0.02,0.09,0.99] [-0.02,0.09,0.99]

E1 (%) 0.219 0.284 0.372 0.306

E2 (%) 0.327 0.315 0.365 0.449

E3 (%) 0.286 0.329 0.633 0.547

pose of the camera by the proposed methods, then computed the relative errors E1 , E2 and E3 of the three side lengths of the book with respect to the ground truth taken manually. The results are shown in Table 1. The reconstructed 3D structure of the book is shown Fig.6. The results are realistic with good accuracy.

ωc y x

Fig. 6. Pose estimation and 3D reconstruction of the book image

6

Conclusion

In this paper, we proposed and proved the possibility to recover the pose of the camera from a single image of one circle or two general pairs of parallel lines. Compared with previous techniques, less conditions are required by the proposed method. Thus the results in the paper may find wide applications. Since the method utilizes the least information in computation, it is important to adopt some robust techniques to fit the conics and lines.

Acknowledgment The work is supported in part by the Canada Research Chair program and the National Natural Science Foundation of China under grant no. 60575015.

References 1. Ansar, A., Daniilidis, K.: Linear pose estimation from points or lines. IEEE Trans. Pattern Anal. Mach. Intell. 25(5), 578–589 (2003) 2. Chen, H.H.: Pose determination from line-to-plane correspondences: Existence condition and closed-form solutions. IEEE Trans. Pattern Anal. Mach. Intell. 13(6), 530–541 (1991)

372

G. Wang, Q.M.J. Wu, and Z. Ji

3. Criminisi, A., Reid, I., Zisserman, A.: Single view metrology. International Journal of Computer Vision 40(2), 123–148 (2000) 4. Dhome, M., Richetin, M., Lapreste, J.T.: Determination of the attitude of 3D objects from a single perspective view. IEEE Trans. Pattern Anal. Mach. Intell. 11(12), 1265–1278 (1989) 5. Fischler, M.A., Bolles, R.C.: Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartomated cartography. Communications of the ACM. 24(6), 381–395 (1981) 6. Gao, X.S., Tang, J.: On the probability of the number of solutions for the P4P problem. J. Math. Imaging Vis. 25(1), 79–86 (2006) 7. Hartley, R.I., Zisserman, A.: Multiple View Geometry in Computer Vision, 2nd edn. Cambridge University Press, Cambridge (2004) 8. Horaud, R., Conio, B., Leboulleux, O., Lacolle, B.: An analytic solution for the perspective 4-point problem. CVGIP 47(1), 33–44 (1989) 9. Hu, Z.Y., Wu, F.C.: A note on the number of solutions of the noncoplanar P4P problem. IEEE Trans. Pattern Anal. Mach. Intell. 24(4), 550–555 (2002) 10. Jiang, G., Quan, L.: Detection of concentric circles for camera calibration. In: Proc. of ICCV, pp. 333–340 (2005) 11. Kim, J.S., Gurdjos, P., Kweon, I.S.: Geometric and algebraic constraints of projected concentric circles and their applications to camera calibration. IEEE Transactions on Pattern Analysis and Machine Intelligence 27(4), 637–642 (2005) 12. Liu, Y., Huang, T.S., Faugeras, O.D.: Determination of camera location from 2-D to 3-D line and point correspondences. IEEE Trans. Pattern Anal. Mach. Intell. 12(1), 28–37 (1990) 13. Meng, X., Li, H., Hu, Z.: A new easy camera calibration technique based on circular points. In: Proc. of BMVC (2000) 14. Nist´er, D., Stew´enius, H.: A minimal solution to the generalised 3-point pose problem. J. Math. Imaging Vis. 27(1), 67–79 (2007) 15. Quan, L., Lan, Z.: Linear n-point camera pose determination. IEEE Trans. Pattern Anal. Mach. Intell. 21(8), 774–780 (1999) 16. Schmid, C., Zisserman, A.: Automatic line matching across views. In: Proc. of CVPR, pp. 666–671 (1997) 17. Wang, G.H., Hu, Z.Y., Wu, F.C., Tsui, H.T.: Single view metrology from scene constraints. Image Vision Comput. 23(9), 831–840 (2005) 18. Wang, G.H., Tsui, H.T., Hu, Z.Y., Wu, F.C.: Camera calibration and 3D reconstruction from a single view based on scene constraints. Image Vision Comput. 23(3), 311–323 (2005) 19. Wang, G.H., Wang, S., Gao, X., Li, Y.: Three dimensional reconstruction of structured scenes based on vanishing points. In: Proc. of PCM, pp. 935–942 (2006) 20. Zhang, Z.: A flexible new technique for camera calibration. IEEE Trans. Pattern Anal. Mach. Intell. 22(11), 1330–1334 (2000)

An Occupancy–Depth Generative Model of Multi-view Images Pau Gargallo, Peter Sturm, and Sergi Pujades INRIA Rhˆ one-Alpes and Laboratoire Jean Kuntzmann, France [email protected]

Abstract. This paper presents an occupancy based generative model of stereo and multi-view stereo images. In this model, the space is divided into empty and occupied regions. The depth of a pixel is naturally determined from the occupancy as the depth of the first occupied point in its viewing ray. The color of a pixel corresponds to the color of this 3D point. This model has two theoretical advantages. First, unlike other occupancy based models, it explicitly models the deterministic relationship between occupancy and depth and, thus, it correctly handles occlusions. Second, unlike depth based approaches, determining depth from the occupancy automatically ensures the coherence of the resulting depth maps. Experimental results computing the MAP of the model using message passing techniques are presented to show the applicability of the model.

1

Introduction

Extracting 3D information from multiple images is one of the central problems of computer vision. It has applications to photorealistic 3D reconstruction, image based rendering, tracking and robotics among others. Many successful methods exist for each application, but a satisfactory general formalism is still to be defined. In this paper we present a simple probabilistic generative model of multi-view images that accurately defines the natural relationship between the shape and color of the 3D world and the observed images. The model is constructed with the philosophy that if one is able to describe the image formation process with a model, then Bayesian inference can be used to invert the process and recover information about the model from the images. The approach yields to a generic widely applicable formalism. The price to pay for such a generality is that the results for each specific application will be poor compared to those of more specialized techniques. There are mainly two approaches to the stereo and multi-view stereo problems. In small-baseline situations, the 3D world is represented by a depth map on a reference image and computing depth is regarded as a matching problem [1]. In wide-baseline situations, it is often more convenient to represent the shape of the objects by a surface or an occupancy function [2]. Depth and occupancy are Y. Yagi et al. (Eds.): ACCV 2007, Part II, LNCS 4844, pp. 373–383, 2007. c Springer-Verlag Berlin Heidelberg 2007 

374

P. Gargallo, P. Sturm, and S. Pujades

obviously highly related, but most of the algorithms concentrate on finding one of the two. The main problem with either approach are occlusions. The fact that a 3D point is not always visible from all the cameras makes the extraction of 3D information from images hard. The two main approaches to solve this issue are to treat occluded points as outliers [3] or to explicitly model the geometrical reason for the occlusion. Making a accurate generative model for multi-view images, as we wish to do, necessarily involves modeling occlusions geometrically, because geometric occlusions really exist in the true image formation process. In fact, geometric occlusions are so important that there are reconstruction techniques, like those based on the visual hull, that use only the occlusion information [4,5]. Geometric occlusions can be modeled effectively in depth based approaches by computing a depth map for every input image [6,7,8]. This requires to add constraints, so that the multiple depth maps are coherent and form a single surface. These constrains are not necessary in shape based approaches that implicitly incorporate them because they compute a single model for all the images. Shape based approaches usually deal with geometric occlusions in an alternating way [9,10,11]. They first compute the visibility given the current estimate of the shape; and then modify the shape according to some criteria. This procedure disregards the fact that the visibility will change while modifying the shape. A voxel carving technique carving inside the object, or a shrinking surface evolution are consequences of this oversight. Our recent work [12] avoids these problems by explicitly taking into account the visibility changes during the surface evolution. This is nothing but the continuous version of the model presented in this paper. The model presented in this paper explicitly characterizes the relationship between depth and shape and profits of the benefits of both worlds. The shape occupancy automatically gives coherence to the depth maps. Properly deriving the depth maps from the occupancy implicitly encodes the geometric occlusions. Our model is very related both in objective and approach to the recent work of Hern´ andez et al. [13]. In that work depth cues, are probabilistically integrated for inferring occupancy. The principal difference between their model and the one presented here is that they make some independence assumptions that we do not. In particular, they assume the depth of different pixels to be independent. This greatly simplifies the inference and good results are achieved. In our model, as in the real world, depth is determined by the occupancy and therefore the depth of different pixels are not independent. This create a huge very loopy factor graph representation of the join probability of occupancy and depth. Inference in such a graph is hard as we will see in the experiments.

2

The Model

This section presents the occupancy-depth model. We first introduce the random variables involved in the generative process. Then we decompose their joint probability distribution into simpler terms and give a form to each of them.

An Occupancy–Depth Generative Model of Multi-view Images

2.1

375

Occupancy, Depth and Color Variables

Consider a discretization of the 3D space in a finite set of sites S ⊂ R3 . A given site x ∈ S, can be in the free space or inside an object. This defines the occupancy of the site that will be represented by a binary random variable ux (1 meaning occupied and 0 meaning free). The occupancy of the whole space will be represented by the random process u : S → {0, 1}, which defines the shape of the objects in the space. The shape of the objects is not enough to generate images. Their appearance is also needed. In the simplest case, under the constant brightness assumption, the appearance can be represented by a color at each point on the surface of the objects. As we do not know the shape of the objects right now, we will need to define the color of all sites in S, even if only the color of the sites lying on the surface is relevant. The color will be represented by a random process C : S → R3 . The depth of a pixel is defined as the distance (measured along the camera’s z-axis) between the optical center of the camera and the 3D point observed at the pixel. Given the occupancy of the space and the position and calibration of a camera i, the depth Dpi of a pixel p is determined as the depth of the first occupied point in its viewing ray. The observed color at that pixel will be denoted by Ipi . This color should ideally correspond to the color of the site observed at that pixel. i.e. the point of the viewing ray of p which is at depth Dpi . 2.2

Decomposition

Having defined all the variables depths, colors, occupancies and observed colors – we will now define their joint probability distribution. To do so, we first decompose the distribution terms representing the natural dependence between the variables. One can think of this step as defining the way the data (the images) were generated. The proposed decomposition is   p(Dpi |u) p(Ipi |Dpi , C) . (1) p(u, C, D, I) = p(u)p(C|u) i,p

i,p

Fig. 1. Bayesian network representation of the joint probability decomposition

376

P. Gargallo, P. Sturm, and S. Pujades

It is represented graphically in Figure 1. Each term of the decomposition corresponds to a variable and, thus, to a node of the network. The arrows represent the statistical dependencies between the variables. In other words, the order that one has to follow to generate random samples of the model from scratch. Therefore, the data generation process is as follows. First one builds the objects of the world by generating an occupancy function. Then one paints them by choosing the space colors. Finally, one takes pictures of the generated world: one first determines which points are visible from the camera by computing the depth of the pixels, and then sets the color of the pixels to be the color of the observed 3D points. In the following sections, we define each of the terms of the decomposition (1). 2.3

World Priors

Not all the possible occupancy functions are equally likely a priori. One expects the occupied points to be gathered together forming objects, rather than randomly scattered over the 3D space. To represent such a belief, we choose the occupancy u to follow a Markov Random Field distribution. This gives the following prior,    ψ(ux , uy ) (2) p(u) ∝ exp − x,y

where the sum extends to all the neighboring points (x, y) in a grid discretization S of the 3D space. The energy potentials are of the form ψ(ux , uy ) = α|ux − uy |, so that they penalize neighboring sites of different occupancies by a cost α. This prior is isotropic in the sense that two neighboring points are equally likely to have the same occupancy regardless of their position and color. From experience, we know that the discontinuities or edges in images often correspond to discontinuities in the occupancy (the occluding contours). Therefore, one could be tempted to use the input images to derive a smoothing prior for the occupancy that is weaker at the points projecting to image discontinuities. While effective, this would not be correct from a Bayesian point of view, as one would be using the data to derive a prior for the model. We will now see how to obtain this anisotropic smoothing effect in a more theoretically well funded way. In the proposed decomposition, the prior on the color of the space depends on the occupancy. This makes it possible to express the following idea. Two neighboring points that are both either occupied or free, are likely to have similar colors. The colors of two points with different occupancies are not necessarily related. This can be expressed by the MRF distribution    φ(Cx , Cy , ux , uy ) (3) p(C|u) ∝ exp − x,y

with

 φ(Cx , Cy , ux , uy ) =

(Cx − Cy ) if ux = uy 0 otherwise

(4)

An Occupancy–Depth Generative Model of Multi-view Images

377

where  is some robust penalty function, that penalize the difference of colors of neighboring points with the same occupancy. Now, combining the prior on the occupancy with the prior on the color we have    ¯ x , Cy , u x , u y ) (5) ψ(C p(u, C) ∝ exp − x,y



with ¯ x , Cy , u x , u y ) = ψ(C

(Cx − Cy ) if ux = uy α otherwise

(6)

If we are given the color of the space, then p(u|C) ∝ p(u, C) is a color driven smoothing prior on the occupancy. Neighboring points with the same color are more likely to have the same occupancy than neighboring points with different colors. As the color will be estimated from the images, the color discontinuities will coincide with the edges in the images. Thus, this term will represent our experience based knowledge that object borders coincide with image edges. 2.4

Pixel Likelihood

The color Ipi observed at a pixel should be equal to the color of the 3D point visible at that pixel, up to the sensor noise and other unmodeled effects, e.g. specularities. If we denote the color of the observed 3D point as C(Dpi ), we have    p(Ipi |Dpi , C) ∝ exp −ρ Ipi − C(Dpi )

(7)

where ρ is some robust penalty function.

Fig. 2. Bayesian network of the model associated to a single viewing ray. The occupancy variables ui along the viewing ray determine de depth d. The color of the image I corresponds to the color C at depth d.

Note that unlike traditional stereo algorithms, here there are no occlusions to be taken into account by the function ρ. We are matching the color of a pixel with the color of the observed scene point, not with the color of pixels in the other images. The observed scene point is, by definition, non-occluded, so no occlusion problem appears here.

378

P. Gargallo, P. Sturm, and S. Pujades

Depth Marginalization. The likelihood (7) of a pixel depends only on its depth and not on the whole occupancy function. However, the relationship between occupancy and depths is simple and deterministic. Therefore, it is easy to marginalize out the depth and express the likelihood directly in terms of the occupancy of the points on the viewing ray of the pixel. To simplify the notation we will do the computations for a single pixel. Figure2 shows the Bayesian network associated to a single pixel. The points of its viewing ray will be denoted by the natural numbers {0, 1, 2, · · · }, ordered by increasing distance to the camera. Their occupancy is a vector u such that ui is the occupancy of the i-th point in the viewing ray. The depth will be denoted by d. With this language, the probability of a depth given the occupancy is  (1 − ui )ud (8) p(d|u) = i
This equation encodes the fact that if d is the depth of a pixel, then occupancy of the points with lower depths must be zero and the occupancy of the point at depth d must be one. It is easy to check that p(d|u) is 1 only when d is the actual depth determined by u and 0 otherwise. Now, if we note the likelihood of the pixel having a depth d by L(d) = p(Ipi |Dpi = d, C) and the likelihood of a pixel given the occupancy u by L(u), then    p(I|d, C)p(d|u) = L(d) (1 − ui )ud . (9) L(u) = p(I|u, C) = d

d

i
Note that the summand is null for all depths d except for the one that corresponds to the occupancy u.

3

Inference

The last section presented the generative occupancy–depth model by defining the joint probability of occupancy, color, depth and image pixels. In this section we will present an algorithm for inverting the process and recover occupancy from multiple images. Given a set of observed images {I i } the goal is to find the posterior probability of occupancy and color, p(u, C|I). In a tracking application, for example, one may be interested in computing the occupancy marginals at each point in the 3D space. This can be used as input for a volumetric 3D tracker. Alternatively, in a novel view synthesis application one may be more interested in finding the most probable world in order to render it from other points of view. As we will see these are difficult tasks, challenging the most recent inference algorithms. The main problem is the interdependence of the occupancies of the points in a viewing ray, which creates high order cliques, in addition to the extrem loopiness of the network. We present here a first try of solving the inference problem by using EM and message passing.

An Occupancy–Depth Generative Model of Multi-view Images

379

The optimization is done by alternating between the optimization of the occupancy and the color. In the E-step, the probabilities of the occupancies are computed using message passing. Depending on the goal, the sum-product or the max-product algorithm is used. In the M-step, the color estimation is improved by maximizing its expected log-posterior. It is interesting to note that simpler optimization techniques like iterative conditional modes will lead to algorithms strongly similar to voxel carving or surface evolution. In this case, one will start with an initial guess of the occupancy and will try to change the occupancy of the voxels at the border of the objects in order to improve the posterior. The message passing algorithm presented below is smarter in the sense that it will make decisions about the occupancy of a whole viewing ray at a time. 3.1

Factor Graph Representation

In the E-step, for a fixed coloring of the space, C, the posterior probability p(u|I, C) must be computed (or maximized). This distribution can be represented as a factor graph. Again, for notational simplicity, we detail here the factor graph corresponding to a single pixel,  ¯ i , uj )) . exp(−ψ(u (10) p(u|I, C) ∝ L(u) ij

Figure 3 shows the factor graph for a single pixel and sketches the one corresponding to a pair of pixels in different images.

Fig. 3. Factor graph representation of the distribution for a single pixel (left) and for two pixel with intersecting viewing rays (right)

The graph contains two types of factors. The likelihoods of the pixels L are huge factors connecting all the points in the viewing ray of each pixel. The ¯ are standard pairwise factors. Notice the extreme smoothing potentials exp(−ψ) loopiness of the complete graph where the viewing rays of different pixels intersect to each other. 3.2

Message Passing

Inference will be done by message passing in the factor graph. Messages will be sent from the likelihood and smoothing factors to the occupancy variables and

380

P. Gargallo, P. Sturm, and S. Pujades

vice-versa. One can visualize this process as a dialog between the images and the occupancy. Each image tells the occupancy what to be. The occupancy gathers all the messages and replies to each image with a summary of the messages sent by the other images. The process continues until a global agreement is found. We derive here the reweighted message passing equations [14,15] for the occupancy factor graph. The equations are presented for the sum-product algorithm, but the max-product equivalents can be obtained easily by replacing all the sums in the equations by maximums. Using the notation of Minka [14], the posterior probability of the occupancy p(u|I, C) will be approximated by a fully factorized variational distribution q(u) = i qi (ui ). The belief qi of an occupancy variable ui is the product of the messages that it receives. That is 

qi (ui ) =

ma→i (ui ) ,

(11)

a∈N (i)

where N (i) are the indices of the factors connected to ui . The message that a factor fa sends to a variable is given by ma→i (ui )αa =





fa (ua )αa

ua \ui

nj→a (uj ) ,

(12)

j∈N (a)\i

where ua is the set of variables connected to the factor fa and N (a) their indices. αa is the weight of the factor and it is a parameter of the message passing algorithm. Finally the replying message from a variable to a factor is ni→a (ui ) = qi (ui )ma→i (ui )−αa .

(13)

The pair-wise smoothing potentials are simple and a direct implementation of these formulas is straightforward. The likelihood factors, however, need more attention. These factors link a lot of variables. A direct implementation of the equations above will involve computing sums over all the possible configurations of occupancy along each viewing ray. The number of configurations grows exponentially with the number of sites and becomes quickly intractable. Luckily, the likelihood factor L is simple enough that the sum can be simplified analytically. The message that a pixel likelihood factor sends to the occupancy of the grid point i is mL→ui (x)αL =



L(u)αL

u\ui



nj→L (uj ) .

(14)

j=i

Substituting L(u) from equation (9) we get  d u\ui

L(d)αL

 j
(1 − uj )ud

 j=i

nj→L (uj ) .

(15)

An Occupancy–Depth Generative Model of Multi-view Images

381

And now we can split and simplify the sum over d into 3 sums with d smaller, equal and bigger than i respectivelly,      L(d)αL nj→L (0) nd→L (1) nj→L (uj ) d
+

j
αL

ui L(i)



nj→L (0)

j
+

(1 − ui )



 

d>i

 nj→L (uj )

uj

uj

j>d



L(d)αL

j>d∧j=i

nj→L (0) nd→L (1)

j
(16)   j>d

 nj→L (uj ) .

uj

Finally, defining τ (d) = L(d)αL



nuj →L (0)nud →L (1)

j
  j>d

 nuj →L (uj ) ,

(17)

uj

we have 

1

mL→ui (ui )αL = ui

nui →L (ui )

τ (d) +

d
ui nui →L (1)

τ (i) +

1 − ui  τ (d), nui →L (0) d>i

(18) which can be computed in linear time. 3.3

Color Estimation

In the M-step the color of the space is computed by maximizing the expected log-posterior, ln p(u, C|I)q = ln p(I|u, C)q + ln p(u, C)q + const.

(19)

where ·q denotes the expectation with respect to u assuming that it follows the variational distribution q. Again, the expectation is a sum over all possible occupancy configurations. Simplifications similar to the ones done above yield to    qi (0) qd (1) ρ I − C(d) (20) ln p(I|u, C)q = − d i
and ln p(u, C)q = −

   qi (0) qj (0) + qi (1) qj (1) (Ci − Cj ) ij

   + qi (0) qj (1) + qi (1) qj (0) α .

(21)

The optimization is done by a gradient descent procedure. The initialization of the whole EM procedure is as follows. The messages are all initialized to uniform distributions. The color is initialized at each site by computing the mean of the color of its projections to the input images.

382

P. Gargallo, P. Sturm, and S. Pujades

Fig. 4. Results for the single pixel (top-left), the multi-view (top-right) and stereo pair (bottom) experiments

4

Experimental Validation

In order to validate the generative model and to test the possibility of inverting the process using the inference method described above, we performed three types of experiments. The first experiment was done to test the performance of the reweighted message passing algorithm. A single pixel camera model was used and different likelihood function L(d) were manually chosen. The ground truth posterior of the occupancy was computed by sampling. First image of Figure 4 shows an example result. In general, both the sum-product and the max-product performed poorly, only roughtly approximating the accual marginals and MAPs. Although this poor performance, we tested the algorithm with real images with the hope that the interconnections between different viewing rays will constrain the problem and possibly get better results. Last row of Figure 4 shows the results obtained for the four Middlebury stereo pairs [1]. The displayed depthmaps were computed from the inferred occupancy of frontoparallel layers placed in front of the cameras. The results are generally correct, but large errors are present, specially in the textureless regions. Holes appear in these regions. A possible explanation for these holes is that, as we are smoothing the occupancy of the layers (which are mostly empty) the likelihood term in the textureless regions is too weak to prevent the smoothing from erasing any occupancy on the layer. The last test was done with wide angle multi-view stereo data, to test the exact same algorithm in a very different setup. Figure 4 shows preliminary results of the inference for the dataset dinoSparseRing [2]. The shape of the dinosaur is globally recovered, but here also there are numerous holes in the object. We did not perform a numerical evaluation of the results because the scores will be dominated by the errors described above and, therefore, uninteresting.

An Occupancy–Depth Generative Model of Multi-view Images

5

383

Conclusion

We presented a generative model of multi-view images, developped an inference algoritm to compute occupancy from images and test it experimentally. The results of the experiments are not comparable to the ones obtained by the stateof-the-art techniques, but served well to validate the model. Future work is needed for finding better inference methods and to determine to which degree the errors are due to the inference method or to the generative model itself.

References 1. Scharstein, D., Szeliski, R.: A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. Int. J. Comput. Vision 47(1-3), 7–42 (2002) 2. Seitz, S.M., Curless, B., Diebel, J., Scharstein, D., Szeliski, R.: A comparison and evaluation of multi-view stereo reconstruction algorithms. In: Computer Vision and Pattern Recognition 2006, Washington, DC, USA, pp. 519–528 (2006) 3. Strecha, C., Fransens, R., Gool, L.V.: Combined depth and outlier estimation in multi-view stereo. In: Computer Vision and Pattern Recognition 2006, Washington, pp. 2394–2401. IEEE Computer Society Press, Los Alamitos (2006) 4. Laurentini, A.: The visual hull concept for silhouette-based image understanding. IEEE Trans. Pattern Anal. Mach. Intell. 16(2), 150–162 (1994) 5. Solem, J.E., Kahl, F., Heyden, A.: Visibility constrained surface evolution. In: Computer Vision and Pattern Recognition 2005, San Diego, USA, pp. 892–899 (2005) 6. Kang, S.B., Szeliski, R., Chai, J.: Handling occlusions in dense multi-view stereo. In: Computer Vision and Pattern Recognition 2001, Kauai, Hawaii, pp. 103–110 (2001) 7. Kolmogorov, V., Zabih, R.: Multi-camera scene reconstruction via graph cuts. In: European Conference on Computer Vision 2002, London, UK, pp. 82–96 (2002) 8. Gargallo, P., Sturm, P.: Bayesian 3d modeling from images using multiple depth maps. In: Computer Vision and Pattern Recognition 2005, San Diego, vol. 2, pp. 885–891 (2005) 9. Faugeras, O.D., Keriven, R.: Complete dense stereovision using level set methods. In: Burkhardt, H., Neumann, B. (eds.) ECCV 1998. LNCS, vol. 1407, pp. 379–393. Springer, Heidelberg (1998) 10. Kutulakos, K.N., Seitz, S.M.: A theory of shape by space carving. Int. J. Comput. Vision 38(3), 199–218 (2000) 11. Paris, S., Sillion, F.X., Quan, L.: A surface reconstruction method using global graph cut optimization. Int. J. Comput. Vision 66(2), 141–161 (2006) 12. Gargallo, P., Prados, E., Sturm, P.: Minimizing the reprojection error in surface reconstruction from images. In: Proceedings of the International Conference on Computer Vision, Rio de Janeiro, Brazil, IEEE Computer Society Press, Los Alamitos (2007) 13. Hern´ andez, C., Vogiatzis, G., Cipolla, R.: Probabilistic visibility for multi-view stereo. In: Computer Vision and Pattern Recognition 2007, Minneapolis (2007) 14. Minka, T.: Divergence measures and message passing. Technical report, Microsoft Research (2005) 15. Wainwright, M., Jaakkola, T., Willsky, A.: Map estimation via agreement on trees: message-passing and linear programming. Information Theory, IEEE Transactions on 51, 3697–3717 (2005)

Image Correspondence from Motion Subspace Constraint and Epipolar Constraint Shigeki Sugimoto1 , Hidekazu Takahashi2 , and Masatoshi Okutomoi1 Department of Mechanical and Control Engineering, Graduate School of Sceience and Engineering, Tokyo Institute of Technology. 2–12–1 O-okayama, Meguro-ku, Tokyo, 152–8550 Japan {shige,hidekazu,mxo}@ok.ctrl.titech.ac.jp 2 Hidekazu Takahashi is currently with Diesel Inj. Eng. Dept. 1, DENSO Corporation. 1–1 Showa-cho, Kariya-shi, Aichi, 448-8661 Japan hidekazu [email protected] 1

Abstract. In this paper, we propose a novel method for inferring image correspondences on the pair of synchronized image sequences. In the proposed method, after tracking the feature points in each image sequence over several frames, we solve the image corresponding problem from two types of geometrical constraints: (1) the motion subspace obtained from the tracked feature points of a target sequence, and (2) the epipolar constraints between the two cameras. Dissimilarly to the conventional correspondence estimation based on image matching using pixel values, the proposed approach enables us to obtain the correspondences even though the feature points, that can be seen from one camera view, but can not be seen (occluded or outside of the view) from the other camera. The validity of our method is demonstrated through the experiments using synthetic and real images.

1

Introduction

Image correspondence estimation is fundamental to various vision applications, such as object/scene recognition, 3D structure/motion from sequential images, and stereo vision. In computer vision literatures, in general, correspondences are obtained from pixel values: each correspondence is estimated by evaluating the similarity measures computed from image pixel values [1,2,3,4]. The pixel-based approach is usually effective and efficient for tracking feature points in an image sequence, since the neighboring image frames in the sequence are similar each other, and there is always a high similarity on every pair of corresponding points. On the other hand, in the case of wide-baseline views, the pixel-based approach becomes inefficient or fails, even though the epipolar geometry between two images are known, because of (i) a small overlapped view (or no overlapped view), (ii) occlusions, or (iii) dissimilarities on the pairs of corresponding points. Especially in the former two cases, in principle, the pixelbased approach cannot obtain the correspondences on scene points visible in one view but invisible in the other. Y. Yagi et al. (Eds.): ACCV 2007, Part II, LNCS 4844, pp. 384–393, 2007. c Springer-Verlag Berlin Heidelberg 2007 

Image Correspondence from Motion Subspace Constraint

385

F Camera

Camera

Moving object

Another sequence

Target sequence Correspondence

Unknown Unknown

Tracked feature point

Correspondence

Unknown

Correspondence

Fig. 1. Correspondence between two wide-baseline images. We consider the image correspondences as a track correspondence, while we don’t care whether the track in the target sequence is visible or not.

Our goal in this paper is to obtain image correspondences on each pair of wide-baseline synchronized image sequences, as illustrated by Fig. 1. Based on the facility of the pixel-based approach for tracking feature points in an image sequence, we consider the image correspondence problem as a track correspondence problem; we infer an unknown feature track (image points in all frames) in a target image sequence, which corresponds to one of the tracks of visible feature points in the other sequence, while we don’t care whether the unknown track in the target sequence is visible or not. Our basic idea for solving this problem is as follows. Under the assumption that distinct feature points in both sequences are tracked separately over several frames, and that all feature points belong to one rigid object, the unknown track in the target sequence should be constrained to the motion subspace [5] derived from the collection of visible tracks in the target sequence. Also, under the assumption that the two wide-baseline camera views are weakly calibrated, and that the fundamental matrix is consistent while the images are captured, every image point of the unknown track should be constrained to the fundamental equation with a visible point in the other sequence. In this paper, we show that the correspondence problem can be algebraically solved from the equations formed by the two types of constraints mentioned above. In addition, we present an optional way for refining the correspondences via 3D reconstruction provided that the two views are strongly calibrated (both intrinsic and extrinsic parameters are known). The validity of the proposed method is demonstrated through the experiments using synthetic and real images.

386

2

S. Sugimoto, H. Takahashi, and M. Okutomoi

Related Work

The motion subspace theorem is a fundamental aspect in factorization methods [6,7] for structure from motion (SFM), and widely used in motion segmentation techniques [5,8,9]. Our proposed method presents another use of the theorem. There have been a few papers [10,11,12] which present solutions for similar correspondence problems. If the two views are strongly calibrated, feature tracks between two neighboring images give the structure and the motion of the scene [13] without a scale factor. In [10], the scale factor is recovered from the known extrinsic parameters of the two camera views. In this case, the image correspondences can be estimated from 3D reconstruction results. Instead of recovering the scale factor, Dornaika et al. [11] show that each correspondence point in the target sequence is given as the cross point of the two epipolar lines, derived from the motion between two neighboring images and the known extrinsic parameters of the two views. However, the estimation of the motion between two neighboring frames is very sensitive to tracking errors [14], and roundabout for obtaining the correspondences compared with our method. On the other hand, Ho et al. [12,15] show that the point collection of all tracks of the feature points in both two sequences belong to a four dimensional subspace. The correspondences between the two views are inferred from the basis of the subspace. However, for computing the basis, it is required to obtain more than four track correspondences between the two views. Our method has the advantage that the unknown correspondences are obtained even in an extreme case where no correspondence is given beforehand, such that one view sees one side of the object while the other sees another side.

3

Image Correspondence from Two Geometric Constraints

We track moving feature points over K images of two synchronized image sequences separately. Let M and N be the number of the tracked feature points in a target and the other sequence, respectively. Let ukm = (ukm , vkm )T be the image coordinates of the m-th feature point in the k-th image of the target  )T be the image coordinates of the n-th feature sequence, and ukn = (ukn , vkn point in the k-th image of the other. Our goal is to infer unknown image coordi T ) , (k = 1, · · · , K), (n = 1, · · · , N ) nates in the target sequence, ukn = (ukn , vkn  each of which corresponds to ukn . 3.1

Motion Subspace Constraint

We collect the image coordinates of the m-th feature point in the target sequence into a single vector: T

pm = [u1m , v1m , u2m , v2m , . . . , uKm , vKm ] .

(1)

Image Correspondence from Motion Subspace Constraint

387

Then the 2K dimensional vectors of all feature points are collected into a single matrix: W = [p1 , p2 , . . . , pM ]T .

(2)

Subtracting a mean vector p¯G from each row vector gives: ¯ = [p¯1 , p¯2 , . . . , p¯M ]T , W where p¯m = pm − pG , pG =

(3) 1 M

M 

pm .

(4)

m=1

¯W ¯ T , and {e1 , · · · , e2K } Let λ1 ≤ · · · ≤ λ2K be the eigenvalues of M = W be the orthonormal basis of the corresponding eigenvectors. If affine camera projection is assumed, pm should belong to a three dimensional affine subspace, so-called a ’motion-subspace’ [8]: pm = pG + α1m e1 + α2m e2 + α3m e3 ,

(5)

where α1m , α2m , and α3m denote the coefficients of the basis vectors. In general, camera projection can not be expressed by affine projection. For more general camera projection, we exploit a higher dimensional motion subspace: pm = pG + α1m e1 + α2m e2 + · · · + ανm eν , where (3 ≤ ν ≤ 2M ).

(6)

If all feature points in both sequences belong to a single moving object, the 2K dimensional vector of an unknown track,    pn = [u1n , v1n , u2n , v2n , . . . , uKn , vKn ] , T

(7)

which corresponds to one of the feature tracks in the other sequence, should belong to the motion subspace spanned by {pG , e1 , e2 , · · · , eν }. We refer to these constraints as the motion subspace constraints. 3.2

Inferring Wide-Baseline Correspondence

We can write the motion subspace constraints into the form: pn = pG + β1n e1 + β2n e2 + · · · + βνn eν ,

(8)

where the basis of the subspace, {pG , e1 , e2 , . . . , eν }, is given by the tracks of visible feature points in the target sequence. Consequently, unknowns are β1n , β2n , · · · , βνn . On the other hand, image corresponding points in the wide-baseline views should hold the epipolar constraints between the two views. Therefore we can write: ⎡  ⎤ ⎡ ⎤ ukn f11 f12 f13      ⎦ 1 F ⎣ vkn ukn vkn = 0, where F = ⎣ f21 f22 f23 ⎦ . (9) 1 f31 f32 f33

388

S. Sugimoto, H. Takahashi, and M. Okutomoi

Therein F denotes the fundamental matrix of the two views. We extract the k-th frame from (8). 

ukn (k) (k) (k) = pG + β1n e1 + β2n e2 + · · · + βνn e(k)  ν vkn (k)

(k)

(10)

(k)

where pG , e1 , · · · , eν denote the 2×1 vectors composed of the (2k − 1)-th and 2k-th row elements of pG , e1 , · · · , eν , respectively. Substituting (10) for (9) gives the equation of the epipolar constraints with respect to the n-th feature point in the k-th image frame, as follows: ⎧ ⎫ ⎡ ⎤ 1 ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎨  ⎬  ⎢ β1n ⎥ ⎢ ⎥ (k) (k) (k) T ˜ kn f1 pG e1 · · · eν u (11) ⎢ .. ⎥ + f2 = 0, ⎪ ⎪ ⎣ . ⎦ ⎪ ⎪ ⎪ ⎪ ⎩ ⎭ βνn where

⎤ f11 f12 f1 = ⎣ f21 f22 ⎦ , f31 f32

⎤ f13 f2 = ⎣ f23 ⎦ . f33

⎡





  ˜ T u kn = ukn vkn 1 ,

⎡

Then we collect Eq. (11) of all k, (k = 1, · · · , K) as follows: ⎧ ⎫ ⎡ ⎤ 1 ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎨  ⎬   ⎢ β1n ⎥  ⎢ ⎥ T F1 pG e1 · · · eν ⎢ . ⎥ + F2 = 0 , Mn ⎪ ⎪ ⎣ .. ⎦ ⎪ ⎪ ⎪ ⎪ ⎩ ⎭ βνn where

⎡

⎢ ⎢ MnT = ⎢ ⎣

˜ T u 1n

0

˜ T u 2n ..

.

⎤

⎡

⎦

⎢ ⎢ F1 = ⎢ ⎣

0 ⎥⎥⎥ , ˜ T u Kn

⎤

f1 f1

0

..

.

0 ⎥⎥⎥ , ⎦

f1

(12)

⎤ f2 ⎢ f2 ⎥ ⎢ ⎥ F2 = ⎢ . ⎥ . ⎣ .. ⎦ ⎡

f2

Finally we obtain: An xn = bn , where

⎤ β1n ⎥ ⎢ xn = ⎣ ... ⎦ ,

(13)

⎡





An = MnT F1 e1 · · · eν ,

bn = −MnT F1 pG − MnT F2 .

βνn Note that Eq.(13) has K equations with ν unknowns, β1n , · · · , βνn . Therefore we can compute the unknowns if we choose a number for ν from 3 ∼ K. In consequence, we can obtain all track correspondences between the two widebaseline views by solving Eq.(13) for all n.

Image Correspondence from Motion Subspace Constraint

4

389

Correspondence Refinement Via 3D Reconstruction

In this section, we present an optional way for the refinement of the correspondences via 3D reconstruction provided that the two views are strongly calibrated (both intrinsic and extrinsic parameters are known). Once the target tracks that correspond to the visible tracks in the other sequence are obtained, the intrinsic and extrinsic parameters of the two views provide the 3D position of every point at every frame. Let Xkn = (xkn , ykn , zkn )T be the world coordinates of the n-th feature point in the k-th frame. We can write [8]: ⎤ ⎡ xkn (14) Xkn = ⎣ ykn ⎦ = τk + an ik + bn jk + cn kk , zkn where τk , {ik , jk , kk }, respectively, denote the object origin and orthonormal basis in the k-th image frame, and {an , bn , cn } denotes the object coordinates of the n-th point. Collecting Xkn for all k, (k = 1, · · · , K) gives:   T X2n · · · XKn = τ + an i + bn j + cn k, Xn = X1n (15) where T T τ = (τ1T , · · · , τK ) ,

j=

T T (j1T , · · · , jK ) ,

i = (iT1 , · · · , iTK )T , T T k = (k1T , · · · , kK ) .

Therefore the collection of (15) for (n = 1, · · · , N ),   X = X1 X2 · · · XN ,

(16)

(17)

should belong to a three dimensional affine subspace. Subtracting a mean vector XG from each row vector gives:   ¯1 X ¯2 · · · X ¯N , ¯ = X X (18) N  ¯ n = Xn − XG , XG = 1 where X Xn . N n=1

(19)

This means that the 3D positions, estimated by the track correspondences, can ¯ to a three dimensional subspace. Conbe refined by fitting the collection X sequently, we can refine the correspondences by re-projecting the refined 3D positions into the images. Additionally, it is possible to determine the number of basis, ν, by evaluating the residuals (the sums of the squared distances of the data points to the fitted subspaces) resulted from the 3D position fittings with different ν-numbers. In our experiments, we study the effect of the difference of ν to the fitting residual.

390

S. Sugimoto, H. Takahashi, and M. Okutomoi

0.3 meters Right camera 800

Object

0

5400

X[mm] 120 degrees Z

800

400

0

-400

Y[mm]

5000 4800

5 meters

4600

Y Left camera

X

Fig. 2. Camera and object configuration in simulation

5

-800 1200

5200

Motion

4400

1st frame 50th frame 100th frame trajectory

Z[mm]

Fig. 3. Sphere motion (in millimeter)

Experimental Result

In this section, we show the validity of the proposed method using synthetic and real images. 5.1

Simulation

Our setup for two cameras and an object is shown in Fig. 2, where the view angle of each camera was 7.8 degrees, and the focus length was 35 millimeters. The object was a sphere with 1.5 meter radius. Fig. 3 shows the object motion in our synthetic image sequences. We arranged 200 feature points on the sphere, and recorded the tracks of 71 and 77 feature points over 100 frames in the left and right sequences, respectively. We added zero-mean Gaussian noise of standard deviation σ (in pixels) on the image positions of the feature tracks. Fig. 4 shows the results of the proposed method described in Sec. 3, when ν = 3, σ = 1, and the left and right sequences were, respectively, the target and the other. The red points ’+’ denote the visible points in the 50-th frame of the left image, and the green points ’×’ denote the corresponding points inferred from the visible points in the right frame. We can easily see that the object shape is a sphere, and that some inferred points correspond to the points visible in the left sequence. Fig. 5 shows the correspondence precision (in pixels) with respect to σ when the left image sequence was the target. If σ = 0, the proposed method gains no tracking error. Otherwise, our method reduces tracking errors on visible points, since RMSE in the correspondences is smaller than σ. We also studied the relationship of the numbers of ν and an affine camera effect, which denotes what extent the camera model approximates an affine model to. We tried ν for 3 ∼ K, and selected the number that takes the minimal fitting residual on the 3D-reconstruction subspace (see Sec. 4). The affine camera effect was evaluated by the value computed by dividing the mean depth of the

Image Correspondence from Motion Subspace Constraint

391

Left Image Right Image

Fig. 4. Inferred correspondences

7

Left sequence Mean of determined base number

RMS error of estimated feature position [pix]

1.2 1 0.8 0.6 0.4 0.2 0

0

0.5

1

1.5

2

2.5

Standard dev. of additive tracking error [pix]

Fig. 5. Image correspondence precision (RMSE) with respect to standard deviation σ of additional tracking errors (ν = 3)

3

Std. dev. of additive tracking error

σ =0.2 σ =0.6 σ =1.0 σ =1.4 σ =1.8 σ =2.0

6 5 4 3 2 1 0 4

6

8

10

12

14

16

18

20

22

24

(Depth of object center)/(Object thikness in depth direction)

Fig. 6. Selected basis number: For each setup, we performed the ν-selection for 20 times for each σ, and plotted its mean

scene points by the thickness of the object. We varied the value by changing the camera-object setup while the motions of the feature points in the image sequences are as similar as possible. For each setup, we tried selecting ν 20 times for each tracking noise level σ, and we plotted the mean of ν that take the minimal residuals. This experimental result is shown in 6. If the affine camera effect is smaller (i.e. the camera model is more approximated to the perspective projection model), or the additive tracking noise is smaller, a larger number of ν is selected. In a large tracking noise level, the position errors of the feature points are represented by a higher dimensional part of the motion subspace. This makes the fitting residual on the 3D-reconstruction subspace become larger, if ν is a large number. Considered that tracking errors in real image sequences are relatively large, we set ν = 3 for general cases.

392

S. Sugimoto, H. Takahashi, and M. Okutomoi

1st camera is reference 2nd camera is reference

marker

Fig. 7. Target object

Fig. 8. 3D reconstruction result in wire frame view

Fig. 9. 3D reconstruction result: The side, front, and top views are shown in the top row. Actual shapes in the similar views are shown in the bottom row.

5.2

Real Images

In our experiments using real images, we used a toy car with 83 markers as shown in Fig.7. Two wide-baseline views observed the object from the left and the right. We tracked 42 markers in the left view, and 43 markers in the right view, over 100 frames in each sequence, where only two markers were tracked in both views. Fig. 8 shows the 3D reconstruction results by switching the role of the sequences each other, where ν = 3. The top row of Fig. 9 shows the three views (side, front, top) of the reconstruction results, while the bottom row shows the corresponding actual views of the object.

6

Conclusions

In this paper, we have proposed a novel method for inferring image correspondences on a pair of synchronized image sequences. We have solved a wide-baseline

Image Correspondence from Motion Subspace Constraint

393

image corresponding problem using motion subspace constraints and epipolar constraints. The validity of the proposed method has been demonstrated through the experiments using synthetic and real images. The proposed method may benefit from subspace fitting techniques, such as RANSAC [16] and GPCA [17], for fitting tracks in a single sequence to multiple subspaces. We will incorporate with these robust methods in the future work.

References 1. Lucas, B.D., Kanade, T.: An iterative image registration technique with an approach to stereo vision. In: Image Understanding Workshop, pp. 121–130 (1981) 2. Freeman, W., Adelson, E.: The design and use of steerable flters. PAMI 13(9), 891–906 (1991) 3. Lazebnik, S., Schmid, C., Ponce, J.: Sparse texture representation using affineinvariant neighborhoods. In: CVPR, pp. 319–324 (2003) 4. Lowe, D.G.: Distinctive image features from scale-invariant keypoints. IJCV 60(2), 91–110 (2004) 5. Kanatani, K.: Motion segmentation by subspace separation and model selection. In: ICCV, vol. 2, pp. 301–306 (2001) 6. Tomasi, C., Kanade, T.: Shape and motion from image streams under orthography: A factorization method. IJCV 9(2), 137–154 (1992) 7. Poelman, C.J., Kanade, T.: A paraperspective factorization method for shape and motion recovery. PAMI 19(3), 206–218 (1997) 8. Kanatani, K.: Motion segmentation by subspace separation: Model selection and reliability evaluation. IJIG 2(2), 179–197 (2002) 9. Costeria, J.P., Kanade, T.: A multibody factorization method for independently moving objects. IJCV 29(3), 159–179 (1998) 10. Weng, J., Huang, T.S.: Complete structure and motion from two monochular sequences without stereo correspondence. In: ICPR, pp. 651–654 (1992) 11. Dornaika, F., Chung, R.: Stereo correspondence from motion correspondence. CVPR, 70–75 (1999) 12. Ho, P.K., Chung, R.: Stereo-motion with stereo and motion in complement. PAMI 22(2), 215–220 (2000) 13. Faugeras, O.D., Lustman, F., Toscani, G.: Motion and structure from point and line matches. In: ICCV (1987) 14. Zhang, Z.: Determining the epipolar geometry and its uncertainty: A review. IJCV 27(2), 161–198 (1998) 15. Ho, P.-K., Chung, R.: Use of affine camera model and all stereo pairs in stereomotion. In: IEEE International Conference on Intelligent Vehicles, pp. 323–328. IEEE Computer Society Press, Los Alamitos (1998) 16. Fischer, M.A.: Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography. Comm. ACM 26(6), 381–395 (1981) 17. Vidal, R., Ma, Y., Sastry, S.: Generalized Principal Component Analysis (GPCA). PAMI 27(12) (2005)

Efficient Registration of Aerial Image Sequences Without Camera Priors Shobhit Niranjan1,2 , Gaurav Gupta2 , Amitabha Mukerjee1 , and Sumana Gupta1 1

Indian Institute of Technology Kanpur, India {nshobhit,sumana,amit}@iitk.ac.in 2 Aurora Integrated Systems, Kanpur, India [email protected]

Abstract. We present an efficient approach for finding homographies between sequences of aerial images. We propose a two-step approach: a) initially solving for image-plane rotation and scale parameters without using correspondence (under affine assumption), and b) using these parameters to constrain the full homography search, and c) extending the results to full perspective projection. No flight meta-data, camera priors, or any other user defined information is used for the task. Based on the perspective parameters estimated, the aerial images are stitched with the best matching image based on a probabilistic model, to compose a high resolution aerial image mosaic. While retaining the improved asymptotic worst-case complexity of [6], we demonstrate significant performance improvements in practice.

1 Introduction Registration of aerial images with other aerial (or satellite images) is an important problem for GIS applications, surveillance and monitoring systems [2] [3]. Frame-toreference and frame-to-frame registration of video data is complex due to lack of stable features, because typical frame does not cover large enough area, and computationally expensive. In this work, we propose a computationally efficient algorithm that estimates part of the registration parameters (rotation and scaling) without correspondence, and uses these to estimate the remaining parameters (translation). The current automated registration techniques can be classified into a) area-based ([4] [5]), and b) feature-based [2] techniques. In area-based algorithms, a small window of points in the sensed image is compared statistically with windows of the same size in the reference image or in frequency domain using Fast Fourier Transform(FFT). Window correspondence is based on the similarity between two given windows, usually the normalized cross correlation. A majority of the area-based methods have the limitation of registering only images with small misalignment, and therefore, the images must be roughly aligned with each other initially. The correlation measures become unreliable when the images have multiple modalities and the gray-level characteristics vary. In contrast, The feature-based methods are more robust and more suitable in case of images with small misalignment [2]. There are two critical procedures generally involved 

This work was being done as part of his B.Tech thesis at Dept. of EE, IIT Kanpur.

Y. Yagi et al. (Eds.): ACCV 2007, Part II, LNCS 4844, pp. 394–403, 2007. c Springer-Verlag Berlin Heidelberg 2007 

Efficient Registration of Aerial Image Sequences Without Camera Priors

395

in the feature-based techniques: feature extraction and feature correspondence. Previous attempts either use an initial estimate of camera motion parameters (metadata) available as priors [2], [3], or else use computationally intensive methods for establishing accurate correspondences and deterministic homography computation [14]. Nielsen [6] proposed a probabilistic approach of computing homography from corresponding pairs, that depended on an estimate of range within which motion parameters lie. He proved that bounds on computational complexity could be determined based on these estimates and algorithm parameters. Xiong [1] presented an approach that does not require correspondences to be established between features in two images. However, it estimates only rotation and scale parameters, with affine geometrical transform assumption. Here we present a novel adaptation of Nielsen’s approach by reducing the search for rotation using Xiong’s approach. This results in a computationally efficient framework for registering images. While Xiong’s notion was limited to affine transformations, we assume it provides a good estimate of scale and rotation for perspective transformation as well.

2 Background Surveillance with non-stationary camera involves both rotation and translation of camera axis. Under rigid motion assumption the motion of a camera can be described by translation T and Rotation R. Let P = (X, Y, Z) be a point in the 3D world while p = (x, y) be the perspective projection of P in the image plane, it’s position at time t (xt , yt ) and at t + 1, (xt+1 , yt+1 ) can be correlated as:         xt+1 t a11 a12 xt = x + (1) yt+1 ty a21 a22 yt   a11 a12 A= a21 a22 where (xt+1 , yt+1 ) is the new transformed coordinate of (xt , yt ). The matrix A can be combination of rotation, scale, or shear known as perspective transform. Here we assume that camera intrinsic parameters are already known so we know the relationship between 3D world P = (X, Y, Z) and its projection in image plane p = (x, y). If f be the focal length of the camera then p is give by,    X x fZ p= = y f YZ We present a registration technique which efficiently computes homography matrix, without any priors i.e. camera motion parameters. We have studied various features which are used for registration including Harris corner detector, KLT corner detector and extending visual saliency (Itti’s) model for feature extraction. Initially, rotation and scale parameters are estimated using registration technique devised by Xiong et. al. [1]. The algorithm in [1] differs from conventional feature based image registration algorithms, as it does not establish correspondence between images. We extend the approach in [1] for computing perspective homography, making use of the geometrical filtering of features in feature space [6] and generate mosaic using computed homography.

396

S. Niranjan et al.

(a)

(b)

(c)

Fig. 1. (a) Corners obtained after using Harris Corner Detector algorithm, (b) Detected features using Itti’s Model (c) Typical angle histogram - In the X-axis each column of image represents bin of angle difference and in the Y-axis occurrence of that bin is shown

3 Feature Extraction In general, the feature detection process involves computing the response R of one or multiple detectors (filters/operators) to the input image(s), followed by the analysis of R to isolate points or pixel that satisfy devised constraints. There are several kinds of feature used for matching (e.g. Corners). We compare Harris corner detector [8], KLT feature detector [9], Visually salient points [11] and SIFT features [15]. Harris corner detector [8] algorithm computes a matrix, which averages the first derivatives of the signal on a window:  2  x2 + y 2 Ix Ix Iy ⊗ (2) exp Ix Iy Iy2 2σ 2 where Ix and Iy are the gradient (derivatives) in the x and y direction. The eigen values of this matrix are the principal curvatures of the auto-correlation function. If these two curvatures are high, an interest point is present. We have also integrated in our approach recently formulated Visual Saliency model for identifying reliable features in images. Visual attention is a biological mechanism, used by primates to compensate for the inability of their brains to process the huge amount of visual information gathered by the two eyes [10]. The Caltech’s Hypothesis [10] elaborated by Itti-Koch [11] represents descriptions on how the visual attention model works in our brain and process visual information. According to the hypothesis seven elementary features which are computed from an RGB color image and which belong to three main cues, namely intensity, color, and orientation are extracted. In a second step, each feature map is transformed into a conspicuity map which highlights the parts of the scene that strongly differ from their surroundings. Salient points are subsequently determined either by Winner Take All (WTA) algorithm, or dividing the saliency map into sufficient grids and thresholding local maximas in intensity image of conspicuity map.. Fig. 1(a) and fig. 1(b) shows the visually salient points detected on a pair of images.

Efficient Registration of Aerial Image Sequences Without Camera Priors

397

4 Registration Algorithm Overview 4.1 Estimation of Registration Parameters Using Angle Histogram Method This approach is based on the work Xiong et. al. [1] which demonstrates efficient registration of two aerial images without feature correspondence. Features are detected on pair of images (observed and reference) using any of the above feature extraction methods. Circular patches are then created around the features. For a given patch p(i, j)(i = 1, 2, ..., m), the covariance matrix is defined as (3) COVp = E(X − mx )(X − mx )T     mx i i is the centroid of the image where X = is the position of the pixel; mx = mx j j patch p(i, j), the first order moment. It is then verified that patches are oriented in the first place by thresholding the eigen values of covariance matrix. The direction of eigenvector V1 is defined as the orientation of image patch p(i, j). By applying this approach, we can compute orientations for all image patches on reference and observed images. Taking any two patches in both images from set of patches (nt patches in I1 and nf patches in second image I2 ) we compute orientation differences Δϕ = {Δϕl , l = 1, 2, ..., nt nf }and find nt correspondence patches on reference image. For these nt pairs of correspondence patches, the value of orientation differences will be the rotation angle. Δϕl = |ϕjt − ϕif |, i = 1, 2, ..., nf (4) If we create a histogram for the orientation differences, the count for orientation difference between the two images will show a significant peak. We estimate the value of the scaling through a series of voting processes. By varying the size of the image patches and computing the angle histograms, we can obtain a series of angle histograms. Choose the one which has the highest maximum peak. Let At and Af denote the patch sizes on observed and reference images corresponding to the histogram Hh which has the highest maximum peak. The value of the scaling between observed and reference images can be computed by  At (5) s= Af The orientation difference Δϕ corresponding to the histogram Hh is the rotation angle ϕ between observed and reference images. We have found this algorithm to work very well for images with weak perspective; however, with strong perspective, the angle we estimate is not the exact angle of rotation. Considering two images which differ by rotation of camera about more than one axis, the circular patches around features in the first image are transformed to sheared shape in the second image. Therefore, when we compute orientation angle of features in second image, the histogram formed still has a unique peak, with a rise in other peaks (flattening of distribution). What we found is that this algorithm still reliably estimates angle of rotation about camera axis, thus providing a reliably accurate restriction to be input into geometrical filtering proposed by Nielsen [6].

398

S. Niranjan et al.

(a)

(b)

Fig. 2. Geometrical Filtering (a) General constraints applied by Nielsen with comparatively loser bounds because of no prior information, (b) Constraints provided by first part of proposed method provide much tighter bounds and significant reduction in search space

4.2 Homography Estimation Using Geometrical Filtering The above algorithm estimates rotation and scale; for completion of registration we need to find homography extending the approach of angle histogram with refined search for translation assuming perspective transformation and known initial estimate of rotation and scale. The proposed algorithm in [6] is fully automatic method for mosaicing which handles large scale (∼ = 4) and arbitrary rotation values while keeping the running time attractive for responsive applications. The method is based on Monte Carlo Las Vegas algorithms that combine both randomization and geometric feature selection. The basic idea of geometric filtering is to constrain the properties of feature matchings. Nielsen proposes that we can input geometric constraints like scale value (e.g.,in range [ 14 , 4]) and rotations (e.g. between [−60deg, 45deg] and a precision tolerance  (each feature can be moved into a ball centered at it of radius . For calculation of score, different methods can be used as described in [6] such as Pixel Cross correlation, the Haussdorf matching, Normalized cross correlation and bottleneck matching. We use Normalized Cross correlation and also probabilistic model. We use information of rotation and scale to reduce the search range in feature space and search only from α − δ to α + δ and s − ε to s + ε where α and s are known from previous method. For any planar transformation T we associate a characteristic vector v(T ) = (scale(T ), angle(T )) where scale(T ) is the scale of T and angle(T ) is the rotation angle about the x axis. Given two feature sets S1 = {L1 . . . Ln1 }(n1 = |S1 |) and S2 = {R1 . . . Rn2 }(n2 = S2 ) from two images I1 and I2 , the aim is to estimate an approximate collineation H, which could match as may features as possible. A point p1 in S1 is said to match point p2 in S2 for a collineation H if d2 (p2 , Hp1 ) ≤  for some prescribed  ≥ 0, where d2 (., .) is the Euclidean distance. α is the percentage of points required to match up to an absolute error  (that is max{α|S1 , α|S2 } points at least). We are looking for a (α, ) match i.e. a transformation H that matches at least αn points up to some error tolerance . This error tolerance can be set by trial and error depending on image data set being registered.The algorithm thus is a)Extract stable features in the images (reduced corners based on eigen values, b)Generate a set Γ of homographies satisfying geometric constraints and

Efficient Registration of Aerial Image Sequences Without Camera Priors

399

matching at least a given fraction of the point sets, c)Score each homography using probabilistic model and choose one with highest quality (e.g. score). Geometric Filtering. Geometric Filtering is a novel concept introduced by Nielsen to cut down the search space by considering the geometric constraints imposed by the selection of features in and image to be matched to its corresponding points in the reference image. It extends the combinatorial Monte Carlo approach using properties of the Euclidean/Projective space of point features. Transformations are estimated from point to point alignments. Once we have chosen a pair (L1 , L2 ) ∈ S1 ∗ S1 of features in I1 geometric filtering tests only some of the pairs (R1, R2) ∈ S2 ∗ S2 of I2 using the following constraint. Each detected feature point p lies in a ball B(p, μ). In order to take into account the fuzziness of the features μ is set to be 4 . Let p∗ denote the “visual” feature point that the feature detection algorithm approaches to p (p∗ ∈ B(p, μ)). Between any two feature points we have |d2 (p∗1 , p∗2 ) − d2 (p1 , p2 ) < 2μ| . Instead of comparing (L1 , L2 ) to all pairs (R1 , R2 ) first a point is chosen denoted by R1 ∈ I2 and the second point is chosen inside the ring whose center is R1 with minimum circle B(R1 , d2 (L1 , L2 ) − 2 ∗ μ) and width 4 ∗ μ. If the transformation involves uniform scale also, then length of line varies. To accommodate for variation in length the algorithm requires as input from the previous module a range of values in which the scale lies which have already been estimated. Given, that the scale lies in the range [scalemin , scalemax ] then we consider all pairs of points in S2 whose distance is between (d12 − ) ∗ scalemin to (d12 + ) ∗ scalemax . Since we have obtained a good estimate of scale, we use 5 − 10% tolerance about this scale value as scalemin and scalemax. The value of tolerance was set by trial and error, and this range worked fine on various image subsets we tested. Computing Homographies For Perspective Transform. Every time a k tuple is sampled from feature set S1 and matched with a possible k tuple in S2 we need to compute the homography. This computation depends on the number of features matched. We represent the features in set S1 as Li and the features in set S2 as Ri both represented in cartesian coordinates. The simplest case is when the transformation is with only similitude of the plane i.e. the transformation is a combination of translation, rotation and uniform scaling - affine transformation. In this case there are four unknowns and we can compute the homography by matching 2 points. However, in case of perspective transformation there are 8 unknowns for which we search for 4 pairs of tuple denoted as L1 toL4 in S1 and R1 toR4 in S2 . Then let P = (LT1 LT2 LT3 LT4 ) and Q = (R1T R2T R3T R4T ) be 3x4 matrices. Then perspective homography can be computed using pseudo inverse method [6]: H = Q ∗ PT ∗ (P ∗ PT )−1 For computing the best matching homography we evaluate its score by probabilistic approach proposed in [17]. After computing the homography, we apply the transform to the reference image and find its inlier and outlier features w.r.t. the other image. Let total number of features be nf in I1 and ni be the total number of inliers. The event that this homography is matching homography is represented by a binary variable m0, 1. Now a particular feature is an inlier or outlier f i 0, 1 is assumed to be an independent Bernoulli, so that the total number of inliers is Binomial: p(f(1:nf ) |m = 1) = B(ni ; nf , p1 )

400

S. Niranjan et al.

(a)

(b)

Fig. 3. Prior work on affine image registration (a) clear artifacts visible because of inaccurate computation of homography with affine assumption, (b) same images stitched with homography assuming perspective transform

p(f(1:nf ) |m = 0) = B(ni ; nf , p0 ) where p1 is the probability a feature is an inlier given a correct homography, and p0 is the probability a feature is an inlier given a false image match. We chose p1 = 0.8 and p0 = 0.02 after some trial and error. Now using Bayes’ rule it has been proved in [17], choosing pmin = 0.95 for our implementation, that computed homography is acceptable if, 1 B(ni ; nf , p1 ) > 1 B(ni ; nf , p0 ) pmin − 1

(a)

(b)

Fig. 4. (a) aerial view academic area, (b) orthographic view of academic area (courtesy Google), the registration parameters are given in Table 2

5 Image Mosaicing Images aligned after undergoing geometric corrections require further processing to eliminate distortions and discontinuities. Alignment of images may be imperfect due to registration errors resulting from incompatible model assumptions, dynamic scenes etc. Image Composition is the technique, which modifies the image gray levels in the vicinity of an alignment boundary to obtain a smooth transition between images by removing these seams and creating a blended image by determining how pixels in an overlapping area should be presented. The term image spline refers to digital techniques for making these adjustments [19]. The images to be splined are first decomposed into

Efficient Registration of Aerial Image Sequences Without Camera Priors

(a)

401

(b)

(c) Fig. 5. (a) Mosaic with affine asumption shows misalignments and distortion in regions of strong perspective, (b) Mosaic of same set as (a) constructed with perspective assumption (c) Mosaic with perspective assumption of another set of strongly perspective image set

a set of band-pass filtered component images (image components in a spatial frequency band). Next the component images in each spatial frequency band are assembled into a corresponding band-pass mosaic. In this step, component images are merged using a weighted average within a transition zone, which is proportional in size to the wavelengths represented in the band. Finally by summation of these band- passed mosaic images the desired image mosaic is generated. The spline is thus matched to the scale of features within the images. Figure 3(a) is the mosaic constructed without using the blending algorithm, and has clear visual artifacts along seam lines. Figure 3(b) has these artifacts eliminated using blending.

6 Results The system has been implemented on Visual C++ 6.0 using OpenCV library [20] from Intel. We have performed timing and performance analysis of proposed methodology with different feature detectors and analyzed their significance to accurate registration. The reported values are subjective since the results depend on the image data set. Since the Nielsen method is based on pattern matching, it is nontrivial to establish the logical relevance of the extracted feature points with the information contained in the image [6]. For the purpose of validation of presented algorithm, it was applied to pairs of test images, for which ground truthing was performed manually. KLT and Harris features take

402

S. Niranjan et al. Table 1. Performance Evaluation Dimensions Features Nielsen’s algorithm (sec) Self algorithm (sec) 728x388 100 15 5 728x388 150 27 8 728x388 175 34 10 800x600 100 19 5 800x600 150 30 8.7 Table 2. Ground truth result of Histogram method

Ground Truthing Histogram Method

Angle 34.062o 33.6752o

Scale 1.4512 1.445

equivalent computation times, but feature extraction based on Itti’s model consumes significantly higher time. KLT features were observed to be more stable for aerial images than Harris features. Salient points (Itti’s model) have also been observed to be stable, and are well distributed over entire images. Comparison with Nielsen algorithm with default parameters and proposed algorithms are given in Table 1. There is an overall decrease in computation time, and we are looking at the possibility of achieving this framework in real time. We also tested our registration algorithm with aerial images and orthographic images from Google Earth [21]. Manual registration is performed using standard Matlab functions Results obtained from algorithm for the same are listed in Table 2. Fig. 5(b) and 5(c) are the resulting mosaics with two sets of aerial images. We can clearly see distortions in image shown in Fig. 5(a), constructed from same set of images as Fig. 5(b), but with affine assumption [22]. Algorithm with affine assumption fails to compute correct homography in regions of strong perspective transform, with clear misalignments and distortions in constructed mosaic. Proposed algorithm works even for Figure 5(c), which uses image set of significantly higher perspective geometry, resulting in mosaic with invisible seam lines.

7 Conclusion Unmanned Aerial Vehicles(UAVs) equipped with lightweight,inexpensive cameras have grown in popularity by enabling new applications of UAV technology. Beginning with an investigation of previous registration and mosaicing works,this paper discusses the challenges of registering UAV based video sequence. A novel approach of estimating registration parameters is then presented. Future work for the improvement of proposed algorithm involves a frequency domain approach which could be used to determine a rough estimate of the parameters, which would help the registration algorithm to prune its search for registration parameters. Also with availability of coarse/fine metadata, significant improvements in performance can be achieved.

Efficient Registration of Aerial Image Sequences Without Camera Priors

403

References 1. Xiong, Y., Quek, F.: Automatic Aerial Image Registration Without Correspondence. In: ICVS 2006. The 4th IEEE International Conference on Computer Vision Systems, St. Johns University, Manhattan, New York City, New York, USA (January 5-7, 2006) 2. Sheikh, Y., Khan, S., Shah, M., Cannata, R.W.: Geodetic Alignment of Aerial Video Frames. VideoRegister03, ch. 7 (2003) 3. Wildes, R., Hirvonen, D., Hsu, S., Kumar, R., Lehman, W., Matei, B., Zhao, W.: Video Registration: Algorithm and quantitative evaluation. In: Proc. International Conference on Computer Vision, vol. 2, pp. 343–350 (2001) 4. Li, H., Manjunath, B.S., Mitra, S.K.: A contour based appraoch to multisensor image registration. IEEE Trans. Image Processing, 320–334 (March 1995) 5. Cideciyan, A.V.: Registration of high resolution images of the retina. In: Proc. SPIE, Medical Imaging VI: Image Processing, vol. 1652, pp. 310–322 (February 1992) 6. Nielsen, F.: Adaptive Randomized Algorithms for Mosaicing Systems. Transactions of the Institute of Electronics, Information, and Communication Engineers (IEICE), Information and Systems E83-D (7), 1386–1394 (2000) 7. Canny, J.: A computational approach to edge detection. IEEE PAMI, 679–698 (1986) 8. Harris, J.C., Stephens, M.: A combined corner and edge detector. In: Proc. 4th Alvey Vision Conf., pp. 189–192 (1988) 9. Tomasi, C., Kanade, T.: Detection and Tracking of Point Features. CMU Technical Report CMU-CS-91-132 (April 1991) 10. Itti, L.: Models of Bottom-Up and Top-Down Visual Attention. PhD thesis, Pasadena, California (2000) 11. Itti, L., Koch, C.: Nature Reviews Neuroscience 2(3), 194–203 (2001) 12. Ouerhani, N.: Visual Attention: Form Bio-Inspired Modelling to Real-Time Implementation, PhD thesis (2003) 13. Backer, G., Mertsching, B.: Two selection stages provide efficient object-based attentional controlfor dynamic vision. In: International Workshop on Attention and Performance in Computer Vision (2004) 14. Zoghliami, I., Faugeras, O., Deriche, R.: Using geometric corners to build a 2D mosaic from a set of images. In: CVPR Proceedings (1997) 15. Brown, M., Lowe, D.G.: Invariant features from interest point groups. In: BMVC 2002. British Machine Vision Conference, Cardiff, Wales, September 2002, pp. 656–665 (2002) 16. Maji, S., Mukerjee, A.: Motion Conspicuity Detection: A Visual Attention model for Dynamic Scenes. Report on CS497, IIT Kanpur, avialable at www.cse.iitk.ac.in/report-repository/2005/Y2383 497-report.pdf 17. Brown, M., Lowe, D.G.: Recognising panoramas. In: ICCV 2003. International Conference on Computer Vision, Nice, France, October 2003, pp. 1218–1225 (2003) 18. Kyung, Lacroix, S.: A Robust Interest Point Matching Algorithm. IEEE (2001) 19. Hsu, C., Wu, J.: Multiresolution mosaic. In: Image Processing, 1996, Proceedings, International Conference on, Iss., September 16-19, 1996, vol. 3, pp. 743–746 (1996) 20. Intel Open Source Computer Vision Libraray, http://www.intel.com/technology/computing/opencv/index.htm 21. http://www.earth.google.com 22. Gupta, G.: Feature Based Aerial Image Regsitration and Mosaicing. Bachelors Dissertation, IIT Kanpur (December 2006)

Simultaneous Appearance Modeling and Segmentation for Matching People Under Occlusion Zhe Lin, Larry S. Davis, David Doermann, and Daniel DeMenthon Institute for Advanced Computer Studies University of Maryland, College Park, MD 20742, USA {zhelin,lsd,doermann,daniel}@umiacs.umd.edu

Abstract. We describe an approach to segmenting foreground regions corresponding to a group of people into individual humans. Given background subtraction and ground plane homography, hierarchical parttemplate matching is employed to determine a reliable set of human detection hypotheses, and progressive greedy optimization is performed to estimate the best configuration of humans under a Bayesian MAP framework. Then, appearance models and segmentations are simultaneously estimated in an iterative sampling-expectation paradigm. Each human appearance is represented by a nonparametric kernel density estimator in a joint spatial-color space and a recursive probability update scheme is employed for soft segmentation at each iteration. Additionally, an automatic occlusion reasoning method is used to determine the layered occlusion status between humans. The approach is evaluated on a number of images and videos, and also applied to human appearance matching using a symmetric distance measure derived from the KullbackLeiber divergence.

1

Introduction

In video surveillance, people often appear in small groups, which yields occlusion of appearances due to the projection of the 3D world to 2D image space. In order to track people or to recognize them based on their appearances, it would be useful to be able to segment the groups into individuals and build their appearance models. The problem is to segment foreground regions from background subtraction into individual humans. Previous work on segmentation of groups can be classified into two categories: detection-based approaches and appearance-based approaches. Detection-based approaches model humans with 2D or 3D parametric shape models (e.g. rectangles, ellipses) and segment foreground regions into humans by fitting these models. For example, Zhao and Nevatia [1] introduce an MCMC-based optimization approach to human segmentation from foreground blobs. Following this work, Smith et al. [2] propose a similar trans-dimensional MCMC model to track multiple humans using particle filters. Later, an EM-based approach is proposed by Rittscher et al. [3] for foreground blob segmentation. On the other Y. Yagi et al. (Eds.): ACCV 2007, Part II, LNCS 4844, pp. 404–413, 2007. c Springer-Verlag Berlin Heidelberg 2007 

Simultaneous Appearance Modeling and Segmentation

405

hand, appearance-based approaches segment foreground regions by representing human appearances with probabilistic densities and classifying foreground pixels into individuals based on these densities. For example, Elgammal and Davis [4] introduce a probabilistic framework for human segmentation assuming a single video camera. In this approach, appearance models must first be acquired and used later in segmenting occluded humans. Mittal and Davis [5] deal with the occlusion problem by a multi-view approach using region-based stereo analysis and Bayesian pixel classification. But this approach needs strong calibration of the cameras for its stereo reconstruction. Other multi-view-based approaches [6][7][8] combine evidence from different views by exploiting ground plane homography information to handle more severe occlusions. Our goal is to develop an approach to segment and build appearance models from a single view even if people are occluded in every frame. In this context, appearance modeling and segmentation are closely related modules. Better appearance modeling can yield better pixel-wise segmentation while better segmentation can be used to generate better appearance models. This can be seen as a chicken-and-egg problem, so we solve it by the EM algorithm. Traditional EM-based segmentation approaches are sensitive to initialization and require appropriate selection of the number of mixture components. It is well known that finding a good initialization and choosing a generally reasonable number of mixtures for the traditional EM algorithm remain difficult problems. In [15], a sample consensus-based method is proposed for segmenting and tracking small groups of people using both color and spatial information. In [13], the KDE-EM approach is introduced by applying the nonparametric kernel density estimation method in EM-based color clustering. Later in [14], KDE-EM is applied to single human appearance modeling and segmentation from a video sequence. We modify KDE-EM and apply it to our problem of foreground human segmentation. First, we represent kernel densities of humans in a joint spatial-color space instead of density estimation in a pure color space. This can yield more discriminative appearance models by enforcing spatial constraints on color models. Second, we update assignment probabilities recursively instead of using a direct update scheme in KDE-EM; this modification of feature space and update equations results in faster convergence and better segmentation accuracy. Finally, we propose a general framework for building appearance models from occluded humans and matching them using full or partial observations.

2

Human Detection

In this section, we briefly introduce our human detection approach (details can be found in [16]). The detection problem is formulated as a Bayesian MAP optimization [1]: c∗ = arg maxc P (c|I), where I denotes the original image, c = {h1 , h2 , ...hn } denotes a human configuration (a set of human hypotheses). {hi = (xi , θi )} denotes an individual hypothesis which consists of foot position

406

Z. Lin et al.

(a)

(b)

(c)

(d)

(e)

(f)

Fig. 1. An example of the human detection process. (a) Adaptive rectangular window, (b) Foot candidate regions Rf oot (lighter regions), (c) Object-level likelihood map by hierarchical part-template matching, (d) The initial set of human hypotheses overlaid on the Canny edge map, (e) Human detection result, (f) Shape segmentation result.

xi and corresponding model parameters θi (which are defined as the indices of part-templates). Using Bayes Rule, the posterior probability can be decomposed (c) ∝ P (I|c)P (c). We asinto a joint likelihood and a prior as: P (c|I) = P (I|c)P P (I) sume a uniform prior, hence the MAP problem reduces to maximizing the joint likelihood. The joint likelihood P (I|c) is modeled as a multi-hypothesis, multiblob observation likelihood. The multi-blob observation likelihood has been previously explored in [9][10]. Hierarchical part-template matching is used to determine an initial set of human hypotheses. Given the (off-line estimated) foot-to-head plane homography [3], we search for human foot candidate pixels by matching a part-template tree to edges and binary foreground regions hierarchically and generate the objectlevel likelihood map. Local maxima are chosen adaptively from the likelihood map to determine the initial set of human hypotheses. For efficient implementation, we perform matching only for pixels in foot candidate regions Rf oot . Rf oot is defined as: Rf oot = {x|γx ≥ ξ}, where γx denotes the proportion of foreground pixels in an adaptive rectangular window W (x, (w0 , h0 )) determined → by the human vertical axis − v x (estimated from the homography mapping). The window coverage is efficiently calculated using integral images. Then, a fast and efficient greedy algorithm is employed for optimization. The algorithm works in a progressive way as follows: starting with an empty configuration, we iteratively add a new, locally best hypothesis from the remaining set of possible hypotheses until the termination condition is satisfied. The iteration is terminated when the joint likelihood stops increasing or no more hypothesis can be added. Fig. 1 shows an example of the human detection process.

Simultaneous Appearance Modeling and Segmentation

3

407

Human Segmentation

3.1

Modified KDE-EM Approach

KDE-EM [13] was originally developed for figure-ground segmentation. It uses nonparametric kernel density estimation [11] for representing feature distributions of foreground and background. Given a set of sample pixels {xi , i = 1, 2...N } (with a distribution P), each represented by a d-dimensional feature vector as xi = (xi1 , xi2 ..., xid )t , we can estimate the probability Pˆ (y) of a new pixel y with feature vector y = (y1 , y2 , ..., yd )t belonging to the same distribution P as: pˆ(y ∈ P) =

N  d  yj − xij 1 k( ), N σ1 ...σd i=1 j=1 σj

(1)

where the same kernel function k(·) is used in each dimension (or channel) with different bandwidth σj . It is well known that a kernel density estimator can converge to any complex-shaped density with sufficient samples. Also due to its nonparametric property, it is a natural choice for representing the complex color distributions that arise in real images. We extend the color feature space in KDE-EM to incorporate spatial information. This joint spatial-color feature space has been previously explored for feature space clustering approaches such as [12], [15]. The joint space imposes spatial constraints on pixel colors hence the resulting density representation is more discriminative and can tolerate small local deformations. Each pixel is represented by a feature vector x = (X t , C t )t in a 5D space, R5 , with 2D spatial coordinates X = (x1 , x2 )t and 3D normalized rgs color1 coordinates C = (r, g, s)t . In Equation 1, we assume independence between channels and use a Gaussian kernel for each channel. The kernel bandwidths are estimated as in [11]. In KDE-EM, the foreground and background assignment probabilities fˆt (y) and gˆt (y) are updated directly by weighted kernel densities. We modify this by updating the assignment probabilities recursively on the previous assignment probabilities with weighted kernel densities (see Equation 2). This modification results in faster convergence and better segmentation accuracy, which is quantitatively verified in [17] in terms of pixel-wise segmentation accuracy and number of iterations needed for foreground/background segmentation. 3.2

Foreground Segmentation Approach

Given a foreground regions Rf from background subtraction and a set of initial human detection hypotheses (hk , k = 1, 2, ...K), the problem of segmentation is equivalent to the K-class pixel labeling problem. The label set is denoted as F1 , F2 , ...FK . Given a pixel y, we represent the probability of pixel y belonging to human-k as fˆkt (y), where t = 0, 1, 2... is the iteration index. The assignment K probabilities fˆkt (y) are constrained to satisfy the condition: k=1 fˆkt (y) = 1. 1

r = R/(R + G + B), g = G/(R + G + B), s = (R + G + B)/3.

408

Z. Lin et al.

Algorithm 1. Initialization by Layered Occlusion Model initialize R00 (y) = 1 for all y ∈ Rf for k = 1, 2, ...K − 1 - for all y ∈ Rf t −1 0 - fˆk0 (y) = Rk−1 (y)e−1/2(Y −Y0,k ) V (Y −Y0,k ) and Rk0 (y) = 1 − ki=1 fˆi0 (y) endfor 0 0 0 (y) = RK−1 (y) for all y ∈ Rf and return fˆ10 , fˆ20 , ..., fˆK set fˆK where Y denotes the spatial coordinates of y, Y0,k denotes the center coordinates of object k, and V denotes the covariance matrix of the 2D spatial Gaussian distribution.

Layered Occlusion Model. We introduce a layered occlusion model into the initialization step. Given a hypothesis of an occlusion ordering of detections, we build a layered occlusion representation iteratively by calculating the foreground probability map fˆk0 for the current layer and its residual probability map Rk0 for pixel y). Suppose the occlusion order (from front to back) is given by F1 , F2 , ...FK ; then the initial probability map is calculated recursively from the front layer to the back layer by assigning 2D anisotropic Gaussian distributions based on the location and scales of each detection hypothesis. Occlusion Reasoning. The initial occlusion ordering is determined by sorting the detection hypotheses by their vertical coordinates and the layered occlusion model is used to estimate initial assignment probabilities. The occlusion status is updated at each iteration (after the E − step) by comparing the evidence of occupancy in the overlap area between different human hypotheses. For two human hypotheses hi and hj , if they have overlap area Ohi ,hj , we re-estimate the occlusion   ordering between the two as: hi occlude hj if x∈Oh ,h fˆit (x) > x∈Oh ,h fˆjt (x) i

j

i

j

(i.e. hi better accounts for the pixels in the overlap area than hj ), hj occlude hi otherwise, where fˆit and fˆjt are the foreground assignment probabilities of hi and hj . At each iteration, every pair of hypotheses is compared in this way if they have a non-empty overlap area. The whole occlusion ordering is updated by exchanges if and only if an estimated pairwise occlusion ordering differs from the previous ordering.

4

Partial Human Appearance Matching

Appearance models represented by kernel probability densities can be compared by information theoretic measures such as the Battacharyya Distance or the Kullback Leiber Distance for tracking and matching objects in video. Recently, Yu et al. [18] introduce an approach to construct appearance models from a video sequence by a key frame method and show robust matching results using a pathlength feature and the Kullback-Leiber distance measure. But this approach only handles un-occluded cases. Suppose two appearance models, a and b are represented as kernel densities in a joint spatial-color space. Assuming a as the reference model and b as the test

Simultaneous Appearance Modeling and Segmentation

409

Algorithm 2. Simultaneous Appearance Modeling and Segmentation for Occlusion Handling Given a set of sample pixels {xi , i = 1, 2...N } from the foreground regions Rf , we iteratively estimate the assignment probabilities fˆkt (y) of a foreground pixel y ∈ Rf belonging to Fk as follows: Initialization : Initial probabilities are assigned by the layered occlusion model. M − Step : (Random Pixel Sampling) We randomly sample a set of pixels (we use η = 5% of the pixels) from the foreground regions Rf for estimating each foreground appearances represented by weighted kernel densities. E − Step : (Soft Probability Update) For each k ∈ {1, 2, ...K}, the assignment probabilities Fkt (y) are recursively updated as follows: fˆkt (y) = cfˆkt−1 (y)

 fˆ

 k( y

i=1

j=1

N

d

t−1 (xi ) k

j

− xij ), σj

(2)

where N is the number of samples and c is a normalizing constant such that K ˆt k=1 fk (y) = 1. Segmentation : The iteration is terminated when the average segmentation difference of two consecutive iterations is below a threshold: k

y

|fˆkt (y) − fˆkt−1 (y)| nK

(3)

< ,

where n is the number of pixels in the foreground regions. Let fˆk (y) denote the final converged assignment probabilities. Then the final segmentation is determined as: pixel y belong to human-k, i.e. y ∈ Fk , k = 1, ...K, if k = arg maxk∈{1,...K} fˆk (y).

model, the similarity of the two appearances can be measured by the KullbackLeiber distance as follows [12][18]:  pˆa (y) dy, (4) pb ||ˆ pa ) = pˆa (y)log b DKL (ˆ pˆ (y) where y denotes a feature vector, and pˆa and pˆb denote kernel pdf functions. For simplification, the distance is calculated from samples instead of the whole feature set. We need to compare two kernel pdfs using the same set of samples in the feature space. Given Na samples xi , i = 1, 2..., Na from the appearance model a and Nb samples yk , k = 1, 2..., Nb from the appearance model b, the above equation can be approximated by the following form [18] given sufficient samples from the two appearances: Nb 1  pˆb (yk ) , p ||ˆ p )= log a DKL (ˆ Nb pˆ (yk )

(5)

Nb  Na  d d 1  ykj − xij 1  ykj − yij k( ), pˆb (yk ) = k( ). Na i=1 j=1 σj Nb i=1 j=1 σj

(6)

b

a

k=1

pˆa (yk ) =

Since we sample test pixels only from the appearance model b, pˆb is evaluated by its own samples and pˆb is guaranteed to be equal or larger than pˆa for any

410

Z. Lin et al.

convergence 0.1 0.09

segmentation difference

0.08 0.07 0.06 0.05 0.04 0.03 0.02 0.01 0 1

2

3

4 5 iteration index

6

7

convergence 0.1 0.09

segmentation difference

0.08 0.07 0.06 0.05 0.04 0.03 0.02 0.01 0 0

2

4 6 iteration index

8

10

convergence 0.07

segmentation difference

0.06 0.05 0.04 0.03 0.02 0.01 0 1

2

3 iteration index

4

5

Fig. 2. Examples of the detection and segmentation process with corresponding convergence graphs. The vertical axis of the convergence graph shows the absolute segmentation difference between two consecutive iterations given by Equation 3.

samples yk . This ensures that the distance DKL (ˆ pb ||ˆ pa ) ≥ 0, where the equality holds if and only if two density models are identical. The Kullback-Leiber pb ||ˆ pa ) = DKL (ˆ pa ||ˆ pb ). For distance is a non-symmetric measure in that DKL (ˆ obtaining a symmetric similarity measure between the two appearance models, we define the distance of the two appearances as follows: Dist(ˆ pb , pˆa ) = b a a b min(DKL (ˆ p ||ˆ p ), DKL (ˆ p ||ˆ p )). It is reasonable to choose the minimum as the distance measure since it can preserve the balance between (full-full), (fullpartial), (partial-partial) appearance matching, while the symmetrized distance pb ||ˆ pa ) + DKL (ˆ pa ||ˆ pb )) would only be effective for (full-full) appearance DKL (ˆ matching and does not compensate for occlusion.

5

Experimental Results and Analysis

Fig. 2 shows examples of the human segmentation process for small human groups. The results show that our approach can generate accurate pixel-wise

Simultaneous Appearance Modeling and Segmentation

411

Fig. 3. Experiments on different degrees of occlusion between two people

segmentation of foreground regions when people are in standing or walking poses. Also, the convergence graphs show that our segmentation algorithm converges to a stable solution in less than 10 iterations and gives accurate segmentation of foreground regions for images with discriminating color structures of different humans. The cases of falling into local minimum with inaccurate segmentation is mostly due to the color ambiguity between different foreground objects or misclassification of shadows as foreground. Some inaccurate segmentation results can be found in human heads and feet in Fig. 2 and Fig. 3, and can be reduced by incorporating human pose models as in [17]. We also evaluated the segmentation performance with respect to the degree of occlusion. Fig. 3 shows the segmentation results given images with varying degrees of occlusion when two people walk across each other in an indoor environment. Note that the degree of occlusion does not significantly affect the segmentation accuracy as long as reasonably accurate detections can be achieved. Finally, we quantitatively evaluate our segmentation and appearance modeling approach to appearance matching under occlusion. We choose three frames from a test video sequence (containing two people in the scene) and perform segmentation for each of them. Then, the generated segmentations are used to estimate partial or full human appearance models as shown in Fig. 4. We evaluate the two-way Kullback-Leiber distances and the symmetric distance for each pair of appearances and represent them as affinity matrices shown in Fig. 4. The elements of the affinity matrices quantitatively reflect the accuracy of matching. We also conducted matching experiments using different spatial-color space combinations, 3D (r, g, s) color space, 4D (x, r, g, s) space, 4D (y, r, g, s) space, and 5D (x, y, r, g, s) space. The affinity matrices show that 3D (r, g, s) color space and 4D (y, r, g, s) space produce much better matching results than the other two. This is because color variation is more sensitive in the horizontal direction than in the vertical direction. The color-only feature space obtains good matching performance for this example because the color distributions are significantly different between appearances 1 and 2. But, in reality, there are often cases in which two different appearances have similar color distributions with completely different spatial layouts. On the other hand, 4D (y, r, g, s) joint spatial-color feature space (color distribution as a function of the normalized

412

Z. Lin et al.

Fig. 4. Experiments on appearance matching. Top: appearance models used for matching experiments, Middle: two-way Kullback-Leiber distances, Bottom: symmetric distances.

human height) enforces spatial constraints on color distributions, hence it has much more discriminative power.

6

Conclusion

We proposed a two stage foreground segmentation approach by combining human detection and iterative foreground segmentation. The KDE-EM framework is modified and applied to segmentation of groups into individuals. The advantage of the proposed approach lies in simultaneously segmenting people and building appearance models. This is useful when matching and recognizing people when only occluded frames can be used for training. Our future work includes the application of the proposed approach to human tracking and recognition across cameras.

Acknowledgement This research was funded in part by the U.S. Government VACE program.

Simultaneous Appearance Modeling and Segmentation

413

References 1. Zhao, T., Nevatia, R.: Tracking Multiple Humans in Crowded Environment. In: CVPR (2004) 2. Smith, K., Perez, D.G., Odobez, J.M.: Using Particles to Track Varying Numbers of Interacting People. In: CVPR (2005) 3. Rittscher, J., Tu, P.H., Krahnstoever, N.: Simultaneous Estimation of Segmentation and Shape. In: CVPR (2005) 4. Elgammal, A.M., Davis, L.S.: Probabilistic Framework for Segmenting People Under Occlusion. In: ICCV (2001) 5. Mittal, A., Davis, L.S.: M2Tracker: A Multi-View Approach to Segmenting and Tracking People in a Cluttered Scene. International Journal of Computer Vision (IJCV) 51(3), 189–203 (2003) 6. Fleuret, F., Lengagne, R., Fua, P.: Fixed Point Probability Field for Complex Occlusion Handling. In: ICCV (2005) 7. Khan, S., Shah, M.: A Multiview Approach to Tracking People in Crowded Scenes using a Planar Homography Constraint. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3951, Springer, Heidelberg (2006) 8. Kim, K., Davis, L.S.: Multi-Camera Tracking and Segmentation of Occluded People on Ground Plane using Search-Guided Particle Filtering. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3951, Springer, Heidelberg (2006) 9. Tao, H., Sawhney, H., Kumar, R.: A Sampling Algorithm for Detecting and Tracking Multiple Objects. In: ICCV Workshop on Vision Algorithms (1999) 10. Isard, M., MacCormick, J.: BraMBLe: A Bayesian Multiple-Blob Tracker. In: ICCV (2001) 11. Scott, D.W.: Multivariate Density Estimation. Wiley Interscience, Chichester (1992) 12. Elgammal, A.M., Davis, L.S.: Probabilistic Tracking in Joint Feature-Spatial Spaces. In: CVPR (2003) 13. Zhao, L., Davis, L.S.: Iterative Figure-Ground Discrimination. In: ICPR (2004) 14. Zhao, L., Davis, L.S.: Segmentation and Appearance Model Building from An Image Sequence. In: ICIP (2005) 15. Wang, H., Suter, D.: Tracking and Segmenting People with Occlusions by A Simple Consensus based Method. In: ICIP (2005) 16. Lin, Z., Davis, L.S., Doermann, D., DeMenthon, D.: Hierarchical Part-Template Matching for Human Detection and Segmentation. In: ICCV (2007) 17. Lin, Z., Davis, L.S., Doermann, D., DeMenthon, D.: An Interactive Approach to Pose-Assisted and Appearance-based Segmentation of Humans. In: ICCV Workshop on Interactive Computer Vision (2007) 18. Yu, Y., Harwood, D., Yoon, K., Davis, L.S.: Human Appearance Modeling for Matching across Video Sequences. Special Issue on Machine Vision Applications (2007)

Content-Based Matching of Videos Using Local Spatio-temporal Fingerprints Gajinder Singh, Manika Puri, Jeffrey Lubin, and Harpreet Sawhney Sarnoff Corporation

Abstract. Fingerprinting is the process of mapping content or fragments of it, into unique, discriminative hashes called fingerprints. In this paper, we propose an automated video identification algorithm that employs fingerprinting for storing videos inside its database. When queried using a degraded short video segment, the objective of the system is to retrieve the original video to which it corresponds to, both accurately and in real-time. We present an algorithm that first, extracts key frames for temporal alignment of the query and its actual database video, and then computes spatio-temporal fingerprints locally within such frames, to indicate a content-match. All stages of the algorithm have been shown to be highly stable and reproducible even when strong distortions are applied to the query.

1

Introduction

With the growing popularity of free video publishing on the web, there exist innumerable copies of copyright videos floating over the internet unrestricted. A robust content-identification system that detects perceptually identical video content thus, benefits popular video-sharing websites and peer-to-peer (P2P) networks of today, by detecting and removing all copyright infringing material from their databases and prevent any such future uploads made by their users. Fingerprinting offers a solution to query and identify short video segments from a large multimedia repository using a set of discriminative features called fingerprints. Challenges in designing such a system are: 1. Accuracy to identify content even when it is altered either naturally because of changes in video formats, resolution, illumination settings, color schemes or maliciously by introducing frame letterbox, camcorder recording or cropping. This is addressed by employing robust fingerprints that are invariant to such common distortions. 2. Speed to allow the fingerprinting system to determine a content-match with small turn-around times, which is crucial for real-time applications. A common denominator of all fingerprinting techniques is their ability to capture and represent high-dimensional, perceptually relevant multimedia content, in the form of short robust hashes [1], essential for fast retrieval. In content-based video identification, literature reports of approaches that compute features such as, mean-luminance [1], centroid of gradient [2], rank-ordered image intensity Y. Yagi et al. (Eds.): ACCV 2007, Part II, LNCS 4844, pp. 414–423, 2007. c Springer-Verlag Berlin Heidelberg 2007 

Content-Based Matching of Videos

415

distribution [3] and centroid of gradient orientations [4], over fixed-sized partitions of video frames. The limitation of such features is that they encode complete frame information and therefore fail to identify videos when presented with queries having partially cropped or scaled data. This motivates the use of a local fingerprinting approach. Sivic and Zisserman [5] in their text-retrieval approach for object recognition make use of maximally stable extremal regions (MSERs), proposed by Matas[6], as representations of each video frame. Since their method clusters semantically similar content together in its visual vocabulary, it is expected to offer poor discrimination, for example, between different seasons of the same TV programme having similar scene settings, camera capture positions and actors. However, a video fingerprinting system is expected to provide good discrimination between such videos. Similar to [5], Nist´er and Stew´enius propose an object recognition algorithm that extracts and stores MSERs on a group of images of an object, captured under different viewpoint, orientation, scale and lighting conditions [7]. During retrieval, a database image is scored depending on the number of MSER correspondences it shares with the given query image. Only the top scoring hits are then scanned further. Hence, fewer MSER pairs decrease the possibility of a database hit to figure out within the top ranked images. Since a fingerprinting system needs to identify videos even when queried with short distorted clips, both [5] and [7] become unsuitable. This is because, strong degradations such as, blurring, cropping, frame-letterboxing, result into fewer number of MSERs in a distorted image as compared to its original. Such degradations, thus, have a direct impact on the algorithm performance because of change in representation of the frame. Massoudi et al., proposes an algorithm that first slices a query video in terms of shots, extracts key-frames and then performs local fingerprinting [8]. A major drawback of this approach is the fact that even the most common forms of video processing such as blurring and scaling, disturb the key-frame and introduce misalignment between the query and database frames. In the proposed approach, key-frames correspond to local peaks of maximum intensity change across the video sequence. Such frames are hence, reproducible under most distortions. For feature extraction, information of each key-frame is localized as a set of affine covariant 3D regions, each of which is characterized using a spatio-temporal binary fingerprint. Use of such distortion-invariant fingerprints as an index into the database look-up, fetches appreciable advantages in terms of an easy and efficient database retrieval strategy. During database query, we employ a voting strategy to collate the set of local database hits in order to make a global decision about best-matching video in the database.

2

Proposed Video Fingerprinting Approach

Figure 1 shows various stages of the proposed video fingerprinting framework. The input video is first subjected to a series of video preprocessing steps [4]. These include, changing the source frame rate to a predefined resampling rate (say, 10 fps), followed by converting each video frame to grayscale and finally, resizing all frames to a fixed width(w) and height(h) (in our case,w = 160 and

416

G. Singh et al.

Fig. 1. Framework of proposed localized content-based video fingerprinting approach

h = 120 pixels). Such preprocessing achieves two benefits, one being in robustness against changes in color formats or resizing, while the other in terms of speed for retrieval of large-size videos (by choosing w and h accordingly). To reduce complexity, the frame sequence is examined to extract key-points that correspond to local peaks of maximum change in intensity. Since maximum change in intensity reduces stability of the regions detected in key-frames, we store a few neighboring frames on either side of the key-frame to maintain minimal database redundancy. 2.1

Region Detector

Stable regions are detected on key-frames for localizing their information. Our region detection process is inspired by the concept of Maximally Stable Volumes (MSVs), proposed by Donser and Bischof [9] for 3D segmentation. Extending MSER to the third dimension i.e., time, an MSV is detected by an extremal property of the intensity function in the volume and on its outer boundary. In the present case however, to extract regions stable under the set of distortions, we extend MSERs to the third dimension, that is, resolution. The process of extracting MSVs along with related terminology as adopted in the rest of the paper, is given as follows: Image Sequence. For a video frame F , consider a set of multi-resolution images F1 , F2 , . . . , Fi , . . . , Fs , where Fi is obtained when video frame F is subsampled by a factor 2i−1 . Size of each Fi is the same as F . Thus, a 3D point (x, y, z) in this space corresponds to pixel (x, y) of frame F at resolution z or, equivalently Fz (x, y). Volume. We define Vji as the j th volume such that all 3D points belonging to it have intensities less than (or greater than) i, ∀(x, y, z) ∈ Vji iff Fz (x, y) ≤ i (or Fz (x, y) ≥ i) Connectivity. Volume Vji is said to be contiguous, if for all points p, q ∈ Vji , there exists a sequence p, a1 , a2 , . . . , an , q and pAa1 , a1 Aa2 , . . . , ai Aai+1 , . . . , an Aq. Here, A is an adjacency relation defined such that two pixels p, q ∈ Vji 3 are adjacent (pAq), iff 1 |pi − qi | ≤ 1.

Content-Based Matching of Videos

417

Partial Relationship. Any two volumes Vki and Vlj , are nested, i.e., Vki  Vlj iff i ≤ j (or i ≥ j). Maximally Stable Volumes. Let V1 , V2 , . . . , Vi−1 , Vi , . . . be a sequence of partially ordered set of volumes, such that Vi  Vi+1 . Extremal volume Vi is said to be maximally stable iff v(i) = |Vi+Δ \Vi−Δ |/|Vi | has a local minimum at i , i.e. for changes in intensity of magnitude less than Δ, the corresponding change in region volume is zero. Thus, we represent each video frame as a set of distinguished regions that are maximally stable to intensity perturbation over different scales. The reason for stability of MSVs over MSERs in most cases of image degradations, is that additional volumetric information enables selection of regions with near-identical characteristics across different image resolutions. The more volatile regions (the ones which split or merge), are hence eliminated from consideration. 2.2

Localized Fingerprint Extraction

Consider a frame sequence {F1 , F2 , . . . , Fp , . . .} where Fp denotes the pth frame of the video. For invariance to affine transformations, each MSV is represented in the form of an ellipse. In terms of the present notation, pixels of ith maximally stable volume Vip in frame Fp are made to fit an ellipse denoted by epi . Affine invariance of local image regions makes our fingerprint robust to geometric distortions. Each ellipse epi is represented by (xpi , yip , spi , lxpi , lyip , αpi ) where (xpi , yip ) are its center coordinates, lxpi is the major axis, lyip is the minor axis and αpi is the orientation of the ellipse w.r.t frame axis. A scale factor spi , which depends upon the ratio of ellipse area w.r.t total area of the frame, is used to encode bigger regions around MSVs which are very small. The proposed algorithm expresses each local measurement region of a frame in the form of a spatio-temporal fingerprint that offers a compact representation of videos inside the database. For each ellipse epi extracted from frame F p , the process of fingerprint computation (similar to [1]) has been elaborated below: – Since small regions are more prone to perturbations, each region is blown-up by a factor si . A rectangular region rip that encloses ellipse epi and has an area of (lxpi × spi , lyip × spi ), is detected. – For a scale invariant fingerprint, we divide rip into fixed number of R × C blocks. – Let mean luminance of block (r, c) ∈ rip be denoted by Lpi (r, c) where r = [1, . . . , R] and c = [1, . . . , C]. We choose a spatial filter [-1 1] and a temporal filter [−α 1] for storing the spatio-temporal dimension of a video. In order to reduce susceptibility to noise, we compute a fingerprint between rip and rip+step , which is same as region rip but shifted by step frames. The R×(C −1) bit fingerprint Bip is equal to bit ’1’ if Qpi (r, c) is positive and bit ’0’ otherwise where, (r, c + 1) − Lp+step (r, c)) − α(Lpi (r, c + 1) − Lpi (r, c)) (1) Qpi (r, c) = (Lp+step i i Encoding mean luminance makes our fingerprint invariant to photometric distortions. In the current implementation, α = 0.95 and step = 10.

418

2.3

G. Singh et al.

Database Organization and Lookup

Localized content of each video frame is stored inside the database look-up table (LUT) [1] using 32-bit signatures, as computed by equation 1. Each LUT entry in turn stores pointers to all video clips with regions having the same fingerprint value. In order to have an affine invariant representation of the video frame, independent of different query distortions, we store the geometric and shape information of stable region ei along with its fingerprint inside the database. This is done by transforming coordinates of the original frame center, denoted ˆ Yˆ ). The new axis has by (cx, cy), onto a new reference axis, denoted by (X, the property of projecting ellipse epi onto a circle, with the ellipse center being ˆ Yˆ ) the origin of this axis and ellipse major and minor axes aligned with (X, respectively. The coordinates of the original frame center w.r.t this new reference ˆ pi ). axis are denoted by (cx ˆ pi , cy ˆ pi ) is given by: The transformation between (cx, cy) and (cx ˆ pi , cy cx ˆ pi = ((cx − xpi )cos(−αpi ) − (cy − yip )sin(−αpi ))/(lxpi × spi )

(2)

cy ˆ pi = ((cx − xpi )sin(−αpi ) + (cy − yip )cos(−αpi ))/(lyip × spi )

(3)

Apart from the frame center, transformation of three points c1, c2, c3, located at corners of a prefixed square SQ (in our case, of size 100 × 100) centered at (cx, cy), is also stored inside the database. Let their coordinates w.r.t reference ˆ Yˆ ) be denoted as c1 ˆ pi , c2 ˆ pi , c3 ˆ pi . These three points are stored for their role axis (X, in the verification stage. We define sif tp as the 128-dimensional SIFT descriptor [10] of SQ.   ˆ pi , cy ˆ pi , epi ), sif tp ) is therefore expressed The fingerprint database p ( i (Bip , cx th as a union of fingerprint entries for each i MSV belonging to the pth video frame where each MSV entry inside the database consists of fields such as, its binary fingerprint Bip , ellipse parameters ei , frame center’s coordinates w.r.t reference ˆ Yˆ ) and SIFT descriptor of SQ given by sif tp . axis (X, During database retrieval, for a query video frame E q , we first generate its ellipses and their corresponding fingerprints using equation 1. Thus, the query  frame can be expressed as j {Bjq , eqj }. Expressing in terms of notation, we use q to denote query and p to denote database parameters. Each of the fingerprints of MSVs belonging to the query frame are used to probe the database for potential candidates. That is, ∀j we query the database to get the candidate set given by   p ˆ pi , cy ˆ pi , epi ), sif tp ). p ( i (Bi , cx There exists a possibility for every entry in the candidate set of being the correct-database match that we are looking for during database retrieval. Hence, we propose a hypothesis for the query frame E q of being the same as original frame Fp stored inside the database. This will happen when ellipses eqj and epi point to identical regions in their respective frames. For every candidate hit produced from the database, we compute the coordinates of the query frame’s center by using: cx ˇ p,q ˆ pi × sqj × lxqj )cos(αqj ) − (cy ˆ pi × sqj × lyjq )sin(αqj ) + xqj i,j = (cx

(4)

ˆ pi × sqj × lxqj )sin(αqj ) + (cy ˆ pi × sqj × lyjq )cos(αqj ) + yjq cy ˇ p,q i,j = (cx

(5)

Content-Based Matching of Videos

419

To speed-up query retrieval, we evaluate only a subset of the entire set of candidates by filtering out spurious database hits. For every candidate clip entry, we associate a score sci,j,p,q which is defined as: sci,j,p,q = f ac × (lxpi × lyip × spi × spi ÷ (w × h)) + (1 − f ac) × (log(N ÷ Njq )) (6) where N is the total number of entries present in the database and Njq is the number of database hits generated for query fingerprint Bjq . The first term in Equation 6, (lxpi ×lyip ×spi ×spi ÷(w×h)), signifies that greater the area represented by the fingerprint of the database image, higher is its score. The second term, log(N ÷ Njq ), assigns higher scores to unique fingerprints Bjq that produce fewer database hits. A factor f ac ∈ [0, 1] is used for assigning appropriate weight to each of the two terms in equation 6. We have chosen f ac = 0.5 in the current implementation. 2.4

Scoring and Polling Strategy

As stated earlier, an essential requirement of a video fingerprinting system is speed. For this purpose, we have an additional stage for scoring each database result, followed by a poll to collate all local information and arrive at the final decision. Consider a video’s frame sequence as a 3D space, with its third dimension given by frame number. We divide this space into bins (in our case, of size 2 × 2 × 10), where each bin is described by a three tuple b ≡ (b1, b2, b3). Thus we merge database hits, (1) who have their hypothetical image centers close to each other, and (2) belong to neighboring frames of the same video considering the fact that movements of region across them is appreciably small. The scoring process and preparation of candidate clip for the verification stage, is as follows: ˇ p,q ˇ p,q – ∀ ellipses eqj and epi , add the score sci,j,p,q to the bin in which (cx i,j , cy i,j , q) p p p,q p,q p,q ˇ ˇ ˇ ˆ ˆ ˆp falls. For each such entry, also calculate c1i,j , c2i,j , c3i,j by using c1i , c2 , c3 and ellipse parameters of eqj in Equations similar to 4 and 5. – Pick the entries that fall within the top n scoring bins for the next stage of verification. – Every database hit which polled into a particular bin and added to its score, gives information of the affine transformation by which the database frame can be aligned to the query frame. We compute the average transformation ˇ p,q ˇ p,q ˇ p,q of bin b, denoted by Hb , by taking an average of all c1 i,j , c2i,j , c3i,j that polled to bin b. – Inverse of Hb is applied on sif tnum number of query frames viz. {E q , E q+1 , . . . . . . , E q+sif tnum } to produce a series of frames {Ebq , Ebq+1 , . . . , Ebq+sif tnum }. Now, these frames are hypothetically aligned to the ones stored within the database and which polled to bin b. To carry out a check on sif tnum (in our case, sif tnum = 5) number of frames imposes a tougher constraint on making a decision of the correct database video. 2.5

Verification

From the top n candidates obtained from the previous stage, the correct database hit is found using a SIFT-based verification. Let esif tqb be the sift descriptor of

420

G. Singh et al.

ˆ centered at (cx, cy) in query’s frame E q with its sides aligned to the square SQ b the frame’s axis. The two-step verification procedure is: – ∀p database frames that voted to bin b, calculate Bhattacharyya distance betnum , . . . , esif tq+sif } tween descriptors of the aligned query {esif tqb , esif tq+1 b b and its database hit {sif tp , sif tp+1 , . . . , sif tp+sif tnum }. – If the distance is less than an empirically chosen threshold T , declare a match to database frame p.

3

Experimental Validation

Results presented in this section have been calculated for a database populated with 1200 videos of 4 minutes each (in total, 80 hours of data). Video material stored in the database contains clips taken from different seasons of TV programmes, music videos, animated cartoons and movies. Analysis of our algorithm (method 1) and its comparison with approaches described in [8] (method 2) and [1] (method 3) has been carried out using 2400 clips, each of duration 10 seconds and picked from different locations of their respective original video. Robustness testing of the proposed algorithm has been evaluated under video degradations such as, spatial blur (gaussian blurring of radius 4), contrast modification (query clips with 25% reduced contrast), scaling (resolution of each video frame is changed by a factor 1.5), frame letterbox (refers to the black bands, of width 100 pixels, appended around a video frame), DivX compression (alters bitrate by a factor of 4), temporal cropping (query clips having only 10% of frames present in the original video), spatial cropping (30% of spatial information is cropped in each video frame) and camcorder recording (refers to query clips captured using a camcorder). Different parameters of our method are: top n candidate bins that are chosen based on scoring and polling and, threshold T used in the verification stage. By setting empirical values of parameters, n = 10 and T = 0.2 (justified later in the section), results shown in figure 2 have been computed. Figure 2(a) demonstrates the repeatability of key-frames detected by our algorithm, as compared to method 2. As reflected by the graph, even under strong distortions such as spatial crop, percentage of key-frames re-detected in the query does not fall below 80%. Out of all the query clips performed on the database, percentage of queries which were retrieved correctly, is shown in Figure 2(b). Use of local, affine and scale invariant fingerprints makes our approach better than method 3, which fails for distortions like spatial cropping. At the same time, ability to encode reproducible key-frames makes our system more robust as compared to method 2. Variations in performance of the proposed system as a result of changing parameters n and T , is demonstrated in figure 3. Every row of figure 3 shows three different graphs for each type of distortion. Due to paucity of space, we demonstrate graphs only for the first five types of degradations following the order mentioned above. First column of figure 3 shows the efficacy of our proposed scoring scheme. On the y-axis, it plots the percentage of query fingerprints that produce database

Content-Based Matching of Videos

421

Table 1. Query retrieval timings for different distortions Distortion

Feature Time Query Time Verification Time

Spatial Blur Contrast Modification Scaling Frame Letterbox DivX Compression Temporal Cropping Spatial Cropping Camcorder Capture

(a)

0.4984 0.3173 0.4308 0.3886 0.4396 0.3239 0.4874 0.3729

1.0103 0.3229 0.9442 0.9117 1.0622 0.6657 1.2064 0.9975

1.4660 0.9977 1.4236 1.1160 1.5445 0.9350 1.1237 0.7923

(b)

Fig. 2. Comparison of different fingerprinting algorithms (a) Repeatability of keyframes and, (b) Percentage of queries that are correctly retrieved from the database

hits of the correct video, incorrect video or result in no hits at all (misses). On the x-axis, such analysis is repeated for different values of top n scoring bins as they are varied from 10 to 100. It can be noticed that in all cases of distortions, the yaxis distribution remains unchanged even as more bins are sent to the verification stage. In other words, correct database hits always fall within the top 10 scoring bins, even when the query is altered significantly. Examining a scenario when the size of database is increased, number of false database hits generated by a query fingerprint are expected to go up. However, due to invariance of the proposed scoring strategy to the number of database hits, it is established that while correct hits are always scored at the top, false hits fall within lower scoring bins. Such property proves scalability of our proposed fingerprinting scheme. Second column of figure 3 on the y-axis plots the probability of query and database hit being the same (Psame ) or different (Pdif f ). The x-axis shows the corresponding Bhattacharyya distance between their SIFT descriptors. As reflected by the graphs, the point where Psame and Pdif f intersect (i.e., probability of the query and database candidate being identical becomes less than the two being different) is the threshold T . Third column of figure 3 plots the distribution of queries versus the number of key-frames it takes for them to produce a correct hit. Curve for queries with

422

G. Singh et al.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

(o)

Fig. 3. An evaluation of changes in retrieval results as parameters of the system are varied for different query distortions, Row 1: Spatial Blurring, Row 2: Contrast Modification, Row 3: Scaling, Row 4: Frame Letterboxing and Row 5: DivX Compression. First column {(a),(d),(g),(j),(m)} show the percentage of query fingerprints that are correct, incorrect or a miss, for different number n of top scoring bins scanned by the verification stage. Second column {(b),(e),(h),(k),(n)} show the probability of two videos being identical (Psame ) or different (Pdif f ) and Bhattacharyya distance between their SIFT descriptors. Third column {(c),(f),(i),(l),(o)} plot probability of the number of iterations it takes to complete the retrieval process.

Content-Based Matching of Videos

423

higher distortions spans more number of iterations. In our case, even though distortions are stronger than the ones used in literature, a large percentage of queries lead to hits within the first 10 iterations. Table 1 tabulates the retrieval timings for different types of distorted queries. Given a query, the total time it takes for its retrieval can be split into: time taken to detect stable regions in a key-frame and compute their fingerprints (Feature Time), time taken to obtain database hits for each fingerprint and score them accordingly (Query Time) and finally time taken to process top 10 candidates for SIFT-based verification (verification time). All queries have been executed on a 1.6 GHz Pentium IV machine equipped with 1GB RAM and take a maximum of 3 seconds for retrieval under all distortions.

4

Conclusions

In this paper, a new video fingerprinting scheme has been proposed. Primary novelties of this approach are: extraction of repeatable key-frames, introduction of affine-invariant MSVs that are stable over different frame resolutions and, a scoring and polling strategy to merge local database hits and bring the ones belonging to the correct video within the top rank order. With all the desirable properties of a video fingerprinting paradigm, our system can be used as a robust perceptual hashing solution to identify pirated copies of copyright videos on the internet. We are currently working on new ways of making the algorithm foolproof against distortions such as spatial cropping.

References 1. Oostveen, J., Kalker, T., Haitsma, J.: Feature extraction and a database strategy for video fingerprinting. In: International conference on recent advances in visual information systems, pp. 117–128 (2002) 2. Hampapur, A., Bolle, R.: Videogrep: video copy detection using inverted file indices. Technical report, IBM research (2001) 3. Hua, X.-S., Chen, X., Zhang, H.-J.: Robust video signature based on ordinal measure. ICIP 1, 685–688 (2004) 4. Lee, S., Yoo, C.-D.: Video fingerprinting based on centroids of gradient distortions. In: ICASSP, pp. 401–404 (2006) 5. Sivic, J., Zisserman, A.: Video google: A text retrieval approach to object matching in videos. ICCV 2, 1–8 (2003) 6. Matas, J., Chum, O., Martin, U., Pajdla, T.: Robust wide baseline stereo from maximally stable extremal regions. BMVC 1, 384–393 (2002) 7. Nist´er, D., Stew´enius, H.: Scalable recognition with a vocabulary tree. CVPR 2, 2161–2168 (2006) 8. Massoudi, A., Lefebvre, F., Demarty, C.-H., Oisel, L., Chupeau, B.: A video fingerprint based on visual digest and local fingerprints. ICIP, 2297–2300 (2006) 9. Donser, M., Bischof, H.: 3D segmentation by maximally stable volumes (MSVs). ICPR, 63–66 (2006) 10. Lowe, D.: Distinctive image features from scale-invariant keypoints. IJCV 60(2), 91–110 (2004)

Automatic Range Image Registration Using Mixed Integer Linear Programming Shizu Sakakubara1, Yuusuke Kounoike2, Yuji Shinano1, and Ikuko Shimizu1 1

Tokyo Univ. of Agri. & Tech. 2 Canon Inc.

Abstract. A coarse registration method using Mixed Integer Linear Programming (MILP) is described that finds global optimal registration parameter values that are independent of the values of invariant features. We formulate the range image registration problem using MILP. Our algorithm using MILP formulation finds the best balanced optimal registration for robustly aligning two range images with the best balanced accuracy. It adjusts the error tolerance automatically in accordance with the accuracy of the given range image data. Experimental results show that this method of coarse registration is highly effective.

1 Introduction Automatic 3D model generation of real world objects is an important technique in various fields. Range sensors are useful for generating 3D models because they directly measure the 3D shape of objects. The range image measured by a range sensor reflects the partial 3D shape of the object expressed in the coordinate system corresponding to the pose and position of the sensor. Therefore, to obtain the whole shape of the object, many range images measured from different viewpoints must be expressed in a common coordinate system. Estimation of relative poses and positions of sensors from range images is called range image registration. In general, the registration process is divided into two phases: coarse registration and fine registration. For the fine registration, the ICP (Iterative Closest Point) algorithm by Besl and McKay [1] and its extensions [11] are widely used. However, they need sufficiently good initial estimates to achieve fine registration because their error functions have a number of local minima. Therefore, many coarse registration methods have been developed for obtaining good initial estimates automatically. Many of these coarse registration methods [2] match invariant features under rigid transformation. For example, spin images [7] are matched based on their crosscorrelation. A splash [12], a point signature [4], and a spherical attribute image [6] are used as indices representing surface structures in a hash table for matching. For the modeling of buildings, planar regions [5] and circular features [3] are used for matching. These methods are effective if the features can be sufficiently discriminated and their values can be accurately calculated. However, they cannot guarantee global optimality. In addition, registration has been formalized as a discrete optimization task of finding the maximum strict sub-kernel in a graph [10]. While the globally optimal solution can be obtained with this method, the solution depends on the quality of the features. Y. Yagi et al. (Eds.): ACCV 2007, Part II, LNCS 4844, pp. 424–434, 2007. c Springer-Verlag Berlin Heidelberg 2007 

Automatic Range Image Registration Using MILP

425

However, invariant features such as curvatures are difficult to calculate stably because they greatly affected by occlusion and discretization of the object surface. In this paper, we propose a novel coarse registration method using Mixed Integer Linear Programming (MILP) that guarantees the global optimality. It aligns two range images independent of the values of invariant features. Since MILP problems are NPhard problems, they cannot be solved to optimality in a reasonable amount of time when they exceed a certain size. However, progress in algorithms, software, and hardware has dramatically enlarged this size. Twenty years ago, a mainframe computer was needed to solve problems with a hundred integer variables. Now it is possible to solve problems with thousands of integer variables on a personal computer [8]. The advances related to MILP have made our proposed algorithm possible.

2 Definition of Best Balanced Optimal Registration Consider the registration of two sets of points in R3 . Let V 1 = {v11 , v21 , · · · , vn1 1 } be the source point set and V 2 = {v12 , v22 , · · · , vn2 2 } the target one. Assume that these points are corrupted by additive noises and they are independent random variables with zero mean and unknown variance. The task of registration is to estimate parameter values of a rigid transformation T such that the T aligns the two point sets. We define the rigid transformation as T (R, t; v) = Rv + t where R is a 3 × 3 rotation matrix and t is a translation vector. To deal with occlusion and discretization of range images, we introduce a more accurate point set   2 −T (R, t; vi1 )||∞ ≤  , MAPS(T, φ, )= vi1 : ||vφ(i) where the function φ(i) denotes the index of the corresponding target point for each source point and the constant  is a threshold value to remove outliers. To obtain robust estimation of the rigid transformation, the number of elements of MAPS(T, φ, ), |MAPS(T, φ, )|, needs to be maximized. Because no a priori correspondence between the source and target points is given, the function φ must be determined. And, we make no assumption about the variance of the noise. Hence, the  value cannot be estimated in advance. However, if a number N is given, we can find the minimal value of  such that |MAPS(T, φ, )| ≥ N . Therefore, we define the N -accuracy optimal registration as the (TˆN , φˆN ) that minimizes (TˆN , φˆN ) = argmin{ : |MAPS(T, φ, )| ≥ N }, T,φ

and define the optimal registration error ˆN by N -accuracy optimal registration as ˆN = min{ : |MAPS(T, φ, )| ≥ N }. If 1 ≥ 2 , then max |MAPS(T, φ, 1 )| ≥ max |MAPS(T, φ, 2 )|. On the other hand, if N1 ≤ N2 , then min{ : |MAPS(T, φ,)| ≥ N1 } ≤ min{ : |MAPS(T, φ, )| ≥ N2 }. A well-balanced registration with a large number of elements in MAPS(T, φ, ) with a small  value is desired. Therefore, we define best balanced optimal registration ˆ that attains N ˆ such that as the (Tˆ, φ) ˆ = argmax{|MAPS(TˆN , φˆN , max )| : (TˆN , φˆN ) = argmin{ : |MAPS(T, φ, )| ≥ N }}, N 3≤N≤κ

T,φ

(1)

426

S. Sakakubara et al.

where κ is the maximum value for N and max = max{ˆ N }. The lower bound for N is three because a rigid transformation can be estimated using no less than three point pairs. To avoid that max might be too large, we give an upper bound ¯. In best balanced optimal registration, the N -accuracy optimal registration (TˆN , φˆN ) ˆ and for each N from three to κ is evaluated using |MAPS(TˆN , φˆN , max )|. We call N ˆNˆ the best balanced values. They depend on the accuracy of the original point sets. In general, this accuracy is unknown in advance. Our proposed method finds the best balanced optimal registration automatically. We assume that similarity sij between any pair of points vi1 and vj2 is given. Because the feature values of the corresponding points are similar, we assume that point pairs that have explicitly low similarity are not corresponding and remove them from the candidates for corresponding point pairs. Note that the precise values of the features are not needed because feature values are used only to remove obviously noncorresponding pairs.

3 Mixed Integer Linear Programming Based Registration In this section, we will first introduce the general form of Mixed Integer Linear Programming(MILP) problems. Next, we will give an MILP formulation of N -accuracy optimal registration. We also give an ILP formulation to evaluate each N -accuracy registration by counting |MAPS(TˆN , φˆN , max )|. Note that this step is not necessary to be formalized by ILP because it can be done by many other ways. Then, we will describe our algorithm for obtaining the best balanced optimal registration. Finally, we will address the problem size issue and our approach to overcoming it. 3.1 Mixed Integer Linear Programming Problems An MILP problem is the minimization or maximization of a linear objective function subject to linear constraints and to some of the variables being integers. Here, without loss of generality, we consider a maximization problem. Explicitly, the problem has the form (MILP) max c x sub. to Ax ≤ b, l ≤ x ≤ u, xi : integer, i = 1, . . . , p,

where A is an m×n matrix, c, x, l, and u are n-vectors, and b is an m-vector. Elements of c, l, u, b, and A are constant, and x is a variable vector. In this MILP problem, we assume 1 ≤ p ≤ n. If p = n, then the problem is called an Integer Linear Programming(ILP) problem. If all of the variables need not be integers, then the problem is called a Linear Programming(LP) problem. The set S = {x : Ax ≤ b, l ≤ x ≤ u, xi : integer, i = 1, . . . , p} is called the feasible region and an x ∈ S is called a feasible solution. A feasible solution x∗ that satisfies c x∗ ≥ c x, ∀x ∈ S is called an optimal solution. The value of c x∗ is called the optimal value.

Automatic Range Image Registration Using MILP

427

MILP problems are NP-hard in general. Therefore, heuristic methods for solving them are widely used. However, in this paper, we apply exact methods. Owing to recent progress in MILP techniques, relatively large-scale MILP problems can be solved to optimality with commercial solvers. Throughout this paper, the solutions for MILP problems are optimal ones. 3.2 MILP Formulation of the N -Accuracy Optimal Registration MILP formulation of N -accuracy optimal registration requires that the constraints on the rigid transformation be written in a linear form. Since this cannot be done straightforwardly, we introduce pseudo rotation matrix R and pseudo translation vector t and give linear constraints so that R and t are close to R and t. Symbols denoted with prime ( ) are obtained using R and t . def Initially, to simplify the notation, we define an index set for point set V as I(V ) = def

{i : vi ∈ V }, and define ABS(x) = (|x1 |, |x2 |, |x3 |) for a vector x = (x1 , x2 , x3 ) . We introduce a corresponding point pair vector p = (p11 , · · · , p1n2 , p21 , · · · , pn1 1 , · · · , pn1 n2 ) as one of the decision variables. Each element pij is a 0-1 integer variable that is designed to be 1 if source point vi1 corresponds to target point vj2 , that is, j = φ(i), and is designed to be zero otherwise. The other decision variables are the elements of R and t . All elements of these variables are continuous. We denote the ij-th element  of R as rij (i, j = 1, 2, 3) and the i-th element of t as ti (i = 1, 2, 3). We assume that   ≤ r¯ij , and ti ≤ ti ≤ t¯i . There are trivial upper and lower bounds are given: r ij ≤ rij  bounds: −1 ≤ rij ≤ 1, (i, j = 1, 2, 3). As mentioned above, the constraints on the elements of the rotation matrix cannot be straightforwardly written in linear form. Therefore, we formulate this condition indirectly. Let vi1 , vk1 ∈ V 1 and vj2 , vl2 ∈ V 2 and d(vi1 , vk1 ) be the distance between vi1 and vk1 and and d(vj2 , vl2 ) is the distance between vj2 and vl2 . If |d(vi1 , vk1 ) − d(vj2 , vl2 )| > l is valid, then either j = φ(i) or l = φ(k) needs to be satisfied. This condition can be formulated as pij + pkl ≤ 1, (i, k ∈ I(V 1 ), j, l ∈ I(V 2 ), |d(vi1 , vk1 )−d(vj2 , vl2 )| > l ). This condition can be further rewritten with fewer  pkl ≤ 1, (i, k ∈ I(V 1 ), j ∈ I(V 2 )). constraints as pij + 1 )−d(v 2 ,v 2 )|> l∈I(V 2 ),|d(vi1 ,vk l j l

Here, √ l depends on . Since a rigid transformation √ error within ± is allowed, l less than 2 3 should be allowed. Therefore, l ≥ 2 3 needs to be satisfied. 2 We assume one-to-one correspondence between subsets of V 1 and  V . That is, two 1 conditions need to be satisfied: pij ≤ 1, (i ∈ I(V )) and pij ≤ 1, (j ∈ j∈I(V 2 )

i∈I(V 1 )

I(V 2 )). The minimal  number of elements of MAPS(T, φ, ) is given as a constant N .Hence,  pij ≥ N. i∈I(V 1 ) j∈I(V 2 )

For the points in MAPS(T, φ, ), only when pij = 1, ABS(vj2 − R vi1 − t ) ≤ e(i ∈ I(V 1 ), j ∈ I(V 2 )), where e = (, , ) . When pij is set to zero, these conditions need to be eliminated. They can be written by introducing a continuous vector

428

S. Sakakubara et al.

M = (m1 , m2 , m3 ) that is determined so as to always satisfy ABS(vj2 −R vi1 −t ) ≤ M, (i ∈ I(V 1 ), j ∈ I(V 2 )). Using this vector, we can formulate the condition for the points in MAPS(T, φ, ) as ABS(vj2 − R vi1 − t ) ≤ M(1 − pij ) + e, (i ∈ I(V 1 ), j ∈   pij vj2 − R vi1 − t ) ≤ M(1 − pij )+ I(V 2 )) and further rewrite it as ABS( j∈I(V 2 )

j∈I(V 2 )

e, (i ∈ I(V )). If several variables could be fixed in advance, the MILP problem could be solved more quickly. To fix pij to zero in advance, we use the similarity: pij = 0, (i ∈ I(V 1 ) j ∈ I(V 2 ), sij < s ), where s is a given parameter. If similarity sij is small enough, the pair vi1 and vj2 is removed from the putative corresponding point pairs. The objective of N -accuracy optimal registration is to minimize . Therefore, we can give an MILP formulation as follows. 1

(P1 ) min  pij ≤ 1, (i ∈ I(V 1 )),

sub. to j∈I(V 2 )

pij ≤ 1, (j ∈ I(V 2 )), i∈I(V 1 )

pij ≥ N, i∈I(V 1 ) j∈I(V 2 )

pi,j vj2 − R vi1 − t ≥ −M(1 − j∈I(V 2 )

j∈I(V 2 )

pij vj2 j∈I(V



−R

vi1



− t ≤ M(1 −

2)

l∈I(V

pij ) + e, (i ∈ I(V 1 )),

j∈I(V 2 )

pij + r ij

pij ) −e, (i ∈ I(V 1 )),

pkl ≤ 1, (i, k ∈ (I(V 1 ), j ∈ I(V 2 )), 2 ),|d(v1 ,v1 )−d(v2 ,v2 )|> l i j k l

  ≤ rij ≤ r¯ij , (i = 1, 2, 3, j = 1, 2, 3), ti ≤ ti ≤ t¯i ,

(i = 1, 2, 3),

pij = 0 (i ∈ I(V ), j ∈ I(V ), sij < s ), pij ∈ {0, 1} (i ∈ I(V 1 ), j ∈ I(V 2 )). 1

2

Note that l is a given parameter and affects the optimal value of (P1 ). Here, we define Si and ˆi as the feasible region and the optimal value of the (P1 ) with a given parameter li . If l1 ≥ l2 , S1 ⊃ S2 , because the smaller l value forces the more elements of p to be zero, that is, it restricts the feasible region. Therefore, if l1 ≥ l2 , ˆ1 ≤ ˆ 2 . ˆN , max )| 3.3 ILP Formulation to Count |MAPS(TˆN , φ In this section, we describe ILP problem for evaluating (TˆN , φˆN ) by counting |MAPS(TˆN , φˆN , max )|. Its formulation uses almost the same components described in the previous section, except for the following ones. – The rigid transformation is given. In the formulation in this subsection, rotation matrix R and translation vector t with constant elements are used. They are calculated using a corresponding point pair vector introduced by R and t in our algorithm. – max is not a variable but a constant. – N is not given: the number of correspondences is maximized in this formulation.

Automatic Range Image Registration Using MILP

429

We can give an ILP formulation for counting |MAPS(TˆN , φˆN , max )| as follows using emax = (max , max , max ) . (P2 ) max

pij i∈I(V 1 ) j∈I(V 2 )

sub. to

pij ≤ 1, (i ∈ I(V 1 )), j∈I(V 2 )

pij ≤ 1, (j ∈ I(V 2 )), i∈I(V 1 )

pi,j vj2

−

pij vj2

−

Rvi1

− t ≥ −M(1 −

j∈I(V 2 )

j∈I(V

pij ) − emax (i ∈ I(V 1 )),

j∈I(V 2 )

Rvi1

− t ≤ M(1 −

2)

j∈I(V

pij ) + emax (i ∈ I(V 1 )), 2)

pij = 0 (i ∈ I(V 1 ), j ∈ I(V 2 ), sij < s ),

pij ∈ {0, 1} (i ∈ I(V 1 ), j ∈ I(V 2 )).

3.4 Algorithm for Best Balanced Optimal Registration Now, we describe our algorithm for obtaining the parameter values for the best balanced optimal registration. Equation (1) is solved in two phases. ˆ N . The p ˆ N that attains Phase 1. For N from 5 to κ, find ˆN and obtain (TˆN , φˆN ) as p  ˆN ≤ ¯ is stored in list L. The minimum N value is set to five to avoid trivial solutions. ˆ N and count Phase 2. For each (ˆ pN , ˆN ) ∈ L, make (TˆN , φˆN ) from p    ˆ N : (ˆ pN , ˆN ) ∈ L}. Then, select the N |MAPS(TˆN , φˆN , max )|, max = max{ˆ   ˆ ˆ that attains argmax{|MAPS(TN , φN , max )| : N such that(ˆ pN , ˆN ) ∈ L}. N

In phase 1, To solve the problem P1 , an l needs to be given. However, the value of l affects the optimal value of the problem. The minimum ˆN value is found by solving P1 repeatedly by narrowing its range. ˆ by We find the corresponding point pair vector that attains the best balanced value N a two-phase algorithm as follows. ) [Algorithm] MILP-based Registration(¯ L = ∅; /* Phase 1 */ for(N = 5; N ≤ κ; N ++){ N = ¯; /* N : upper bound for ˆN */ N = 0.0; /* N : lower bound for ˆ N */ N = ¯; while(N > N ){ /* Narrowing process to find ˆ N */ √ l = 2 3N ; Solve problem (P1 ) with given parameters N and l ; ˆ N be the optimal solution and ˆ /* Let p N be the optimal value of problem (P1 ) */ if ( No feasible solution is found ) {break;} if ( ˆN < N ){ if ( ˆN > N ){ N = ˆ N ; } N = N ;

430

S. Sakakubara et al. }else{ if ( ˆN > ¯) {break; } if ( ˆN < N ){ N = ˆ N ; } N = N ; } N = (N − N )/2 + N ;

} L = L ∪ {(ˆ pN , ˆN )} ;

} if(L = ∅){return “No solution”; /* ¯ value is too small */ } /* Phase 2 */ ˆ = 0; max = max{ˆ N : (ˆ pN , ˆ N ) ∈ L}; N while(L = ∅){ Extract (ˆ pN , ˆN ) from L; /* L = L\{(ˆ pN , ˆ N )} */  ˆ ˆ ˆ N and count |MAPS(TˆN , φˆN , max )| by solving problem (P2 ) with Make (TN , φN ) from p a given parameter max ; /* Let Ntp = |MAPS(TˆN , φˆN , max )| */ ˆ = Ntp ; p ˆ < Ntp ){ N ˆ Nˆ = p ˆ N ;} if(N } ˆ, p ˆ Nˆ ); return (N

This algorithm has three particular advantages. – Although ¯  is initially given, l is adjusted to an appropriate value in accordance with the accuracy of the point sets automatically. – If the value of ¯ is too small to find a solution, it outputs “No solution.” If ¯ is too big, it outputs a feasible solution at the cost of time. – Because the optimal values of (P1 ) are used, the narrowing process is completed faster than that in a binary search. 3.5 Problem Size Issue and How to Overcome It Unfortunately, the MILP solver and computers still do not have enough power to solve the MILP problem (P1 ) for all points sets obtained from normal size range images. Therefore, we preselect the feature points from V 1 and V 2 . Since our algorithm evaluates only the distances between corresponding point pairs, it is robust against noise and the preselection method. When the algorithm is applied to the preselected point sets V˙ 1 ⊂ V 1 and V˙ 2 ⊂ V 2 , the robustness of the registration can be improved to solve the ILP problem (P2 ) with V˙ 1 and V 2 . The current MILP solver and computers have enough power to solve this size of problem (P2 ) within a reasonable time.

4 Experiments We tested the robustness of our method by applying it to three synthetic datasets and one real dataset. We used the ILOG CPLEX (ver.10.1) MILP solver installed on a PC

Automatic Range Image Registration Using MILP

431

with a Pentium D 950 CPU (3.4 GHz), 2 GB of RAM, and linux 2.6. We used GNU GLPK to generate files for our formalization. We use the algorithm of Umeyama [13] to estimate the rigid transformation from the point correspondences. We used 3D models of “Stanford Bunny”[14], “Horse” [16], and “Armadillo”[14] to generate synthetic range images. We generated 18 synthetic range images for each model with the size 200 × 200 pixels by rotating 20-degree rotation steps around the Y axis. Then, to generate five noisy datasets for each model, the Z coordinate of each point was perturbed by adding Gaussian noise with zero mean and a standard deviation of σ = 0.02, 0.04, 0.06, 0.08, or 1.00. We also applied our method to the real range images of “Pooh” [15]. The selected feature points had curvedness [9] values that were the maximal with a constraint on thedistance between feature points. The curvedness c of a point is calculated by c = κ21 + κ22 /2 where κ1 and κ2 are two main curvatures of the point. The number of feature points for the each range image was 50 for “Stanford Bunny” and “Horse”, and 70 for “Armadillo” and “Pooh”. The similarity sij between feature points vi1 and vj2 was defined using curvatures values of the local surfaces around the points. It is set to −100 if the shape of the surfaces, such as convex or concave, was not identical with each other, or else it is calculated based on the curvedness of points ci and cj as sij = 1/ ci − cj . We applied our method to all adjacent pairs in the range image sequences. For all  = 1 (i, j = 1, 2, 3). For the synthetic datasets, κ = 10, s = 10, rij = −1, and r¯ij   images, ¯ = 0.15, M = (100, 100, 100) , ti = −10, and t¯i = 10 (i = 1, 2, 3). We also used ¯ = 0.25 for the pairs whose solutions could not be obtained by ¯ = 0.15. For the real images, ¯ = 0.25, M = (1000, 1000, 1000), ti = −100, and t¯i = 100 (i = 1, 2, 3). We also applied ¯  = 0.50 for the pairs whose solutions could not be obtained by ¯  = 0.25. In order to improve accuracy of calculation, each points vi = (xi , yi , zi ) of “Pooh” dataset are translated near the origin of the coordinates ◦ ◦ by following: vir − ((min{xj : j ∈ I(V 0 )} + max{xj : j ∈ I(V 0 )})/2, (min{yj : ◦ ◦ ◦ j ∈ I(V 0 )} + max{yj : j ∈ I(V 0 )})/2, (min{zj : j ∈ I(V 0 )} + max{zj : j ∈ ◦ ◦ I(V 0 )})/2) , where V 0 are point set of 0◦ . The experimental results for the synthetic range image datasets are shown in Table 1( “b”, “h”, and “a” in the column “Dataset” indicate “Stanford Bunny”, “Horse” and “Armadillo”, respectively, followed by the numbers indicating the standard deviation of the added noise). The parameter ¯ = 0.25 was used to calculate the results for pairs 180◦ –200◦ of h00, 180◦ –200◦ and 280◦ –300◦ of h04, 180◦ –200◦ of h06, 280◦ –300◦ of h08, 20◦ –40◦ and 280◦ –300◦ of h10, and 60◦ –80◦ of all “Armadillo” datasets, and ¯ = 0.15 for the other pairs. Table 1 shows that the rigid transformation errors did not always increase with the standard deviation σ of the added Gaussian noise. There are two main reasons that our method is robust against the measurement noise: First, our method does not need accurate values for the invariant features. We use them to only select the feature points and to reduce the putative corresponding point pairs. Second, the rigid transformation is estimated using more than five corresponding pairs of feature

432

S. Sakakubara et al.

Table 1. Error evaluation of MILP-based registration. “Error of angle” is the absolute angle between the true rotation angle and the estimated one in degrees. “Error of axis” is the deviation in the estimated rotation axis in degrees. “Error of translation” is the norm of the error of the translation vector. Dataset σ

Time [sec.]

b00 b02 b04 b06 b08 b10 h00 h02 h04 h06 h08 h10

0.00 0.02 0.04 0.06 0.08 0.10 0.00 0.02 0.04 0.06 0.08 0.10

4578 4587 4906 5406 4700 4877 5334 5213 4273 4264 3202 3525

Error Error Error of Dataset σ of angle of axis translation [degree]

[degree]

0.343 0.321 0.466 0.247 0.366 0.421 0.281 0.272 0.354 0.364 0.417 0.344

1.89 1.74 2.00 1.30 1.67 1.47 2.14 1.94 1.97 2.19 2.55 3.04

Time [sec.]

0.328 0.311 0.390 0.298 0.364 0.291 0.587 0.552 0.507 0.485 0.625 0.592

a00 a02 a04 a06 a08 a10

0.00 119514 0.02 118699 0.04 124356 0.06 127393 0.08 118821 0.10 109337

Error Error Error of of angle of axis translation [degree]

[degree]

0.215 0.231 0.205 0.320 0.244 0.393

0.83 0.88 0.95 0.94 0.84 0.96

0.244 0.227 0.224 0.282 0.219 0.262

Table 2. Error evaluation of data set “Pooh”. “Error of angle” is the absolute angle between the true rotation angle and the estimated one in degrees. “Error of axis” is the angle in degrees between the rotation axis estimated from the given pair and the rotation axis estimated from all pairs. “Error of translation” is the norm of the error between the translation vector estimated from the given pair and the translation vector estimated from all pairs. Angle Time[sec.] Error of angle[degree] Error of axis[degree] Error of translation average 19496 1.950 0.072 0.719

points. If we can find only five pairs of points that are not so noisy by chance, we can accurately estimate rigid transformation. The computational time for “Armadillo” was higher than that of others because the numbers of MILP variables and constraints are proportional to the number of feature points. Table 2 shows the results for the real range image dataset of “Pooh”. The parameter ¯ = 0.50 was used to calculate the results for pairs 20◦ -40◦ , 60◦ –80◦ , 100◦ –120◦, 220◦ –240◦, and 320◦ –140◦, and ¯ = 0.25 was used for the other pairs. The registration results for “Pooh” are shown in Figure 1. While the errors for some pairs were relatively large, the results are good enough for coarse registration.

Fig. 1. Results for “Pooh”

Automatic Range Image Registration Using MILP

433

5 Concluding Remarks Our proposed coarse registration method using Mixed Integer Linear Programming (MILP) can find global optimal registration without using the values of the invariant features. In addition, it automatically adjusts the error tolerance depending on the accuracy of the given range image data. Our method finds the best consistent pairs from all possible point pairs using an MILP solver. While such solvers are powerful tools, all of the constraints should be written in linear form. This means that constraints on the rotation matrix cannot be applied directly. Therefore, we selected a relevant number of consistent pairs in the sense of the distances between point pairs, which gives the constraints on the rotation matrix indirectly. The number of corresponding point pairs and the distances between them are automatically balanced by our algorithm using two different MILP formulations. Future work will focus on reducing the computational time and improving the selection of the feature points.

References 1. Besl, P.J., McKay, N.D.: A Method for Registration of 3-D Shapes. IEEE Trans. on PAMI 14(2), 239–256 (1992) 2. Campbell, R.J., Flynn, P.J.: A Survey of Free-Form Object Representation and Recognition Techniques. CVIU 81, 166–210 (2001) 3. Chen, C.C., Stamos, I.: Range Image Registration Based on Circular Features. In: Proc. 3DPVT, pp. 447–454 (2006) 4. Chua, C.S., Jarvis, R.: 3D Free-Form Surface Registration and Object Recognition. IJCV 17(1), 77–99 (1996) 5. He, W., Ma, W., Zha, H.: Automatic Registration of Range Images Based on Correspondence of Complete Plane Patches. In: Proc. 3DIM, pp. 470–475 (2005) 6. Higuchi, K., Hebert, M., Ikeuchi, K.: Building 3-D Models from Unregistered Range Images. GMIP 57(4), 315–333 (1995) 7. Johnson, A.E., Hebert, M.: Using Spin Images for Efficient Object Recognition in Cluttered 3D Scenes. IEEE Trans. on PAMI 21(5), 433–449 (1999) 8. Johnson, E.L., Nemhauser, G.L., Savelsbergh, M.W.P.: Progress in Linear ProgrammingBased Algorithms for Integer Programming: An Exposition. INFORMS Journal on Computing 12(1), 2–23 (2000) 9. Koenderink, J.J.: Solid Shape. MIT Press, Cambridge (1990) ˇ ara, R., Okatahi, I.S., Sugimoto, A.: Globally Convergent Range Image Registration by 10. S´ Graph Kernel Algorithm. In: Proc. 3DIM, pp. 377–384 (2005) 11. Rusinkiewicz, S., Levoy, M.: Efficient Variants of the ICP Algorithm. In: Proc. 3DIM, pp. 145–152 (2001) 12. Stein, F., Medioni, G.: Structural indexing: Efficient 3-D object recognition. IEEE Trans. on PAMI 14(2), 125–145 (1992) 13. Umeyama, S.: Least-Square Estimation of Transformation Parameters Between Two Point Patterns. IEEE Trans. on PAMI 13(4), 376–380 (1991)

434

S. Sakakubara et al.

14. Stanford 3D Scanning Repository, http://www-graphics.stanford.edu/data/3Dscanrep/ 15. The Ohio State University Range Image Repository, http://sampl.ece.ohio-state.edu/data/3DDB/RID/minolta/ 16. Georgia Institute of Technology Large Geometric Models Archive, http://www-static.cc.gatech.edu/projects/large models/

Accelerating Pattern Matching or How Much Can You Slide? Ofir Pele and Michael Werman School of Computer Science and Engineering The Hebrew University of Jerusalem {ofirpele,werman}@cs.huji.ac.il

Abstract. This paper describes a method that accelerates pattern matching. The distance between a pattern and a window is usually close to the distance of the pattern to the adjacement windows due to image smoothness. We show how to exploit this fact to reduce the running time of pattern matching by adaptively sliding the window often by more than one pixel. The decision how much we can slide is based on a novel rank we define for each feature in the pattern. Implemented on a Pentium 4 3GHz processor, detection of a pattern with 7569 pixels in a 640 × 480 pixel image requires only 3.4ms.

1

Introduction

Many applications in image processing and computer vision require finding a particular pattern in an image, pattern matching. To be useful in practice, pattern matching methods must be automatic, generic, fast and robust.

(a)

(b)

(c)

(d)

Fig. 1. (a) A non-rectangular pattern of 7569 pixels (631 edge pixel pairs). Pixels not belonging to the mask are in black. (b) A 640 × 480 pixel image in which the pattern was sought. (c) The result image. All similar masked windows are marked in white. (d) The two found occurrences of the pattern in the image. Pixels not belonging to the mask are in black. The method suggested in this paper reduced the Pele and Werman pattern matching method[1] running time from 21ms to only 3.4ms. Y. Yagi et al. (Eds.): ACCV 2007, Part II, LNCS 4844, pp. 435–446, 2007. c Springer-Verlag Berlin Heidelberg 2007 

436

O. Pele and M. Werman

(a)

(b)

(c)

(d)

Fig. 2. (a) A non-rectangular pattern of 2197 pixels. Pixels not belonging to the mask are in black. (b) Three 640x480 pixel frames out of fourteen in which the pattern was sought. (c) The result. Most similar masked windows are marked in white. (d) Zoom in of the occurrences of the pattern in the frames. Pixels not belonging to the mask are in black. The method suggested in this paper reduced the Pele and Werman pattern matching method[1] running time from 22ms to only 7.2ms. The average number of samples per window reduced from 19.7 to only 10.6.

Pattern matching is typically performed by scanning the entire image, and evaluating a distance measure between the pattern and a local rectangular window. The method proposed in this paper is applicable to any pattern shape, even a non-contiguous one. We use the notion of “window” to cover all possible shapes. There are two main approaches to reducing the computational complexity of pattern matching. The first approach reduces the time spent on each window. The second approach reduces the number of windows visited. In this work we concentrate on the second approach. We suggest sliding more than one pixel at a time. The question that arises is: how much can you slide? The answer depends on the pattern and on the image. For example, if the pattern and the image are black and white checked boards of pixels, the distance of the pattern to the current window and to the next window will be totally different. However, if the pattern is piecewise smooth, the

Accelerating Pattern Matching or How Much Can You Slide?

437

(a-zoom) (a)

(b)

(c)

(d)

Fig. 3. (a) A rectangular pattern of 1089 pixels. (b) A noisy version of the original 640x480 pixel image. The pattern that was taken from the original image was sought in this image. The noise is Gaussian with a mean of zero and a standard deviation of 25.5. (c) The result image. The single similar masked window is marked in white. (d) The occurrence of the pattern in the zoomed in image. The method suggested in this paper reduced the Pele and Werman pattern matching method[1] running time from 19ms to only 6ms. The average number of samples per window reduced from 12.07 to only 2. The image is copyright by Ben Schumin and was downloaded from: http://en. wikipedia.org/wiki/Image:July 4 crowd at Vienna Metro station.jpg.

distances will be similar. We describe a method which examines the pattern and decides how much we can slide in each step. The decision is based on a novel rank we define for each feature in the pattern. We use a two stage method on each window. First, we test all the features with a high rank. Most of the windows will not pass and we will be able to slide more than one pixel. For the windows that passed the test we perform the simple test on all the features. A typical pattern matching task is shown in Fig. 1. A non-rectangular pattern of 7569 pixels (631 edge pixel pairs) was sought in a 640 × 480 pixel image. Using the Pele and Werman method[1] the running time was 21ms. Using our method the running time reduced to only 3.4ms. All runs were done on a Pentium 4 3GHz processor. Decreasing the number of visited windows is usually achieved using an image pyramid[3]. By matching a coarser pattern to a coarser level of the pyramid, fewer windows are visited. Once the strength of each coarser resolution match is calculated, only those that exceed some threshold need to be compared for the next finer resolution. This process proceeds until the finest resolution is reached. There are several problems with the pyramid approach. First, important details of the objects can disappear. Thus, the pattern can be missed. For example, in Fig. 1 if we reduce the resolution to a factor of 0.8, the right occurrence of the pattern is found, but the left one is missed. Using the smaller images the running time decreases from 21ms to 18ms (without taking into account the time spent on decreasing the resolution). Using our approach, both occurrences of the patterns are found in only 3.4ms. Note that smoothness can change

438

O. Pele and M. Werman

(a)

(b)

(a-zoom)

(c)

(d)

Fig. 4. (a) A non-rectangular pattern of 3732 pixels (3303 edge pixel pairs). Pixels not belonging to the mask are in black. (b) A 2048 × 1536 pixel image in which the pattern was sought. The area where the pattern was found is marked in white. (c) The occurrence of the pattern in the image zoomed in. (d) The occurrence of the pattern in the image zoomed in, with the exact found outline of the pattern painted in white. The method suggested in this paper reduced the Pele and Werman pattern matching method[1] running time from 437ms to only 51ms. Note the large size of the image. The average number of samples per window reduced from 27 to only 3.7.

between different local parts of the pattern. The pyramid approach is global, while our approach is local and thus more distinctive. The second problem of pyramid approach is the memory overhead. This paper is organized as follows. Section 2 presents the LUp rank for pixels and for pairs of pixels. Section 3 describes a method that uses the LUp rank for accelerating pattern matching. Section 4 presents extensive experimental results. Finally, conclusions are drawn in Section 5.

2

The LUp Rank

In this section we define a novel smoothness rank for features, the LUp rank. The rank is later used as a measure that tells us how much we can slide for each

Accelerating Pattern Matching or How Much Can You Slide?

439

(a)

(b) Fig. 5. (a) A 115 × 160 pattern (2896 edge pixel pairs). (b) A 1000 × 700 pixel image in which the pattern was sought. The most similar window is marked in white. The method suggested in this paper reduced the Pele and Werman pattern matching method[1] running time from 51ms to only 9.2ms. The average number of samples per window reduced from 23 to only 3. The images are from the Mikolajczyk and Schmid paper[2].

pattern. This is first defined for pixels and then for pairs of pixels. Finally, we suggest ways of calculating the LUp rank. 2.1

The LUp Rank for Pixels

In this sub-section we use the Thresholded Absolute Difference Hamming distance that was suggested by Pele and Werman[1]. This distance is the number of different corresponding pixels between a window and a pattern, where the corresponding pixels are defined as different if and only if their absolute intensity difference is greater than a predefined pixel similarity threshold, q; i.e. The distance between the set of pixels, A, applied to the pattern and the current window is defined as (δ returns 1 for true and 0 for false):  δ (|pattern(x, y) − window(x, y)| > q) (1) T ADA (pattern, window) = (x,y)∈A

We first define the LU rank for a pattern pixel as: LU(pattern, (x, y)) = max s.t: R

∀ 0 ≤ rx , ry ≤ R pattern(x, y) = pattern(x − rx , y − ry )

(2)

Now, if we if we assess the similarity between a pixel in the pattern with an LU rank of R, to a pixel in the window, we get information about all the windows which are up to R pixels to the right and down to the current window. Using this information we can slide in steps of R + 1 pixels, without losing accuracy.

440

O. Pele and M. Werman

The requirement for equality in Eq. 2 is relaxed in the definition of the LUp rank. In this rank the only requirement is that the absolute difference is not too high: LU p (pattern, (x, y)) = max s.t: R

∀ 0 ≤ rx , ry ≤ R |pattern(x, y) − pattern(x − rx , y − ry )| ≤ p

(3)

Note that the LU and LU0 ranks for pixels are equivalent. 2.2

The LUp Rank for Pairs of Pixels

In this sub-section we use the Monotonic Relations Hamming distance that was suggested by Pele and Werman[1]. This distance is the number of pairs of pixels in the current window that does not have the same relationship as in the pattern; i.e. the basic features of this distance are pairs of pixels and not pixels. Pixel relations have been successfully applied in many fields such as pattern matching[1], visual correspondence[4] and keypoint recognition[5]. Each pattern is defined by a set of pairs of pixels which are close, while the intensity difference is high. We assume without loss of generality that in the pattern the first pixel in each pair has a higher intensity value than the second pixel. The distance between the set of pairs, A, applied to the pattern and the current window is defined as (δ returns 1 for true and 0 for false):  δ (window(x1 , y1 ) ≤ window(x2 , y2 )) (4) M RA (pattern, window) = [(x1 ,y1 ), (x2 ,y2 )]∈A

Given a pattern and pair of pixels, [(x1 , y1 ), (x2 , y2 )] such that the first pixel has a higher intensity value than the second pixel, i.e. pattern(x1 , y1 ) > pattern(x2 , y2 ), we define the pair’s LUp rank as: LU p (pattern, [(x1 , y1 ), (x2 , y2 )]) = max s.t: ∀ 0 ≤ rx , ry ≤ R R

pattern(x1 − rx , y1 − ry ) > pattern(x2 − rx , y2 − ry ) + p

(5)

The requirement that the relation must be bigger in at least p is added for stability. Now, if we assess the similarity between a pair of pixels in the pattern with an LUp rank of R, to a pair of pixels in the window, we get information about all the windows which are up to R pixels to the right and down to the current window. Figure 6 illustrates this. Using this information we can slide in steps of R + 1 pixels, without losing accuracy. 2.3

How to Calculate the LUp Rank

We suggest two methods of calculating the LUp rank of all features (pixels or pairs of pixels) in the pattern. The first is to calculate the rank for each feature. ¯ the average LUp rank and by |A| the feature set size, then If we denote by R

Accelerating Pattern Matching or How Much Can You Slide? Image 4

5

6

7

8

2

3

4

5

6

7

8

9

6

7

8

9

2 3 4 5

4

90

1

1

0 60

11

6

6

5

3

40

0

9 0

3

1

0

2

2

0

1

3

40

0

3

4

0

20

2

2

1

1

Pattern 0

441

(b)

(a) 2

3

4

5

6

7

8

0

9

1

2

3

4

5

1 2 3 4 5 6

6

5

4

3

2

1

0

1

0

0

(c)

(d)

Fig. 6. The pair of pixels in the pattern (marked with two circles): [(3, 4), (1, 1)], has LU10 rank of 1 (Pattern(3, 4) > Pattern(1, 1) + 10 and Pattern(3, 3) > Pattern(1, 0) + 10), etc). Thus, when we test whether Image(3, 4) > Image(1, 1), we get an answer to these 4 questions (all coordinates are relative to the window’s coordinates): 1. In the window of (a), is Window(3, 4) > Window(1, 1) as in the pattern? 2. In the window of (b), is Window(2, 4) > Window(0, 1) as in the pattern? 3. In the window of (c), is Window(3, 3) > Window(1, 0) as in the pattern? 4. In the window of (d), is Window(2, 2) > Window(0, 0) as in the pattern?

¯ 2 ). The second method is to test which the average time complexity is O(|A|R features have each LUp rank. This can be done quickly by finding the 2d min and max for each value of R. The Gil and Werman[6] method does this with a time complexity of O(1) per pixel. If we denote by Rmax the maximum R value, then the time complexity is O(|A|Rmax ). A combined approach can also be used. Note that the computation of the LUp is done offline for each given pattern. Moreover, the size of the pattern is usually much smaller than the size of the image; thus the running time of this stage is negligible. In this paper we simply calculate the LUp rank for each feature.

3

The Pattern Matching Method

The problem of pattern matching can be formulated as follows: given a pattern and an image, find all the occurrences of the pattern in the image. We define a window as a match, if the Hamming distance (i.e. Eq. 1 or Eq. 4) is smaller or equal to the image similarity threshold. In order to reduce the running time spent on each window we use the Pele and Werman[1] sequential sampling algorithm. The sequential algorithm random samples corresponding features sequentially and without replacement from the

442

O. Pele and M. Werman

window and pattern and tests them for similarity. After each sample, the algorithm tests whether the accumulated number of non-similar features is equal to a threshold, which increases with the number of samples. We call this vector of thresholds the rejection line. If the algorithm touches the rejection line, it stops and returns non-similar. If the algorithm finishes sampling all the features, it has computed the exact distance between the pattern and the window. Pele and Werman[1] presented a Bayesian framework for sequential hypothesis testing on finite populations. Given an allowable bound on the probability of a false negative the framework computes the optimal rejection line; i.e. a rejection line such that the sequential algorithm parameterized with it has the minimum expected running time. Pele and Werman[1] also presented a fast near-optimal framework for computing the rejection line. In this paper, we use the near-optimal framework. The full system we use for pattern matching is composed of an offline and an online part. The offline part gets a pattern and returns the characteristic LUp rank, two sets of features and the two corresponding rejection lines. One set contains all the pattern features. The second set contains all the pattern features from the first set that have an LUp rank greater or equal to the characteristic LUp rank. The online part slides through the image in steps of the characteristic LUp rank plus one. On each window it uses the sequential algorithm to test for similarity on the second set of features. If the sequential algorithm returns nonsimilar, the algorithm slides the characteristic LUp rank plus one pixels right or the characteristic LUp rank plus one rows (at the end of each row). If the sequential algorithm returns similar (which we assume is a rare event), the window and all the windows that would otherwise be skipped are tested for similarity. The test is made again using the sequential algorithm, this time on the set that contains all the pattern features.

4

Results

The proposed method was tested on real images and patterns. The results show that the method accelerates pattern matching, with a very small decrease in robustness to rotations. For all other transformations tested - small scale change, image blur, JPEG compression and illumination - there was no decrease in robustness. First we describe results that were obtained using the Thresholded Absolute Difference Hamming distance (see Eq. 1). Second, we describe results that were obtained using the Monotonic Relations Hamming distance (see Eq. 4). 4.1

Results Using the Thresholded Absolute Difference Hamming Distance

We searched for windows with a Thresholded Absolute Difference Hamming distance lower than 0.4×|A|. The sequential algorithm was parameterized using the near-optimal method of Pele and Werman[1] with input of a uniform prior and

Accelerating Pattern Matching or How Much Can You Slide?

443

a false negative error bound of 0.1%. In all of the experiments, the p threshold for the LUp rank was set to 5. The characteristic LU5 rank for each pattern was set to the maximum LU5 rank found for at least 30 pattern pixels. Note that the online part first tests similarity on the set of pixels with a LU5 rank greater or equal to the characteristic LU5 rank. The same relative similarity threshold is used; i.e. if the size of this small set is |As | we test whether the Thresholded Absolute Difference Hamming distance is lower than 0.4 × |As |. Results that show the substantial reduction in running time are shown in Figs. 2 and 3. 4.2

Results Using the Monotonic Relations Hamming Distance

The pairs that were used in the set of each pattern were pairs of pixels belonging to edges, i.e. pixels that had a neighbor pixel, where the absolute intensity value difference was greater than 80. Two pixels, (x2 , y2 ), (x1 , y1 ) are considered neighbors if their l∞ distance: max(|x1 − x2 |, |y1 − y2 |) is smaller or equal to 2. We searched for windows with a Monotonic Relations Hamming distance lower than 0.25 × |A|. The sequential algorithm was parameterized using the nearoptimal method of Pele and Werman[1] with input of a uniform prior and a false negative error bound of 0.1%. In all of the experiments, the p threshold for the LUp rank was set to 20. The characteristic LU20 rank for each pattern was set to the maximum LU20 rank found for at least 30 pairs of pixels from the set of all edge pixel pairs. Note that the online part first tests similarity on the set of pairs of edge pixels with a LU20 rank greater or equal to the characteristic LU20 rank. The same relative similarity threshold is used; i.e. if the size of this small set is |As | we test whether the Monotonic Relations Hamming distance is lower than 0.25 × |As |. Results that show the substantial reduction in running time are shown in Figs. 1, 4 and 5. To illustrate the performance of our method, we ran the tests that were also conducted in the Pele and Werman paper[1]. All the data for the experiments were downloaded from http://www.cs.huji.ac.il/~ofirpele/hs/ all images.zip. Five image transformations were evaluated: small rotation; small scale change; image blur; JPEG compression; and illumination. The names of the datasets used are rotation; scale; blur ; jpeg; and light respectively. The blur, jpeg and light datasets were from the Mikolajczyk and Schmid paper[2]. scale dataset contains 22 images with an artificial scale change from 0.9 to 1.1 in jumps of 0.01; and rotation dataset contains 22 images with an artificial inplane rotation from -10◦ to 10◦ in jumps of 1◦ . For each collection, there were ten rectangular patterns that were chosen from the image with no transformation. In each image we considered only the window with the minimum distance as similar, because we knew that the pattern occurred only once in the image. We repeated each search of a pattern in an image 1000 times. There are two notions of error: miss detection error rate and false detection error rate. As we know the true homographies between the images, we know where the pattern pixels are in the transformed image. We denote a correct match as one that covers at least 80% of the transformed pattern pixels. A false match is one that covers less than 80% of the transformed pattern pixels. Note

444

O. Pele and M. Werman <11%

100%

Miss Detection Rate

<1%

Our Method 10.9

Original Method 1 2 3 4 5 6 7 8 9 10

pattern number

pattern number

1 2 3 4 5 6 7 8 9 10

0.7

0.1

-10◦ -8◦ -6◦ -4◦ -2◦

2◦

0◦

rotation

4◦

6◦

0.3 0.5

-10◦ -8◦ -6◦ -4◦ -2◦

0◦

1 0.4 2 3 4 5 6 7 8 9 0.2 10 0.9 0.92 0.94 0.96 0.98

1

8◦ 10◦

2◦

rotation

4◦

6◦

8◦ 10◦

pattern number

pattern number

1 4.2 2 3 4 5 6 7 8 9 10 0.9 0.92 0.94 0.96 0.98

0%

1

1.02 1.04 1.06 1.08 1.1

scale

1.02 1.04 1.06 1.08 1.1

scale

Fig. 7. The miss detection error rates of our method on the rotation and scale tests. There is a slight decrease in performance with our method on the rotation test. On the scale test the performance is the same for both methods. Note that our approach runs much faster and that on all other tests (light, jpeg and blur ) the miss detections error rates were exactly the same.

that there is also an event of no detection at all if our method does not find any window with a Monotonic Relations Hamming distance lower than 0.25 × |A|. The miss detection error rate is the percentage of searches of a pattern in an image that does not yield a correct match. The false detection error rate is the percentage of searches of a pattern in an image that yields a false match. The detection and miss detection error rates were the same as in the Pele and Werman method[1], except in the rotation test where there was a slight decrease in performance (see Fig. 7). In the light and jpeg tests, the performance was perfect; i.e. 0% miss detection rate and 0% false detection rate. In the blur test, only one pattern was not found correctly in the most blurred image. The miss detection rate and false detection rate for this specific case was 99.6%. In all other patterns and images in the blur test, the miss detection rate and false detection rate was 0%. In the scale test, there was only one pattern with false detection in two images with scale 0.9 and 0.91. In the rotation test, there was only one pattern with false detection in images with rotation smaller than -2◦ or larger than +2◦ . Miss detection rates in the scale and rotation tests (see Fig. 7) were dependent on the pattern. If the scale change or rotation was not too big, the pattern was found correctly.

Accelerating Pattern Matching or How Much Can You Slide?

1 2 3 4 5 6 7 8 9 10

#avg samples

40

30

20

35

1 2 3 4 5 6 7 8 9 10

30

#avg samples

50

445

25 20 15 10

10 5 0 0

1

2

3

4

0 0

5

(a) darkness level

3

4

5

25 20 15 10 5

35

1 2 3 4 5 6 7 8 9 10

30

#avg samples

1 2 3 4 5 6 7 8 9 10

30

#avg samples

2

(b) blur level

35

25 20 15 10 5

0 0

1

2

3

4

0 -10◦-8◦ -6◦ -4◦ -2◦ 0◦ 2◦ 4◦ 6◦ 8◦ 10◦

5

(d) rotation

(c) JPEG compression level 35

1 2 3 4 5 6 7 8 9 10

30

#avg samples

1

25 20 15

Original method Our method

10 5 0 0.9

0.94

1

1.06

1.1

(e) scale

Fig. 8. Average number of samples per window for each of the ten patterns with our method (dotted lines) and with the Pele and Werman method[1] (solid lines). In all of the tests our approach ran much faster. Note that as our approach skips windows, the average of samples per window can be even smaller than one. It is also noteworthy that the running time in our approach depends on the characteristic LU20 rank. For example, in the light test - (a), finding pattern number 6 (marked as a pink star) using the original method took the most time, while using our method it is one of the patterns that was found the fastest. This can be explained by the fact that this pattern has a characteristic LU20 rank of two, which is high compared to the other patterns.

446

O. Pele and M. Werman

We also measured the average samples taken for each window using our method and the Pele and Werman method (see Fig. 8). In all of the tests our approach ran much faster. Note that as our approach skips windows, the average of samples per window can be even smaller than one. Further, the running time in our approach depends on the characteristic LU20 rank.

5

Conclusions

This paper described a method to accelerate pattern matching by adaptively sliding the window often by more than one pixel. We assigned a novel rank to pixels and edge pixel pairs that tells us how many pixels we can slide through the image. We suggested a pattern matching method that uses this rank. Extensive testing showed that the pattern matching was accelerated without losing almost any accuracy. Faster than real time results were presented, where patterns under large illumination changes, blur, occlusion, gaussian noise, etc, were detected in several milliseconds. To the best of our knowledge, the running time results presented in this paper are the fastest published. An interesting extension of this work would be to use this novel rank to accelerate additional methods of pattern matching.

References 1. Pele, O., Werman, M.: Robust real time pattern matching using bayesian sequential hypothesis testing. Technical Report 973, Department of Computer Science, The Hebrew University of Jerusalem (2007), http://www.cs.huji.ac.il/∼ ofirpele/hs/TR.html 2. Mikolajczyk, K., Schmid, C.: A performance evaluation of local descriptors. IEEE Trans. Pattern Analysis and Machine Intelligence 27(10), 1615–1630 (2005) 3. Gonzalez, R.C., Woods, R.E.: Digital Image Processing. Addison-Wesley Longman Publishing Co., Inc., Boston, MA, USA (2001) 4. Zabih, R., Woodfill, J.: Non-parametric local transforms for computing visual correspondence. In: Eklundh, J.-O. (ed.) ECCV 1994. LNCS, vol. 800, pp. 151–158. Springer, Heidelberg (1994) 5. Lepetit, V., Fua, P.: Keypoint recognition using randomized trees. IEEE Trans. Pattern Analysis and Machine Intelligence 28(9), 1465–1479 (2006) 6. Gil, J., Werman, M.: Computing 2-d min, median, and max filters. IEEE Trans. Pattern Analysis and Machine Intelligence 15(5), 504–507 (1993)

Detecting, Tracking and Recognizing License Plates Michael Donoser, Clemens Arth, and Horst Bischof Institute for Computer Graphics and Vision Graz University of Technology {donoser,arth,bischof}@icg.tu-graz.ac.at Abstract. This paper introduces a novel real-time framework which enables detection, tracking and recognition of license plates from video sequences. An efficient algorithm based on analysis of Maximally Stable Extremal Region (MSER) detection results allows localization of international license plates in single images without the need of any learning scheme. After a one-time detection of a plate it is robustly tracked through the sequence by applying a modified version of the MSER tracking framework which provides accurate localization results and additionally segmentations of the individual characters. Therefore, tracking and character segmentation is handled simultaneously. Finally, support vector machines are used to recognize the characters on the plate. An experimental evaluation shows the high accuracy and efficiency of the detection and tracking algorithm. Furthermore, promising results on a challenging data set are presented and the significant improvement of the recognition rate due to the robust tracking scheme is proved.

1

Introduction

There is a need for intelligent traffic management systems in order to cope with the constantly increasing traffic on todays roads. Video based traffic surveillance is one of the key parts of such installations. Beside detection and tracking of vehicles, identification by license plate recognition is important for a variety of applications like access-control, security or traffic monitoring. In general, license plate recognition systems consist of two separate parts. First, plates are detected within a single frame of a traffic video sequence and second, character recognition is applied to identify the characters on the plate. Different methods have been proposed for the detection of license plates. Shapiro et al. [1] use a mixture of edge detection and vertical pixel projection for their detection module. In the work of Jia et al. [2] color images were segmented by the Mean Shift algorithm into candidate regions and subsequently classified as plate or not. The AdaBoost algorithm was used by Dlagnekov and Belongie [3] for license plate detection on rather low resolution video frames. Matas and Zimmermann [4] proposed a different approach for the localization of license plates. Instead of using properties of the plate directly, the algorithm tries to find all character-like regions in the image by analyzing an interest region detection result. Y. Yagi et al. (Eds.): ACCV 2007, Part II, LNCS 4844, pp. 447–456, 2007. c Springer-Verlag Berlin Heidelberg 2007 

448

M. Donoser, C. Arth, and H. Bischof

Also several approaches for recognizing the characters on the plate after successful detection were proposed. Shapiro et al. [1] use adaptive iterative thresholding and analysis of connected components for segmentation. The classification task is then performed with two sets of templates. Rahman et al. [5] used horizontal and vertical intensity projection for segmentation and template matching for classification. Dlagnekov and Belongie [3] use the normalized cross correlation for classification by analyzing the whole plate, hence skipping segmentation. Although much scientific work has focused on recognizing license plates from traffic video sequences, surprisingly little work has been done on integrating a tracking scheme to gather additional representations of the plate for improving the recognition rate. Furthermore, in the few systems that apply a tracking scheme only simple and unstable approaches are used, as e. g. by Dlagnekov and Belongie [3] who perform tracking by simply repeating detection for building correspondences. The main contribution of this paper is a novel framework which unifies detection, tracking and recognition of license plates in a robust and efficient way. The underlying idea is to base detection, tracking and character segmentation on the same principles which allows to provide segmentations of the individual characters for recognition in addition to accurate and robust license plate localization results in subsequent frames. The framework is presented in detail in Section 2 and an experimental evaluation is shown in Section 3.

2

License Plate Recognition Framework

This section describes the entire framework for detection, tracking and recognition of license plates from traffic video sequences. The introduced system detects newly appearing license plates from the sequence by a novel algorithm which is based on the analysis of Maximally Stable Extremal Region (MSER) [6] detection results. The concept, introduced in Section 2.1, does not require any learning scheme and is capable of detecting different types of international plates. After detection of a plate it is robustly tracked through the sequence by a modified version of the MSER tracking framework [7] as shown in Section 2.2. Therefore, for each appearing car in the video sequence a set of license plate representations is collected which is used to improve the subsequent character recognition. Finally, Section 2.3 describes how support vector machines are used to recognize the characters on the different representations of each plate and how the results are merged by a voting scheme to achieve the final recognition result. 2.1

License Plate Detection

The first step of every license plate recognition system is the detection of the license plates within a single frame of the traffic video sequence. We propose a novel detection scheme which is motivated by the work of Matas et al. [6] who presented a method to learn category-specific extremal regions, i. e. the characters on a license plate, to perform detection. In contrast to their approach

Detecting, Tracking and Recognizing License Plates

449

our scheme does not require any learning scheme and is able to detect different types of international license plates without adaption. Our detection algorithm is based on analyzing the results of a Maximally Stable Extremal Region (MSER) detection [8]. MSERs denote a set of distinguished regions and have proven to be one of the best interest point detectors in computer vision [9]. All of these regions are defined by an extremal property of the intensity function in the region and on its outer boundary. Special properties form their superior performance as stable local detector. The set of MSERs is closed under continuous geometric transformations and is invariant to affine intensity changes. Furthermore, MSERs are detected in every scale. We predominantly exploit these properties for segmentation purposes. In general, two variants of MSER detection can be distinguished denoted as MSER+ and MSER-. While MSER+ detects bright regions with darker boundary, MSER- finds dark regions with brighter boundary. Figure 1(a) shows an image from a traffic video sequence and Figure 1(b) and Figure 1(c) illustrate the corresponding MSER detection results as binary images. As can be seen, the license plate itself is identified as MSER+, whereas the characters on the plate are detected as MSER-.

(a) Input frame

(b) MSER+

(c) MSER-

Fig. 1. MSER detection results can be used for detecting license plates in video sequences. MSER+ finds the license plate, whereas MSER- identifies the individual characters on it.

The underlying idea of our novel license plate detection scheme is to analyze the MSER+ and MSER- detection results. We are looking for a larger MSER+ region (license plate) that contains a set of smaller MSER- regions (characters). Such a combination is considered as license plate detection result. Furthermore, we verify the detection by checking if the MSER- regions are approximately equal sized, if their center points approximately lie on a line and if the height of the MSER+ is in the range of the average MSER- height. After verification, the MSER+ is returned as license plate localization result and additionally segmentations of the characters are provided by the corresponding MSER- detections. Although the detection process is simple, it is effective and allows stable and accurate detection of license plate in complex scenes. An exemplary result is shown in Figure 2(a). In this example 226 MSER- and 688 MSER+ regions are found, but only one set fulfills the proposed criterions.

450

M. Donoser, C. Arth, and H. Bischof

(a) Proposed algorithm

(b) AdaBoost result

Fig. 2. License plate detection results. Figure (a) shows the single detection of the proposed algorithm indicated by the white border. 914 MSERs are detected, but only one set of regions fulfills the described criterion. Figure (b) shows the wrong and multiple detections of an AdaBoost algorithm.

For comparison, Figure 2(b) shows the result of an AdaBoost detector based on Haar-like features [10]. As can be seen the boosting framework returns wrong and multiple detections which need to be significantly post-processed, as e. g. by using non maximum suppression to remove multiple detections. Our result is also more accurate as the bounding box provided by the boosting variant. Furthermore, training of an AdaBoost based classifier is a rather complex procedure, and in this case was especially trained on Austrian license plates. Our approach does not require any learning scheme and is able to identify different types of international plates, because the simple criterion is fulfilled for almost all of them. Figure 3 shows detection results for different international types.

(a) Input image

(b) Characters (MSER-)

(c) Plates (MSER+)

Fig. 3. Detection results on international license plates where (b) shows the segmented signs detected as MSER- and (c) the plate detected as MSER+. As can be seen the detection algorithm can be applied to any international type, because the criterion used in the detection algorithm is valid for all cases.

Detecting, Tracking and Recognizing License Plates

2.2

451

Tracking of License Plates

After detection of a newly appearing car in the traffic video sequence by its license plate, a robust tracker is applied to increase the number of character representations for the subsequent recognition step. In general, any tracker can be used but we propose to apply a modified version of the MSER tracking framework introduced by Donoser and Bischof [7] which has some advantages in contrast to other tracking schemes. First, it is efficient and stable and can be adapted to our specific requirements. Second, it provides an accurate segmentation of the license plate and third, in addition to the tracked plate it also returns segmentations of the individual characters on the plate (MSERs), thus tracking and segmentation are handled simultaneously. The MSER tracking framework was designed to improve the stability and speed of MSER detection results in video sequences. The tracker has to be initialized by passing the region to be tracked Rt detected in image It of the sequence to the framework. The first step of the algorithm is to propagate the center point of the region Rt to the next image It+1 and to crop a region-of-interest (ROI) around it from the image. Then a data-structure named component tree [11] is built for this ROI. Every node of the component tree contains one candidate i for the tracking and the algorithms looks for the node which is most region Ct+1 similar to the region Rt . The best fit is identified by comparing feature vectors i that are built for each of the nodes of the component tree Ct+1 and the input i region Rt . The candidate Ct+1 with the lowest Euclidean distance between its feature vector and the one of the region Rt is taken as tracked representation. The features calculated are mean gray value, region size, center of mass, width and height of the bounding box and stability. All of these features are computed incrementally [12] during creation of the component tree. Thus, no additional computation time is required. After detection of the new representation the described steps can be repeated for tracking the region through the entire sequence. The original MSER tracking framework was designed for tracking arbitrarily shaped region, but we adapt the method to our specific requirements of tracking license plates. In our framework, the MSER tracking algorithm is initialized by the result of the license plate detection algorithm presented in Section 2.1. Because we focus on license plates we reformulate the feature comparison approach by replacing the Euclidean distance computation by a simpler, but more effective equation based on comparison of two distinct features, the size and the rectangularity of the region. Thus, in our framework the tracked representation i ) for every candidate region is found by calculating a distance value Δ(Rt , Ct+1 i Ct+1 by  i      i  abs |Rt | − Ct+1 i + 1 − ϑ Ct+1 , (1) Δ(Rt , Ct+1 ) = |Rt | where |. . .| denotes the size of the region and ϑ(. . .) is the rectangularity. Then, i i with the lowest Δ(Rt , Ct+1 ) value is taken as final reprethe candidate Ct+1 sentation. Tracking is considered as valid as long the minimum distance value Δ is below a fixed threshold. By repeating the presented steps the license plate

452

M. Donoser, C. Arth, and H. Bischof

(a) Frame 1

(b) Frame 15

Fig. 4. Illustration of license plate tracking. The images show a traffic scene and the segmented characters on the license plate are highlighted in white.

is tracked through the sequence and the detected MSER- regions within the tracked plate are provided as segmentations of the individual characters. Furthermore, the tracking scheme is also used for discarding false positives of the detection step by removing non-moving or unstable plate tracks. Figure 4 shows two frames of a traffic video sequence, where accurate license plate detections are provided in addition to the segmentation of the eight characters on the plate. 2.3

Character Recognition

The final step of our framework is to recognize the individual characters on the detected plates based on support vector machines (SVMs). SVMs were first described by Vapnik [13] and have proven to be an efficient tool for classification tasks as in optical character recognition (OCR) of handwritten digits or license plate contents [14]. For an introduction to SVMs and other kernel methods see for example [15]. In general, SVMs are designed for binary classification problems. Because character recognition is a multi-class problem we apply a method based on a combination of several binary SVMs. The strategy is called one-vs-one where for each pair of output classes an individual SVM is trained, resulting in a total number of n · (n − 1)/2 classifiers. Then all classifiers are evaluated, the votes are summed up and the class with the maximum number is chosen. First the provided character segmentations are aligned, and then the one-vsone approach is used to classify each character independently of all the others. The presented tracking approach provides several license plate representations for every car and therefore, we also have several classification results for every character on the plate. A majority voting scheme is then used to determine the final character recognition result for every car. Figure 5 shows a sequence of tracked license plates, the segmented characters and the corresponding classification results. As can be seen, the single image based recognition provides wrong assignments, but the subsequent majority voting scheme ensures that the final character classification is correct. Therefore, the power of our framework clearly comes from the stable tracking approach

Detecting, Tracking and Recognizing License Plates

(a) Tracking Sequence

453

(b) Classification Results

Fig. 5. Recognition is improved by using several license plate representations provided by the tracking scheme. The final result based on a majority voting for this plate sequence is G~19VAB which matches with the real plate number.

which allows to combine a sequence of single image based recognition results into the final classification. 2.4

Framework

Our framework is able to analyze traffic video sequences in real-time. It detects newly appearing cars by localization of their license plate. After a one-time detection, the plate is robustly tracked and several representations of the license plate are collected. Until tracking fails the available repeated segmentations of each character on the plate are used to improve the recognition rate along the complete tracking sequence and to determine a final result by the majority voting scheme. The running times of the individual steps of the concept for analysis of a video sequence of size 352 × 288 are shown in Table 1. Table 1. Running time per image of the individual steps of the framework for analyzing a video stream of size 352 × 288

Running time

3

Detection

Tracking

Recognition

70ms

5ms

6ms

Experimental Evaluation

We evaluated our framework on a challenging traffic video sequence in the type of Figure 5(a) which was filmed from a footbridge. The provided resolution was 352 × 288 and therefore the characters on the plate only have an average size of 9 × 6 pixels in the sequence. Section 3.1 describes how the necessary training data for character recognition was acquired. Finally, Section 3.2 demonstrates that promising results are

454

M. Donoser, C. Arth, and H. Bischof

achieved on the challenging data set and shows how the recognition rate is significantly improved by using several plate representations provided by the tracking scheme. 3.1

Character Image Database

The support vector machines are trained on approximately 2700 manually labeled images of characters which were automatically extracted from high resolution images of parked cars using the license plate detection algorithm. According to the resolution of the test video sequence we resized all character images to 9 × 6 pixels. 3.2

Recognition Results

To evaluate the quality of the proposed framework we used it for recognizing the license plates of 109 cars passing the video sequence area. Due to the low resolution and changing lighting conditions the average recognition rate for independent classification of every character in every detected license plate is only 80.74%. As a consequence, in a single image based license plate recognition approach on average only 23.23% of the cars are totally correct classified. But this rate is significantly improved by analyzing additional plate representations provided by the tracking scheme and combining the corresponding recognition results by the presented majority voting scheme. Figure 6 analyzes the increase of the recognition rate by using more representations. As the number of tracked plate representations gets close to 70, the percentage of totally correct classifications reaches more than 90%. By using all available representations for recognizing the plate of every car (4880 plates) our framework classifies 94.65% of all cars totally correct which is a

Fig. 6. Illustration of improvement in recognition rate achieved by analyzing several license plate representations provided by the tracking scheme

Detecting, Tracking and Recognizing License Plates

455

Table 2. Recognition rates of totally correct classified plates (plate level) and correctly classified characters (character level) for a single image based approach in comparison to the tracking based classification

Character level Plate level

Single image approach

Tracking approach

80.74% 23.23%

97.16% 94.65%

promising result for such a challenging data set. Please note, that no postprocessing, like checking the validity of license plates according to syntax restrictions, is used. Table 2 analyzes the recognition rate on plate level, i. e. the number of totally correct classified plates, and character level, i. e. the number of correctly classified characters, for the single image based approach in comparison to our tracking variant. As can be seen analyzing several representations instead of a single one significantly improves the recognition rate.

4

Conclusion

This paper introduced a novel framework which allows detection, tracking and recognition of license plates. Detection is handled by analysis of Maximally Stable Extremal Region (MSER) detection results and does not require any learning scheme. We introduced a robust tracking scheme, which provides accurate license plate localizations and segmented characters simultaneously. The experimental evaluation showed that promising results are achieved on a challenging data set and that the robust tracking approach significantly improves the recognition rate. Furthermore, due to the high efficiency of the individual components, the framework can be used for real-time traffic video sequence analysis. To make the proposed framework applicable in industrial scenarios, we also ported it to an DSP-based embedded platform. Although the experiments were all performed on a desktop computer, the results also hold for a fully embedded implementation.

References 1. Shapiro, V., Gluhchev, G., Dimov, D.: Towards a multinational car license plate recognition system 17(3), 173–183 (2006) 2. Jia, W., Zhang, H., He, X., Piccardi, M.: Mean shift for accurate license plate localization. In: ITSC. Proceedings of the IEEE Conference on Intelligent Transportation Systems, pp. 566–571. IEEE Computer Society Press, Los Alamitos (2005) 3. Dlagnekov, L., Belongie, S.: Recognizing cars. Technical Report CS2005-0833, UCSD University of California, San Diego (2005) 4. Matas, J., Zimmermann, K.: Unconstrained licence plate and text localization and recognition. In: ITSC. Proceedings of the IEEE Conference on Intelligent Transportation Systems, Vienna, Austria, pp. 572–577. IEEE Computer Society Press, Los Alamitos (2005)

456

M. Donoser, C. Arth, and H. Bischof

5. Rahman, C., Badawy, W., Radmanesh, A.: A real time vehicle’s license plate recognition system. In: AVSS. Proceedings of the IEEE Conference on Advanced Video and Signal Based Surveillance, pp. 163–166. IEEE Computer Society Press, Los Alamitos (2003) 6. Matas, J., Zimmermann, K.: Unconstrained licence plate detection. In: ITSC. Proceedings of International Conference on Intelligent Transportation Systems, pp. 572–577 (2005) 7. Donoser, M., Bischof, H.: Efficient maximally stable extremal region (MSER) tracking. In: CVPR. Proceedings of Conference on Computer Vision and Pattern Recognition, pp. 553–560 (2006) 8. Matas, J., Chum, O., Urban, M., Pajdla, T.: Robust wide baseline stereo from maximally stable extremal regions. In: BMVC. Proceedings of British Machine Vision Conference, pp. 384–393 (2002) 9. Mikolajczyk, K., Schmid, C.: A performance evaluation of local descriptors. IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI) 27(10), 1615– 1630 (2005) 10. Zhang, H., Jia, W., He, X., Wu, Q.: Learning-based license plate detection using global and local features. In: ICPR. Proceedings of International Conference on Pattern Recognition, pp. 1102–1105 (2006) 11. Najman, L., Couprie, M.: Quasi-linear algorithm for the component tree. In: SPIE Vision Geometry XII, vol. 5300, pp. 98–107 (2004) 12. Matas, J., Zimmermann, K.: A new class of learnable detectors for categorisation. In: SCIA. Proceedings of Scandinavian Conference of Image Analysis, pp. 541–550 (2005) 13. Vapnik, V.N.: The nature of statistical learning theory. Springer-Verlag New York, Inc., New York, NY, USA (1995) 14. Zheng, L., He, X.: Number plate recognition based on support vector machines. In: AVSS 2006. Proceedings of the IEEE International Conference on Video and Signal Based Surveillance, Washington, DC, USA, p. 13. IEEE Computer Society Press, Los Alamitos (2006) 15. Sch¨ olkopf, B., Smola, A.J.: Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. MIT Press, Cambridge, MA, USA (2001)

Action Recognition for Surveillance Applications Using Optic Flow and SVM Somayeh Danafar* and Niloofar Gheissari Institute for studies in Thoretical Physics and Mathematics(IPM), Mathematics school, Tehran, Iran [email protected], [email protected]

Abstract. Low quality images taken by surveillance cameras pose a great challenge to human action recognition algorithms. This is because they are usually noisy, of low resolution and of low frame rate. In this paper we propose an action recognition algorithm to overcome the above challenges. We use optic flow to construct motion descriptors and apply a SVM to classify them. Having powerful discriminative features, we significantly reduce the size of the feature set required. This algorithm can be applied to videos with low frame rate without scarifying efficiency or accuracy, and is robust to scale and view point changes. To evaluate our method, we used a database consisting of walking, running, jogging, hand clapping, hand waving and boxing actions. This grayscale database has images of low resolution and poor quality. This image database resembles images taken by surveillance cameras. The proposed method outperforms competing algorithms evaluated on the same database. Keywords: Action recognition, motion recognition, human motion, motion analysis, surveillance.

1 Introduction In this paper, we address the problem of human action recognition based on optic flow. Human action recognition is a challenging and essential subject in computer vision. In fact, many computer vision applications such as surveillance, humancomputer interaction, video retrieval and scene understanding involve recognizing human action. Some major challenges in action recognition for surveillance applications [5] include noisy [6,11] [12]images of low resolution and low frame rate. In addition, changes in illumination and view point pose a great challenge to most human action recognition algorithms. In addition, they are problematic for most of segmentation algorithms. Hence, those action recognition algorithms that rely on segmenting body parts fail to recognize the correct actions. There are two general approaches to human action recognition. The first approach is based upon the shape and the movement of different body parts. Wang et al. [16] proposed computing a mean contour to represent static contour information. They *

Corresponding author.

Y. Yagi et al. (Eds.): ACCV 2007, Part II, LNCS 4844, pp. 457–466, 2007. © Springer-Verlag Berlin Heidelberg 2007

458

S. Danafar and N. Gheissari

also build a model composed of 14 rigid body parts each of which presented by a truncated cone. An advantage of using body parts is that we can detect local variations in shape and movement which might in turn lead to different actions. However, an accurate segmentation and labeling of human body parts is necessary which is not a trivial task especially for surveillance cameras. In contrast to the above methods, the second approach is holistic and one look at the human body as a whole and tries to recognize actions based on its dominant movement. A pioneering algorithm in this category was proposed by Efros [3]. They build their motion descriptors using blurred optic flow and rely on spatio-temporal correlations to match the descriptors to the database. They reported their results on a database of images taken at distance. A recently introduced algorithm by Yilmaz and Shah [17] tracks the 2D contour of a human over time to construct spatio-temporal volumes. They analyze different geometrical properties of volumes to describe motion. This algorithm is invariant to view point and appears to perform well. However, they applied their algorithm to close up images and therefore we cannot objectively compare their results to ours. An advantage of the above holistic approaches is that they do not need labeling of different body parts. However, the local variations in shape or movement are overlooked. Since local variations might be due to different actions, they are important to be taken into account. To overcome the limitations of both approaches and to take advantage of their strength, we propose a hybrid method in this paper. We first find a bounding box around a human subject as a coarse segmentation and then refine the boundaries using k-means clustering. We spilt up the human body to head, torso and legs (based on some heuristic ratios). Then we use histograms of horizontal and vertical components of the optic flow field as our action descriptors. Having powerful discriminative features, we significantly reduce the size of the feature set require. We observed that by customizing the number of bins we can capture enough spatial patterns of optical flows to recognize actions. In addition, since we use a coarse to fine robust optic flow computation method (Black and Anandan [2]), we can reliably estimate optic flow for large displacements. Histograms of optical flows are robust relative to noise and changes in illumination. We evaluated our algorithm on a grayscale database of low resolution images and achieve above 85 percent accuracy. This image database resembles low quality images taken by surveillance cameras. This paper is organized as follows. In section 2 we present our human action recognition algorithm. This method has several ingredients including human body segmentation, optic flow computation, local motion representation, and action classification using SVM. Each ingredient is described in the following sub-sections. We used the database introduced by [13] to evaluate our algorithm. The proposed algorithm outperforms the competing methods on the same database (section 3).

2 The Proposed Action Recognition Algorithm Here we first discuss our database and then in the following subsections, we describe the details of the proposed action recognition algorithm.

Action Recognition for Surveillance Applications Using Optic Flow and SVM

459

We used the database used by [13] as it is the largest database available for this application. This database contains 2391 sequences of six types of human actions. That is 1. 2. 3. 4. 5. 6.

Walking Running Jogging. Hand clapping. Hand waving. Boxing

The above actions were performed by 25 people in four different scenarios: 1. 2. 3. 4.

Outdoor (s1), Outdoors with scale variations (s2), Outdoors with different clothes (s3), Indoors (s4).

Some samples of this database are shown in Figure 1. Walking

Jogging

Running

Boxing

Hand waving

Hand clapping

S1

S2

S3

S4

Fig. 1. Action database: different actions and scenarios

In this paper only 150 frames of each sequence are used. This is because the actions are periodic and consequently we can extract sufficient information using only a portion of each sequence. In addition, our experiments show that the algorithm needs only limited training samples. 2.1 Human Body Segmentation In some frames we only have background and there is no human in the scene. We expected that such frames have nominal optic flow field. However, we observed that due to noise and significant illumination change, the optic flow of background was significant compared to the human motion.

460

S. Danafar and N. Gheissari

To detect these frames we first employed a heuristic method to find a bounding box around the human body. We applied Harris corner detector [6] to extract interest points in each image. Background images (grasses and empty rooms) had almost no interest point compared to the large number of interest points in images containing humans. Using coordinates of the extracted interest points, we created a bounding box around human subjects (Figure 2). This is only a coarse segmentation and will be refined later. Alternatively, a robust and fast foreground from background separation algorithm can be used. Then in order to accurately subtract human body from the background in the mentioned bounding boxes, we use a k-means clustering algorithm to cluster pixels inside the bonding box into two classes. This results in more accurate boundaries of the human body. Applying k-means only to the pixels inside the bounding box, also improves the speed of clustering task significantly.

Fig. 2. (left) Detected points by Harris corner detector (right) Constructed bounded area from detected points. Database is borrowed from [13].

2.2 Optic Flow Computation The optic flow field is defined as the set of apparent velocities of the brightness patterns in the image plane [8]. To compute optic flow usually two assumptions are made, namely the brightness is assumed to be constant in a small neighborhood and the motion is considered to be small. Other assumptions also could be that the motion is locally smooth or image gradients are constant. The first two assumptions lead us to the first order optic flow equation

u . I x + v. I y = I t

(1)

where u and v are optic flow components in horizontal and vertical directions. In addition, Ix and Iy and It are derivatives of the brightness function with respect to x, y (image coordinates) and t (time). 1 In this research, we used the method proposed by Black and Anandan [2] . They incorporate a robust estimation method to overcome violations of the brightness constancy and spatial smoothness assumptions due to multiple motions. This method estimates parametric motion models within a region and computes piecewise-smooth flow fields. One reason for choosing this algorithm is that it uses a coarse to fine approach which makes the computation robust to large displacement. 1

We acknowledge the source code provided by them.

Action Recognition for Surveillance Applications Using Optic Flow and SVM

461

2.3 Local Motion Representation

In this section, we describe the proposed motion descriptor based on histograms of optic flows. We normalized the optic flow values of the segmented human body region using the global minimum and maximum values of optic flow over all images in the training set. Using normalized histograms makes motion descriptors robust relative to noise and outliers. Zhu et al. [18] also used histograms of optic flow as motion descriptors for human action recognition. However, they only applied their method to recognize left swings and right swings in Tennis games. To categorize forward and backward movements in the same class of actions, we only consider the absolute values of horizontal and vertical components of optic flow. For this purpose, let Fx and Fy be the horizontal and vertical components of the optic flow field F inside a bounding box. Motivated by Efros et al [3], we split up Fx and Fy, into four channels, Fx+ and Fy+ for optical flows with positive values and Fx- , Fyfor optical flows with negative values. Then we construct two optic flow channels: positive channel and negative one. As described in section 1, we use a hybrid approach to take advantage of both holistic and body part based approaches. To consider local information, we partition the bounding box of human body into 3 parts from top to bottom. The first 1/5 of the window from the top approximately contains the head and neck. The next 2/5 of the window has body and hands. Finally the bottom 2/5 of the window contains the legs (Figure 3). This heuristic approach can capture local information of optic flow without requiring exact segmentation which is a challenging task. We consider the combination of 15 bin histogram of head-neck part,15 bin histogram of body-hands part, and the 21 bin histogram of legs part ( Fhx, Fhy, Fbx, Fby, Ffx, Ffy) as local motion descriptors. Figure 4 and Figure 5 respectively show the motion descriptors for a walking and a hand waving sequence.

Fig. 3. Bounding box and its partitions based on ratios

2.4 Motion Classification

Efros et al. [3] used nearest neighbor classifier for human action recognition. This approach often has the drawback of poor generalization capabilities in real-world conditions.

462

S. Danafar and N. Gheissari

Fig. 4. Local motion features of head, body and legs parts in horizontal (top three figures) and in vertical direction (bottom three figures) for walking

Fig. 5. Local motion features of head, body and legs parts in horizontal (top three figures) and in vertical direction (bottom three figures) for hand waving

KNN classifiers suffer from so called “jig-jag” problem along the decision boundary, if the training set is small or features are high dimensional. This problem is particularly crucial here, since in surveillance have consequently a small training set (because of low frame rate). In addition, in KNN classifiers the number of neighbors (K) should be set. To avoid this problem we used SVM2 [15] in our research. SVM lends itself well to generalization and can avoid the over-fitting phenomena. SVM has been successfully used in many different recognition tasks. It is widely known that SVM is relatively slow because it uses quadratic programming for optimization. However, we use a small training set and as a result we mitigate computational burden. Since SVM is a kernel based method, selecting the appropriate type of kernel is crucial. We tested 2

We modified and used the source code provided in [8].

Action Recognition for Surveillance Applications Using Optic Flow and SVM

463

different kernels including polynomial, intersection, radial basis, linear and χ . We concluded that intersection kernel outperforms the others in our application. Therefore, we used the intersection kernel as:[14] 2

K(X,Y) =

∑ min( x , y ) i

i

for i = 1,.., m .

(2)

where xi and yi are feature vectors extracted from the data. This kernel has no free parameters and so can be adapted to real-time requirements. In the next section, we report the results of classifying the proposed motion descriptors using SVM.

3 Experiments and Evaluation Following Schuldt et al [13], we divided all sequences into a training set (8 persons), a validation set (8 persons), and a test set (9 persons) . The classifier was trained on a training set while the validation set was used to optimize the parameters for each method. Results of our experiments are shown in Figure 6. We achieved 85 percent total success rate. The algorithm seems to be stable in recognizing different actions and its performance varies between 75 to 92 percent. In Figure 7 and Figure 8, we have shown the results of three competing algorithms (introduced respectively by Schuldt et al. [13], Yeo et al [1], Dollar et al. [3]and Niebles et al [10]) on the same database. In both [13][10]and [3] the authors used local space-time features extracted from video. Schuldt at al. and Dollar et al. combined space-time features with a local kernel in SVM. In contrast, Neibels et al. achieved used a probabilistic Latent Semantic Analysis (PLSA) in an unsupervised learning framework. Yeo et al used a completely different approach based on a novel motion correlation measure. The results reported by Yeo et al exclude the indoor scenario.

Fig. 6. Confusion matrix achieved by our algorithm. Results are reported in percents.

464

S. Danafar and N. Gheissari

Comparing Figure 6 with Figures 7 and 8 shows the success of the proposed method. It is due to the discriminative motion descriptors which capture sufficient conceptual information to recognize action. In addition, we use a hybrid method in contrast with the local methods presented in [13] and [10]. This means we benefit from advantages of both global and local methods.

1

Fig. 7. (left)Confusion matrix achieved by Yeo et al [1]. Results are reported in percents.,(right) Confusion matrix achieved by Schuldt et al. [13]. Results are reported in percents.

6 1

Fig. 8. (left) Confusion matrix achieved by Dollar et al [3 ]. (right) Confusion matrix achieved by Niebles et al [10]. Results are reported in percents.

4 Conclusion In this paper, we proposed an action recognition algorithm suitable for surveillance applications. We use histograms of vertical and horizontal optic flow as motion descriptors that are robust to noise and outliers. We use a coarse partitioning of human body to capture local motion information which in turn makes the algorithm capable of detecting local variations. We apply a SVM to classify the descriptors. SVM has strong generalization capability in real world applications and performs well

Action Recognition for Surveillance Applications Using Optic Flow and SVM

465

with limited training samples. Therefore, this algorithm can be applied to videos with low frame rate. To evaluate our method, we applied it to a database of grey level low resolution images. This database consists of walking, running, jogging, hand clapping, hand waving and boxing image sequences. This database includes noisy images taken at different view points, scales and illumination conditions. We achieved about 85 percent of success rate in total. The success rates for different actions vary from 75 percent (for hand clapping) to 92 percent (for running and jogging). This means unlike the competing methods we achieve a reasonable and relatively stable performance for all types of action.

References 1. Ahammad, P., Yeo, C., Sastry, S.S., Ramchandran, K.: Compressed Domain Real-time Action Recognition. In: MMSP. Proceedings of 8th IEEE Workshop on Multimedia Signal Processing, pp. 33–36. IEEE Computer Society Press, Los Alamitos (2006) 2. Black, M.J., Anandan, P.: The Robust Estimation of Multiple Motions: Parametric and Piecewise-Smooth Flow Fields. Computer Vision and Image Understanding (CVIU) 63, 75–104 (1996) 3. Dollar, P., Rabaud, V., Cottrel, G., Belongie, S.: Behavior Recognition Via SpatioTemporal Features. In: Proceedings 2nd Joint IEEE International Workshop on VS-PETS, Beijing, pp. 65–72. IEEE Computer Society Press, Los Alamitos (2005) 4. Efros, A.A., Berg, A.C., Mori, G., Malik, J.: Recognizing Action at a Distance. In: ICCV 2003. IEEE International Conference on Computer Vision, Nice, France, pp. 726–733. IEEE Computer Society Press, Los Alamitos (2003) 5. Gavrila, D.M.: The Analysis of Human Motion and its Application for Visual Surveillance. In: Proceedings of the Second IEEE Workshop on Visual Surveillance, p. 3. IEEE Computer Society Press, Los Alamitos (1999) 6. Gavrila, D.M.: The Visual Analysis of Human Movement: A Survey. Computer Vision and Image Understanding (CVIU) 23, 82–98 (1999) 7. Harris, C., Stephens, M.: A combined Corner and Edge Detector. In: Proceedings of the 4th Alvey Vision Conference, Manchester, UK, pp. 147–151 (1988) 8. Horn, B.K., Schunck, B.G.: Determining Optic Flow. Artificial Intelligence 17, 185–203 (1981) 9. Chih-Jen, L., Chin-chung, C.: LIBSVM–A Library for Support Vector Machines (2007) 10. Niebles, J.C., Wang, H., Fei-fei, L.: Unsupervised Learning of Human Action Categories Using Spatial-Temporal Words. In: BMVC (2006) 11. Niu, W., Long, J., Han, D., Wang, Y.-F.: Human Activity Detection and Recognition for Video Surveillance. In: Proceedings of the IEEE Multimedia and Expo Conference, pp. 719–722. IEEE Computer Society Press, Los Alamitos (2004) 12. Robertson, N., Reid, I., Brady, M.: Behavior Recognition and Explanation for Video Survillance. In: ICDP 2006. IEE Conf. Imaging for Crime Detection and Prevention (2006) 13. Schuldt, C., Laptev, I., Caputo, B.: Recognizing Human Actions: A Local SVM Approach. In: ICPR 2004, pp. 32–36 (2004) 14. Swain, M.J., Ballard, D.H.: Color Indexing. International Journal of Computer Vision 7, 11–32 (1991)

466

S. Danafar and N. Gheissari

15. Vapnik, V.: The Nature of Statistical Learning Theory. Springer, New York (1995) 16. Wang, L., Ning, H., Hu., W.: Fusion of Static and Dynamic Body Biometrics for Gait Recognition. In: International Conference on Computer Vision, Nice, France, pp. 1449– 1454 (2003) 17. Yilmaz, A., Shah, M.: Actions As Objects: A Novel Action Representation. In: IEEE CVPR, San Diego, IEEE Computer Society Press, Los Alamitos (2005) 18. Zhu, G., Changsheng, X., Gao, W., Huang, Q.: Action Recognition in Broadcast Tennis Video Using Optical Flow and SVM. In: Proceedings of HCI/ECCV (2006)

The Kernel Orthogonal Mutual Subspace Method and Its Application to 3D Object Recognition Kazuhiro Fukui1 and Osamu Yamaguchi2 2

1 Department of Computer Science, University of Tsukuba, Japan Corporate Research and Development Center, Toshiba corporation, Japan

Abstract. This paper proposes the kernel orthogonal mutual subspace method (KOMSM) for 3D object recognition. KOMSM is a kernel-based method for classifying sets of patterns such as video frames or multi-view images. It classifies objects based on the canonical angles between the nonlinear subspaces, which are generated from the image patterns of each object class by kernel PCA. This methodology has been introduced in the kernel mutual subspace method (KMSM). However, KOMSM is different from KMSM in that nonlinear class subspaces are orthogonalized based on the framework proposed by Fukunaga and Koontz before calculating the canonical angles. This orthogonalization provides a powerful feature extraction method for improving the performance of KMSM. The validity of KOMSM is demonstrated through experiments using face images and images from a public database.

1

Introduction

This paper introduces the kernel orthogonal mutual subspace method (KOMSM) for 3D object recognition. KOMSM is an appearance-based method for classifying a set of patterns such as video frames or multi-view images. As the set of such patterns generally has highly nonlinear structure, we have to tackle a nonlinear classification problem of multiple sets of patterns. The kernel mutual subspace method (KMSM) is one suitable method for this task. KMSM is a nonlinear extension of the mutual subspace method (MSM)[3] by using the kernel trick. MSM classifies sets of patterns based on the canonical angles θ between linear class subspaces, which represent the distribution of the training set of each class respectively as shown in Fig.1. In this method a w×h image pattern is represented as a vector in w×h-dimensional space (called input space I). Although MSM has the ability to handle the variability of patterns to achieve higher performance compared to other methods[1], its performance drops significantly when the pattern distributions have highly nonlinear structure. In such cases class distributions cannot be represented by a linear subspace without overlapping each other. The kernel mutual subspace method (KMSM)[4,5] has been proposed in order to solve this problem. In this method an input pattern x is mapped into a high (in some cases infinite) dimensional feature space F via a nonlinear map φ. Y. Yagi et al. (Eds.): ACCV 2007, Part II, LNCS 4844, pp. 467–476, 2007. c Springer-Verlag Berlin Heidelberg 2007 

468

K. Fukui and O. Yamaguchi

...... Subspace P

Subspace Q

KL expansion

...... KL expansion

θ...

Canonical angles

Fig. 1. Similarity between two distributions of view patterns in the input space I

Consequently, KMSM carries out the MSM on the linear subspaces 1 generated from the mapped patterns {φ(x)} using the Karhunen-Lo`eve (KL) expansion, also known as principal component analysis (PCA). KMSM works well since each subspace can be generated without overlapping with another subspace in the feature space F . However its classification performance is still insufficient for many applications in practice, because the nonlinear class subspaces are generated independently of each other [1]. There is no reason to assume a priori that it is the optimal nonlinear class subspace in terms of classification performance, while each nonlinear class subspace represents the distribution of the training patterns well in terms of a least-mean-square approximation. This suggests that there is room for improving the performance of KMSM. In order to improve the performance of KMSM the kernel constrained mutual subspace method (KCMSM)[11] has been proposed. In this method, each nonlinear class subspace is projected onto a discrimination space called the constraint subspace. This projection extracts a common subspace of all the nonlinear class subspaces from each nonlinear class subspace, so that the canonical angles between nonlinear class subspaces are enlarged to approach orthogonal relation. KCMSM could significantly improve the performance of KMSM in many applications, such as face recognition and 3D object recognition[11]. This implies that the concept of orthogonalization is essential for improving the performance of KMSM. Therefore, in this paper we apply the framework proposed by Fukunaga and Koontz[2], which achieves perfect orthogonalization, while the orthogonalization achieved in KCMSM is only approximate. We orthogonalize the nonlinear class subspaces using this framework and then apply KMSM to the orthogonalized nonlinear class subspaces. Fukunaga and Koontz’s method has been applied to related linear subspace methods including MSM, improving their performance [1,7,8,9]. In Fukunaga and Koontz’s method orthogonalization is achieved by applying a whitening transformation matrix to the training patterns or orthonormal basis vectors of each class subspace. Thus, the main task we need to solve is to construct the whitening transformation matrix for orthogonalizing the linear subspaces in the feature space F . 1

Note that a linear subspace in the feature space F is an nonlinear subspace in the input space I.

The Kernel Orthogonal Mutual Subspace Method

P2

P1

469

P3 P2 Whitening transformation

P3 P1

Fig. 2. Concept of orthogonalization of subspaces by the whitening transformation

This paper is organized as follows. Section 2 describes the calculation of the canonical angles. In Section 3, we compute the whitening transformation. In Section 4, we define the kernel whitening transformation and orthogonalize nonlinear class subspaces. Then we construct the KOMSM by applying the KMSM to the orthogonalized nonlinear class subspaces. Section 5 demonstrates the effectiveness of KOMSM through experiments. We conclude in section 6.

2

Canonical Angles Between Subspaces

Given an dp -dimensional linear subspace P and an dq -dimensional linear reference subspace Q in the n-dimensional input space, the canonical angles {0 ≤ θ1 , . . . , θdp ≤ π2 } between P and Q (for convenience dp ≤ dq ) are uniquely defined as [12]: cos(θi ) = max max u i vi ui ∈P vi ∈Q

(1)

subject to :    u i ui = vi vi = 1; ui uj = 0; vi vj = 0; i = j, i = 1∼dp , j = 1∼dp .

Let Φi and Ψi denote the i-th n-dimensional orthonormal basis vectors of the subspaces P and Q, respectively. These orthonormalbasis vectors can be l  obtained as the eigenvectors of the correlation matrix i=1 xi xi calculated from the l learning patterns {x} of each class. A practical method of finding the canonical angles is by computing the matrix X=A B, where A = [Φ1 , . . . , Φdp ] and B=[Ψ1 , . . . , Ψdq ]. Let {κ1 , . . . , κdp } (κ1 ≥, . . . , ≥ κdp ) be the singular values of the matrix X. The canonical angles {θ1 , . . . , θdp } can be obtained as {cos−1 (κ1 ), . . . , cos−1 (κdp )}. In practice, we t consider the value, S[t] = 1t i=1 cos2 θi , as the similarity between two subspaces. The value S reflects the structural similarity between two subspaces.

3

Orthogonalization by the Whitening Transformation

In this section, we will describe how to calculate the whitening matrix O for orthogonalizing r d-dimensional linear class subspaces with the orthonormal basis vectors ei (i = 1∼d) in the n-dimensional input space I.

470

K. Fukui and O. Yamaguchi

At first, we define the projection corresponding to the projection onto matrix d the class i subspace Pi by Pi = j=1 ej e orthonormal j where ej is the j-th  basis vector of Pj . Then we define the total projection matrix G= ri=1 Pi . Using the eigenvectors and the eigenvalues of the total projection matrix G, the v×n whitening matrix O is defined by the following equation: O = Λ−1/2 H ,

(2)

where v = r×d, (v = n, if v > n), Λ is the v×v diagonal matrix with the i-th highest eigenvalue of the matrix G as the i-th diagonal component, and H is the n×v matrix whose i-th column vector is the eigenvector of the matrix G corresponding to the i-th highest eigenvalue. We can confirm that the matrix O can whiten the matrix G so that r class subspaces are orthogonalized as proved in [8]. Fig.2 shows the concept of orthogonalization by the whitening transformation. We can see that the orthogonalization is achieved by whitening, i.e. homogenizing the variances in all directions: All transformed basis vectors are orthogonal to each other when the multiple of the number r of classes and the dimension m of each class is smaller than the dimension n of the input space[8]. On the other hand, since the transformed basis vectors of an input subspace are no longer orthogonal, all transformed basis vectors need to be orthogonalized again by Gram-Schmidt orthogonalization.

4

The Proposed Method

In this section, we will first describe the concept of nonlinear subspaces, and then construct the kernel whitening matrix in the feature space F . 4.1

Nonlinear Subspace by Kernel PCA

The nonlinear function φ maps the pattern x = (x1 , . . . , xn ) of an n-dimensional input space I onto an f -dimensional feature space F : φ : n → f , x → φ(x). To perform PCA on the mapped patterns, we need to calculate the inner product (φ(x) · φ(y)) between the function values. However, this calculation is difficult, because the dimension f of the feature space F can be very high, possibly infinite. However, if the nonlinear map φ is defined through a kernel function k(x, y) which satisfies Mercer’s conditions, the inner products (φ(x) · φ(y)) can be calculated from the inner products (x · y). This technique is known as the “kernel  trick”.2  . A common choice is to use an exponential function: k(x, y)=exp − ||x−y|| σ2 The function φ with this kernel function maps an input pattern onto an infinite feature space F . The PCA of the mapped patterns is called kernel PCA[10]. 4.2

Kernel Whitening Transformation

In the following, we will explain how to generate the kernel whitening matrix Oφ from all the basis vectors of r d-dimensional nonlinear class subspaces Vk (k =

The Kernel Orthogonal Mutual Subspace Method

471

1 ∼ r), that is, r×d basis vectors. This calculation corresponds to the kernel PCA for the basis vectors of all the classes. Assume that a class k nonlinear subspace Vk is generated from l learning patterns xki (i = 1 ∼ l). The d basis vectors eki (i = 1∼d), which expand the subspace Vk , are defined by the following equation: eki =

l 

akij φ(xkj ) ,

(3)

j=1

where the coefficient akij is the j-th component of the eigenvector ai corresponding to the i-th largest eigenvalue λi of the l×l kernel matrix K defined by kij =(φ(xki ), φ(xkj )). ai is normalized to satisfy λi (ai · ai )=1. Next, assume that E is the matrix where all basis vectors are arranged as the column component: E = [e11 , . . . , e1d , . . . , er1 , . . . , erd ] .

(4)

Then, we solve the eigenvalue problem of the matrix Q defined by the following equation: Qb = βb

(5)

Qij = (Ei · Ej ),

(i, j = 1∼r×d) ,

where Ei means the i-th column component of the matrix E. In the above equation, the inner product between i-th basis vector eki of class k and j-th ∗ basis vector ekj of class k ∗ can be actually calculated as the linear combination ∗ ∗ of a kernel function value k(xk , xk ) of xk and xk . l l   ∗ ∗ ∗ akis φ(xks ) · akjt φ(xkt )) (eki · ekj ) = ( s=1

=

(6)

t=1

l  l 

∗

∗

akis akjt (φ(xks ) · φ(xkt ))

(7)

s=1 t=1

=

l  l 

∗

∗

akis akjt k(xks , xkt ) .

(8)

s=1 t=1

The i-th row vector Oφi of the kernel whitening matrix Oφ can be represented as the linear combination of the vectors Ej  (j = 1 ∼ r×d) using the eigenvector bi corresponding to the eigenvalue βi as the combination coefficient. O φi =

r×d  b √ij Ej  , βi j=1

(9)

where the vector bi is normalized to satisfy that βi (bi · bi ) is equal to 1. The row vectors of Oφ with eigenvalues β lower than a threshold value are discarded,

472

K. Fukui and O. Yamaguchi

since their reliability is low. Moreover, assume that E[j] is the η(j)-th basis vector of the class ζ(j). Then the above equation can be changed as follows: O φi =

r×d  j=1

=

l  b  ζ(j) √ij a φ(xζ(j) ) s βi s=1 η(j)s

r×d l  j=1 s=1

b ζ(j)  √ij aζ(j) ) . η(j)s φ(xs βi

(10)

(11)

Although we cannot calculate this vector Oφi , the inner product with the mapped vector φ(x) can be calculated. 4.3

Whitening Transformation of the Mapped Patterns

The mapped vector φ(x) is transformed by the kernel whitening matrix. This can be calculated from an input vector x and all r×l learning vectors xks (s = 1∼l, k = 1∼r) using the following equation: r×d  l  bij ζ(j) √ aζ(j) ) · φ(x)) (Oφi · φ(x))= η(j)s (φ(xs β i j=1 s=1

=

r×d  l  bij ζ(j) √ aζ(j) , x) . η(j)s k(xs β i j=1 s=1

(12)

(13)

Finally, the whitening transformed vector χ(φ(x))(= Oφ φ(x)) of the mapped vector φ(x) is represented as (z1 , z2 , . . . , zno ) , zi = (φ(x)·Oφi ), 1≤i≤no≤r×d, where no is the row number of Oφ mentioned above. 4.4

The KOMSM Algorithm

We construct the KOMSM by applying the linear MSM to the linear subspaces generated from the whitening transformed vectors {χ(φ(x))} in the feature space F as follows. In the learning stage: 1. The nonlinear mapped φ(xki ) of all the patterns xki (i = 1 ∼ l) belonging to class k(= 1∼c) are transformed by the kernel whitening matrix Oφ . 2. The basis vectors of the d-dimensional linear orthogonal reference k(= 1∼c) O subspace Pk φ are obtained as the eigenvectors of the correlation matrix generated from the whitening transformed pattern set {χ(φ(xk1 )), . . . , χ(φ(xkl ))}, corresponding to the d highest values. In the recognition stage: O

1. The linear input orthogonal subspace Pinφ is also generated from the whitenin ing transformed pattern set {χ(φ(xin 1 )), . . . , χ(φ(xl ))}.

The Kernel Orthogonal Mutual Subspace Method

473 O

2. The canonical angles between the linear orthogonal input subspace Pinφ and O all the linear orthogonal reference subspaces Pk φ (k = 1∼c) are calculated as the similarity. 3. Finally the object class is determined as the linear orthogonal reference subspace with the highest similarity S, given that S is above a threshold value. In the above process, it is possible to replace the generation process of the nonlinear orthogonal subspaces with the following processes. Firstly the input subspace and the reference subspaces are generated from the set of the nonlinear mapped patterns. Next the basis vectors of the generated subspaces are transformed by the kernel whitening matrix, and then the whitening transformed basis vectors are orthogonalized by the Gram-Schmidt method.

5

Evaluation Experiments

We compared the performances of KOMSM with previous methods (MSM[3], CMSM[6], OMSM[8], KMSM[4], KCMSM[11]) using the public data base of multi-view images (Cropped-close128 of ETH-80)[13] and the data set of front face images collected by ourselves. 5.1

3D Object Recognition (Experiment-I)

Thirty similar models were selected as the evaluation data from the ETH-80 database as shown in Fig.3. The images of each model were captured from 41 views as shown in Fig.3. All images are cropped, so that they contain only the object without any border area. The odd numbered images (21 frames) and the even numbered images (20 frames) were used for training and evaluation, respectively. We prepared 10 datasets for each model by making the start frame number i changes from 1 to 10 where 10 frames from i-th frame to i + 9-th is one set. The total number of the evaluation trials is 9000(=10×30×30). The evaluation was performed in terms of the recognition rate and the equal error rate (EER), which represents the error rate at the point where the false accept rate (FAR) is equal to the false reject rate (FRR). The experimental conditions are as follows. We converted the 180×180 pixel color images to 15×15 pixels monochrome images and use them as the evaluation data. Thus, the dimension n of a pattern is 225(=15×15). The dimensions of the input subspace and the reference subspaces were set to 7 for all methods. The whitening matrix O and the kernel whitening matrix Oφ were generated from thirty 20-dimensional linear class subspaces and thirty 20-dimensional nonlinear class subspaces respectively. The row numbers of O and Oφ were set to 100 and 550, respectively, on the basis of the eigenvalues β in Eq.(6). The dimensions of the constraint subspaces of the CMSM and KCMSM were set to 200 and 400, respectively, as used in [11]. The length of input vectors was not normalized for all the methods. A Gaussian kernel with σ 2 = 1e + 6 was used for all the kernel methods.

474

K. Fukui and O. Yamaguchi

Fig. 3. Left: Evaluation data, Bottom: cow, Middle: dog, Bottom: horse. This figure shows five of ten models. Right: All view-patterns of dog1, the rows indicated by arrows are used as training data. Table 1. Recognition rate and EER of each method (%) (Experiment-I) Method MSM CMSM-200 OMSM-100 KMSM KCMSM-400 KOMSM-550 Rate 78.6 92.33 89.6 96.33 99.67 99.67 EER 16.6 4.7 7.7 5.2 1.0 1.0

Table 1 shows the recognition rate and EER of each method. The recognition of multiple view images is typically a nonlinear problem. This is clearly shown by the experimental results that the performance of the nonlinear methods (KMSM, KCMSM and KOMSM) is superior to that of the linear methods (MSM, CMSM and OMSM). The performance of MSM was improved by the nonlinear extension of MSM to KMSM where the recognition rate increased from 78.6% to 96.3% and the EER decreased from 16.6% to 5.2%. KOMSM further improved the performance of KMSM where the recognition rate increased from 96.3% to 99.6% and the EER decreased from 5.7% to 1.0%. This confirms the effectiveness of the orthogonalization of the nonlinear subspaces, which serves as a feature extraction step in the feature space F . 5.2

Recognition of Face Image (Experiment-II)

We conducted the evaluation experiment of all the methods using the face images of 50 persons captured under 10 kinds of lighting. We cropped the 15 ×15 pixel

Fig. 4. Face images: From left, Lighting1∼Lighting10

The Kernel Orthogonal Mutual Subspace Method

475

Table 2. Recognition rate and EER of each method (%)(Experiment-II) Method MSM CMSM-200 OMSM KMSM KCMSM-1050 KOMSM Recognition rate 91.74 91.30 97.09 91.15 97.40 97.42 EER 12.0 7.5 6.3 11.0 4.3 3.5

face images from the 320 × 240 pixel input images based on the positions of pupils and nostrils. The normalized face patterns of subjects 1-25 in lighting conditions L1-L10 were used for generating the difference subspace D, the kernel difference subspace Dφ , the whitening matrix O and the kernel whitening matrix Oφ . The face patterns extracted from the images of the other subjects, 26-50, in lighting conditions L1-L10 were used for evaluation. The number of the data of each person is 150∼180 frames for each lighting condition. The data was divided into 15∼18 sub datasets by every 10 frames. The input subspaces were generated from these sub datasets. The dimension of the input subspace and reference subspaces were set to 7 for all the methods. The difference subspace D and the whitening matrix O were generated from 25 60-dimensional linear subspaces of 1∼25 persons. The kernel difference subspace Dφ and the kernel whitening matrix Oφ were generated from 25 60-dimensional nonlinear class subspaces. The row numbers of O and Oφ were set to the full dimensions, 225 and 1500, respectively. The dimensions of the generalized difference subspace and the kernel generalized difference subspace were set to 200 and 1050, respectively. We used a Gaussian kernel with σ 2 = 1.0 for all nonlinear methods. Table 2 shows the recognition rate and the EER of each method. The difference between the recognition rates of OMSM and KOMSM was small, while the EER decreased from 6.3% to 3.5%. This implies that the data sets used in this task do not exhibit highly nonlinear structure. The good performance of KOMSM and KCMSM are also demonstrated in this experiment. In particular the EER of KOMSM is very low. Although the performance of KOMSM and KCMSM are at the same level regardless of their different principals of orthogonalization, KOMSM has an advantage in selecting parameters compared to KCMSM. KOMSM does not have any parameters to be tuned. In contrast, the dimension of the constraint subspace used in KCMSM needs to be carefully selected in prior experiments, since the performance of KCMSM strongly depends on this value.

6

Conclusion

In this paper we have introduced the kernel orthogonal mutual subspace method (KOMSM) and applied it to 3D object recognition. The essence of the KOMSM is to orthogonalize nonlinear subspaces based on Fukunaga and Koontz’s framework before applying the KMSM. We have confirmed that this orthogonalization provides a strong feature extraction method for the KMSM and that the performance of the KMSM is improved significantly. This was shown by evaluation

476

K. Fukui and O. Yamaguchi

on multi-view image sets of 3D objects as well as frontal face images. In future work, we will attempt to find the efficient computation of the eigen problems of the matrixes K and Q with the sizes which are proportional to the numbers of the classes and the training patterns.

References 1. Oja, E.: Subspace methods of pattern recognition. Research Studies Press (1983) 2. Fukunaga, K., Koontz, W.C.G.: Application of the Karhunen-Loeve expansion to feature selection and ordering. IEEE Trans. Computers C-19, 311–318 (1970) 3. Maeda, K., Watanabe, S.: A pattern matching method with local structure: Trans. IEICE J68-D(3), 345–352 (1985) (in Japanese) 4. Sakano, H., Mukawa, N., Nakamura, T.: Kernel Mutual Subspace Method and its Application for Object Recognition. Electronics and Communications in Japan 88, 45–53 (2005) 5. Wolf, L., Shashua, A.: Kernel principal angles for classification machines with applications to image sequence interpretation. In: CVPR 2003, pp. 635–642 (2003) 6. Fukui, K., Yamaguchi, O.: Face recognition using multi-viewpoint patterns for robot vision. In: ISRR. 11th International Symposium of Robotics Research, pp. 192–201 (2003) 7. Nagao, K., Sohma, M.: Weak orthogonalization of face and perturbation for recognition. In: CVPR 1998, pp. 845–852 (1998) 8. Kawahara, T., Nishiyama, M., Yamaguchi, O.: Face recognition by orthogonal mutual subspace method. CVIM-151 151, 17–24 (2005) (in Japanese) 9. Kim, T.-K., Kittler, J., Cipolla, R.: Incremental Learning of Locally Orthogonal Subspaces for Set-based Object Recognition. In: BMVC 2006, pp. 559–568 (2006) 10. Sch¨ olkopf, B., Smola, A., M¨ uller, K.-R.: Nonlinear principal component analysis as a kernel eigenvalue problem. Neural Computation 10, 1299–1319 (1998) 11. Fukui, K., Stenger, B., Yamaguchi, O.: A framework for 3D object recognition using the kernel constrained mutual subspace method. In: Narayanan, P.J., Nayar, S.K., Shum, H.-Y. (eds.) ACCV 2006. LNCS, vol. 3851, pp. 315–324. Springer, Heidelberg (2006) 12. Chatelin, F.: Eigenvalues of matrices. John Wiley & Sons, Chichester (1993) 13. Leibe, B., Schiele, B.: Analyzing appearance and contour based methods for object categorization. In: CVPR 2003, pp. 409–415 (2003)

Viewpoint Insensitive Action Recognition Using Envelop Shape Feiyue huang and Guangyou Xu Department of Computer Science and Technology, Tsinghua University, 100084, Beijing, China [email protected], [email protected]

Abstract. Action recognition is a popular and important research topic in computer vision. However, it is challenging when facing viewpoint variance. So far, most researches in action recognition remain rooted in view-dependent representations. Some view invariance approaches have been proposed, but most of them suffer from some weaknesses, such as lack of abundant information for recognition, dependency on robust meaningful feature detection or point correspondence. To perform viewpoint and subject independent action recognition, we propose a representation named “Envelop Shape” which is viewpoint insensitive. “Envelop Shape” is easy to acquire from silhouettes using two orthogonal cameras. It makes full use of two cameras’ silhouettes to dispel influence caused by human body’s vertical rotation, which is often the primary viewpoint variance. With the help of “Envelop Shape”, we obtained inspiring results on action recognition independent of subject and viewpoint. Results indicate that “Envelop Shape” representation contains enough discriminating features for action recognition. Keywords: Viewpoint Insensitive, Envelop Shape, Action Recognition, HMM.

1 Introduction Human action recognition is an active area of research in computer vision. There have been several surveys which tried to summarize and classify previous existing approaches on this area [1], [2], [3], [4]. In this paper, we develop a general approach to recognize actions independent of viewpoint and subject, focusing on discovering viewpoint invariance for action recognition. In our research work, we define human action recognition system as a system made up of three modules: Preprocessing, Pose Estimation and Recognition. Preprocessing module includes human detection and tracking, it extracts low level representation for pose estimation. Pose Estimation module is the process of identifying and representing how a human body and/or individual limbs are configured in a single frame. Recognition module uses results of pose estimation of frames to classify actions. Here we define posture as a kind of representation of human body in a single frame, for example, horizontal and vertical histograms of silhouette [5], vector of distances from boundary pixels to the centroid [6]. In our opinion, posture representation is one of the most basic and key issues in action recognition system. Y. Yagi et al. (Eds.): ACCV 2007, Part II, LNCS 4844, pp. 477–486, 2007. © Springer-Verlag Berlin Heidelberg 2007

478

F. huang and G. Xu

It is well known that a good representation for classification should have such measurement property whose values are similar for objects in the same category while very different for objects in the different categories. So this leads to the idea of seeking distinguishing features that are invariant to irrelevant transformations of the input [7]. In the case of recognition of human action, we argue that a good feature representation should be able to tolerate variations in viewpoint, human subject, background, illumination and so on. Among them, the most important invariance is viewpoint invariance. We can perform training and recognition according to given environments and specialized persons. But in order to perform natural human action recognition, we can’t limit human body’s movement and rotation at any time which inevitably leads to variable viewpoint. It is a challenge to find a viewpoint invariant posture representation for action recognition. There have been some proposed approaches on viewpoint invariant action recognition. Campbell et al. proposed a complex 3D gesture recognition system based on stereo data [8]. Seitz and Dyer described an approach to detect cyclic motion that is affine invariant [9]. Cen Rao did a lot of research work on view invariant analysis of human activities [10], [11]. He used trajectory of hand centroid to describe an action performed by one hand. He discovered affine invariance of trajectory and his system can work automatically. Vasu Parameswaran also focused on approaches for view invariant human action recognition [12], [13]. He chose six joints of the body and calculated their 3D invariants of each posture. So each posture can be represented by a parametric surface in 3D invariance space. Daniel et al. introduced Motion History Volumes as a free viewpoint representation for action recognition, which needs multiple calibrated cameras [14]. Though there have been some research work on viewpoint invariant action recognition, there are still many problems to be solved. Most approaches depend on robust meaningful feature detection or point correspondence, which, as we know, are often hard to implement. And there is a tradeoff to be insensitive to viewpoint is that some useful information for discriminating different actions is often eliminated. How to make representation insensitive to viewpoint while still keeping appropriate discriminating information for recognition appears to be the key issue. In this regard, we propose a posture representation named “Envelop Shape”. Under the assumption of affine camera projection model, we prove it from both theory and experiments that such representation is viewpoint insensitive for action recognition. “Envelop Shape” is easy to acquire from low level features, which can be obtained from silhouettes of subjects by using two orthogonal cameras. It conveys more information compared to previous view invariant representation for action recognition. And it does not rely on any meaningful feature detection or point correspondence, which as we know is often difficult and sensitive to errors. With the help of our proposed representation, we develop our action recognition system. Experiment results show that our system has impressive discriminating ability for actions independent of subject and viewpoint. The remainder of this paper is organized as follows. In section 2, we present our viewpoint insensitive representation. We propose the implementation of our action recognition system and give experiment results in section 3. We conclude in section 4.

Viewpoint Insensitive Action Recognition Using Envelop Shape

479

2 View Invariant Posture Representation In human action recognition, representation is a basic and key issue. A good system should have viewpoint invariance to perform natural action recognition. So a direct idea is to discover viewpoint invariant representation which means that measurements using this representation will keep almost the same even under different viewpoints. 2.1 Viewpoint in Action Recognition Viewpoint transformation can be separated into two parts, translation and rotation. In action recognition, almost all representations have translation invariance, so we only consider rotation invariance. Figure 1 shows the coordinate in our system. In this coordinate, the Y-axis is vertical. There are three kinds of rotation terms used to describe the rotation in the coordinate: roll, pitch and yaw. Roll, pitch and yaw describe the rotation around the Z-axis (α), X-axis (β), and Y-axis (γ), respectively.

Fig. 1. Coordinate in our system

Fig. 2. Two cameras’ configuration

It is quite often that actor makes some kind of action with his body yawing, when roaming in front of the fixed camera, for example the actor on the stage or teacher in front of the blackboard. In this case the yaw motion of the body causes variation of the viewpoint, but actions still make sense. In this regard, we classify human postures into a same category if only yaw rotation exists and we classify human postures in

480

F. huang and G. Xu

different categories if there exist the other two kinds of rotation terms, roll or pitch. For example, when a human is standing compared with lying on the ground, the rotation term is roll or pitch, and we regard them as different postures. However, if a human only turns his body facing another direction, he is thought to be acting the same posture. With above discussion, we can conclude that we need only consider invariance on yaw rotation for most viewpoint invariant action recognition. 2.2 Envelop Shape Representation In the practical situation of human action recognition, because the depth range of human body is usually small compared with the distance between human and the camera, the affine camera model can be used. To acquire viewpoint invariant representation for action recognition, a two cameras’ configuration is proposed as Figure 2. The image planes of two cameras are both parallel to the vertical axis Y, and the optical axes are orthogonal. Let us consider a horizontal section plane of human body, projections of all points on this section plane into the image plane 1 are on the line l and projections of all points on this section into the image plane 2 are on the line l’. The line l are the epipolar line of point p’ and the line l’ are the epipolar line of point p. To discover the yaw rotation invariance, we need only to analyze the 2D horizontal section shape’s projection on X-axis and Y-axis in different rotations.

Fig. 3. 2D Shape projection on X-axis and Y-axis in different rotations

In Figure 3, let us suppose a 2D shape “S” whose projection segments in original coordinate XY are AB and BC, so it is in the rectangle ABCD. In another coordinate X’Y’ which rotates at an angle θ, its projection segments will be in segment EF and FG. Let us define the original projection segment’s length as x and y, the new projection segment’s length as x’ and y’. We can get the following relationships:

x ' ≤ x cos θ + y sin θ y ' ≤ y cos θ + x sin θ

(1)

Let us define value “r” as equation 2, so we can get equation 3. r =

x2 + y2

r ' = x' 2 + y ' 2 ≤ x 2 + y 2 + 2 xy sin 2θ ≤ x 2 + y 2 + 2 xy ≤ 2r

(2) (3)

Let r0 be the minimal value of “r” s among all rotations. Then at any rotation, the r value will meet the following expression:

Viewpoint Insensitive Action Recognition Using Envelop Shape

r0 ≤ r ≤

2 r0

481

(4)

This is a quite small value range compared to the unlimited range of ratios between x’ and x or y’ and y, which indicates that we find a view insensitive representation of human body. At each horizontal section plane, we can calculate an “r” value using equation (2). From a single frame of human posture, we get a vector of “r” value. Since this vector can envelop the human body silhouette inside, we call this representation of vector of “r” values as “Envelop Shape”. Here we show some “Envelop Shape” images of synthesized human body model data at different viewpoints. Figure 4 shows two kinds of synthetic postures rotated on vertical axis Y at eight different angles, the first two rows are silhouettes of images in two cameras, and the third rows are “Envelop Shape” images. We can see that the Envelop Shapes does really change few facing viewpoint variance.

Fig. 4. Two kinds of postures at different viewpoints

Though we propose our cameras’ configuration as Figure 2, that is, two cameras should be placed with image planes both parallel to the vertical axis Y and optical axes orthogonal. It does not need accurate calibration. As we know, accurate calibration is often complex. It is enough when the cameras are placed approximately meeting this need of cameras placement. That means we do not need to spend a lot of time in configuring the accurate placement. As we mentioned above, this kind of representation is just view insensitive, so approximate value can also work. We will show our experiments in section 3.The videos are collected with just rough placement of two cameras, while we will see that the result is still inspiring good. Here is a brief description of our algorithm to generate Envelop Shape representation. 1. Extract silhouettes of human body from two cameras’ video data. 2. Perform a scale normalization of the silhouettes. 3. Use expression (2) to calculate “r” value at each height of the silhouettes, the x and y are the corresponding width of the two normalized silhouettes at this height. Envelop Shape representation has following advantages: 1. It keeps information on two dimensional degrees of freedom, vertical axis and horizontal plane. It has more information than some simple view invariance representation such as trajectory projection which is in fact one dimensional. Therefore, it has better discriminating ability, also it is view insensitive.

482

F. huang and G. Xu

2. It is easy to obtain. Only silhouettes are required as input, which are easier to extract than meaningful feature detection, tracking or point correspondence.

3 Action Recognition Experiment Results With the help of Envelop Shape representation, we deploy our arbitrary viewpoint action recognition system in a smart classroom. Figure 5 shows system flow diagram. We first use “Pfinder” algorithm to extract human silhouette [15]. With two cameras’ silhouette video sequences as original input, we generate “envelop shape” vector of each frame. We then use principal component analysis (PCA) to perform dimensional reduction.

Fig. 5. Action Recognition System Flow Diagram

For each video, after preprocessing and posture representation steps are completed, time sequential feature vectors can be obtained. There are many algorithms for classifying time sequential feature vectors, such as Hidden Markov Model[16], Coupled Hidden Markov Model[17] or stochastic parsing[18] and so on. Here we use continuous Hidden Markov Model for action training and recognizing. For action recognition experiment, we collected our own database of action video sequences. It contains seven different actors. Each actor performs nine natural actions which are “Point To”, “Raise Hand”, “Wave Hand”, “Touch Head”, “Communication”, “Bow”, “Pick Up”, “Kick” and “Walk”. Each action is performed by every actor three times repetitively at three arbitrary view point. Figure 6 and Figure 7 show examples of our experiment data. Each figure contains two groups of sampled action sequences. Each group contains five rows, the first two rows are images of two cameras, the next two rows are silhouettes extracted using “Pfinder” and the last row is normalized Envelop Shape vector images. (Each action sequence contains about 30 frames of images. Figure 6 and 7 only show partial sampled frames of each sequence.) As we can see, action sequences are collected at arbitrary view point, which means that our experiment is viewpoint independent. Experiment parameters are as follows: the dimension of input vector (which is, output after PCA dimensional reduction of Envelop Shape vector) is 8, the number of states of each HMM Model is 5 and the number of mixture Gaussian Models for observation is 10. With our video database, we carry out subject dependent and subject independent experiments separately.

Viewpoint Insensitive Action Recognition Using Envelop Shape

Fig. 6. Two groups of “point to” action sequences

Fig. 7. Two groups of “walk” action sequences

483

484

F. huang and G. Xu

In the case of subject independent action recognition, for each type of actions, we train an entire HMM models for all actors in train set and use the entire HMM models for recognition. For each action, we use first five actors’ video sequences as training sets and the last two actors’ sequences as test sets. That is to say, for each action, there are 45 sequences as train sets and 18 sequences as test sets. Table 2 shows the correct recognition rate. Table 1. Subject dependent recognition result

Point. Raise. Wave Touch Comm Bow Pick Kick Walk

Actor1 6 3 6 3 6 3 6 2 6 2 6 3 6 3 6 3 6 2

Actor2 6 3 6 3 6 3 6 3 5 3 6 3 6 3 6 3 6 3

Actor3 6 3 6 3 6 3 6 3 6 3 6 3 6 3 6 3 6 3

Actor4 6 3 6 3 5 3 6 3 6 3 6 3 6 3 6 3 6 3

Actor5 6 3 6 3 6 3 6 3 5 2 6 3 6 3 6 3 6 3

Actor6 6 3 6 3 6 3 6 3 6 3 6 3 6 3 6 3 6 3

Actor7 6 3 6 3 6 3 6 3 6 3 6 3 6 3 6 3 6 3

Aver. % 100 100 100 95.2 97.6 100 100 95.2 95.2 90.5 100 100 100 100 100 100 100 95.2

Table 2. Subject independent recognition result

Point. Raise. Wave Touch Comm Bow Pick Kick Walk

Train set(%) 97.8 100 95.6 95.6 88.9 100 100 100 100

Test set(%) 100 100 88.9 94.4 83.3 100 94.4 100 94.4

Table 3. Comparison with view variant action recognition methods

Point. Raise. Wave Touch Comm Bow Pick Kick Walk

Envelop Shape (%) 100 94.4 100 100 88.9 94.4 94.4 88.9 83.3 83.3 100 100 94.4 100 100 94.4 94.4 94.4

Horizontal projection Motion Feature [19] (%) of silhouettes (%) 44.4 61.1 33.3 38.8 27.7 61.1 38.8 55.5 33.3

88.9 94.4 61.1 83.3 61.1 94.4 72.2 83.3 72.2

38.8 33.3 55.5 22.2 27.7. 38.8 33.3 38.8 44.4

100 94.4 94.4 100 88.9 94.4 83.3 88.9 100

Viewpoint Insensitive Action Recognition Using Envelop Shape

485

In order to make our approach on view invariant action recognition more convincing, we also tried some comparison experiments. In table 3, we give our results compared with view dependent methods. The table’s first row shows three kinds of methods we used. The first method is our proposed approach, “Envelop Shape”. As the second method, we use vector of horizontal projection of silhouettes (that is, the “x” in expression (2)) as input, which is view variant. As the third method, we refer to [19] which uses motion features and is a view dependent method. Below each method, there are two sub columns. The first column gives recognition rates of subject independent action recognition for any view points. The second column gives average recognition rates at a specified view point. We can see that in view independent scenario, only “Envelop Shape” method performs well, however the other two methods perform poor since they are view dependent.

4 Conclusion With the help of “Envelop Shape” representation, we set up an action recognition system despite of viewpoint variance. Experiment shows that with the help of Envelop Shape representation, our system has achieved a high correct recognition rate in the case of free view point actions. Results indicate that "Envelop Shape" representation contains enough discriminating features even for subject-independent action recognition. “Envelop Shape” representation is view insensitive, and compared to previous approaches, it is easier to acquire and has more abundant information. It does not need any meaningful feature detection or point correspondence, which as we know is often difficult to get and sensitive to errors. However as a view insensitive representation, it loses some view variant information sometimes important for certain action recognition. For example, we can not distinguish left or right hand’s movement only with this representation. Some view variant information may help for solving this kind of problem. How to combine this representation and other view variant information? It is further work to accomplish. Acknowledgements. This work is supported by National Science Foundation of China under grant No 60673189 and No 60433030.

References 1. Cedras, C., Shah, M.: Motion-based recognition: a survey. Image and Vision Computing 13(2), 129–155 (1995) 2. Aggarwal, J.K., Cai, Q.: Human motion analysis: a review. Computer Vision and Image Understanding 73(3), 428–440 (1999) 3. Moeslund, T.B., Granum, E.: A survey of computer vision-based human motion capture. Computer Vision and Image Understanding 81(3), 231–268 (2001) 4. Wang, L., Hu, W., Tan, T.: Recent Developments in Human Motion Analysis. Pattern Recognition 36(3), 585–601 (2003)

486

F. huang and G. Xu

5. Leo, M., D’Orazio, T., Spagnolo, P.: International Multimedia Conference. In: Proceedings of the ACM 2nd international workshop on Video surveillance & sensor networks, pp. 124–130. ACM Press, New York (2004) 6. Wang, L., Tan, T., Ning, H., Hu, W.: Silhouette analysis-based gait recognition for human identification. IEEE Trans on Pattern Analysis and Machine Intelligence 25(12), 1505– 1518 (2003) 7. Duda, R.O., Hart, P.E., Stock, D.G.: Pattern Classification, p. 11 8. Campbell, L.W., Becker, D.A., Azarbayejani, A., Bobick, A.F., Pentland, A.: Invariant Features for 3D Gesture Recognition. In: Proceedings of International Conference on Automatic Face and Gesture Recognition, pp. 157–162 (1996) 9. Seitzl, S.M., Dyerl, C.R.: View-Invariant Analysis of Cyclic Motion. International Journal of Computer Vision (1997) 10. Rao, C., Yilmaz, A., Shah, M.: View-Invariant Representation And Recognition of Actions. International Journal of Computer Vision 50(2) (2002) 11. Rao, C., Shah, M., Mahmood, T.S.: Action Rectionition based onView Invariant Spatiotemporal Analysis. In: ACM Multimedia 2003, November 2-8, Berkeley, CA USA (2003) 12. Parameswaran, V., Chellappa, R.: Using 2D Projective Invariance for Human Action Recognition. International Journal of Computer Vision (2005) 13. Parameswaran, V., Chellappa, R.: Human Action Recognition Using Mutual Invariants, Computer Vision and Image Understanding (2005) 14. Weinland, D., Ronfard, R., Boyer, E.: Free Viewpoint Action Recognition using Motion History Volumes, Computer Vision and Image Understanding (2006) 15. Wren, C., Azarbayejani, A., Darrell, T., Pentland, A.: Pfinder: real-time tracking of the human body. IEEE Trans on Pattern Analysis and Machine Intelligence 19(7), 780–785 (1997) 16. Yamato, J., Ohya, J., Ishii, K.: Recognizing human action in time-sequetial images using hidden markov model. In: Proc. 1992 IEEE Conf. on Computer Vision and Pattern Rec., pp. 379–385. IEEE Press, Los Alamitos (1992) 17. Brand, M., Oliver, N., Pentland, A.: Coupled hidden Markov models for complex action recognition. In: Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition, Puerto Rico, IEEE Computer Society Press, Los Alamitos (1997) 18. Ivanov, Y.A., Bobick, A.F.: Recognition of visual activities and interactions by stochastic parsing. IEEE Transactions on Pattern Analysis and Machine Intelligence 22(8), 852–872 (2000) 19. Masoud, O., Papanikolopoulos, N.: A method for human action recognition. Image and Vision Computing 21, 729–743 (2003)

Unsupervised Identification of Multiple Objects of Interest from Multiple Images: dISCOVER Devi Parikh and Tsuhan Chen Carnegie Mellon University {dparikh,tsuhan}@cmu.edu Abstract. Given a collection of images of offices, what would we say we see in the images? The objects of interest are likely to be monitors, keyboards, phones, etc. Such identification of the foreground in a scene is important to avoid distractions caused by background clutter and facilitates better understanding of the scene. It is crucial for such an identification to be unsupervised to avoid extensive human labeling as well as biases induced by human intervention. Most interesting scenes contain multiple objects of interest. Hence, it would be useful to separate the foreground into the multiple objects it contains. We propose dISCOVER, an unsupervised approach to identifying the multiple objects of interest in a scene from a collection of images. In order to achieve this, it exploits the consistency in foreground objects - in terms of occurrence and geometry - across the multiple images of the scene.

1 Introduction Given a collection of images of a scene, in order to better understand the scene, it would be helpful to be able to identify the foreground separate from the background clutter. We interpret foreground to be the objects of interest, the objects that are found frequently across the images of the scene. In a collection of images of offices, for instance, we may find a candy box in some office image. However, we would perceive it to be part of the background clutter because most office scenes don’t have candy boxes. Most interesting scenes contain multiple objects of interest. Office scenes contain monitors, keyboards, chairs, desks, phones, etc. It would be useful if, given a collection of images of offices, we can identify the foreground region from the background clutter/objects and further more, separate the identified foreground into the different objects. This can then be used to study the interactions among the multiple objects of interest in the scene, learn models for these objects for object detection, track multiple objects in a video, etc. It is crucial to approach this problem in an unsupervised manner. First, it is extremely time consuming to annotate images containing multiple objects. Second, human annotation could introduce subjective biases as to which objects are the foreground objects. Unsupervised approaches on the other hand require no hand annotations, truly capture the properties of the data, and let the objects of interest emerge from the collection of images. In our approach we focus on rigid objects. We exploit two intuitive notions. First, the parts of the images that occur frequently across images are likely to belong to the foreground. And second, only those parts of the foreground that are found at geometrically consistent relative locations are likely to belong to the same rigid object. Y. Yagi et al. (Eds.): ACCV 2007, Part II, LNCS 4844, pp. 487–496, 2007. c Springer-Verlag Berlin Heidelberg 2007 

488

D. Parikh and T. Chen

Several approaches in literature address the problem of foreground identification. First of, we differentiate our work from image segmentation approaches. These approaches are based on low level cues and aim to separate a given image into several regions with pixel level accuracies. Our goal is higher level, where we wish to separate the local-parts of the images that belong to the objects of interest from those that lie on background clutter using cues from multiple images. To re-iterate, several image segmentation approaches aim at finding regions that are consistent within a single image in color, texture, etc. We are however interested in finding objects in the scene that are consistent across multiple images in occurrence and geometry. Several approaches for discovering the topic of interest have been proposed such as discovering main characters [1] or objects and scenes [2] in movies or celebrities in collections of news clippings [3]. Recently, statistical text analysis tools such as probabilistic Latent Semantic Analysis (pLSA) [4] and Latent Dirichlet Allocation (LDA) [5] have been applied to images for discovering object and scene categories [6,7,8]. These use unordered bag-of-words [9] representation of documents to automatically (unsupervised) discover topics in a large corpus of documents/images. However these approaches, which we loosely refer to as popularity based approaches, do not incorporate any spatial information. Hence, while they can identify the foreground separate from the background, they can not further separate the foreground into multiple objects. Hence, these methods have been applied to images that contain only one foreground object. We further illustrate this point in our results. These popularity based approaches can separate the multiple objects of interest only if they are provided images that contain different number of these objects. For the office setting, in order to discover the monitor and keyboard separately, plSA, for instance, would require several images with just the monitor, and just the keyboard (and also a specified number of topics of interest). This is not a natural setting for images of office scenes. Leordeanu, et al. [10] propose an approach to unsupervised learning of the object model from its low resolution video. However, this approach is also based on co-occurrence and hence can not separate out multiple objects in the foreground. Several approaches have been proposed to incorporate spatial information in the popularity based approaches [11,12,13,14], however, only with the purpose of robustly identifying the single foreground object in the image, and not for separation of the foreground into multiple objects. Russell, et al. [15], through their approach of breaking an image down into multiple segments and treating each segment individually, can deal with multiple objects as a byproduct. However, although from multiple segmentations, they rely on consistent segmentations of the foreground objects. Further on the object detection/recognition front instead of object discovery, object localization approaches could be considered, with a stretch of argument, to provide rough foreground/background separation. Part-based approaches, as is ours, however towards this goal of object localization, have been proposed such as [16,17] which use spatial statistics of parts to obtain objects masks. However, these are supervised approaches for single objects. Unsupervised part-based approaches for learning the object models for recognition have also been proposed, such as [18,19]. However they deal with single objects.

Unsupervised Identification of Multiple Objects of Interest from Multiple Images

Images of a particular scene category

Feature extraction

489

Correspondences

Foreground identification Multiple objects (foreground)

Recursive clustering

Interaction between pairs of features

Fig. 1. Flow of dISCOVER for unsupervised identification of multiple objects of interest image1 ba b

a

image2 a(a)

=A

A d(B, )

a(ba) =

B

Fig. 2. An illustration of the geometric consistency metric used to retain good correspondences

The rest of the paper is organized as follows. Section 2 describes our algorithm dISCOVER, followed by experimental results in Section 3 and conclusion in Section 4.

2 dISCOVER Our approach, dISCOVER, is summarized in Fig. 1. The input to dISCOVER is a collection of images taken from a particular scene, and the desired output is the identified foreground separated into the multiple objects it contains. 2.1 Feature Extraction Given the collection of images taken from a particular scene, local features describing interest points/parts are extracted in all the images. These features may be appearance based features such as SIFT [20], shape based features such as shape context [21], geometric blur [22], or any such discriminative local descriptors as may be suitable for the objects under consideration. In our current implementation, we use the Derivative of Gaussian interest point detector, and SIFT features as our local descriptors. 2.2 Correspondences Having extracted features from all images, correspondences between these local parts are to be identified across images. For a given pair of images, potential correspondences are identified by finding k nearest neighbors of each feature point from one image in the other image. We use Euclidean distance between the SIFT descriptors to determine the nearest neighbors. The geometric consistency between every pair of correspondences is computed to build a geometric consistency adjacency matrix.

490

D. Parikh and T. Chen

Suppose we wish to compute the geometric consistency between a pair of correspondences shown in Fig. 2 involving interest regions a and b in image1 and A and B in image2 . All interest regions have a scale and orientation associated with them. Let φa be the similarity transform that transforms a to A. βA is the transformed ba , the relative location of b with respect to a in image1 , using φa . β is thus the estimated location of B in the image2 based on φa . If a and A, as well as b and B are geometrically consistent, the distance between β and B, d(B, β) would be small. A score that decreases exponentially with increasing d(B, β) is used to quantify the geometric consistency of the pair of correspondences. To make the score symmetric, a is similarly mapped to α using the transform φb that maps b to B, and the score is based on max(d(B, β), d(A, α)). This metric provides us with invariance only to scale and rotation, the assumption being that the distortion due to affine transformation in realistic scenarios is minimal among local features that are closely located on the same object. Having computed the geometric consistency score between all possible pairs of correspondences, a spectral technique is applied to the geometric consistency adjacency matrix to retain only the geometrically consistent correspondences [23]. This helps eliminate most of the background clutter. This also enables us to deal with incorrect low-level correspondences among the SIFT features that can not be reliably matched, for instance at various corners and edges found in an office setting. To deal with multiple objects in the scene, an iterative form of [23] is used. However, it should be noted that due to noise, affine and perspective transformations of objects, etc. correspondences of all parts even on a single object do not always form one strong cluster and hence are not entirely obtained in a single iteration, instead they are obtained over several iterations. 2.3 Foreground Identification Only the feature points that find geometrically consistent correspondences in most other images are retained. This is in accordance to our perception that the objects of interest are those that occur frequently across the image collection. Also, this post processing step helps to eliminate the remaining background features that may have found geometrically consistent correspondences in another image by chance. Using multiple images gives us the power to be able to eliminate these random errors which would not be consistent across images. However, we do not require features to be present in all images either in order to be retained. This allows us to handle occlusions, severe view point changes, etc. Since these affect different parts of the objects across images, it is unlikely that a significant portion of the object will not be matched in many images, and hence be eliminated by this step. Also, this enables us to deal with different number of objects in the scene across images, again, the assumption being that the objects that are present in most images are the objects of interest (foreground), while those that are present in a few images are part of the background clutter. This proportion can be varied to suit the scenario at hand. We now have a reliable set of foreground feature points and a set of correspondences among all images. An illustration can be seen in Fig. 3 where only a subset of the detected features and their correspondences are retained. It should be noted that the approach being unsupervised, there is no notion of an object yet. We only have a cloud of features in each image which have all been identified as foreground and

Unsupervised Identification of Multiple Objects of Interest from Multiple Images

491

Features discarded as no geometrically consistent correspondences in any image (background) Features discarded as geometrically consistent correspondences not found across enough images (occlusions, etc.) Features retained Geometric consistency adjacency matrix:

Fig. 3. An illustration of the correspondences and features retained during feature selection. The images contain two foreground objects, and some background. An illustration of the geometric consistency adjacency matrix of the graph that would be built for this set up is also shown.

correspondences among them. The goal is to now separate these features into different groups, where each group corresponds to a foreground object in the scene. 2.4 Interaction Between Pairs of Features In order to separate the cloud of retained feature points into clusters, a graph is built over the feature points, where the weights on the edge between the nodes represents the interaction between the pair of features across the images. The metric used to capture the interaction between the pairs of features is the same geometric consistency as computed in Section 2.2, except now averaged across all pairs of images that contain these features. While the geometric consistency could contain errors for a particular pair of images due to errors in correspondences, etc. averaging across all pairs suppress the contribution of these erroneous matchings and amplifies the true interaction among the pairs of features. If the geometric consistency between two feature points is high, they are likely to belong to the same rigid object. On the other hand, features that belong to different objects would be geometrically inconsistent because the different objects are likely to be found in different configurations across images. An illustration of the geometric consistency adjacency matrix can be seen in Fig. 3. Again, there is no concept of an object yet. The features in Fig. 3 are arranged in an order that correspond to the objects, and each object is shown to have only two features, only for illustration purposes. 2.5 Recursive Clustering Having built the graph capturing the interaction between all pairs of features across images, recursive clustering is performed on this graph. At each step, the graph is clustered into two clusters. The properties of each cluster are analyzed, and one or both of the clusters are further separated into two clusters, and so on. If the variance in the

492

D. Parikh and T. Chen

(a)

(b)

Fig. 4. (a) A subset of the synthetic images used as input to dISCOVER (b) Background suppressed for visualization purposes

adjacency matrix corresponding to a certain cluster (subgraph) is very low but with a high mean, it is assumed to contain parts from a single object, and is hence not divided further. Since the statistics of each of the clusters formed are analyzed to determine if it should be further clustered or not, the number of foreground objects need not be known a priori. This is an advantage as compared to plSA or parametric methods such as fitting a mixture Guassians to the foreground features spatial distribution. dISCOVER is nonparametric. We use normalized cuts [24] to perform the clustering. The code provided at [25] was used.

3 Results 3.1 Synthetic Images dISCOVER uses two aspects: popularity and geometric consistency. These can be loosely thought of as first order as well as second order statistics. In the first set of experiments, we use synthetic images to demonstrate the inadequacy of any of these alone. To illustrate our point - we consider 50×50 synthetic images as shown in Fig. 4(a). The images contain 2500 distinct intensity values, of which 128, randomly selected from the 2500, always lie on the foreground objects and the rest is background. We consider each pixel in the image to be an interest point, and the descriptor of each pixel is the intensity value of the pixel. To make visualization clearer, we display only the foreground pixels of these images in Fig. 4(b). It is evident from these that there are two foreground objects of interest. We assume that the objects undergo pure translation only. We now demonstrate the use of pLSA, as an example of an unsupervised popularity based foreground identification algorithm, on 50 such images. Since pLSA requires negative images without the foreground objects we also input 50 random negative images to pLSA, which dISCOVER does not need. If we specify pLSA to discover 2 topics, the result obtained is shown in Fig 5. It can be seen that it can identify the foreground from the background, but is unable to further separate the foreground into multiple objects. One may argue that we could further process these results and fit a mixture of Gaussians (for instance) to further separate the foreground into multiple objects. However this would require us to know the number of foreground objects a priori and also the distribution of features on the objects need not be Gaussian as in these images. If we specify pLSA to discover 3 topics instead, with the hope that it might separate the foreground into 2 objects, we find that it randomly splits the background into 2 topics, while still maintaining a single foreground topic, as seen in Fig. 5. This is because pLSA simply incorporates occurrence (popularity) and no spatial information.

Unsupervised Identification of Multiple Objects of Interest from Multiple Images Image

pLSA: 2 topics pLSA: 3 topics

493

dISCOVER

Fig. 5. Comparison of results obtained using pLSA with those obtained using dISCOVER

Hence, pLSA is inherently missing the information required to perceive the features on one of the foreground objects any different than those on the second object and hence separate them. On the other hand, dISCOVER does incorporate this spatial/geometric information and hence can separate the foreground objects. Since the input images are assumed to allow only translation of the foreground objects and the descriptor is simply the intensity value, we alter the notion of geometric consistency than that described in Section 2.2. In order to compute the geometric consistency between a pair of correspondences, we compute the distance between the pairs of features in both images. The geometric consistency decreases exponentially as the discrepancy in the distances increases. The result obtained by dISCOVER is shown in Fig. 5. We successfully identify the foreground from the background and further separate the foreground into multiple objects. Also, dISCOVER does not require any parameters to be specified, such as number of topics or foreground objects in the images. The inability of a popularity based approach to obtain the desired results illustrates the need for geometric consistency in addition to popularity. In order to illustrate the need for considering popularity and not just geometric consistency, let us consider the following analysis. If we consider all pairs of images such as those shown in Fig. 4 and keep all features that find correspondences that are geometrically consistent with at least one other feature in atleast one other image, we would retain approximately 2300 of the background features. This is because even for background, it is possible to find at least some geometrically consistent correspondences. However the background being random, this would not be consistent across several images. Hence, instead of retaining features that have geometrically consistent correspondences in one other image, if we now retain only those that have geometrically consistent correspondences in at least two other images, only about 50 of the background features are retained. As we use more images, we can eliminate the background features entirely. dISCOVER being an unsupervised approach, the use of multiple images to prune out background clutter is crucial. Hence, this demonstrates the need for considering popularity in addition to geometric consistency. 3.2 Real Images In the following experiments with real images, while we present results on specific objects, it is important to note that the recent advances in object recognition that deal with object categories complement the proposed work. Since any particular features are not an integral part of dISCOVER, it can be applied to object categories by using appropriate features. However, the focus of our work is to identify the multiple objects

494

D. Parikh and T. Chen

Accuracy ofdISC O VER

Fig. 6. A subset of images provided as input to dISCOVER 1 0.8 0.6 0.4 0.2 0

(a)

5

10 15 20 25 # inputimages used

30

(b)

Fig. 7. (a) Visual results obtained by dISCOVER. The cloud of features retained as foreground and further clustered into groups. Each group corresponds to an object in the foreground. (b) Quantitative results obtained using dISCOVER.

of interest in the scene, and not object categorization. Hence, to illustrate our algorithm, we show results on specific objects (however with considerable variations) using SIFT. We first illustrate dISCOVER on a collection of 30 real images as shown in Fig. 6. Note the variation in orientation, scale and view-point of objects as well as in lighting conditions along with the highly cluttered backgrounds. We now use the descriptors as well as geometric consistency notions as described in our approach in Section 2. The results obtained are shown in Fig. 7(a). All background features have been successfully eliminated and the foreground features have been accurately clustered into multiple objects. In order to quantify the results obtained, we hand labeled the images with the foreground objects. This being a staged scenario where the objects were intentionally placed, the ground truth foreground objects of interest were known and hence such an analysis is possible. The portion of features that were assigned to their appropriate cluster in the foreground was computed as the accuracy of dISCOVER. The accuracy is shown in Fig. 7(b) with varying number of images used as input. It can be seen that while we need multiple images for accurate unsupervised multiple-object foreground identification, our accuracy reaches its optimum with a fairly small number of images. Let us now consider a real scene where the objects are not staged. Consider a collection of 30 images such as those shown in Fig. 8. These are images of an office taken at different times. Note the change in view-points, scale of objects and varying lighting conditions. We run dISCOVER on these images, and the result obtained is as shown in Fig. 9. The monitor, keyboard and CPU are identified to be the foreground objects.

Unsupervised Identification of Multiple Objects of Interest from Multiple Images

495

Fig. 8. A subset of images provided as input to dISCOVER

Fig. 9. Results obtained by dISCOVER. The cloud of features retained as foreground and further clustered into groups. Each group corresponds to an object in the foreground.

This seems reasonable. The mouse is not identified to be the foreground object because very few features were detected on the mouse, which were not stable across images mainly due to the lighting variation and pose changes. The photo frame and the CPU are clustered together. This is because these objects are stationary in all the input images and hence are found at identical locations with respect to each other (whenever present) across images, and are hence perceived to be one object. This is an artifact of dISCOVER being an unsupervised algorithm. Also, the bag next to the CPU is not retained. This is because the bag is occluded in most images, and hence is considered to be background. Overall, the foreground is successfully separated from the background, and is further clustered into the different objects of interest it contains.

4 Conclusion We propose dISCOVER, which, given a collection of images of a scene, identifies the foreground and further separates the foreground into the multiple objects of interest it contains - all in an unsupervised manner. It relies on occurrence based popularity cues as well as geometry based consistency cues to achieve this. Future work includes loosening the geometric consistency notion to deal with non-rigid objects, learning models for the identified objects of interest for detection and studying interactions among the multiple objects in the scene to provide context for robust object detection.

Acknowledgments We thank Andrew Stein and Dhruv Batra for code to compute geometrically compatible correspondences among images.

496

D. Parikh and T. Chen

References 1. Fitzgibbon, A., Zisserman, A.: On affine invariant clustering and automatic cast listing in movies. In: Heyden, A., Sparr, G., Nielsen, M., Johansen, P. (eds.) ECCV 2002. LNCS, vol. 2350, Springer, Heidelberg (2002) 2. Sivic, J., Zisserman, A.: Video data mining using configurations of viewpoint invariant regions. CVPR (2004) 3. Berg, T., Berg, A., Edwards, J., White, R., Teh, Y., Learned-Miller, E., Forsyth, D.: Names and faces in the news. CVPR (2004) 4. Hofmann, T.: Unsupervised learning by probabilistic latent semantic analysis. Machine Learning (2001) 5. Blei, D., Ng, A., Jordan, M.: Latent dirichlet allocation. Journal of Machine Learning Research (2003) 6. Fei-Fei, L., Perona, P.: A bayesian hierarchical model for learning natural scene categories. CVPR (2005) 7. Quelhas, P., Monay, F., Odobez, J., Gatica, D., Tuytelaars, T., Van Gool, L.: Modeling scenes with local descriptors and latent aspects. ICCV (2005) 8. Sivic, J., Russell, B., Efros, A., Zisserman, A., Freeman, W.: Discovering objects and their location in images. ICCV (2005) 9. Csurka, G., Bray, C., Dance, C., Fan, L.: Visual categorization with bags of keypoints. In: Pajdla, T., Matas, J(G.) (eds.) ECCV 2004. LNCS, vol. 3021, Springer, Heidelberg (2004) 10. Leordeanu, M., Collins, M.: Unsupervised learning of object models from video sequences. CVPR (2005) 11. Liu, D., Chen, T.: Semantic-shift for unsupervised object detection. In: CVPR. Workshop on Beyond Patches (2006) 12. Li, Y., Wang, W., Gao, W.: A robust approach for object recognition. In: PCM (2006) 13. Fergus, R., FeiFei, L., Perona, P., Zisserman, A.: Learning object categories from Googles image search. In: ICCV (2005) 14. Lazebnik, S., Schmid, C., Ponce, J.: Beyond bags of features: spatial pyramid matching for recognizing natural scene categories. CVPR (2006) 15. Russell, B., Efros, A., Sivic, J., Freeman, W., Zisserman, A.: Using multiple segmentations to discover objects and their extent in image collections. CVPR (2006) 16. Marszałek, M., Schmid, C.: Spatial weighting for bag-of-features. CVPR (2006) 17. Leibe, B., Leonardis, A., Schiele, B.: Combined object categorization and segmentation with an implicit shape model. In: Pajdla, T., Matas, J(G.) (eds.) ECCV 2004. LNCS, vol. 3021, Springer, Heidelberg (2004) 18. Fergus, R., Perona, P., Zisserman, A.: Object class recognition by unsupervised scaleinvariant learning. CVPR (2003) 19. Weber, M., Welling, M., Perona, P.: Unsupervised learning of models for recognition. In: Vernon, D. (ed.) ECCV 2000. LNCS, vol. 1842, Springer, Heidelberg (2000) 20. Lowe, D.: Distinctive image features from scale-invariant keypoints. IJCV (2004) 21. Belongie, S., Malik, J., Puzicha, J.: Shape context: a new descriptor for shape matching and object recognition. In: NIPS (2000) 22. Berg, A., Malik, J.: Geometric blur for template matching. CVPR (2001) 23. Leordeanu, M., Hebert, M.: A spectral technique for correspondence problems using pairwise constraints. In: ICCV (2005) 24. Shi, J., Malik, J.: Normalized cuts and image segmentation. In: PAMI (2000) 25. Shi, J.: http://www.cis.upenn.edu/jshi/software/

Fast 3-D Interpretation from Monocular Image Sequences on Large Motion Fields Jong-Sung Kim and Ki-Sang Hong Division of Electrical and Computer Engineering, POSTECH, Pohang, Korea {kimjs,hongks}@postech.ac.kr http://iip.postech.ac.kr

Abstract. This paper proposes a fast method for dense 3-D interpretation to directly estimate a dense map of relative depth and motion from a monocular sequence of images on large motion fields. The NagelEnkelmann technique is employed in the variational formulation of the problem. Diffusion-reaction equations are derived from the formulation so as to approximate the dense map on large motion fields and realize an anisotropic diffusion to preserve the discontinuities of the map. By combining the ideas of implicit schemes and multigrid methods, we present a new implicit multigrid block Gauss-Seidel relaxation scheme, which dramatically reduces the computation time for solving the largescale linear system of diffusion-reaction equations. Using our method, we perform fast 3-D interpretation of image sequences with large motion fields. The efficiency and effectiveness of our method are experimentally verified with synthetic and real image sequences.

1

Introduction

The 3-D interpretation is one of fundamental problems in computer vision. Reconstructed 3-D information such as scene depth and motion can be used in vision applications. Most methods for 3-D interpretation first compute sparse or dense correspondences such as image features or optical flow. They depend on the accuracy of generally ill-posed matching or estimation process [3,7]. As an alternative, direct methods [2,11,12] have been introduced to estimate the structure and motion of a static scene without prior matching and estimation. However, the requirement of static scenes limits the applicability of the direct methods in a variety of environments, e.g., where viewed objects move independently of viewing systems. This limitation has been first studied as a dense 3-D interpretation problem [8,10], where a dense map of relative depth and motion is estimated from the spatio-temporal change of intensity images due to smallrange image motion. It has been formulated within the variational framework and partial differential equation (PDE) systems were solved to obtain the functional minimization. In this framework, each pixel has six variables representing 3-D motion. Thus, there are more than two million variables to evaluate with a 640 × 480 video image. For this reason, the computation cost for dense 3-D Y. Yagi et al. (Eds.): ACCV 2007, Part II, LNCS 4844, pp. 497–506, 2007. c Springer-Verlag Berlin Heidelberg 2007 

498

J.-S. Kim and K.-S. Hong

interpretation has been too expensive for time-critical applications, even though it has been applied to small motion fields. In this paper we present a fast method for dense 3-D interpretation. Our method performs fast 3-D interpretation on large motion fields. It was developed in the following way: First, dense 3-D interpretation was formulated with a variational PDE model based on the instantaneous motion model [6] and the constant brightness assumption. The Nagel-Enkelman technique [9] was employed in the variational model to perform our method on large motion fields. This model is used to derive diffusion-reaction equations, which approximate the dense map between two images and realize an anisotropic diffusion to preserve discontinuities in the map. Then, by combining implicit schemes [1,13] and multigrid methods [4,5], a new implicit multigrid block Gauss-Seidel relaxation scheme is devised to quickly solve the diffusion-reaction equations. The efficiency and effectiveness of our method is verified with synthetic and real image sequences. This paper is organized as follows. Section 2 introduces the variational PDE model for dense 3-D interpretation. Section 3 describes the discretization for numerically solving the variational PDE model. Section 4 explains our numerical scheme based on implicit schemes and multigrid methods. Sections 5 and 6 present experimental results and conclusions.

2

Variational PDE Model

A 3-D point X = (X, Y, Z) on a moving object with linear velocity v = (v1 , v2 , v3 ) and angular velocity w = (w1 , w2 , w3 ) satisfies the differential ˙ = v +[w]× X and its projection xc = (x, y, f ) through a calibrated equation X camera (focal length f ) has the form xc = f (X/Z, Y /Z, 1) . The 2-D coordinate vector (x, y) is denoted by x to simplify the notation. Under the instantaneous motion model [6], the optical flow u = (u, v) can be written as u = P t + Qw,

(1)

where t = (t1 , t2 , t3 ) is Z −1 v and the matrices P and Q are defined by     1 f 0 −x −xy f 2 + x2 −f y , P = , Q= xy fx 0 f −y f −f 2 + y 2

(2)

respectively. As we see in Eq. (1), only the direction of translational motion and the depth of the scene up to a scaling factor [6,8,10] can be reconstructed by ˆ = ||t||−1 t and Zˆ−1 = ||t||, where || · || denotes the l2 -norm of a vector. v We perform dense 3-D interpretation between two images I1 (x) ≡ I1 (x, y) and I2 (x) ≡ I2 (x, y). Just as in optical flow, the Nagel-Enkelman model is applied to dense 3-D interpretation between two frames as follows:    2 I1 (x) − I2 (x + P t + Qw) (3) E (t, w) = Ω

+α

3 3        ∇ti D (∇I1 ) ∇ti + β ∇wi D (∇I1 ) ∇wi dx, i

i

Fast 3-D Interpretation from Monocular Image Sequences

where D (∇I1 ) is the regularized projection matrix, defined by 

   1 −∂I1 /∂y −∂I1 /∂y 2 +ν I , D (∇I1 ) = ∂I1 /∂x ∂I1 /∂x ||∇I1 ||2 + 2ν 2

499

(4)

where I is an identity matrix, ν a contrast parameter which is set to inhibit smoothing of the flow field across the edges of I1 at locations where ||∇I1 ||  ν, and α and β smoothing parameters. A piecewise smooth flow field is estimated by using that projection matrix in the regularization. We will describe the details for estimating the translational motion t, but our model can be applied to the simultaneous estimation of the translational motion t and rotational motion w without any modification. We seek t minimizing the energy functional of Eq. (3) by solving a solution of the diffusion-reaction equations derived through the calculus of variations. The diffusion-reaction equations are given by the following PDE system ⎧ ∂I2 ⎪ ⎨ αdiv (D(∇I1 )∇t1 ) + (I1 (x) − I2 (x + P t + Qw)) ∂t1 = 0 2 αdiv (D(∇I1 )∇t2 ) + (I1 (x) − I2 (x + P t + Qw)) ∂I (5) ∂t2 = 0 ⎪ ⎩ αdiv (D(∇I )∇t ) + (I (x) − I (x + P t + Qw)) ∂I2 = 0 1 3 1 2 ∂t3 with boundary conditions ∂t1 /∂n = 0, ∂t2 /∂n = 0, ∂t3 /∂n = 0, where n is the unit vector normal to the boundary ∂Ω of the image domain Ω. In Eq. (5), the firs terms realize a discontinuity-preserving anisotropic diffusion process and the second terms enforce the constant brightness constraint in the estimation of t. The former are called diffusion terms and the latter reaction terms. As we see, the diffusion terms are linear w.r.t. t but the reaction terms are nonlinear. To apply efficient linear methods to this problem, we approximate the nonlinear reaction terms as follows: Given initial estimates t0 , current estimates t is t = t0 + Δt, where Δt is the error between t and t0 , i.e., Δt = t−t0 . Then, our 3-D brightness constraint equations can be written as   (6) I1 (x) − I2 x + P t0 + Qw + J t0 − J t = 0, where J is the Jacobian of I2 w.r.t. t, defined by J ≡ ∂I2 /∂t. If we assume that t0 = 0 and w = 0 in Eq. (6), we can also acquire the 3-D brightness constraint equations for small motion fields, I1 (x) − I2 (x) − Jt = 0,

(7)

which have been used in all 3-D interpretation methods [8,10]. Finally, the successively approximated diffusion-reaction equations are given by ⎧     ⎨ αdiv (D(∇I1 )∇t1 ) + I1 (x) − I2 x + P t0 + Qw + J t0 − J t J1 = 0 αdiv (D(∇I1 )∇t2 ) + I1 (x) − I2 x + P t0 + Qw + J t0 − J t J2 = 0 (8) ⎩ αdiv (D(∇I1 )∇t3 ) + I1 (x) − I2 x + P t0 + Qw + J t0 − J t J3 = 0 which are our diffusion-reaction equations for dense 3-D interpretation. To solve a solution of these equations, final estimates t should be iteratively updated with initial estimates t0 for every pixels. Therefore, the large-scale linear system of Eq. (8) should be solved in each iteration.

500

3

J.-S. Kim and K.-S. Hong

Discretization

The unknown functions, t1 (x, y), t2 (x, y), and t3 (x, y), are defined on a pixel grid of cell size hx × hy . We denote by h the index for the cell size. We denote by t1i , t2i , and t3i the approximation to t1 , t2 , and t3 , respectively, at some pixel i with i = 1, . . . , N . All spatial gradients are approximated by central differences, and the divergences are estimated using the eight neighboring pixels, denoted by j, with j = 1, . . . , 8. We denote by Ni the set of neighbors of pixel i and dmni the element (m, n) of the projection matrix D (∇I1i ) for some pixel i. Then, the diffusivity coefficients, cji , with j = 1, . . . , 8 in some pixel i, can be computed with c1i = c5i =

d111 +d11i , 2 d125 +d12i , 4

11i 22i c2i = d112 +d , c3i = d223 +d , c4i = 2 2 d126 +d12i d127 +d12i c6i = − , c = − , c8i = 7i 4 4

d224 +d22i 2 d128 +d12i . 4

(9)

We denote by Jmi the the element (m) of the Jacobian in some pixel i, defined by (J1i , J2i , J3i ) = (f ∂I1i /∂x, f ∂I1i /∂y, −f x − f y). We describe the finite difference approximation to the diffusion-reaction equations in Eq. (8) by using this discretization. By assuming that t0 = 0 due to small motion and w = 0 in Eq. (8), we obtain the discrete diffusion-reaction equations for small motion, given by the elliptic PDE system ⎧  t1j −t1i 2 ⎪ ⎨ α j∈Ni cji h2 + J1i esi − J1i t1i − J2i J1i t2i − J3i J1i t3i = 0, t2j −t2i 2 t2i − J3i J2i t3i = 0, α j∈Ni cji h2 + J2i esi − J1i J2i t1i − J2i (10)  ⎪ t −t ⎩α 3j 3i 2 + J3i esi − J1i J3i t1i − J2i J3i t2i − J3i t3i = 0, j∈Ni cji h2 for i = 1, . . . , N , where esi ≡ I1 (xi ) − I2 (xi ). This constitutes a large-scale linear system of equations for the 3N unknowns t1i , t2i , and t3i . Large motions are handled by considering the unknowns as the time evolution functions t (x, y, n) with a time variable n. We solve an evolution solution by calculating the asymptotic state (n → ∞) of Eq. (8). The associated discrete diffusion-reaction equations are given by the parabolic PDE system ⎧    t −t t −t0 2 ⎪ t1i − J2i J1i t2i − J3i J1i t3i , ⎪ 1i τ 1i = α j∈Ni cji 1jh2 1i + J1i edi t0i − J1i ⎨ 0  0  t2i −t2i t2j −t2i 2 = α c + J e t − J 2i di 1i J2i t1i − J2i t2i − J3i J2i t3i , i j∈Ni ji τ h2 ⎪  0 ⎪ t3j −t3i ⎩ t3i −t03i = α  2 + J3i edi ti − J1i J3i t1i − J2i J3i t2i − J3i t3i , j∈Ni cji τ h2 (11)     for i = 1, . . . , N , where edi t0i ≡ I1 (xi ) − I2 xi + Pi t0i + Qi wi − Ji t0i , and τ the size of the time-step. The above system can be solved by iteratively applying the gradient descent method but a small time-step is required at each iteration of the gradient descent method, which drastically increases the computation time. Instead, we present a new efficient numerical scheme for quickly finding an evolution solution to the parabolic PDE system.

Fast 3-D Interpretation from Monocular Image Sequences

4

501

Efficient Numerical Scheme

The equations in Eq. (10) constitute a linear elliptic PDE system for the 3N unknowns t1i , t2i , and t3i , with a sparse matrix structure. Given a linear system Az = b and a decomposition A = D − L − U with diagonal matrix D, lower triangular matrix L, and upper triangular matrix U , then one iteration of the Gauss-Seidel relaxation [4] is z n+1 = (D − L)−1 (U z n + b), where n is the iteration index of relaxation. The three unknowns of each pixel are coupled with each other. Such a coupling yields to a block Gauss-Seidel relaxation (BGSR). In each iteration, we simultaneously update the three unknowns, t1i , t2i , and t3i , by solving the following 3 × 3 linear system at pixel i: = Q−1 tn+1 i si y si

n+1/2

n+1/2

where Qsi and y si ⎛ α  Qsi = ⎝

h2

n+1/2

y si

(12)

are defined by

2 cji + J1i J1i J2i J1i J3i

j∈Ni

⎛

and

,

α h2

⎜ ⎜ = ⎜ hα2 ⎝ α h2

α h2



J2i J1i 2 j∈Ni cji + J2i J2i J3i

α h2

⎞ J3i J1i ⎠ J J  3i 2i 2 j∈Ni cji + J3i

  ⎞  n+1 + j∈N + cji tn1j + J1i esi j∈Ni− cji t1j i ⎟    ⎟ n+1 n − cji t + cji t + e + J ⎟, 2i si 2j 2j j∈Ni j∈Ni ⎠    n+1 n + j∈N + cji t3j + J3i esi j∈N − cji t3j i

(13)

(14)

i

where Ni− = {j ∈ Ni |i(j) < i} and Ni+ = {j ∈ Ni |i(j) > i}. The i(j) returns the corresponding pixel index of j. The explicit scheme for solving a solution to the resulting parabolic PDE system in Eq. (11) can be written as z n+1 = z n + τ (Az n − bn ), where bn denotes that this source term should be updated at each iteration n, unlike the case for small motion. This explicit scheme is only stable for small timesteps. The implicit scheme is stable no matter how large the time-step is [1,13]. In the implicit scheme, we obtain a solution at each iteration by solving the linear system (I − τ A) z n+1 = z n − τ bn . Since the matrix (I − τ A) has a sparse matrix structure equal to the structure of the matrix A, we can employ BGSR as a basic iterative solver for the implicit scheme. Thus, one iteration reads z n+1 = 1  −1  1 n n n τI −D+L τ z − U z − b . In each iteration step of the implicit block Gauss-Seidel relaxation (IBGSR), the three coupled unknowns, t1i , t2i , and t3i , in some pixel i at time n, are simultaneously updated by solving the 3 × 3 linear system: n+1/2 = Q−1 , (15) tn+1 i di y di n+1/2

are defined by where Qdi and y di ⎛1 α  2 J J j∈Ni cji +J1i τ + h2 2i 1i α 1 2 ⎝ J1i J2i Qdi = j∈Ni cji + J2i τ + h2 J1i J3i J2i J3i

1 τ

+

α h2

J3i J1i J J 3i 2i j∈Ni

⎞ ⎠ cji +

2 J3i

(16)

502

J.-S. Kim and K.-S. Hong

and

⎛

n+1/2

y di

1 n τ t1i

+

⎜ ⎜ = ⎜ τ1 tn2i + ⎝ 1 n τ t3i +

α h2 α h2 α h2



 ⎞ cji tn1j + J1i edi (tni ) ⎟    ⎟ n+1 + j∈N + cji tn2j + J2i edi (tni ) ⎟ . (17) j∈Ni− cji t2j i ⎠    n+1 n n − cji t + cji t + J + e (t ) 3i di i 3j 3j j∈N j∈N j∈Ni−

i

cji tn+1 + 1j



j∈Ni+

i

We can obtain two different solutions to our model by iteratively solving the 3 × 3 linear system of Eqs. (12) or (15) in each pixel. The convergence of relaxation is improved by employing the multigrid methods [5]. Let us denote the linear systems described above as Ah z h = bh . We ˜ the current approximate solution z, and z ¯ the smoothed solution denote by z obtained by BGSR or IBGSR. Let us denote by H the new cell size Hx × Hy on the coarse grid. The prolongation for coarse-to-fine grid transfer is represented by P H→h , and the restriction for fine-to-coarse grid transfer by Rh→H . The procedure of the two-grid iteration is composed of three steps; pre-smoothing, coarse-grid correction, and post-smoothing. The details can be described as follows. Two-Grid Iteration Algorithm ¯h 1. (Pre-smoothing) Compute a smoothed approximate solution z by applying μ1 iterations of a relaxation method to a current ap˜h. proximate solution z ¯ h with the error eh approxi2. (Coarse-grid correction) Correct z mated with the coarse grid as follows: (a) Compute the residual on ¯ h . (b) Restrict the residual r h the fine grid rh with rh = bh − Ah z H h→H h to the coarse grid as r = R r . (c) Compute the error on the coarse grid eH by solving a linear system AH eH = rH . (d) Prolong the error eH to the fine grid as eh = P H→h eH . (e) Correct the ¯ h by z ¯ h = z h + eh . approximate solution z ˜ h by apply3. (Post-smoothing) Compute the new approximation z h ¯ . ing μ2 iterations of the relaxation method to z The two-grid iteration is repeated recursively down to some coarsest grid. The recursive procedure is the multigrid method. One iteration of a multigrid method is called a cycle, and the exact structure of a cycle depends on the the value of γ, the number of two-grid iterations at each immediate step. The case γ = 1 is called a multigrid V-cycle (MV), while γ = 2 is called a multigrid W-cycle (MW). Let us denote by h the index of cell size on the current grid and by H  the index of cell size on some coarsest grid. Then, the multigrid iteration can be defined recursively by the following two steps: Multigrid Iteration Algorithm ¯ H  obtained by solving 1. If h = H  , then return an exact solution z −1   ¯ H = AH  bH . z ¯ h obtained by recursively applying γ itera2. Otherwise, return an z tions of a multigrid iteration, from h to H  .

Fast 3-D Interpretation from Monocular Image Sequences

503

Table 1. Comparison of the block Gauss-Seidel relaxation and the implicit block Gauss-Seidel relaxation

Block Gauss-Seidel Block Gauss-Seidel (V(1,1)) Block Gauss-Seidel (W(1,1)) Implicit block Gauss-Seidel Implicit block Gauss-Seidel (V(1,1)) Implicit block Gauss-Seidel (W(1,1))

α 10 1 10 1 10 1 10 1 10 1 10 1

n 1356 107 37 3 37 3 302 36 5 2 2 1

ζn 0.499 0.499 0.499 0.485 0.499 0.485 0.499 0.499 0.496 0.418 0.464 0.489

Time [s] 59.8 4.6 8.7 0.7 8.6 0.7 19.0 2.3 1.3 0.5 0.6 0.3

Speedup 1 13 7 85 7 85 3 26 46 120 100 200

Table 2. Comparison of different multigrid implementations of the implicit block Gauss-Seidel relaxation

V(1,1) V(2,2) W(1,1) W(2,2)

5

n 19 17 17 16

Yosemite ζn Time [s] 0.267 4.8 0.263 5.7 0.266 5.3 0.262 6.7

n 16 11 11 9

Flower Garden ζn Time [s] 0.282 4.3 0.285 4.2 0.281 3.7 0.283 4.0

n 17 10 4 4

Rubik Cube ζn Time [s] 0.211 6.4 0.229 5.7 0.287 1.9 0.299 2.5

Experimental Results

All numerical experiments were carried out on a desktop PC with a 3.2-GHz Intel P4 CPU executing C/C++ code. The qualitative analysis of dense 3-D interpretation was conducted by using a display of gray value rendering of a dense map of relative depth as in [8,10]. To quantitatively evaluate the accuracy of 3-Dinterpretation, we have computed the error reduction factor defined by ζ n = en /e0 , where en denotes the sum of squared intensity errors at iteration n. We used four grids for multigrid V-cycle or W-cycle iterations. We denote by V(1,1) or W(1,1) the V-cycle or W-cycle with one pre-smoothing and one post-smoothing iteration at each grid. The contrast parameter of our model, ν was set to 2. In all experiments, we set the field of view of camera to 45◦ . In the first experiment, we compared the performance of BGSR and IBGSR. The previous methods in [8,10] used BGSR for performing dense 3-D interpretation. In each test run, the algorithm was stopped when the error reduction factor was below 0.5. We also evaluated the performance of each relaxation with V(1,1) or W(1,1). The first two frames of the Yosemite sequence were tested for this experiment. Table 1 shows the required number of iterations, n, to reach the

504

J.-S. Kim and K.-S. Hong

Fig. 1. Top: Result of the Yosemite sequence (ν=2.0, α=1, 4.8 s with 19 V(1,1) cycles). Middle: Result of the Flower Garden sequence (ν=2.0, α=1, 3.7 s with 10 W(1,1) cycles). Bottom: Result of the Rubik Cube sequence (ν=2.0, α=10, 1.9 s with 4 W(1,1) cycles).

desired error reduction of ζ n < 0.5, the computation time in seconds, and the speedup factor with respect to BGSR. While IBGSR computed the source term bn in each iteration, it was about 3 times faster than BGSR. While multigrid methods provided a significant acceleration to both relaxation methods, IBGSR reached the desired error reduction faster than BGSR with V(1,1) or W(1,1). In our second experiment, different multigrid implementations of IBGSR were tested. We performed tests with the first two frames of the Yosemite sequence, the first two frames of the Flower Garden sequence of 352×240 pixels, and the first and the fifth frames of the Rubik Cube sequence of 256×240 pixels. The Rubik Cube sequence has a rotating object, and thereby we have estimated the rotational motion w as well as the translational motion t. The algorithm was stopped when ζ n − ζ n−1 ≤ 10−3 . IBGSR was applied to each test set with 4grid V-cycle or W-cycle. Table 2 shows the required number of V or W cycles n, the error reduction factor ζ n , and the computation time for each multigrid implementation. To demonstrate the quality of 3-D interpretation, we display

Fast 3-D Interpretation from Monocular Image Sequences

505

Table 3. Comparison of the previous methods (MDL [8], Diffusion [10]) and our method (IBGSR, V(1,1)) for the Yosemite sequence

MDL Diffusion Our method

n 10000 10000 19

ζn 0.459 0.478 0.267

κ [%] 35.06 36.64 23.65

φ [◦ ] 13.91 17.17 4.11

Time [s] 6061.4 989.5 4.8

Fig. 2. Top: Result of the Teddy sequence (ν=2.0, α=5, 3.8 s with 5 W(2,2) cycles). Bottom: Different views of the reconstructed 3-D structure by our method.

the first frame, the gray value rendering result of the estimated depth map, and the optical flow computed by Eq. (1) for three test set in Figure 1. Note that our method has output the full 3-D information of each test set in a few seconds. In Table 3, we show the performance of our method compared to the previous methods, MDL [8] and Diffusion [10]. For the quantitative analysis, we calculated the average angular error φ and the relative depth error κ, defined by κ = ˆ + b − d||/||d||, where d denotes the ground truth depth, d ˆ the estimated ||ad depth, and a and b the scale and the bias factors of the estimated depth map, respectively. The maximum number of iterations was set to 104 . As we see, the previous methods could not reach the same error reduction as our method. In the last experiment, we experimentally verified that our method allows the movement of both viewing system and viewed objects. This was demonstrated using the Teddy sequence, where an object moves laterally and a textured background moves in the opposite direction. We used the first two frames after resizing to 400 × 340 pixels and set ν=2 and α=5 for our model parameters. W(2,2) was selected for the multigrid implementation. Figure 2 shows the frame 1 of the Teddy sequence, the estimated depth map between the first two frames, and the

506

J.-S. Kim and K.-S. Hong

computed optical flow field. The computation time was 3.8 seconds. That figure also displays different views of the reconstructed 3-D structure by our method.

6

Conclusions

We have presented a fast method for dense 3-D interpretation method without prior estimation of optical flow. The proposed method can conduct fast 3-D interpretation on large motion fields. The Nagel-Enkelman technique was employed in the variational model of our problem, and we have derived diffusion-reaction equations. The idea of implicit schemes and multigrid methods has been combined to develop a new implicit multigrid block Gauss-Seidel relaxation scheme for solving our dense 3-D interpretation problem quickly on large motion fields. We have verified the efficiency and effectiveness of our method through the experimental results with synthetic and real image sequences.

References 1. Alvarez, L.: Images and PDE’s. ICAOS 1996 Images, Wavelets and PDE’s 219, 3–14 (1996) 2. Cohen, M., Irani, M., Anandan, P.: Direct recovery of planar-parallax from multiple frames. IEEE Trans. on Pattern Anal. and Mach. Intell. 24(11), 1528–1534 (2002) 3. Chellappa, R., Qian, G., Srinivasan, S.: Invited paper: Structure from motion: sparse versus dense correspondence methods. In: Proc. IEEE Conf. on Image Processing, pp. 492–499. IEEE Computer Society Press, Los Alamitos (1999) 4. Hackbusch, W.: Applied mathematical sciences, vol. 95. Springer, New York (1993) 5. Henson, V.E., Briggs, W.L., McCormick, S.F.: A multigrid tutorial. In: SIAM (2000) 6. Longuet-Higgins, H.C.: A computer algorithm for reconstructing a scene from two projections. Nature 293, 133–135 (1987) 7. Mitiche, A.: Computational analysis of visual motion. Plenum Press, New York (1994) 8. Mitiche, A., Hadjres, S.: MDL estimation of a dense map of relative depth and 3d motion from a temporal sequence of images. Pattern Analysis and Applications 6, 78–87 (2003) 9. Nagel, H.H., Enkelmann, W.: An investigation of smoothness constraints for the estimation of displacement vector fields from image sequences. IEEE Trans. Pattern Anal. and Mach. Intell. 8, 565–593 (1986) 10. Sekkati, H., Mitiche, A.: Dense 3D interpretation of image sequences: A variational approach using anisotropic diffusion. In: Proc. Int. Conf. on Image Analysis and Processing (2003) 11. Stein, S., Shashua, M.: Model-based brightness constraints: On direct estimation of structure and motion. IEEE Trans. on Pattern Anal. and Mach. Intell. 22(9), 993–1005 (2000) 12. Vedula, S., Baker, S., Rander, P., Collins, R., Kanade, T.: Three-dimensional scene flow. IEEE Trans. on Pattern Anal. and Mach. Intell. 27(3), 475–480 (2005) 13. Weickert, J., Romeny, B.M., Viergever, M.A.: Efficient and reliable schemes for nonlinear diffusion filtering. IEEE Trans. Image Process. 7, 398–410 (1998)

Color-Stripe Structured Light Robust to Surface Color and Discontinuity Kwang Hee Lee, Changsoo Je, and Sang Wook Lee Dept. of Media Technology Sogang University Shinsu-dong 1, Mapo-gu, Seoul 121-742, Korea {Khecr, Vision, Slee}@sogang.ac.kr

Abstract. Multiple color stripes have been employed for structured light-based rapid range imaging to increase the number of uniquely identifiable stripes. The use of multiple color stripes poses two problems: (1) object surface color may disturb the stripe color and (2) the number of adjacent stripes required for identifying a stripe may not be maintained near surface discontinuities such as occluding boundaries. In this paper, we present methods to alleviate those problems. Log-gradient filters are employed to reduce the influence of object colors, and color stripes in two and three directions are used to increase the chance of identifying correct stripes near surface discontinuities. Experimental results demonstrate the effectiveness of our methods. Keywords: Structured light, color stripes, range imaging, active vision, surface color, surface discontinuity, projector-camera system, 3D modeling.

1 Introduction Structured light-based ranging is an accurate yet simple technique for acquiring depth image, and thus has been widely investigated [3-5, 7-15, 18]. There has been a variety of light patterns developed for rapid range imaging and the range resolution achievable in a single video frame has recently been increased sufficiently so that real-time shape capture became a practical reality. [12, 14, 18]. Structured-light methods suitable for realtime ranging uses their specific codifications based on color assignment [3, 5, 8, 11, 12, 14, 15, 18] and spatially windowed uniqueness (SWU) [3, 8, 12, 14, 18] for increasing the number of unique subpatterns in a single projection pattern. This SWU and the use of color have made it possible to design color stripe patterns for high-resolution range imaging in a single video frame. On the other hand, the main disadvantages of a single-frame color stripe pattern compared to sequential BW (black-and-white) stripes are that color stripe identification is affected by object surface color and it is often impossible near surface discontinuities such as occluding boundaries. In general, in a multiple-stripe color pattern, the uniqueness of a spatially windowed subpattern becomes higher as the number of stripes in the subpattern increases. To guarantee sufficient SWU for rapid imaging, a subpattern consists of several adjacent stripe colors. However, some of those adjacent stripes are often unavailable to identify a stripe color of interest near Y. Yagi et al. (Eds.): ACCV 2007, Part II, LNCS 4844, pp. 507–516, 2007. © Springer-Verlag Berlin Heidelberg 2007

508

K.H. Lee, C. Je, and S.W. Lee

surface discontinuities. To the best of our knowledge, little research has been carried out explicitly to alleviate these problems.

(a)

(b)

(c)

Fig. 1. (a) A multiple-color stripe pattern and a unique subpattern with 7-stripe wide spatial window, (b) a subpattern window in projected stripes, and (c) projected stripes on a colored object

In this paper, we present structured-light methods with range estimation robust to object surface colors and discontinuities. We show that the logarithmic gradient (loggradient) [1, 2] can be used for decreasing the influence of surface color reflectance in structured-light range imaging. Our data processing algorithm incorporates the Fourier transform-based filtering that has been used for deriving intrinsic images [6, 16, 17]. By applying the algorithm to a captured image, the object reflectance in the image is discounted to some degree, and the stripe colors of structured light are detected with higher accuracy. In addition, we develop a method of applying stripe patterns in two or three directions simultaneously, decoupling the patterns and estimating depth images independently. This substantially improves the chance of estimating correct depths near surface discontinuities. The rest of this paper is organized as follows. Section 2 presents the problem statement, and Section 3 presents a filtering processing method for decreasing the influence of surface colors in color stripe identification. A method of applying more than one pattern simultaneously for improving depth estimation near surface discontinuities is described in Section 4. Experimental results are shown in Section 5 and Section 6 concludes this paper.

2 Problem Statement Sequential projections of BW patterns are in general inappropriate for rapid structurelight depth imaging and the use of colors has been investigated to reduce the number of pattern projections required for a given range resolution. If a stripe by its color alone in a color stripe pattern, global uniqueness is hard to achieve due to the repeated appearance of the color among M stripes. The use of more colors than three RGB can compromise their distinctiveness due to the reduced color distances between the stripes. In the SWU scheme, colors are sequentially encoded to stripes according to permutations or de Bruijn sequence. A stripe color centered a spatially windowed subpattern can be identified since the combination of the colors in the window is unique globally or at least semi-globally [3, 8, 12, 14, 18]. Figure 1 (a) illustrates this uniqueness of a seven-stripe subpattern. The larger the number of stripes is in the subpattern, the wider the window is and the more unique the subpattern is.

Color-Stripe Structured Light Robust to Surface Color and Discontinuity

509

A wide subpattern has an obvious disadvantage near a surface discontinuity. When the directions of an occluding boundary and stripes are similar, some stripes in the subpattern around a color stripe may be occluded and thus the color stripe is unidentifiable as depicted in Figure 1 (b). This results in a failure in depth estimation near surface discontinuities. The failure also occurs when strong surface colors alter stripe colors significantly as shown in Figure 1 (c). The goal of the research presented in this paper is to develop algorithms for reducing the influence of object colors and discontinuities using log-gradient filters and combined projection patterns in multiple directions.

3 Log-Gradient Processing for Surface Color Reduction As mentioned earlier, surface color hinders detection of stripes. Angelopoulou et al. proposed a spectral log-gradient method [1], and Berwick and Lee used spectral and spatial log-gradients to discount illumination colors [2]. In the research for deriving intrinsic image, several filtering schemes have been proposed [6, 16, 17]. Based on those spectral and spatial operators and filtering algorithms, we develop a method for discounting object colors across the color stripe directions. Let pd be the result signal of an original signal p processed by a filter fd:

pd ≡ f d ∗ p .

(1)

Then, p can be estimated as follows [6]:

§ F ( ¦ f dr ∗ pˆ d ) · ¨ d = x, y ¸ pˆ = F −1 ¨ ¸, r ( ∗ ) F f f ¦ d d ¸ ¨ © d = x, y ¹

(2)

where f dr is the reversed filter of f d : f d ( x, y ) = f dr (− x, − y ) , and F and F-1 denote the Fourier transform and its inverse transform, respectively. The objective of our processing algorithm is to regard the color variation across the stripes as illumination change (i.e., projection color change) and remove the surface reflectance as much as possible. We use log-gradient operators for this purpose. The reflected light intensity is given by the equation: I (λ ) = S (λ ) L ( λ ) ,

(3)

where S, L and Ȝ is the surface reflectance, illumination and wavelength, respectively. Applying log-gradient with respect to the way of stripe transition to the equation 3, ∂ x (ln I ) = ∂ x (ln S ) + ∂ x (ln L) ≅ ∂ x (ln L) ,

(4)

where it is assumed the spatial change of surface reflectance is much slower than that of illumination (through the stripe transition) and can be ignored. In equation 4, we can see spatially static surface reflectance can be easily removed. Since the stripe direction on the surface spatially varies in a scene image due to the surface geometry and triangulation, log-gradient has to be applied w.r.t. the corresponding direction to

510

K.H. Lee, C. Je, and S.W. Lee

each infinitesimal area for removing the surface reflectance component as much as possible. We do this process approximately by differentiating the images rotated from the original image with 17 angles sampled at the step of 10 degrees. In order to operate this process based on a filtering scheme in intrinsic imaging we apply a gradient filter w.r.t. x, Dx to the logarithm of each rotated image:

Fig. 2. The process of 3D modeling based on the proposed method with a combination of two stripe-patterns. Horizontal and vertical color stripe-patterns encoded based on SWU, are combined to a single combination pattern, and it is projected onto the leather surface of a colortextured bag. The scene is captured by a camera, and the scene image is filter-processed with rotations to estimate the two illumination-restored images. Stripes are identified independently in each restoration image, and ranges are obtained by geometric calibration of the projector and camera. Finally the ranges are merged into one, and it is meshed and rendered.

ixϕ ≡ Dx ∗ ln ( Rϕ ( I ) ) ,

(5)

where Rϕ denotes the rotation operator by an angle ϕ . For obtaining depth data it is proper to restore the illumination component of single-directional transitions. Therefore, in our application the restoration does not include the gradient w.r.t. y, iϕy contrary to equation 2:

§ · r ϕ i ϕ ≡ F −1 ¨ F ( Dx ∗ ix ) ¸ . ¨ F( Ddr ∗ Dd ) ¸¸ ¦ ¨ d x y = , © ¹

(6)

Color-Stripe Structured Light Robust to Surface Color and Discontinuity

511

Taking the exponential to the restoration and rotating back each image by −ϕ , the restoration of each rotation is obtained:

Iϕ ≡ R−ϕ ( exp(iϕ ) ) .

(7)

Finally, the illumination-restored image can be constructed by I Φ ( x , y ) where Φ ( x, y ) is the rotation angle which maximizes the gradient magnitude: Φ ≡ arg max (∂ x Iϕ ) 2 + (∂ y Iϕ )2 . ϕ

(8)

With the illumination-restored image, stripe segmentation is expected to be much less insensitive to object surface colors and to result in more accurate depth results.

4 Patterns Combination for Adapting to Surface Discontinuities As asserted in section 2, SWU based stripe identification can lose the correct coordinate of illumination near the surface discontinuity. In a scene where dominant discontinuities exist through the horizontal way, projection of vertical stripes can easily escape the problem of SWU based identification. By combining two sets of stripes with different directions, a single projection pattern containing multidirectional stripes can be generated, and projected into a scene. As a result, it can be noted that for any of single-directional discontinuities there can be a sufficient number of connected stripes with the directions distant to those of discontinuities, and thus the stripes can be correctly identified for most of single-directional discontinuities. In this case, we can find the two colors of the illuminated stripes, I Φ1 and I Φ 2 by estimating Φ1 and Φ 2 for each pixel. We estimate Φ1 and Φ 2 by a similar way as in equation 8, by finding the maximum in the π / 2 period centering at the globally dominant direction ( mΦ1 and mΦ 2 ) of each stripe-pattern. In an appropriate case, stripe patterns with three different directions can be combined into a single pattern to be projected into the scene, and Φ1 , Φ 2 and Φ3 can be found in each smaller range ( π / 3 interval) with the center mΦ1 , mΦ 2 , and mΦ 3 . Figure 2 illustrates the process of 3D modeling based on the proposed method with a combination of two stripe-patterns. The process is mostly similar to those for a single stripe-pattern and three intrinsic patterns.

5 Experimental Results We have made many experiments to validate the proposed techniques. The experimental setup consists of a Sony XC-003 3CCD VGA camera, an Epson color LCD XGA projector and a cubic calibration-object. A permutation-based color-stripe

512

K.H. Lee, C. Je, and S.W. Lee

pattern in [12] is used to generate the three kinds of patterns: single-direction, doubledirection and triple-direction stripe-patterns (see Figure 3). We used the Sobel operator as the gradient filter, and the stripes are segmented by hue thresholding of restored illumination colors normalized by neighbor colors. Each segmented stripe is identified according to the sequence of neighboring consecutive stripes. From the image coordinates and stripe identities, 3D world coordinates are estimated by the geometric calibration [13] which gives the extrinsic and intrinsic parameters of the projector and camera. When the double-direction or triple-direction is used, more than one range image are obtained, and are merged into one by removing erroneous points and by averaging multiple reliable points. Some results are meshed and rendered.

Fig. 3. The three kinds of stripe-patterns: (Left) single-direction, (Middle) double-direction, and (Right) triple-direction

(a)

(d)

(b)

(e)

(c)

(f)

Fig. 4. The results of a crumpled paper which consists of squares of 4 colors (cyan, blue, green and red, clockwise): (a) input image, (b) illumination-restored image by log-gradient processing without rotations (c) illumination-restored image by proposed processing, (d) range result by direct estimation (e) range estimation from (b), and (f) range estimation from (c)

Color-Stripe Structured Light Robust to Surface Color and Discontinuity

(a)

513

(b)

Fig. 5. The stripe segmentation results by (a) direct estimation and by (b) proposed processing

(a)

(b)

(d)

(e)

(c)

(f)

Fig. 6. The results of the leather surface of a color-textured bag: (a) input image, (b) illumination-restored image by proposed processing, (c) scene under white illumination, (d) rendered result by direct estimation, (e) result by proposed method, and (f) a filtered image with a single rotation

Figure 4 shows the experimental results of a crumpled paper which consists of squares of 4 colors (cyan, blue, green and red, clockwise): (a) input image, (b) illumination-restored image by log-gradient processing without rotations (c) illumination-restored image by proposed processing, (d) range result by direct estimation (e) range estimation from (b), and (f) range estimation from (c). Figure 5 compares the stripe segmentation results by direct estimation and by proposed processing. Figure 6 shows the results of the leather surface of a color-textured bag: (a) input image, (b) illumination-restored image by proposed processing, (c) scene under white illumination, (d) rendered result by direct estimation, (e) result by proposed method, and (f) a filtered image with a single rotation. Figure 7 depicts the results of the bag: (a) input image, (b and c) the two illumination-restored images, (e and f) rendered results from (b and c), and (d) merged result from (b and c).

514

K.H. Lee, C. Je, and S.W. Lee

Figure 8 depicts the results of hands: (a) input image, (b and c) the two illumination-restored images, (e and f) rendered results from (b and c), and (d) merged result from (b and c). Figure 9 depicts the results of results of a cow: (a) input image of a scene with double-directional pattern, (b and c) the two illumination-restored images, (e and f) rendered results from (b and c), and (d) merged result from (b and c).

(a)

(d)

(b)

(e)

(c)

(f)

Fig. 7. The results of the bag: (a) input image, (b and c) the two illumination-restored images, (e and f) rendered results from (b and c), and (d) merged result from (b and c)

(a)

(b)

(c)

(d)

(e)

(f)

Fig. 8. The results of hands: (a) input image, (b and c) the two illumination-restored images, (e and f) rendered results from (b and c), and (d) merged result from (b and c)

Color-Stripe Structured Light Robust to Surface Color and Discontinuity

(a)

(d)

(b)

(e)

515

(c)

(f)

Fig. 9. The results of a cow: (a) input image of a scene with double-directional pattern, (b and c) the two illumination-restored images, (e and f) rendered results from (b and c), and (d) merged result from (b and c)

6 Conclusion In this paper, we demonstrated a filtering method for improving the estimation of stripe colors, and proposed a strategy of combining two or three stripe-patterns for increasing the probability of correct stripe identification. Through the experiments, we have shown that the method makes the range acquisition more insensitive to the scene characteristics. Acknowledgments. This work was supported by the Korea Science and Engineering Foundation (KOSEF) Grant (No. R01-2006-000-11374-0) and a Seoul R&BD Program.

References 1. Angelopoulou, E., Lee, S.W., Bajcsy, R.: Spectral gradient: a material descriptor invariant to geometry and incident illumination. In: Proc. ICCV 1999, pp. 861–867 (1999) 2. Berwick, D., Lee, S.W.: Spectral gradients for color-based object recognition and indexing. Computer Vision and Image Understanding 94(1-3), 28–43 (2004) 3. Boyer, K.L., Kak, A.C.: Color-encoded structured light for rapid active ranging. IEEE Trans. PAMI 9(1), 14–28 (1987)

516

K.H. Lee, C. Je, and S.W. Lee

4. Carrihill, B., Hummel, R.: Experiments with the intensity ratio depth sensor. Computer Vision, Graphics, and Image Processing 32, 337–358 (1985) 5. Caspi, D., Kiryati, N., Shamir, J.: Range imaging with adaptive color structured light. IEEE Trans. on PAMI 20(5) (1998) 6. Chung, Y.-C., Wang, J.M., Bailey, R.R., Chen, S.-W., Chang, S.-L., Cherng, S.: PhysicsBased Extraction of Intrinsic Images from a Single Image. ICPR (4), 693–696 (2004) 7. Curless, B., Levoy, M.: Better optical triangulation through spacetime analysis. In: Proc. ICCV, pp. 987–994 (1995) 8. Davies, C.J., Nixon, M.S.: A hough transformation for detecting the location and orientation of three-dimensional surfaces via color encoded spots. IEEE Trans. on Systems, Man, and Cybernetics 28(1B) (1998) 9. Hall-Holt, O., Rusinkiewicz, S.: Stripe Boundary Codes for Real-time Structured-light Range Scanning of Moving Objects. In: Proc. ICCV (2001) 10. Horn, E., Kiryati, N.: Toward optimal structured light patterns. Image and Vision Computing 17 (1999) 11. Huang, P.S., Hu, Q., Jin, F., Chiang, F.: Color-encoded digital fringe projection technique for high-speed three-dimensional surface contouring. Optical Engineering 38(6), 1065– 1071 (1999) 12. Je, C., Lee, S.W., Park, R.-H.: High-Contrast Color-Stripe Pattern for Rapid StructuredLight Range Imaging. In: Pajdla, T., Matas, J(G.) (eds.) ECCV 2004. LNCS, vol. 3021, pp. 95–107. Springer, Heidelberg (2004) 13. McIvor, A.M., Valkenburg, R.J.: Calibrating a Structured Light System, Image and Vision Computing, New Zealand, Lincoln, pp. 167–172 (August 1995) 14. Pagès, J., Salvi, J., Collewet, C., Forest, J.: Optimised De Bruijn patterns for one-shot shape acquisition. Image Vision Computing 23(8), 707–720 (2005) 15. Tajima, J., Iwakawa, M.: 3-D Data Acquisition by Rainbow Range Finder. In: Proc. ICPR, pp. 309–313 (1990) 16. Tappen, M.F., Freeman, W.T., Adelson, E.H.: Recovering Intrinsic Images from a Single Image. In: NIPS 2002, pp. 1343–1350 (2002) 17. Weiss, Y.: Deriving Intrinsic Images from Image Sequences. In: ICCV, pp. 68–75 (2001) 18. Zhang, L., Curless, B., Seitz, S.M.: Rapid Shape Acquisition Using Color Structured Light and Multi-pass Dynamic Programming. In: 3DPVT 2002, pp. 24–37 (2002)

Stereo Vision Enabling Precise Border Localization Within a Scanline Optimization Framework Stefano Mattoccia1,2 , Federico Tombari1,2 , and Luigi Di Stefano1,2 Department of Electronics Computer Science and Systems (DEIS) University of Bologna, Viale Risorgimento 2, 40136 Bologna, Italy Advanced Research Center on Electronic Systems ’Ercole De Castro’ (ARCES) University of Bologna, Via Toffano 2/2, 40135 Bologna, Italy {smattoccia, ftombari, ldistefano}@deis.unibo.it 1

2

Abstract. A novel algorithm for obtaining accurate dense disparity measurements and precise border localization from stereo pairs is proposed. The algorithm embodies a very effective variable support approach based on segmentation within a Scanline Optimization framework. The use of a variable support allows for precisely retrieving depth discontinuities while smooth surfaces are well recovered thanks to the minimization of a global function along multiple scanlines. Border localization is further enhanced by symmetrically enforcing the geometry of the scene along depth discontinuities. Experimental results show a significant accuracy improvement with respect to comparable stereo matching approaches.

1

Introduction and Previous Work

In the last decades stereo vision has been one of the most studied task of computer vision and many proposals have been made in literature on this topic (see [1] for a review). The problem of stereo correspondence can be formulated as follows: given a pair of rectified stereo images, with one being the reference image Ir and the other being the target image It , we need to find for each point pr ∈ Ir its correspondence pt ∈ It which, due to the epipolar constraint, lies on the same scanline as pr and within the disparity range D = [dmin ; dmax ]. The taxonomy proposed by Scharstein and Szelinski [1] for dense stereo techniques subdivides stereo approaches into two categories: local and global. Local approaches determine the stereo correspondence for a point pr by selecting the candidate pt,d , d ∈ D which minimizes a matching cost function CM (pr , pt,d ). In order to decrease the ambiguity of the scores the matching cost is not pointwise but is typically computed over a support which includes pr on Ir and pt,d on It . While the support can be in the simplest cases a static squared window, notable results have been yielded by using a variable support which dynamically adapts itself depending on the surroundings of pr and pt,d [2], [3], [4], [5], [6], [7], [8]. Conversely, most global methods attempt to minimize an energy function computed on the whole image area by employing a Markov Random Field model. Y. Yagi et al. (Eds.): ACCV 2007, Part II, LNCS 4844, pp. 517–527, 2007. c Springer-Verlag Berlin Heidelberg 2007 

518

S. Mattoccia, F. Tombari, and L. Di Stefano

Since this task turns out to be a NP-hard problem, approximate but efficient strategies such as Graph Cuts (GC) [9] and Belief Propagation (BP) [10], [11] have been proposed. In particular, a very effective approach turned out to be the employment of segmentation information and a plane fitting model within a BP-based framework [12], [13], [14]. A third category of methods which lies in between local and global approaches refers to those techniques based on the minimization of an energy function computed over a subset of the whole image area, i.e. typically along epipolar lines or scanlines. The adopted minimization strategy is usually based on Dynamic Programming (DP) or Scanline Optimization (SO) [15], [16], [17], [18] techniques, and some algorithms also exploit DP on a tree [19], [20]. The global energy function to be minimized includes a pointwise matching cost CM (see [1] for details) and a smoothness term which enforces constant disparity e.g. on untextured regions by means of a discontinuity penalty π:    (1) CM pir , pit,d(A) + N (d (A)) · π E (d (A)) = i∈A

with A being the image subset (e.g. a scanline) and N being the number of times the smoothness constraint is violated within the region where the cost function has to be minimized. These approaches achieved excellent results in terms of accuracy in the disparity maps [15] and in terms of very fast, near real-time, computational performances [17]. In order to increase robustness against outliers a fixed support (typically a 3×3 window) can be employed instead of the pointwise matching score. Nevertheless, this approach embodies all the negative aspects of a local window-based method, which are especially evident near depth discontinuities: object borders tend to be inaccurately detected. Hence, a first contribution proposed by this paper is to deploy an SO-based algorithm which embodies, as matching cost CM , a function based on a variable support. The SO framework allows to handle effectively low-textured surfaces while the variable support approach helps preserving accuracy along depth borders. In order to determine the variable support, we adopt a very effective technique based on colour proximity and segmentation [21] recently proposed for local approaches. The accuracy of the SO-based process is also improved by the use of a symmetrical smoothness penalty which depends on the pixel intensities of both stereo images. It will be shown that this approach allows to obtain notable accuracy in the retrieved disparities. Moreover, we propose a refinement step which allows to further increase the accuracy of the proposed method. This step relies on a technique that, exploiting symmetrically the relationship between occlusions and depth discontinuities on the disparity maps obtained assuming alternatively as reference the left and the right image, allows for accurately locating borders. This is shown to be particularly useful to assign the correct disparity values to those points violating the cross-checking constraint. Finally, experimental results show that the proposed approach is able to determine accurate dense stereo maps and it is state-of-theart for what means approaches which do not rely on a global framework.

Stereo Vision Enabling Precise Border Localization

2

519

The Support Aggregation Stage

The first step of the proposed technique computes matching costs based on a variable support strategy proposed in [21] for local algorithms. In particular, given the task of finding the correspondence score between points pr ∈ Ir and pt,d ∈ It , during the support aggregation step each point of Ir is assigned a weight which depends on color proximity from pr as well as on information derived from a segmentation process applied on the colour images. In particular, weight wr (pi , pr ) for point pi belonging to Ir is defined as:  1.0  p i ∈ Sr  (2) wr (pi , pr ) = dc (Ir (pi ),Ir (pr )) otherwise exp − γc with Sr being the segment on which pr lies, dc the Euclidean distance between two RGB triplets and the constant γc a parameter of the algorithm. A null weight is assigned to those points of Ir which lie too far from pr , i.e. whose distance along x or y direction exceeds a certain radius. A similar approach is adopted to assign a weight wt (qi , pt,d ) to each point qi ∈ It . It is interesting to note that this strategy allows to ideally extract two distinct supports at every new correspondence evaluation, one for the reference image and the other for the target image. Once the weights are computed, the matching cost for correspondence (pr , pt,d ) is determined by summing over the image area the product of such weights with a pointwise matching score (the Truncated Absolute Difference (TAD) of RGB triplets) normalised by the weight sum:  wr (pi , pr ) · wt (qi , pt,d ) · T AD(pi , qi ) CM,v (pr , pt,d ) =

pi ∈Ir ,qi ∈It



wr (pi , pr ) · wt (qi , pt,d )

(3)

pi ∈Ir ,qi ∈It

3

A Symmetric Scanline Optimization Framework

The matching cost CM,v (pr , pt,d ) described in the previous section is embodied in a simplified SO-based framework similar to that proposed in [15]. Hence, in the first stage of the algorithm the matching cost matrix CM,v (pr , pt,d ) is computed for each possible correspondence (pr , pd,t ). Then, in the second stage, 4 SO processes are used: 2 along horizontal scanlines on opposite directions and 2 similarly along vertical scanlines. The j-th SO computes the current global cost between pr and pt,d as: j j j (pr , pt,d ) = CM,v (pr , pt,d ) + min(CG (ppr , ppt,d ), CG (ppr , ppt,d−1 ) + π1 , CG j (ppr , ppt,d+1 ) + π1 , cmin + π2 ) − cmin CG

(4)

with ppr and ppt,d being respectively the point in the previous position of pr and pt,d along the considered scanline, π1 and π2 being the two smoothness penalty terms (with π1 ≤ π2 ) and cmin defined as:

520

S. Mattoccia, F. Tombari, and L. Di Stefano j cmin = mini (CG (ppr , ppt,i ))

(5)

For what means the two smoothing penalty terms, π1 and π2 , they are dependent on the image local intensities similarly to what proposed in [22] within a global stereo framework. This is due to the assumption that often a depth discontinuity coincides with an intensity edge, hence the smoothness penalty must be relaxed along edges and enforced within low-textured areas. In particular, we apply a symmetrical strategy so that the two terms depend on the intensities of both Ir and It . If we define the intensity difference between the current point and the previous one along the considered scanline on the two images as:  (pr ) = |Ir (pr ) − Ir (ppr )| (pt,d ) = |It (pt,d ) − It (ppt,d )|

(6)

then π1 is defined as:

⎧ ⎪ ⎪ Π1 ⎨ Π1 /2 π1 (pr , pt,d ) = Π1 /2 ⎪ ⎪ ⎩ Π1 /4

(pr ) < Pth , (pt,d ) < Pth (pr ) ≥ Pth , (pt,d ) < Pth (pr ) < Pth , (pt,d ) ≥ Pth (pr ) ≥ Pth , (pt,d ) ≥ Pth

(7)

where Π1 is a constant parameter of the algorithm, and π2 is defined in the same manner based on Π2 . Finally, Pth is a threshold which determines the presence of an intensity edge. Thanks to this approach, horizontal/vertical edges are taken into account along corresponding scanline directions (i.e. horizontal/vertical) during the SO process, so that edges orthogonal to the scanline direction can not influence the smoothness penalty terms. Once the 4 global costs CG are obtained, they are summed up together and a Winner-Take-All approach on the final cost sum assigns the disparity: dpr ,best = arg min{ d∈D

4

4 

j CG (pr , pt,d )}

(8)

j=1

A First Experimental Evaluation of the Proposed Approach

We now briefly show some results dealing with the use of the approach outlined so far. In particular, in order to demonstrate the benefits of the joint use of the SO-based framework with the variable support-based matching cost CM,v , we compare the results yielded by our method to those attainable by the same SO framework using the pointwise TAD matching cost on RGB triplets, as well as by CM,v in a local WTA approach. The dataset used for experiments is available at the Middlebury website1 . Parameter set is constant for all runs. Truncation parameter for TAD in both 1

vision.middlebury.edu/stereo

Stereo Vision Enabling Precise Border Localization

521

Table 1. Error rates using CM,v within the SO-based framework proposed (first row), a pointwise matching cost (CM,p ) within the same SO-based framework (second row), and CM,v in a local WTA approach (last row) Tsukuba

Venus

Teddy

Cones

N.O. - DISC

N.O. - DISC

N.O. - DISC

N.O. - DISC

CM,v , SO 1.63 - 6.80 0.97 - 9.03 9.64 - 19.35 4.60 - 11,52 3.70 - 13.38 4.19 - 19.27 12.28 - 20.40 5.99 - 13.96 CM,p , SO 10,8 - 21,7 5,08 - 12,5 CM,v , local 2,05 - 7,14 1,47 - 10,5

approaches is set to 80. For what means the variable support, segmentation is obtained by running the Mean Shift algorithm [23] with a constant set of parameters (spatial radius σS = 3, range radius σR = 3, minimum region size minR = 35), while maximum radius size of the support is set to 51, and parameter γc is set to 22. Finally, for what means the SO framework, our approach is run with Π1 = 6, Π2 = 27, Pth = 10, while the pointwise cost-based approach is run with Π1 = 106, Π2 = 312, Pth = 10 (optimal parameters for both approaches). Table 1 shows the error rates computed on the whole image area except for occlusion (N.O.) and in proximity of discontinuities (DISC ). Occlusions are not evaluated here since at this stage no specific occlusion handling approach is adopted by any of the algorithms. As it can be inferred, the use of a variable support in the matching cost yields significantly higher accuracy in all cases compared to the pointwise cost-based approach, the highest benefits being on Tsukuba and Venus datasets. Moreover, benefits are significant also by considering only depth discontinuities, which demonstrate the higher accuracy in retrieving correctly depth borders provided by the use of a variable support within the SO-based framework. Finally, benefits of the use of the proposed SO-based framework are always notable if we compare the results of our approach with those yielded by using the same cost function within a local WTA strategy.

5

Symmetrical Detection of Occluded Areas and Depth Borders

By respectively assuming as reference Ir the left and the right image of the stereo pair, it is possible to obtain two different disparity maps, referred to as DLR and DRL . Our idea is to derive a general method for detecting depth borders and occluded regions by enforcing the symmetrical relationship on both maps between occlusions and depth borders resulting from the stereo setup and the scene structure. In particular, due to the stereo setup, if we imagine to scan any epipolar line of DLR from left side to right side, each sudden depth decrement corresponds to an occlusion in DRL . Similarly, scanning any epipolar line of DRL from right side to left side, each sudden depth increment corresponds to an occlusion in DLR . Moreover, the occlusion width is directly proportional to the amount of

522

S. Mattoccia, F. Tombari, and L. Di Stefano

Fig. 1. Points violating (9) on DLR and DRL (colored points, left and center) are discriminated between occlusions (yellow) and false matches (green) on Tsukuba and Cones datasets. Consequently depth borders are detected (red points, right) [This Figure is best viewed with colors].

each depth decrement and increment along the correspondent epipolar line, and the two points composing a depth border on one disparity map respectively correspond to the starting point and ending point of the occluded area in the other map. Hence, in order to detect occlusions and depth borders, we deploy a symmetrical cross-checking strategy, which detects the disparities in DLR which violate a weak disparity consistence constraint by tagging as invalid all points pd ∈ DLR for which: |DLR (pd ) − DRL (pd − DLR (pd )) | ≤ 1 (9) and analogously detects invalid disparities on DRL . Points referring to disparity differences equal to 1 are not tagged as invalid at this stage as we assume that occlusions are not present where disparity varies smoothly along the epipolar lines, as well as to handle slight discrepancies due to the different view points. The results of this symmetrical cross-checking are shown, referred to Tsukuba and Cones, on the left and center images of Fig. 1, where colored points in both maps represent the disparities violating (9). It is easy to infer that only a subset of the colored regions of the maps is represented by occlusions, while all other violating disparities denote mismatches due to outliers. Hence, after cross-checking the two disparity maps DLR and DRL , it is possible to discriminate on both maps occluded areas from incorrect correspondences (respectively yellow and green points on left and center image, Fig. 1) by means of application of the constraints described previously. Then, putting in correspondence occlusions on one map with homologous depth discontinuities in the

Stereo Vision Enabling Precise Border Localization

523

other map, it is possible to reliably localize depth borders generated by occlusions on both disparity maps (details of this method are not provided here due to the lack of space). Right images on Fig. 1 show the superimposition of the detected borders referred to DLR (in red color) on the corresponding grayscale stereo image. As it can be seen, borders along epipolar lines are detected with notable precision and very few outliers (detected borders which do not correspond to real borders) are present.

Fig. 2. The reliability of assigning disparities to points violating the strong crosschecking (10) along depth borders (green points, left) is increased by exploiting information on depth borders location (red points, center) compared to a situation where this information is not available (right)

6

Refinement by Means of Detected Depth Borders and Segmentation

Depth border detection is employed in order to determine the correct disparity values to be assigned to points violating cross-checking. In particular, a two-step refinement process is now proposed, which exploits successively segmentation and depth border information in order to fill-in, respectively, low textured areas and regions along depth discontinuities. First of all, the following strong cross-checking consistency constraint is applied on all points of DLR :

DLR (pd ) DLR (pd ) = DRL (pd − DLR (pd )) (10) DLR (pd ) = invalid otherwise The first step of the proposed refinement approach employs segmentation information in order to fill-in regions of DLR denoted as invalid after application of (10). In particular, for each segment extracted from the application of the Mean Shift algorithm, a disparity histogram is filled with all valid disparities included within the segment area. Then, if a unique disparity value can be reliably associated with that segment, i.e. if there is a minimum number of valid disparities in the histogram and its variance is low, the mean disparity value of the histogram is assigned to all invalid points falling within the segment area. This allows to

524

S. Mattoccia, F. Tombari, and L. Di Stefano

correctly fill-in uniform areas which can be easily characterized by mismatches during the correspondence search. As this first step is designed to fill-in only invalid points within uniform areas, then a second step allows to fill-in the remaining points by exploiting the previously extracted information on border locations, especially along depth border regions which usually are not characterized by uniform areas. In particular, the assigned disparity value for all invalid points near to depth discontinuities is chosen as the minimum value between neighbours which do not lie beyond a depth border. This allows to increase the reliability of the assigned values compared to the case of no information on borders location, where e.g. the minimum value between neighbouring disparities is selected, as shown in Fig. 2.

7

Experimental Results