Lecture Notes in Computer Science Commenced Publication in 1973 Founding and Former Series Editors: Gerhard Goos, Juris Hartmanis, and Jan van Leeuwen
Editorial Board David Hutchison Lancaster University, UK Takeo Kanade Carnegie Mellon University, Pittsburgh, PA, USA Josef Kittler University of Surrey, Guildford, UK Jon M. Kleinberg Cornell University, Ithaca, NY, USA Friedemann Mattern ETH Zurich, Switzerland John C. Mitchell Stanford University, CA, USA Moni Naor Weizmann Institute of Science, Rehovot, Israel Oscar Nierstrasz University of Bern, Switzerland C. Pandu Rangan Indian Institute of Technology, Madras, India Bernhard Steffen University of Dortmund, Germany Madhu Sudan Massachusetts Institute of Technology, MA, USA Demetri Terzopoulos University of California, Los Angeles, CA, USA Doug Tygar University of California, Berkeley, CA, USA Moshe Y. Vardi Rice University, Houston, TX, USA Gerhard Weikum Max-Planck Institute of Computer Science, Saarbruecken, Germany
4843
Yasushi Yagi Sing Bing Kang In So Kweon Hongbin Zha (Eds.)
Computer Vision – ACCV 2007 8th Asian Conference on Computer Vision Tokyo, Japan, November 18-22, 2007 Proceedings, Part I
13
Volume Editors Yasushi Yagi Osaka University The Institute of Scientific and Industrial Research 8-1 Mihogaoka, Ibaraki, Osaka, 567-0047, Japan E-mail:
[email protected] Sing Bing Kang Microsoft Corporation 1 Microsoft Way, Redmond WA 98052, USA E-mail:
[email protected] In So Kweon KAIST School of Electrical Engineering and Computer Science 335 Gwahag-Ro Yusung-Gu, Daejeon, Korea E-mail:
[email protected] Hongbin Zha Peking University Department of Machine Intelligence Beijing, 100871, China E-mail:
[email protected]
Library of Congress Control Number: 2007938408 CR Subject Classification (1998): I.4, I.5, I.2.10, I.2.6, I.3.5, F.2.2 LNCS Sublibrary: SL 6 – Image Processing, Computer Vision, Pattern Recognition, and Graphics ISSN ISBN-10 ISBN-13
0302-9743 3-540-76385-6 Springer Berlin Heidelberg New York 978-3-540-76385-7 Springer Berlin Heidelberg New York
This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, re-use of illustrations, recitation, broadcasting, reproduction on microfilms or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer. Violations are liable to prosecution under the German Copyright Law. Springer is a part of Springer Science+Business Media springer.com © Springer-Verlag Berlin Heidelberg 2007 Printed in Germany Typesetting: Camera-ready by author, data conversion by Scientific Publishing Services, Chennai, India Printed on acid-free paper SPIN: 12183654 06/3180 543210
Preface
It is our great pleasure to welcome you to the Proceedings of the Eighth Asian Conference on Computer Vision (ACCV07), which held November 18–22, 2007 in Tokyo, Japan. ACCV07 was sponsored by the Asian Federation of Computer Vision. We received 640 abstracts by the abstract submission deadline, 551 of which became full submissions. This is the largest number of submissions in the history of ACCV. Out of these 551 full submissions, 46 were selected for oral presentation and 130 as posters, yielding an acceptance rate of 31.9%. Following the tradition of previous ACCVs, the reviewing process was double blind. Each of the 31 Area Chairs (ACs) handled about 17 papers and nominated five reviewers for each submission (from 204 Program Committee members). The final selection of three reviewers per submission was done in such a way as to avoid conflict of interest and to evenly balance the load among the reviewers. Once the reviews were done, each AC wrote summary reports based on the reviews and their own assessments of the submissions. For conflicting scores, ACs consulted with reviewers, and at times had us contact authors for clarification. The AC meeting was held in Osaka on July 27 and 28. We divided the 31 ACs into 8 groups, with each group having 3 or 4 ACs. The ACs can confer within their respective groups, and are permitted to discuss with pre-approved “consulting” ACs outside their groups if needed. The ACs were encouraged to rely more on their perception of paper vis-a-vis reviewer comments, and not strictly based on numerical scores alone. This year, we introduced the category “conditional accept;” this category is targeted at papers with good technical content but whose writing requires significant improvement. Please keep in mind that no reviewing process is perfect. As with any major conference, reviewer quality and timeliness of reviews varied. To minimize the impact of variation of these factors, we chose highly qualified and dependable people as ACs to shepherd the review process. We all did the best we could given the large number of submissions and the limited time we had. Interestingly, we did not have to instruct the ACs to revise their decisions at the end of the AC meeting—all the ACs did a great job in ensuring the high quality of accepted papers. That being said, it is possible there were good papers that fell through the cracks, and we hope such papers will quickly end up being published at other good avenues. It has been a pleasure for us to serve as ACCV07 Program Chairs, and we can honestly say that this has been a memorable and rewarding experience. We would like to thank the ACCV07 ACs and members of the Technical Program Committee for their time and effort spent reviewing the submissions. The ACCV Osaka team (Ryusuke Sagawa, Yasushi Makihara, Tomohiro Mashita, Kazuaki Kondo, and Hidetoshi Mannami), as well as our conference secretaries (Noriko
VI
Preface
Yasui, Masako Kamura, and Sachiko Kondo), did a terrific job organizing the conference. We hope that all of the attendees found the conference informative and thought provoking. November 2007
Yasushi Yagi Sing Bing Kang In So Kweon Hongbin Zha
Organization
General Chair General Co-chairs
Program Chair Program Co-chairs
Workshop/Tutorial Chair Finance Chair Local Arrangements Chair Publication Chairs Technical Support Staff
Area Chairs
Katsushi Ikeuchi (University of Tokyo, Japan) Naokazu Yokoya (NAIST, Japan) Rin-ichiro Taniguchi (Kyuushu University, Japan) Yasushi Yagi (Osaka University, Japan) In So Kweon (KAIST, Korea) Sing Bing Kang (Microsoft Research, USA) Hongbin Zha (Peking University, China) Kazuhiko Sumi (Mitsubishi Electric, Japan) Keiji Yamada (NEC, Japan) Yoshinari Kameda (University of Tsukuba, Japan) Hideo Saito (Keio University, Japan) Daisaku Arita (ISIT, Japan) Atsuhiko Banno (University of Tokyo, Japan) Daisuke Miyazaki (University of Tokyo, Japan) Ryusuke Sagawa (Osaka University, Japan) Yasushi Makihara (Osaka University, Japan) Tat-Jen Cham (Nanyang Tech. University, Singapore) Koichiro Deguchi (Tohoku University, Japan) Frank Dellaert (Georgia Inst. of Tech., USA) Martial Hebert (CMU, USA) Ki Sang Hong (Pohang University of Sci. and Tech., Korea) Yi-ping Hung (National Taiwan University, Taiwan) Reinhard Klette (University of Auckland, New Zealand) Chil-Woo Lee (Chonnam National University, Korea) Kyoung Mu Lee (Seoul National University, Korea) Sang Wook Lee (Sogang University, Korea) Stan Z. Li (CASIA, China) Yuncai Liu (Shanghai Jiaotong University, China) Yasuyuki Matsushita (Microsoft Research Asia, China) Yoshito Mekada (Chukyo University, Japan) Yasuhiro Mukaigawa (Osaka University, Japan)
VIII
Organization
P.J. Narayanan (IIIT, India) Masatoshi Okutomi (Tokyo Inst. of Tech., Japan) Tomas Pajdla (Czech Technical University, Czech) Shmuel Peleg (The Hebrew University of Jerusalem, Israel) Jean Ponce (Ecole Normale Superieure, France) Long Quan (Hong Kong University of Sci. and Tech., China) Ramesh Raskar (MERL, USA) Jim Rehg (Georgia Inst. of Tech., USA) Jun Sato (Nagoya Inst. of Tech., Japan) Shinichi Sato (NII, Japan) Yoichi Sato (University of Tokyo, Japan) Cordelia Schmid (INRIA, France) Christoph Schnoerr (University of Mannheim, Germany) David Suter (Monash University, Australia) Xiaoou Tang (Microsoft Research Asia, China) Guangyou Xu (Tsinghua University, China)
Program Committee Adrian Barbu Akash Kushal Akihiko Torii Akihiro Sugimoto Alexander Shekhovtsov Amit Agrawal Anders Heyden Andreas Koschan Andres Bruhn Andrew Hicks Anton van den Hengel Atsuto Maki Baozong Yuan Bernt Schiele Bodo Rosenhahn Branislav Micusik C.V. Jawahar Chieh-Chih Wang Chin Seng Chua Chiou-Shann Fuh Chu-song Chen
Cornelia Fermuller Cristian Sminchisescu Dahua Lin Daisuke Miyazaki Daniel Cremers David Forsyth Duy-Dinh Le Fanhuai Shi Fay Huang Florent Segonne Frank Dellaert Frederic Jurie Gang Zeng Gerald Sommer Guoyan Zheng Hajime Nagahara Hanzi Wang Hassan Foroosh Hideaki Goto Hidekata Hontani Hideo Saito
Hiroshi Ishikawa Hiroshi Kawasaki Hong Zhang Hongya Tuo Hynek Bakstein Hyun Ki Hong Ikuko Shimizu Il Dong Yun Itaru Kitahara Ivan Laptev Jacky Baltes Jakob Verbeek James Crowley Jan-Michael Frahm Jan-Olof Eklundh Javier Civera Jean Martinet Jean-Sebastien Franco Jeffrey Ho Jian Sun Jiang yu Zheng
Organization
Jianxin Wu Jianzhuang Liu Jiebo Luo Jingdong Wang Jinshi Cui Jiri Matas John Barron John Rugis Jong Soo Choi Joo-Hwee Lim Joon Hee Han Joost Weijer Jun Sato Jun Takamatsu Junqiu Wang Juwei Lu Kap Luk Chan Karteek Alahari Kazuhiro Hotta Kazuhiro Otsuka Keiji Yanai Kenichi Kanatani Kenton McHenry Ki Sang Hong Kim Steenstrup Pedersen Ko Nishino Koichi Hashomoto Larry Davis Lisheng Wang Manabu Hashimoto Marcel Worring Marshall Tappen Masanobu Yamamoto Mathias Kolsch Michael Brown Michael Cree Michael Isard Ming Tang Ming-Hsuan Yang Mingyan Jiang Mohan Kankanhalli Moshe Ben-Ezra Naoya Ohta Navneet Dalal Nick Barnes
Nicu Sebe Noboru Babaguchi Nobutaka Shimada Ondrej Drbohlav Osamu Hasegawa Pascal Vasseur Patrice Delmas Pei Chen Peter Sturm Philippos Mordohai Pierre Jannin Ping Tan Prabir Kumar Biswas Prem Kalra Qiang Wang Qiao Yu Qingshan Liu QiuQi Ruan Radim Sara Rae-Hong Park Ralf Reulke Ralph Gross Reinhard Koch Rene Vidal Robert Pless Rogerio Feris Ron Kimmel Ruigang Yang Ryad Benosman Ryusuke Sagawa S.H. Srinivasan S. Kevin Zhou Seungjin Choi Sharat Chandran Sheng-Wen Shih Shihong Lao Shingo Kagami Shin’ichi Satoh Shinsaku Hiura ShiSguang Shan Shmuel Peleg Shoji Tominaga Shuicheng Yan Stan Birchfield Stefan Gehrig
Stephen Lin Stephen Maybank Subhashis Banerjee Subrata Rakshit Sumantra Dutta Roy Svetlana Lazebnik Takayuki Okatani Takekazu Kato Tat-Jen Cham Terence Sim Tetsuji Haga Theo Gevers Thomas Brox Thomas Leung Tian Fang Til Aach Tomas Svoboda Tomokazu Sato Toshio Sato Toshio Ueshiba Tyng-Luh Liu Vincent Lepetit Vivek Kwatra Vladimir Pavlovic Wee-Kheng Leow Wei Liu Weiming Hu Wen-Nung Lie Xianghua Ying Xianling Li Xiaogang Wang Xiaojuan Wu Yacoob Yaser Yaron Caspi Yasushi Sumi Yasutaka Furukawa Yasuyuki Sugaya Yeong-Ho Ha Yi-ping Hung Yong-Sheng Chen Yoshinori Kuno Yoshio Iwai Yoshitsugu Manabe Young Shik Moon Yunde Jia
IX
X
Organization
Zen Chen Zhifeng Li Zhigang Zhu
Zhouchen Lin Zhuowen Tu Zuzana Kukelova
Additional Reviewers Afshin Sepehri Alvina Goh Anthony Dick Avinash Ravichandran Baidya Saha Brian Clipp C´edric Demonceaux Christian Beder Christian Schmaltz Christian Wojek Chunhua Shen Chun-Wei Chen Claude P´egard D.H. Ye D.J. Kwon Daniel Hein David Fofi David Gallup De-Zheng Liu Dhruv K. Mahajan Dipti Mukherjee Edgar Seemann Edgardo Molina El Mustapha Mouaddib Emmanuel Prados Frank R. Schmidt Frederik Meysel Gao Yan Guy Rosman Gyuri Dorko H.J. Shim Hang Yu Hao Du Hao Tang Hao Zhang Hirishi Ohno Hiroshi Ohno Huang Wei Hynek Bakstein
Ilya Levner Imran Junejo Jan Woetzel Jian Chen Jianzhao Qin Jimmy Jiang Liu Jing Wu John Bastian Juergen Gall K.J. Lee Kalin Kolev Karel Zimmermann Ketut Fundana Koichi Kise Kongwah Wan Konrad Schindler Kooksang Moon Levi Valgaerts Li Guan Li Shen Liang Wang Lin Liang Lingyu Duan Maojun Yuan Mario Fritz Martin Bujnak Martin Matousek Martin Sunkel Martin Welk Micha Andriluka Michael Stark Minh-Son Dao Naoko Nitta Neeraj Kanhere Niels Overgaard Nikhil Rane Nikodem Majer Nilanjan Ray Nils Hasler
Nipun kwatra Olivier Morel Omar El Ganaoui Pankaj Kumar Parag Chaudhuri Paul Schnitzspan Pavel Kuksa Petr Doubek Philippos Mordohai Reiner Schnabel Rhys Hill Rizwan Chaudhry Rui Huang S.M. Shahed Nejhum S.H. Lee Sascha Bauer Shao-Wen Yang Shengshu Wang Shiro Kumano Shiv Vitaladevuni Shrinivas Pundlik Sio-Hoi Ieng Somnath Sengupta Sudipta Mukhopadhyay Takahiko Horiuchi Tao Wang Tat-Jun Chin Thomas Corpetti Thomas Schoenemann Thorsten Thormaehlen Weihong Li Weiwei Zhang Xiaoyi Yu Xinguo Yu Xinyu Huang Xuan Song Yi Feng Yichen Wei Yiqun Li
Organization
Yong MA Yoshihiko Kawai
Zhichao Chen Zhijie Wang
Sponsors Sponsor Technical Co-sponsors
Asian Federation of Computer Vision IPSJ SIG-CVIM IEICE TG-PRMU
XI
Table of Contents – Part I
Plenary and Invited Talks Less Is More: Coded Computational Photography . . . . . . . . . . . . . . . . . . . . Ramesh Raskar
1
Optimal Algorithms in Multiview Geometry . . . . . . . . . . . . . . . . . . . . . . . . . Richard Hartley and Fredrik Kahl
13
Machine Vision in Early Days: Japan’s Pioneering Contributions . . . . . . Masakazu Ejiri
35
Shape and Texture Coarse-to-Fine Statistical Shape Model by Bayesian Inference . . . . . . . . . Ran He, Stan Li, Zhen Lei, and ShengCai Liao
54
Efficient Texture Representation Using Multi-scale Regions . . . . . . . . . . . . Horst Wildenauer, Branislav Miˇcuˇs´ık, and Markus Vincze
65
Fitting Comparing Timoshenko Beam to Energy Beam for Fitting Noisy Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ili´c Slobodan A Family of Quadratic Snakes for Road Extraction . . . . . . . . . . . . . . . . . . . Ramesh Marikhu, Matthew N. Dailey, Stanislav Makhanov, and Kiyoshi Honda
75 85
Poster Session 1: Calibration Multiperspective Distortion Correction Using Collineations . . . . . . . . . . . . Yuanyuan Ding and Jingyi Yu
95
Camera Calibration from Silhouettes Under Incomplete Circular Motion with a Constant Interval Angle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Po-Hao Huang and Shang-Hong Lai
106
Mirror Localization for Catadioptric Imaging System by Observing Parallel Light Pairs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ryusuke Sagawa, Nobuya Aoki, and Yasushi Yagi
116
Calibrating Pan-Tilt Cameras with Telephoto Lenses . . . . . . . . . . . . . . . . . Xinyu Huang, Jizhou Gao, and Ruigang Yang
127
XIV
Table of Contents – Part I
Camera Calibration Using Principal-Axes Aligned Conics . . . . . . . . . . . . . Xianghua Ying and Hongbin Zha
138
Poster Session 1: Detection 3D Intrusion Detection System with Uncalibrated Multiple Cameras . . . . Satoshi Kawabata, Shinsaku Hiura, and Kosuke Sato Non-parametric Background and Shadow Modeling for Object Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Tatsuya Tanaka, Atsushi Shimada, Daisaku Arita, and Rin-ichiro Taniguchi Road Sign Detection Using Eigen Color . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Luo-Wei Tsai, Yun-Jung Tseng, Jun-Wei Hsieh, Kuo-Chin Fan, and Jiun-Jie Li
149
159
169
Localized Content-Based Image Retrieval Using Semi-supervised Multiple Instance Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Dan Zhang, Zhenwei Shi, Yangqiu Song, and Changshui Zhang
180
Object Detection Combining Recognition and Segmentation . . . . . . . . . . . Liming Wang, Jianbo Shi, Gang Song, and I-fan Shen
189
An Efficient Method for Text Detection in Video Based on Stroke Width Similarity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Viet Cuong Dinh, Seong Soo Chun, Seungwook Cha, Hanjin Ryu, and Sanghoon Sull
200
Multiview Pedestrian Detection Based on Vector Boosting . . . . . . . . . . . . Cong Hou, Haizhou Ai, and Shihong Lao
210
Pedestrian Detection Using Global-Local Motion Patterns . . . . . . . . . . . . . Dhiraj Goel and Tsuhan Chen
220
Poster Session 1: Image and Video Processing Qualitative and Quantitative Behaviour of Geometrical PDEs in Image Processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Arjan Kuijper
230
Automated Billboard Insertion in Video . . . . . . . . . . . . . . . . . . . . . . . . . . . . Hitesh Shah and Subhasis Chaudhuri
240
Improved Background Mixture Models for Video Surveillance Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Chris Poppe, Ga¨etan Martens, Peter Lambert, and Rik Van de Walle
251
Table of Contents – Part I
XV
High Dynamic Range Scene Realization Using Two Complementary Images . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ming-Chian Sung, Te-Hsun Wang, and Jenn-Jier James Lien
261
Automated Removal of Partial Occlusion Blur . . . . . . . . . . . . . . . . . . . . . . . Scott McCloskey, Michael Langer, and Kaleem Siddiqi
271
Poster Session 1: Applications High Capacity Watermarking in Nonedge Texture Under Statistical Distortion Constraint . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fan Zhang, Wenyu Liu, and Chunxiao Liu Attention Monitoring for Music Contents Based on Analysis of Signal-Behavior Structures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Masatoshi Ohara, Akira Utsumi, Hirotake Yamazoe, Shinji Abe, and Noriaki Katayama View Planning for Cityscape Archiving and Visualization . . . . . . . . . . . . . Jiang Yu Zheng and Xiaolong Wang
282
292
303
Face and Gesture Synthesis of Exaggerative Caricature with Inter and Intra Correlations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Chien-Chung Tseng and Jenn-Jier James Lien Pose-Invariant Facial Expression Recognition Using Variable-Intensity Templates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Shiro Kumano, Kazuhiro Otsuka, Junji Yamato, Eisaku Maeda, and Yoichi Sato Gesture Recognition Under Small Sample Size . . . . . . . . . . . . . . . . . . . . . . . Tae-Kyun Kim and Roberto Cipolla
314
324
335
Tracking Motion Observability Analysis of the Simplified Color Correlogram for Visual Tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Qi Zhao and Hai Tao
345
On-Line Ensemble SVM for Robust Object Tracking . . . . . . . . . . . . . . . . . Min Tian, Weiwei Zhang, and Fuqiang Liu
355
Multi-camera People Tracking by Collaborative Particle Filters and Principal Axis-Based Integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Wei Du and Justus Piater
365
XVI
Table of Contents – Part I
Poster Session 2: Camera Networks Finding Camera Overlap in Large Surveillance Networks . . . . . . . . . . . . . . Anton van den Hengel, Anthony Dick, Henry Detmold, Alex Cichowski, and Rhys Hill
375
Information Fusion for Multi-camera and Multi-body Structure and Motion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Alexander Andreopoulos and John K. Tsotsos
385
Task Scheduling in Large Camera Networks . . . . . . . . . . . . . . . . . . . . . . . . . Ser-Nam Lim, Larry Davis, and Anurag Mittal
397
Poster Session 2: Face/Gesture/Action Detection and Recognition Constrained Optimization for Human Pose Estimation from Depth Sequences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Youding Zhu and Kikuo Fujimura
408
Generative Estimation of 3D Human Pose Using Shape Contexts Matching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Xu Zhao and Yuncai Liu
419
An Active Multi-camera Motion Capture for Face, Fingers and Whole Body . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Eng Hui Loke and Masanobu Yamamoto
430
Tracking and Classifying Human Motions with Gaussian Process Annealed Particle Filter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Leonid Raskin, Michael Rudzsky, and Ehud Rivlin
442
Gait Identification Based on Multi-view Observations Using Omnidirectional Camera . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Kazushige Sugiura, Yasushi Makihara, and Yasushi Yagi
452
Gender Classification Based on Fusion of Multi-view Gait Sequences . . . . Guochang Huang and Yunhong Wang
462
Poster Session 2: Learning MAPACo-Training: A Novel Online Learning Algorithm of Behavior Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Heping Li, Zhanyi Hu, Yihong Wu, and Fuchao Wu
472
Optimal Learning High-Order Markov Random Fields Priors of Colour Image . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ke Zhang, Huidong Jin, Zhouyu Fu, and Nianjun Liu
482
Table of Contents – Part I
XVII
Hierarchical Learning of Dominant Constellations for Object Class Recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Nathan Mekuz and John K. Tsotsos
492
Multistrategical Approach in Visual Learning . . . . . . . . . . . . . . . . . . . . . . . . Hiroki Nomiya and Kuniaki Uehara
502
Poster Session 2: Motion and Tracking Cardiac Motion Estimation from Tagged MRI Using 3D-HARP and NURBS Volumetric Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jia Liang, Yuanquan Wang, and Yunde Jia
512
Fragments Based Parametric Tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Prakash C., Balamanohar Paluri, Nalin Pradeep S., and Hitesh Shah
522
Spatiotemporal Oriented Energy Features for Visual Tracking . . . . . . . . . Kevin Cannons and Richard Wildes
532
Synchronized Ego-Motion Recovery of Two Face-to-Face Cameras . . . . . . Jinshi Cui, Yasushi Yagi, Hongbin Zha, Yasuhiro Mukaigawa, and Kazuaki Kondo
544
Optical Flow – Driven Motion Model with Automatic Variance Adjustment for Adaptive Tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Kazuhiko Kawamoto
555
A Noise-Insensitive Object Tracking Algorithm . . . . . . . . . . . . . . . . . . . . . . Chunsheng Hua, Qian Chen, Haiyuan Wu, and Toshikazu Wada
565
Discriminative Mean Shift Tracking with Auxiliary Particles . . . . . . . . . . . Junqiu Wang and Yasushi Yagi
576
Poster Session 2: Retrival and Search Efficient Search in Document Image Collections . . . . . . . . . . . . . . . . . . . . . . Anand Kumar, C.V. Jawahar, and R. Manmatha
586
Human Pose Estimation Hand Posture Estimation in Complex Backgrounds by Considering Mis-match of Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Akihiro Imai, Nobutaka Shimada, and Yoshiaki Shirai
596
Learning Generative Models for Monocular Body Pose Estimation . . . . . Tobias Jaeggli, Esther Koller-Meier, and Luc Van Gool
608
Human Pose Estimation from Volume Data and Topological Graph Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Hidenori Tanaka, Atsushi Nakazawa, and Haruo Takemura
618
XVIII
Table of Contents – Part I
Matching Logical DP Matching for Detecting Similar Subsequence . . . . . . . . . . . . . . Seiichi Uchida, Akihiro Mori, Ryo Kurazume, Rin-ichiro Taniguchi, and Tsutomu Hasegawa
628
Efficient Normalized Cross Correlation Based on Adaptive Multilevel Successive Elimination . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Shou-Der Wei and Shang-Hong Lai
638
Exploiting Inter-frame Correlation for Fast Video to Reference Image Alignment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Arif Mahmood and Sohaib Khan
647
Poster Session 3: Face/Gesture/Action Detection and Recognition Flea, Do You Remember Me? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Michael Grabner, Helmut Grabner, Joachim Pehserl, Petra Korica-Pehserl, and Horst Bischof
657
Multi-view Gymnastic Activity Recognition with Fused HMM . . . . . . . . . Ying Wang, Kaiqi Huang, and Tieniu Tan
667
Real-Time and Marker-Free 3D Motion Capture for Home Entertainment Oriented Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Brice Michoud, Erwan Guillou, Hector Brice˜ no, and Sa¨ıda Bouakaz
678
Tracking Iris Contour with a 3D Eye-Model for Gaze Estimation . . . . . . . Haiyuan Wu, Yosuke Kitagawa, Toshikazu Wada, Takekazu Kato, and Qian Chen
688
Eye Correction Using Correlation Information . . . . . . . . . . . . . . . . . . . . . . . Inho Choi and Daijin Kim
698
Eye-Gaze Detection from Monocular Camera Image Using Parametric Template Matching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ryo Ohtera, Takahiko Horiuchi, and Shoji Tominaga
708
An FPGA-Based Smart Camera for Gesture Recognition in HCI Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yu Shi and Timothy Tsui
718
Poster Session 3: Low Level Vision and Phtometory Color Constancy Via Convex Kernel Optimization . . . . . . . . . . . . . . . . . . . Xiaotong Yuan, Stan Z. Li, and Ran He
728
Table of Contents – Part I
XIX
User-Guided Shape from Shading to Reconstruct Fine Details from a Single Photograph . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Alexandre Meyer, Hector M. Brice˜ no, and Sa¨ıda Bouakaz
738
A Theoretical Approach to Construct Highly Discriminative Features with Application in AdaBoost . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yuxin Jin, Linmi Tao, Guangyou Xu, and Yuxin Peng
748
Robust Foreground Extraction Technique Using Gaussian Family Model and Multiple Thresholds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Hansung Kim, Ryuuki Sakamoto, Itaru Kitahara, Tomoji Toriyama, and Kiyoshi Kogure Feature Management for Efficient Camera Tracking . . . . . . . . . . . . . . . . . . Harald Wuest, Alain Pagani, and Didier Stricker Measurement of Reflection Properties in Ancient Japanese Drawing Ukiyo-e . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Xin Yin, Kangying Cai, Yuki Takeda, Ryo Akama, and Hiromi T. Tanaka
758
769
779
Texture-Independent Feature-Point Matching (TIFM) from Motion Coherence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ping Li, Dirk Farin, Rene Klein Gunnewiek, and Peter H.N. de With
789
Where’s the Weet-Bix? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yuhang Zhang, Lei Wang, Richard Hartley, and Hongdong Li
800
How Marginal Likelihood Inference Unifies Entropy, Correlation and SNR-Based Stopping in Nonlinear Diffusion Scale-Spaces . . . . . . . . . . . . . . Ram¯ unas Girdziuˇsas and Jorma Laaksonen
811
Poster Session 3: Motion and Tracking Kernel-Bayesian Framework for Object Tracking . . . . . . . . . . . . . . . . . . . . . Xiaoqin Zhang, Weiming Hu, Guan Luo, and Steve Maybank
821
Markov Random Field Modeled Level Sets Method for Object Tracking with Moving Cameras . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Xue Zhou, Weiming Hu, Ying Chen, and Wei Hu
832
Continuously Tracking Objects Across Multiple Widely Separated Cameras . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yinghao Cai, Wei Chen, Kaiqi Huang, and Tieniu Tan
843
Adaptive Multiple Object Tracking Using Colour and Segmentation Cues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Pankaj Kumar, Michael J. Brooks, and Anthony Dick
853
XX
Table of Contents – Part I
Image Assimilation for Motion Estimation of Atmospheric Layers with Shallow-Water Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ´ Nicolas Papadakis, Patrick H´eas, and Etienne M´emin
864
Probability Hypothesis Density Approach for Multi-camera Multi-object Tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Nam Trung Pham, Weimin Huang, and S.H. Ong
875
Human Detection AdaBoost Learning for Human Detection Based on Histograms of Oriented Gradients . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Chi-Chen Raxle Wang and Jenn-Jier James Lien
885
Multi-posture Human Detection in Video Frames by Motion Contour Matching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Qixiang Ye, Jianbin Jiao, and Hua Yu
896
A Cascade of Feed-Forward Classifiers for Fast Pedestrian Detection . . . . Yu-Ting Chen and Chu-Song Chen
905
Combined Object Detection and Segmentation by Using Space-Time Patches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yasuhiro Murai, Hironobu Fujiyoshi, and Takeo Kanade
915
Segmentation Embedding a Region Merging Prior in Level Set Vector-Valued Image Segmentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ismail Ben Ayed and Amar Mitiche
925
A Basin Morphology Approach to Colour Image Segmentation by Region Merging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Erchan Aptoula and S´ebastien Lef`evre
935
Detecting and Segmenting Un-occluded Items by Actively Casting Shadows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Tze K. Koh, Amit Agrawal, Ramesh Raskar, Steve Morgan, Nicholas Miles, and Barrie Hayes-Gill
945
A Local Probabilistic Prior-Based Active Contour Model for Brain MR Image Segmentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jundong Liu, Charles Smith, and Hima Chebrolu
956
Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
965
Less Is More: Coded Computational Photography Ramesh Raskar Mitsubishi Electric Research Labs (MERL), Cambridge, MA, USA
Abstract. Computational photography combines plentiful computing, digital sensors, modern optics, actuators, and smart lights to escape the limitations of traditional cameras, enables novel imaging applications and simplifies many computer vision tasks. However, a majority of current Computational Photography methods involve taking multiple sequential photos by changing scene parameters and fusing the photos to create a richer representation. The goal of Coded Computational Photography is to modify the optics, illumination or sensors at the time of capture so that the scene properties are encoded in a single (or a few) photographs. We describe several applications of coding exposure, aperture, illumination and sensing and describe emerging techniques to recover scene parameters from coded photographs.
1 Introduction Computational photography combines plentiful computing, digital sensors, modern optics, actuators, and smart lights to escape the limitations of traditional cameras, enables novel imaging applications and simplifies many computer vision tasks. Unbounded dynamic range, variable focus, resolution, and depth of field, hints about shape, reflectance, and lighting, and new interactive forms of photos that are partly snapshots and partly videos are just some of the new applications found in Computational Photography. In this paper, we discuss Coded Photography which involves encoding of the photographic signal and post-capture decoding for improved scene analysis. With filmlike photography, the captured image is a 2D projection of the scene. Due to limited capabilities of the camera, the recorded image is a partial representation of the view. Nevertheless, the captured image is ready for human consumption: what you see is what you almost get in the photo. In Coded Photography, the goal is to achieve a potentially richer representation of the scene during the encoding process. In some cases, Computational Photography reduces to ‘Epsilon Photography’, where the scene is recorded via multiple images, each captured by epsilon variation of the camera parameters. For example, successive images (or neighboring pixels) may have a different exposure, focus, aperture, view, illumination, or instant of capture. Each setting allows recording of partial information about the scene and the final image is reconstructed from these multiple observations. In Coded Computational Photography, the recorded image may appear distorted or random to a human observer. But the corresponding decoding recovers valuable information about the scene. ‘Less is more’ in Coded Photography. By blocking light over time or space, we can preserve more details about the scene in the recorded single photograph. In this paper we look at four specific examples. Y. Yagi et al. (Eds.): ACCV 2007, Part I, LNCS 4843, pp. 1–12, 2007. © Springer-Verlag Berlin Heidelberg 2007
2
R. Raskar
(a) Coded Exposure: By blocking light in time, by fluttering the shutter open and closed in a carefully chosen binary sequence, we can preserve high spatial frequencies of fast moving objects to support high quality motion deblurring. (b) Coded Aperture Optical Heterodyning: By blocking light near the sensor with a sinusoidal grating mask, we can record 4D light field on a 2D sensor. And by blocking light with a mask at the aperture, we can extend the depth of field and achieve full resolution digital refocussing. (c) Coded Illumination: By observing blocked light at silhouettes, a multi-flash camera can locate depth discontinuities in challenging scenes without depth recovery. (d) Coded Sensing: By sensing intensities with lateral inhibition, a gradient sensing camera can record large as well as subtle changes in intensity to recover a highdynamic range image. We describe several applications of coding exposure, aperture, illumination and sensing and describe emerging techniques to recover scene parameters from coded photographs. 1.1 Film-Like Photography Photography is the process of making pictures by, literally, ‘drawing with light’ or recording the visually meaningful changes in the light leaving a scene. This goal was established for film photography about 150 years ago. Currently, ‘digital photography’ is electronically implemented film photography, refined and polished to achieve the goals of the classic film camera which were governed by chemistry, optics, mechanical shutters. Film-like photography presumes (and often requires) artful human judgment, intervention, and interpretation at every stage to choose viewpoint, framing, timing, lenses, film properties, lighting, developing, printing, display, search, index, and labelling. In this article we plan to explore a progression away from film and film-like methods to something more comprehensive that exploits plentiful low-cost computing and memory with sensors, optics, probes, smart lighting and communication. 1.2 What Is Computational Photography? Computational Photography (CP) is an emerging field, just getting started. We don’t know where it will end up, we can’t yet set its precise, complete definition, nor make a reliably comprehensive classification. But here is the scope of what researchers are currently exploring in this field. – Computational photography attempts to record a richer visual experience, captures information beyond just a simple set of pixels and makes the recorded scene representation far more machine readable. – It exploits computing, memory, interaction and communications to overcome long-standing limitations of photographic film and camera mechanics that have persisted in film-style digital photography, such as constraints on dynamic
Less Is More: Coded Computational Photography
3
range, depth of field, field of view, resolution and the extent of scene motion during exposure. – It enables new classes of recording the visual signal such as the ‘moment’ [Cohen 2005], shape boundaries for non-photorealistic depiction [Raskar et al 2004] , foreground versus background mattes, estimates of 3D structure, ‘relightable’ photos and interactive displays that permit users to change lighting, viewpoint, focus, and more, capturing some useful, meaningful fraction of the ‘light field’ of a scene, a 4-D set of viewing rays. – It enables synthesis of impossible photos that could not have been captured at a single instant with a single camera, such as wrap-around views (‘multiple-centerof-projection’ images [Rademacher and Bishop 1998]), fusion of time-lapsed events [Raskar et al 2004], the motion-microscope (motion magnification [Liu et al 2005]), video textures and panoramas [Agarwala et al 2005]. They also support seemly impossible camera movements such as the ‘bullet time’ (Matrix) sequence recorded with multiple cameras with staggered exposure times. – It encompass previously exotic forms of scientific imaging and data gathering techniques e.g. from astronomy, microscopy, and tomography. 1.3 Elements of Computational Photography Traditional film-like photography involves (a) a lens, (b) a 2D planar sensor and (c) a processor that converts sensed values into an image. In addition, the photography may involve (d) external illumination from point sources (e.g. flash units) and area sources (e.g. studio lights).
Computational Photography
Novel Illumination Light Sources
Novel Cameras
Modulators Generalized Optics
Generalized
Sensor
Processing Ray Reconstruction
Generalized
Optics
4D Incident Lighting
4D Ray Bender
Upto 4D Ray Sampler
4D Light Field Display
Recreate 4D Lightfield
Scene: 8D Ray Modulator
Fig. 1. Elements of Computational Photography
4
R. Raskar
Computational Photography generalizes these four elements. (a) Generalized Optics: Each optical element is treated as a 4D ray-bender that modifies a light field. The incident 4D light field for a given wavelength is transformed into a new 4D lightfield. The optics may involve more than one optical axis [Georgiev et al 2006]. In some cases the perspective foreshortening of objects based on distance may be modified using wavefront coded optics [Dowski and Cathey 1995]. In recent lensless imaging methods [Zomet and Nayar 2006] and coded-aperture imaging [Zand 1996] used for gamma-ray and X-ray astronomy, the traditional lens is missing entirely. In some cases optical elements such as mirrors [Nayar et al 2004] outside the camera adjust the linear combinations of ray bundles that reach the sensor pixel to adapt the sensor to the viewed scene. (b) Generalized Sensors: All light sensors measure some combined fraction of the 4D light field impinging on it, but traditional sensors capture only a 2D projection of this lightfield. Computational photography attempts to capture more; a 3D or 4D ray representation using planar, non-planar or even volumentric sensor assemblies. For example, a traditional out-of-focus 2D image is the result of a capture-time decision: each detector pixel gathers light from its own bundle of rays that do not converge on the focused object. But a Plenoptic Camera [Adelson and Wang 1992, Ren et al 2005] subdivides these bundles into separate measurements. Computing a weighted sum of rays that converge on the objects in the scene creates a digitally refocused image, and even permits multiple focusing distances within a single computed image. Generalizing sensors can extend their dynamic range [Tumblin et al 2005] and wavelength selectivity as well. While traditional sensors trade spatial resolution for color measurement (wavelengths) using a Bayer grid or red, green or blue filters on individual pixels, some modern sensor designs determine photon wavelength by sensor penetration, permitting several spectral estimates at a single pixel location [Foveon 2004]. (c) Generalized Reconstruction: Conversion of raw sensor outputs into picture values can be much more sophisticated. While existing digital cameras perform ‘demosaicking,’ (interpolate the Bayer grid), remove fixed-pattern noise, and hide ‘dead’ pixel sensors, recent work in computational photography can do more. Reconstruction might combine disparate measurements in novel ways by considering the camera intrinsic parameters used during capture. For example, the processing might construct a high dynamic range scene from multiple photographs from coaxial lenses, from sensed gradients [Tumblin et al 2005], or compute sharp images a fast moving object from a single image taken by a camera with a ‘fluttering’ shutter [Raskar et al 2006]. Closed-loop control during photography itself can also be extended, exploiting traditional cameras’ exposure control, image stabilizing, and focus, as new opportunities for modulating the scene’s optical signal for later decoding. (d) Computational Illumination: Photographic lighting has changed very little since the 1950’s: with digital video projectors, servos, and device-to-device communication, we have new opportunities to control the sources of light with as much sophistication as we use to control our digital sensors. What sorts of spatiotemporal modulations for light might better reveal the visually important contents
Less Is More: Coded Computational Photography
5
of a scene? Harold Edgerton showed high-speed strobes offered tremendous new appearance-capturing capabilities; how many new advantages can we realize by replacing ‘dumb’ the flash units, static spot lights and reflectors with actively controlled spatio-temporal modulators and optics? Already we can capture occluding edges with multiple flashes [Raskar 2004], exchange cameras and projectors by Helmholz reciprocity [Sen et al 2005], gather relightable actor’s performances with light stages [Wagner et al 2005] and see through muddy water with coded-mask illumination [Levoy et al 2004]. In every case, better lighting control during capture to builds richer representations of photographed scenes.
2 Sampling Dimensions of Imaging 2.1 Epsilon Photography for Optimizing Film-Like Camera Think of film cameras at their best as defining a ‘box’ in the multi-dimensional space of imaging parameters. The first, most obvious thing we can do to improve digital cameras is to expand this box in every conceivable dimension. This effort reduces Computational Photography to ‘Epsilon Photography’, where the scene is recorded via multiple images, each captured by epsilon variation of the camera parameters. For example, successive images (or neighboring pixels) may have different settings for parameters such as exposure, focus, aperture, view, illumination, or the instant of capture. Each setting allows recording of partial information about the scene and the final image is reconstructed from these multiple observations. Epsilon photography is thus concatenation of many such boxes in parameter space; multiple film-style photos computationally merged to make a more complete photo or scene description. While the merged photo is superior, each of the individual photos is still useful and comprehensible on its own, without any of the others. The merged photo contains the best features from all of them. (a) Field of View: A wide field of view panorama is achieved by stitching and mosaicking pictures taken by panning a camera around a common center of projection or by translating a camera over a near-planar scene. (b) Dynamic range: A high dynamic range image is captured by merging photos at a series of exposure values [Debevec and Malik 1997, Kang et al 2003] (c) Depth of field: All-in-focus image is reconstructed from images taken by successively changing the plane of focus [Agrawala et al 2005]. (d) Spatial Resolution: Higher resolution is achieved by tiling multiple cameras (and mosaicing individual images) [Wilburn et al 2005] or by jittering a single camera [Landolt et al 2001]. (e) Wavelength resolution: Traditional cameras sample only 3 basis colors. But multi-spectral (multiple colors in the visible spectrum) or hyper-spectral (wavelengths beyond the visible spectrum) imaging is accomplished by taking pictures while successively changing color filters in front of the camera, using tunable wavelength filters or using diffraction gratings. (f) Temporal resolution: High speed imaging is achieved by staggering the exposure time of multiple low-framerate cameras. The exposure durations of individual cameras can be non-overlapping ) [Wilburn et al 2005] or overlaping [Shechtman et al 2002].
6
R. Raskar
Taking multiple images under varying camera parameters can be achieved in several ways. The images can be taken with a single camera over time. The images can be captured simultaneously using ‘assorted pixels’ where each pixel is a tuned to a different value for a given parameter [Nayar and Narsimhan 2002]. Simultaneous capture of multiple samples can also be recorded using multiple cameras, each camera having different values for a given parameter. Two designs are currently being used for multi-camera solutions: a camera array [Wilburn et al 2005] and single-axis multiple parameter (co-axial) cameras [Mcguire et al 2005]. Coded Exposure
Temporal 11-D broadband code
Coded Aperture
Spatial 2-D broadband code
Fig. 2. Blocking light to achieve Coded Photography. (Left) Using a 1-D code in time to block and unblock light over time, a coded exposure photo can reversibly encode motion blur (Raskar et al 2006). (Right) Using a 2-D code in space to block parts of the light via a masked aperture, a coded aperture photo can reversibly encode defocus blur (Veeraraghavan et al 2007).
2.2 Coded Photography But there is much more beyond the ‘best possible film camera’. We can virtualize the notion of the camera itself if we consider it as a device that collects bundles of rays, each ray with its own wavelength spectrum and exposure duration. Coded Photography is a notion of an ‘out-of-the-box’ photographic method, in which individual (ray) samples or data sets may or may not be comprehensible as ‘images’ without further decoding, re-binning or reconstruction. Coded aperture techniques, inspired by work in astronomical imaging, try to preserve high spatial frequencies so that out of focus blurred images can be digitally re-focused [Veeraraghavan07]. By coding illumination, it is possible to decompose radiance in a scene into direct and global components [Nayar06]. Using a coded exposure technique, one can rapidly flutter open and close the shutter of a camera in a carefully chosen binary sequence, to capture a single photo. The fluttered shutter encoded the motion in the scene in the observed blur in a reversible way. Other examples include confocal images and techniques to recover glare in the images [Talvala07].
Less Is More: Coded Computational Photography
7
We may be converging on a new, much more capable ‘box’ of parameters in computational photography that we don’t yet recognize; there is still quite a bit of innovation to come! In the rest of the article, we survey recent techniques that exploit exposure, focus, active illumination and sensors. Coding in Time
Coding in Space
Coded Illumination
Coded Sensing
Exposure
Aperture
Inter-View
Gradient Sensor (Differential Encoding)
[Raskar et al 2006]
[Veeraraghavan et al 07]
[Raskar et al 2004]
[Tumblin et al 2005]
Mask, Optical Heterodyning
Intra-view
[Veeraraghavan et al 07]
[Nayar et al 2006]
Fig. 3. An overview of projects. Coding in time or space, coding the incident active illumination and coding the sensing pattern.
3 Coded Exposure In a conventional single-exposure photograph, moving objects or moving cameras cause motion blur. The exposure time defines a temporal box filter that smears the moving object across the image by convolution. This box filter destroys important high-frequency spatial details so that deblurring via deconvolution becomes an illposed problem. We have proposed to flutter the camera’s shutter open and closed during the chosen exposure time with a binary pseudo-random sequence, instead of leaving it open as in a traditional camera [Raskar et al 2006]. The flutter changes the box filter to a broad-band filter that preserves high-frequency spatial details in the blurred image and the corresponding deconvolution becomes a well-posed problem. Results on several challenging cases of motion-blur removal including outdoor scenes, extremely large motions, textured backgrounds and partial occluders were presented. However, the authors assume that PSF is given or is obtained by simple user interaction. Since changing the integration time of conventional CCD cameras is not feasible, an external ferro-electric shutter is placed in front of the lens to code the exposure. The shutter is driven opaque and transparent according to the binary signals generated from PIC using the pseudo-random binary sequence.
8
R. Raskar
Fig. 4. The flutter shutter camera. The coded exposure is achieved by fluttering the shutter open and closed. Instead of a mechanical movement of the shutter, we used a ferro-electric LCD in front of the lens. It is driven opaque and transparent according to the desired binary sequence.
4 Coded Aperture and Optical Heterodyning Can we capture additional information about a scene by inserting a patterned mask inside a conventional camera? We use a patterned attenuating mask to encode the light field entering the camera. Depending on where we put the mask, we can effect desired frequency domain modulation of the light field. If we put the mask near the lens aperture, we can achieve full resolution digital refocussing. If we put the mask near the sensor, we can recover a 4D light field without any additional lenslet array.
Fig. 5. Encoded Blur Camera, i.e. with mask in the aperture, can preserve high spatial images frequencies in the defocus blur. Notice the glint in the eye. In the misfocused photo, on the left, the bright spot appears blurred with the bokeh of the chosen aperture (shown in the inset). In the deblurred result, on the right, the details on the eye are correctly recovered.
Less Is More: Coded Computational Photography
9
Ren et al. have developed a camera that can capture the 4D light field incident on the image sensor in a single photographic exposure [Ren et al. 2005]. This is achieved by inserting a microlens array between the sensor and main lens, creating a plenoptic camera. Each microlens measures not just the total amount of light deposited at that location, but how much light arrives along each ray. By re-sorting the measured rays of light to where they would have terminated in slightly different, synthetic cameras, one can compute sharp photographs focused at different depths. A linear increase in the resolution of images under each microlens results in a linear increase in the sharpness of the refocused photographs. This property allows one to extend the depth of field of the camera without reducing the aperture, enabling shorter exposures and lower image noise. Our group has shown that it is also possible to create a plenoptic camera using a patterned mask instead of a lenslet array. The geometric configurations remains nearly identical [Veeraraghavan2007]. The method is known as ‘spatial optical heterodyning’. Instead of remapping of rays in 4D using microlens array so that they can be captured on a 2D sensor, spatial optical heterodyning remaps frequency components of the 4D lightfield so that the frequency components can be recovered from Fourier transform of the captured 2D image. In microlens array based design, each pixel effectively records light along a single ray bundle. With patterned masks, each pixel records a linear combination multiple ray-bundles. By carefully coding the linear combination, the coded heterodyning method can reconstruct the values of individual ray-bundles. This is reversible modulation of 4D light field by inserting a patterned planar mask in the optical path of a lens based camera. We can reconstruct the 4D light field from a 2D camera image. The patterned mask attenuates light rays inside the camera instead of bending them, and the attenuation recoverably encodes the ray on the 2D sensor. Our mask-equipped camera focuses just as a traditional camera might to capture conventional 2D photos at full sensor resolution, but the raw pixel values also hold a modulated 4D light field. The light field can be recovered by rearranging the tiles of the 2D Fourier transform of sensor values into 4D planes, and computing the inverse Fourier transform. Mask?
Mask
Sensor
Coded Aperture for Full Resolution Digital Refocusing
Sensor
Mask
Sensor
Heterodyne Light Field Camera
Fig. 6. Coding Light Field entering a camera via a mask
10
R. Raskar
5 Coded Illumination By observing blocked light at silhouettes, a multi-flash camera can locate depth discontinuities in challenging scenes without depth recovery. We used a multi-flash camera to find the silhouettes in a scene [Raskar et al 2004]. We take four photos of an object with four different light positions (above, below, left and right of the lens). We detect shadows cast along the depth discontinuities are use them to detect depth discontinuities in the scene. The detected silhouettes are then used for stylizing the photograph and highlighting important features. We also demonstrate silhouette detection in a video using a repeated fast sequence of flashes. Bottom Flash
Top Flash
Left Flash
Right Flash
Ratio images showing shadows and traversal to find edges
Photo
Shadow-Free
Depth Edges
Depth Edges
Fig. 7. Multi-flash Camera for Depth Edge Detection. (Left) A camera with four flashes. (Right) Photos due to individual flashes, highlighted shadows and epipolar traversal to compute the single pixel depth edges.
6 High Dynamic Range Using a Gradient Camera A camera sensor is limited in the range of highest and lowest intensities it can measure. To capture the high dynamic range, one can adaptively exposure the sensor so that the signal to noise ratio is high over the entire image, including in the the dark and brightly lit regions. One approach for faithfully recording the intensities in a high dynamic range scenes is to capture multiple images using different exposures, and then to merge these images. The basic idea is that when longer exposures are used, dark regions are well exposed but bright regions are saturated. On the other hand, when short exposures are used, dark regions are too dark but bright regions are well imaged. If exposure varies and multiple pictures are taken of the same scene, value of a pixel can be taken from those images where it’s neither too dark nor saturated. This type of approach is often referred to as exposure bracketing. At the sensor level, various approaches have also been proposed for high dynamic range imaging. One type of approach is to use multiple sensing elements with different sensitivities within each cell [Street 1998, Handy 1986, Wen 1989, Hamazaki 1996]. Multiple measurements are made from the sensing elements, and they are combined
Less Is More: Coded Computational Photography
11
on-chip before a high dynamic range image is read out from the chip. Spatial sampling rate is lowered in these sensing devices, and spatial resolution is sacrificed. Another type of approach is to adjust the well capacity of the sensing elements during photocurrent integration [Knight 1983, Sayag 1990, Decker 1998] but this gives higher noise. By sensing intensities with lateral inhibition, a gradient sensing camera can record large as well as subtle changes in intensity to recover a high-dynamic range image. By sensing different between neighboring pixels instead of actual intensities, our group has shown that a ‘Gradient Camera’ can record large global variations in intensity [Tumblin et al 2005]. Rather than measure absolute intensity values at each pixel, this proposed sensor measures only forward differences between them, which remain small even for extremely high-dynamic range scenes, and reconstructs the sensed image from these differences using Poisson solver methods. This approach offers several advantages: the sensor is nearly impossible to over- or under-expose, yet offers extremely fine quantization, even with very modest A/D convertors (e.g. 8 bits). The thermal and quantization noise occurs in the gradient domain, and appears as low frequency ‘cloudy’ noise in the reconstruction, rather than uncorrelated highfrequency noise that might obscure the exact position of scene edges.
7 Conclusion As these examples indicate, we have scarcely begun to explore the possibilities offered by combining computation, 4D modeling of light transport, and novel optical systems. Nor have such explorations been limited to photography and computer graphics or computer vision. Microscopy, tomography, astronomy and other optically driven fields already contain some ready-to-use solutions to borrow and extend. If the goal of photography is to capture, reproduce, and manipulate a meaningful visual experience, then the camera is not sufficient to capture even the most rudimentary birthday party. The human experience and our personal viewpoint is missing. Computational Photography can supply us with visual experiences, but can’t decide which one’s matter most to humans. Beyond coding the first order parameters like exposure, focus, illumination and sensing, maybe the ultimate goal of Computational Photography is to encode the human experience in the captured single photo.
Acknowledgements We wish to thank Jack Tumblin and Amit Agrawal for contributing several ideas for this paper. We also thank co-authors and collaborators Ashok Veeraraghavan, Ankit Mohan, Yuanzen Li, Karhan Tan, Rogerio Feris, Jingyi Yu, Matthew Turk. We thank Shree Nayar and Marc Levoy for useful comments and discussions.
References Raskar, R., Tan, K., Feris, R., Yu, J., Turk, M.: Non-photorealistic Camera: Depth Edge Detection and Stylized Rendering Using a Multi-Flash Camera. SIGGRAPH 2004 (2004) T umblin, J., Agrawal, A., Raskar, R.: Why I want a Gradient Camera. In: CVPR 2005, IEEE, Los Alamitos (2005)
12
R. Raskar
Raskar, R., Agrawal, A., Tumblin, J.: Coded exposure photography: motion deblurring using fluttered shutter. ACM Trans. Graph 25(3), 795–804 (2006) Veeraraghavan, A., Raskar, R., Agrawal, A., Mohan, A., Tumblin, J.: Dappled Photography: Mask-Enhanced Cameras for Heterodyned Light Fields and Coded Aperture Refocusing. ACM Siggraph (2007) Nayar, S.K., Narasimhan, S.G.: Assorted Pixels: Multi-Sampled Imaging With Structural Models. In: ECCV. Europian Conference on Computer Vision, vol. IV, pp. 636–652 (2002) Debevec, Malik.: Recovering high dynamic range radiance maps from photographs. In: Proc. SIGGRAPH (1997) Mann, Picard.: Being ’undigital’ with digital cameras: Extending dynamic range by combining differently exposed pictures. In: Proc. IS&T 46th ann. conference (1995) McGuire, M., Matusik, Pfister, Hughes, Durand.: Defocus Video Matting, ACM Transactions on Graphics. Proceedings of ACM SIGGRAPH 2005 24(3) (2005) Adelson, E.H., Wang, J.Y.A.: Single Lens Stereo with a Plenoptic Camera. IEEE Transactions on Pattern Analysis and Machine Intelligence 14(2) (1992) Ng, R.: Fourier Slice Photography, SIGGRAPH (2005) Morimura. Imaging method for a wide dynamic range and an imaging device for a wide dynamic range. U.S. Patent 5455621 (October 1993) Levoy, M., Hanrahan, P.: Light field rendering. In: SIGGRAPH, pp. 31–42 (1996) Dowski Jr., E.R., Cathey, W.T.: Extended depth of field through wave-front coding. Applied Optics 34(11), 1859–1866 (1995) Georgiev, T., Zheng, C., Nayar, S., Salesin, D., Curless, B., Intwala, C.: Spatio-angular Resolution Trade-Offs in Integral Photography. In: proceedings, EGSR 2006 (2006)
Optimal Algorithms in Multiview Geometry Richard Hartley1 and Fredrik Kahl2 Research School of Information Sciences and Engineering, The Australian National University National ICT Australia (NICTA) Centre for Mathematical Sciences, Lund University, Sweden
1
2
Abstract. This is a survey paper summarizing recent research aimed at finding guaranteed optimal algorithms for solving problems in Multiview Geometry. Many of the traditional problems in Multiview Geometry now have optimal solutions in terms of minimizing residual imageplane error. Success has been achieved in minimizing L2 (least-squares) or L∞ (smallest maximum error) norm. The main methods involve Second Order Cone Programming, or quasi-convex optimization, and Branch-andbound. The paper gives an overview of the subject while avoiding as far as possible the mathematical details, which can be found in the original papers. J.E.Littlewood: The first test of potential in mathematics is whether you can get anything out of geometry. G.H.Hardy: The sort of mathematics that is useful to a superior engineer, or a moderate physicist has no esthetic value and is of no interest to the real mathematician.
1
Introduction
In this paper, we describe recent work in geometric Computer Vision aimed at finding provably optimal solutions to some of the main problems. This is a subject which the two authors of this paper have been involved in for the last few years, and we offer our personal view of the subject. We cite most of the relevant papers that we are aware of and apologize for any omissions. There remain still several open problems, and it is our hope that more researchers will be encouraged to work in this area. Research in Structure from Motion since the start of the 1990s resulted in the emergence of a dominant accepted technique – bundle adjustment [46]. In this method, a geometric problem is formulated as a (usually) non-linear optimization problem, which is then solved using an iterative optimization algorithm.
NICTA is funded by the Australian Government’s Backing Australia’s Ability initiative, in part through the Australian Research Council.
Y. Yagi et al. (Eds.): ACCV 2007, Part I, LNCS 4843, pp. 13–34, 2007. c Springer-Verlag Berlin Heidelberg 2007
14
R. Hartley and F. Kahl
Generally, the bundle adjustment problem is formulated as follows. One defines a cost function (also called an objective function) in terms of a set of parameters. Solving the problem involves finding the set of parameters that minimize the cost. Generally, the parameters are associated with the geometry that we wish to discover. Often they involve parameters of a set of cameras, as well as parameters (such as 3D point coordinates) describing the geometry of the scene. The cost function usually involves image measurements, and measures how closely a given geometric configuration (for instance the scene geometry) explains the image measurements. Bundle adjustment has many advantages, which account for its success and popularity. 1. It is quite general, and can be applied to a large range of problems. 2. It is very easy to augment the cost function to include other constraints that the problem must satisfy. 3. We can “robustify” the cost function by minimizing a robust cost function, such as the Huber cost function. 4. Typically, the estimation problem is sparse, so sparse techniques can be used to achieve quite fast run times. One of the issues with bundle adjustment is that it requires a relatively accurate initial estimate of the geometry in order to converge to a correct minimum. This requirement led to one of the other main themes of research in Multiview Geometry through the 1990s - finding reliable initial solutions usually through so-called algebraic techniques. Most well known among such techniques is the 8point algorithm [28] for estimation of the essential or fundamental matrix, which solves the two-view relative orientation problem. Generally, in such methods, one defines an algebraic condition that must hold in the case of a noise-free solution, and defines an algebraic cost function that expresses how far this condition is from being met. Unfortunately, the cost function often is not closely connected with the geometry, and the cost function may be meaningless in any geometric or statistical sense. Multiple minima. One of the drawbacks of bundle adjustment is the possibility of converging to a local, rather than a global minimum of the cost function. The cost functions that arise in multiview optimization problems commonly do have multiple local minima. As an example, in Fig 1 we show graphs of the cost functions associated with the two-view triangulation problem (described later) and the autocalibration problem. The triangulation problem with more than two views is substantially more complex. It has been shown in [43] that the triangulation problem with three views involves solving a polynomial of degree 47, and hence the cost function potentially may have up to 24 minima. For higher numbers of views, the degree of the polynomial grows cubically. Stew´enius and Nist´er have calculated the sequence of degrees of the polynomial to be 6, 47, 148, 336, 638, 1081 for 2 to 7 view triangulation, which implies a large number of potential local minima of the cost function.
Optimal Algorithms in Multiview Geometry
15
Fig. 1. Top: Two-view triangulation cost function, showing up to three minima. The independent variable (x-axis) parametrizes the epipolar plane. On the left, an example with three local minima (two of them with equal cost). On the right, an example with global solution with zero cost (perfect noise-free triangulation), yet having a further local minimum. The graphs are taken from [12]. Bottom: 2-dimensional cross-section of a cost associated with an autocalibration problem. The graphs are taken from [9]. Left to right: top view, side-view, contour plot. Note the great complexity of the cost function, and the expected difficulties in finding a global minimum.
As for autocalibration, progress has been made on this problem in [5], which finds optimal methods of carrying out the projective - affine - Euclidean upgrade, the “stratified” approach to autocalibration. However, this is not quite the same as an optimal solution to autocalibration, which remains an open (very difficult) problem. Optimal methods. The difficulties and uncertainties associated with bundle adjustment, and algebraic methods has led to a theme of research in the last few years that aims at finding guaranteed provably optimal methods for solving these problems. Although such methods have not been successful for all geometric problem, the number of problems that can be solved using such optimal methods continues to grow. It is the purpose of this paper to survey progress in this area.
2
What Is Meant by an Optimal Solution?
In this section, we will argue that what is meant by an optimal solution to a geometric problem is not clearly defined. We consider a problem in which a set of measurements are made and we need to fit a parametrized model of some kind to these measurements. Optimization involves finding the set of parameters of the model that best fit the purpose. The optimization problem is defined in terms of a defined cost function which must be minimized over the set of
16
R. Hartley and F. Kahl
meaningful parameters. However, what particular cost functions merit being called “optimal” will be considered in the rest of this section. To set up the problem, consider a set of measurements, xi to which we wish to ˆ i which are fit a parametrized model. The model gives rise to some model values x defined in some way in terms of the parametrization. The set of residuals δ i are defined as δi = xi −ˆ xi where · represents a suitable norm in the measurement space. For image measurements, this is most reasonably the distance in the image ˆ i . Finally, denote by Δ the vector with components δi . This between xi and x may be called the vector of residuals. 2.1
L2 Error and ML Estimation
Often, it is argued that the optimal solution is the least squares solution, which minimizes the cost function ˆ i 2 , xi − x Δ 2 = i
namely the L2 -norm of the vector of residuals.1 The argument for the optimality of this solution is as follows. If we assume that the measurements are derived from actual values, corrupted with Gaussian noise with variance σ 2 , then the ˆ i is given by probability of the set of measurements xi given true values x ˆ i 2 /(2σ 2 ) . P ({xi } | {ˆ xi }) = K exp −xi − x i
where K is a normalizing constant. Taking logarithms and minimizing, we see that the modelled data that2 maximizes the probability of the measurements ˆ i . Thus, the least-squares solution is the maximialso minimizes i xi − x mum likelihood (ML) estimate, under an assumption of Gaussian noise in the measurements. This is the argument for optimality of the least-squares solution. Although useful, this argument does rely on two assumptions that are open to question. 1. It makes an assumption of Gaussian noise. This assumption is not really justified at all. Measurement errors in a digital image are not likely to satisfy a Gaussian noise model. 2. Maximum likelihood is not necessarily the same as optimal, a term that we have not defined. One might define optimal to mean maximum likelihood, but this is a circular argument. 2.2
L∞ Error
An alternative noise model, perhaps equally justified for image point measurements is that of uniform bounded noise. Thus, we assume that all measurements 1
In future the symbol · represents the 2-norm of a vector.
Optimal Algorithms in Multiview Geometry
17
less than a given threshold distance from the true value are equally likely, but measurements beyond this threshold have probability zero. In the case of a discrete image, measurements more accurate than one pixel from the true value are difficult to obtain, though in fact they may be achieved in carefully controlled situations. If we assume that the measurement error probability model is ˆ ) = K exp (−(x − x ˆ /σ)p ) P (x | x
(1)
where K is a normalizing factor, then as before, the ML estimate is the one ˆ i p . Letting p increase to infinity, the probability that minimizes i xi − x distribution (1) converges uniformly (except at σ) to a uniform distribution for ˆ . Now, taking the p-th root, we see that minimizing (1) is x within distance σ of x equivalent to normalizing the p-norm of the vector Δ of residuals. Furthermore, as p increases to infinity, Δp converges to the L∞ norm Δ∞ . In this way, minimizing the L∞ error corresponds to an assumption of uniform distribution of measurement errors. Looked at another way, under the L∞ norm, all sets of measurements that are within a distance σ of the modelled values are equally likely, whereas a set of measurements where one of the values exceeds the threshold σ has probability zero. Then L∞ optimization finds the smallest noise threshold for which the set of measurements is possible, and determines the ML estimate for this minimum threshold. Note that the L∞ norm of a vector is simply the largest component of the vector, in absolute value. Thus, ˆi , min Δ∞ = min max xi − x i
where the minimization is taken over the parameters of the model. For this reason, L∞ minimization is sometimes referred to as minimax optimization. 2.3
Other Criteria for Optimality
It is not clear that the maximum likelihood estimate has a reasonable claim to being the optimal estimate. It is pointed out in [13] that the maximum likelihood estimate may be biased, and in fact have infinite bias even for quite simple estimation problems. Thus, as the noise level of measurements increases, the average (or expected) estimate drifts away from the true value. This is of course undesirable. In addition, a different way of thinking of the estimation problem is in terms of the risk of making a wrong estimate. For example consider the triangulation problem (discussed in more detail later) in which several observers estimate a bearing (direction vector) to a target from known observation points. If the bearing directions are noisy, then where is the target? In many cases, there is a cost associated with wrongly estimating the position of the target. (For instance, if the target is an incoming ballistic missile, the
18
R. Hartley and F. Kahl
cost of a wrong estimate can be quite high.) A reasonable procedure would be to choose an estimate that minimizes the expected cost. As an example, if the cost of an estimate is equal to the square of the distance between the estimate and the true value, then the expected cost is equal to the mean of the posterior probability distribution of the parameters, P (θ | {xi })2 More discussion of these matters is contained in [13], Appendix 3. We are not, however, aware of any literature in multiview geometry finding estimates of this kind. What we mean by optimality. In this survey, we will consider the estimates that minimize the L2 or L∞ norms of the residual error vector, with respect to a parametrized model as being the optimal. This is reasonable in that it is related in either case to a specific geometric noise model, tied directly to the statistics of the measurements.
3
Polynomial Methods
One approch for obtaining optimal solutions to multiview problems is to compute all stationary points of the cost function and then check which of these is the global minimum. From a theoretical point of view, any structure and motion problem can be solved in this manner as long as the cost function can be expressed as a rational polynomial function in the parameters. This will be the case for most cost functions encountered (though not for L∞ cost functions, which are not differentiable). The method is as follows. A minimum of the cost function must occur at a point where the derivatives of the cost with respect to the parameters vanish. If the cost function is a rational polynomial function, then the derivatives are rational as well. Setting the derivatives to zero leads to a system of polynomial equations, and the solutions of this set of equations define the stationary points of the cost function. These can be checked one-by-one to find the minimum. This method may also be applied when the parameters satisfy certain constraints, such as a constraint of zero determinant for the fundamental matrix, or a constraint that a matrix represents a rotation. Such problems can be solved using Lagrange multipliers. Although this method is theoretically generally applicable, in practice it is only tractable for small problems, for example the triangulation problem. A solution to the two-view triangulation problem was given in [12], involving the solution of a polynomial of degree 6. The three-view problem was addressed in [43]; the solution involves the solution of a polynomial of degree 47. Further work (unpublished) by Stew´enius and Nist´er has shown that the triangulation problem for 4 to 7 views can be solved by finding the roots of a polynomial of degree 148, 336, 638, 1081 respectively, and in general, the degree grows cubically. Since 2
This assumes that the parameters θ are in a Euclidean space, which may not always be the case. In addition, estimation of the posterior probability distribution may require the definition of a prior P (θ).
Optimal Algorithms in Multiview Geometry
19
solving large sets of polynomials is numerically difficult, the issue of accuracy has been addressed in [4]. The polynomial method may also be applied successfully in many minimalconfiguration problems. We do not attempt here to enumerate all such problems considered in the literature. One notable example, however, is the relative orientation (two-view reconstruction) problem with 5 point correspondences. This has long been known to have 10 solutions [7,16].3 The second of these references gives a very pleasing derivation of this result. Recent simple algorithms for solving this problem by finding the roots of a polynomial have been given in [31,26]. Methods for computing a polynomial solution need not result in a polynomial of the smallest possible degree. However recent work using Galois theory [32] gives a way to address this question, showing that the 2-view triangulation and the relative orientation problem essentially require solution of polynomials of degree 6 and 10 respectively.
4
L∞ Estimation and SOCP
In this section, we will consider L∞ optimization, and discuss its advantages vis-a-vis L2 optimization. We show that there are many problems that can be formulated in L∞ and give a single solution. This is the main advantage, and contrasts with L2 optimization, which may have many local minima, as was shown in Fig 1. 4.1
Convex Optimization
We start by considering convex optimization problems. First, a few definitions. Convex set. A subset S of Rn is said to be convex if the line segment joining any two point in S is contained in S. Formally, if x0 , x1 ∈ S, then (1−α)x0 +αx1 ∈ S for all α with 0 ≤ α ≤ 1. Convex function. A function f : Rn → R is convex if its domain is a convex set and for all x, y ∈ domain(f ), and α with 0 ≤ α ≤ 1, we have f ((1 − α)x0 + αx1 ) ≤ (1 − α)f (x0 ) + αf (x1 ). Another less formal way of defining a convex function is so say that a line joining two points on the graph of the function will always lie above the function. This is illustrated in Fig 2. 3
These papers find 20 solutions, not 10, since they are solving for the number of possible rotations. There are 10 possible essential matrices each of which gives two possible rotations, related via the twisted pair ambiguity (see [13]). Only one of these two rotations is cheirally correct, corresponding to a possible realizable solution.
20
R. Hartley and F. Kahl
Fig. 2. Left. Examples of convex and non-convex sets. Middle. The definition of a convex function; the line joining two points lies above the graph of the function. Right. Convex optimization.
Convex optimization. A convex optimization problem is as follows: – Given a convex function f : D → R, defined on a convex domain D ⊂ Rn , find the minimum of f on D. A convex function is always continuous, and given reasonable conditions that ensure a minimum of the function (for instance D is compact) such a convex optimization problem is solvable by known algorithms4 . A further desirable property of a convex problem is that it has no local minima apart from the global minimum. The global minimum value is attained at a single point, or at least on a convex set in Rn where the function takes the same minimum value at all points. For further details, we refer the reader to the book [3]. Quasi-convex functions. Unfortunately, although convex problems are agreeable, they do not come up often in multiview geometry. However, interestingly enough, certain other problems do, quasi-convex problems. Quasi-convex functions are defined in terms of sublevel sets as follows. Definition 1. A function f : D → R is called quasi-convex if its α-sublevel set, Sα = {x ∈ D | f (x) ≤ α} is convex for all α. Examples of quasi-convex and non-quasi-convex functions are shown in Fig 3. Quasi-convex functions have two important properties. 1. A quasi-convex function has no local minima apart from the global minimum. It will attain its global minimum at a single point or else on a convex set where it assumes a constant value. 4
The most efficient algorithm to use will depend on the form of the function f , and the way the domain D is specified.
Optimal Algorithms in Multiview Geometry
21
Fig. 3. Quasi-convex functions. The left two functions are quasi-convex. All the sublevel sets are convex. The function on the right is not quasi-convex. The indentation in the function graph (on the left) means that the sublevel-sets are not convex. All convex functions are quasi-convex, but the example on the left shows that the converse is not true.
2. The pointwise maximum of a set of quasi-convex functions is quasi-convex. This is illustrated in Fig 4 for the case of functions of a single variable. The general case follows directly from the following observation concerning sublevel sets. Sδ (fi ) Sδ (max fi ) = i
i
which is convex, since each Sδ (fi ) is convex.
Fig. 4. The pointwise maximum of a set of quasi-convex functions is quasi-convex
A quasi-convex optimization problem is defined in the same way as a convex optimization problem, except that the function to be minimized is quasi-convex. Nevertheless, quasi-convex optimization problems share many of the pleasant properties of convex optimization. Why consider quasi-convex functions? The primary reason for considering such functions is that the residual of a measured image point x with respect to the projection of a point X in space is a quasi-convex function. In other words, f (X) = d(x, PX) is a quasi-convex function of X. Here, PX is the projection of a point X into an image, and d(·, ·) represents distance in the image.
22
R. Hartley and F. Kahl
Fig. 5. The triangulation problem: Assuming that the maximum reprojection error is less than some value δ, the sought point X must lie in the intersection of a set of cones. If δ is set too small, then the cones do not have a common intersection (left). If δ is set too large, then the cones intersect in a convex region in space, and the desired solution X must lie in this region (right). The optimal value of δ lies between these two extremes, and can be found by a binary search (bisection) testing successive values of δ. For more details, refer to the text.
Specifically, the sublevel set Sδ (f (X)) is a convex set, namely a cone with vertex the centre of projection, as will be discussed in more detail shortly (see Fig 5). As the reader may easily verify by example, however, the sum of quasi-convex functions is not in general quasi-convex. If we take several image measurements then the sum of squares of the residuals will not in general be a quasi-convex 2 function. In other words, an L2 cost function of the form N i=1 fi (X) will not in general be a quasi-convex function of X, nor have a single minimum. On the other hand, as remarked above, the maximum of a set of quasiconvex functions is quasi-convex, and hence will have a single minimum. Specifically, maxi fi (X) will be quasi-convex, and have a single minimum with respect to X. For this reason, it is typically easier to solve the minimax problem minX maxi fi (X) than the corresponding least-squares (L2 ) problem. Example: The triangulation problem. The triangulation problem is the most simple problem in multiview geometry. Nevertheless, in the L2 formulation, it still suffers from the problem of local minima, as shown in Fig 1. In this problem, we have a set of known camera centres Oi and a set of direction vectors vi which give the direction of a target point X from each of the camera centres. Thus, nominally, vi = (X − Oi )/X − Oi . The problem is to find the position of the point X. We choose to solve this problem in L∞ norm, in other words, we seek the point X that minimizes the maximum error (over all i) between vi and the directions given by X − Oi . Consider Fig 5. Some simple notation is required. Define ∠(X − Oi , vi ) to be the angle between the vectors X − Oi and vi . Given a value δ > 0, the set of points Cδ (Oi , vi ) = {X | ∠(X − Oi , vi ) ≤ δ} forms a cone in R3 with vertex Oi , axis vi and angle determined by δ.
Optimal Algorithms in Multiview Geometry
23
We begin by hypothesizing that there exists a solution X to the triangulation problem for which the maximum error is δ. In this case, the point X must lie inside cones in R3 with vertex Oi , axis vi and angle determined by δ. If the cones are too narrow, they do not have a common intersection and there can be no solution with maximum error less than δ. On the other hand, if δ is sufficiently large, then the cones intersect, and the desired solution X must lie in the intersection of the cones. The optimal value of δ is found by a binary search over values of δ to find the smallest value such that the cones Cδ (Oi , vi ) intersect in at least one point. The intersection will be a single point, or in special configurations, a segment of a line. The problem of determining whether a set of cones have non-empty intersection is solved by a convex optimization technique called Second Order Cone Programming (SOCP), for which open source libraries exist [44]. We make certain observations about this problem: 1. Each cone Cδ (Oi , vi ) is a convex set, and hence their intersection is convex. 2. If we define a cost function Cost∞ (X) = max ∠(X − Oi , vi ), i
then the sublevel set Sδ (Cost∞ ) is simply the intersection of the cones Cδ (Oi , vi ), which is convex for all δ. This by definition says that Cost∞ (X) is a quasi-convex function of X. 3. Finding the optimum min Cost∞ (X) = min max ∠(X − Oi , vi ) X
X
i
is accomplished by a binary search over possible values of δ, where for each value of δ we solve an SOCP feasibility problem, (determine whether a set of cones have a common intersection). Such a problem is known as a minimax or L∞ optimization problem. Generally speaking, this procedure generalizes to arbitrary quasi-convex optimization problems; they may be solved by binary search involving a convex feasibility problem at each step. If we have a set of individual cost functions fi (X), perhaps each associated with a single measurement, and each of them quasi-convex, then the maximum of these cost functions maxi fi (X) is also quasiconvex, as illustrated in Fig 4. In this case, the minimax problem of finding minX maxi fi (X) is solvable by binary search. Reconstruction with known rotations. Another problem that may be solved by very similar means to the triangulation problem is that of Structure and Motion with known rotations, which is illustrated in Fig 6. The role of cheirality. It is important in solving problems of this kind to take into account the concept of “cheirality”, which means the requirement that
24
R. Hartley and F. Kahl
Fig. 6. Structure and motion with known rotations. If the orientations of several cameras are all known, then image points correspond to direction vectors in a common coordinate frame. Here, blue (or grey) circles represent the positions of cameras, and black circles the positions of points. The arrows represent direction vectors (their length is not known) from cameras to points. The positions of all the points and cameras may be computed (up to scale and translation) using SOCP. Abstractly, this is the embedding problem for a bi-partite graph in 3D, where the orientation of the edges of the graph is known. The analogous problem for an arbitrary (not bi-partite) graph was applied in [41] to solve for motion of the cameras without computing the point positions.
points visible in an image must lie in front of the camera, not behind. If we subdivide space by planes separating front and back of the camera, then there will be at least one local minimum of the cost function (whether L∞ or L2 ) in each region of space. Since the number of regions separated by n planes grows cubically, so do the number of local minima, unless we constrain the solution so that the points lie in front of the cameras. Algorithm acceleration. Although the bisection algorithm using SOCP has been the standard approach to L∞ geometric optimization problems, there has been recent work for speeding up the computations [39,40,34]. However, it has been shown that the general structure and motion (with missing data) is N P -hard no matter what criterion of optimality of reprojection error is used [33]. 4.2
Problems Solved in L∞ Norm
The list of problems that can be solved globally with L∞ estimation continues to grow and by now it is quite long, see Table 1. In [21], an L∞ estimation algorithm serves as the basis for solving the leastmedian squares problem for various geometric problems. However, the extension to such problems is essentially based on heuristics and it has no guarantee of finding the global optimum.
Optimal Algorithms in Multiview Geometry
25
Table 1. Geometric reconstruction problems that can be solved globally with the L∞ or L2 norm L∞ -norm − Multiview triangulation − Camera resectioning (uncalibrated case) − Camera pose (calibrated case) − Homography estimation − Structure and motion recovery with known camera orientation − Reconstruction by using a reference plane − Camera motion recovery − Outlier detection − Reconstruction with covariance-based uncertainty − Two-view relative orientation − 1D retinal vision L2 -norm − Affine reconstruction from affine cameras − Multiview triangulation − Camera resectioning (uncalibrated case) − Homography estimation − 3D – 3D registration and matching − 3D – 3D registration and matching (unknown pairing)
5
References [11,18,20,6] [18,20] [10,47] [18,20] [11,18,20] [18] [41] [42,20,25,47] [41,21] [10] [2] References [23,45] [1,29] [1] [1] [14] [27]
Branch-and-Bound Theory
The method based on L∞ optimization is not applicable to all problems. In this section, we will describe in general terms a different method that has been used with success in obtaining globally optimal solutions. Branch and bound algorithms are non-heuristic methods for global optimization in non-convex problems. They maintain a provable upper and/or lower bound on the (globally) optimal objective value and terminate with a certificate proving that the solution is within of the global optimum, for arbitrarily small . Consider a non-convex, scalar-valued objective function Φ(x), for which we seek a global minimum over a domain Q0 . For a subdomain Q ⊆ Q0 , let Φmin (Q) denote the minimum value of the function Φ over Q. Also, let Φlb (Q) be a function that computes a lower bound for Φmin (Q), that is, Φlb (Q) ≤ Φmin (Q). An intuitive technique to determine the solution would be to divide the whole search region Q0 into a grid with cells of sides δ and compute the minimum of a lower bounding function Φlb defined over each grid cell, with the presumption that each Φlb (Q) is easier to compute than the corresponding Φmin (Q). However, the number of such grid cells increases rapidly as δ → 0, so a clever procedure must be deployed to create as few cells as possible and “prune” away as many of these grid cells as possible (without having to compute the lower bounding function for these cells). Branch and bound algorithms iteratively subdivide the domain into subregions (which we refer to as rectangles) and employ clever strategies to prune away as many rectangles as possible to restrict the search region.
26
R. Hartley and F. Kahl
Φ(x)
x
l
(a)
u
Φ(x)
Φ(x)
Φ(x)
q1∗
q1∗
q1∗ q2∗
x
l
u
(b)
x
l
u
(c)
x
l
u
(d)
Fig. 7. This figure illustrates the operation of a branch and bound algorithm on a one dimensional non-convex minimization problem. Figure (a) shows the the function Φ(x) and the interval l ≤ x ≤ u in which it is to be minimized. Figure (b) shows the convex relaxation of Φ(x) (indicated in yellow/dashed), its domain (indicated in blue/shaded) and the point for which it attains a minimum value. q1∗ is the corresponding value of the function Φ. This value is the current best estimate of the minimum of Φ(x), and is used to reject the left subinterval in Figure (c) because the minimum value of the convex relaxation is higher than q1∗ . Figure (d) shows the lower bounding operation on the right sub-interval in which a new estimate q2∗ of the minimum value of Φ(x) is found.
A graphical illustration of the algorithm is presented in Fig 7. Computation of the lower bounding functions is referred to as bounding, while the procedure that chooses a domain and subdivides it is called branching. The choice of the domain picked for refinement in the branching step and the actual subdivision itself are essentially heuristic. Although guaranteed to find the global optimum (or a point arbitrarily close to it), the worst case complexity of a branch and bound algorithm is exponential. However, in many cases the properties offered by multiview problems lead to fast convergence rates in practice.
6
Branch-and-Bound for L2 Minimization
The branch-and-bound method can be applied to find L2 norm solutions to certain simple problems. This is done by a direct application of branch-andbound over the parameter space. Up to now, methods used for branching have been quite simple, consisting of simple subdivision of rectangles in half, or in half along all dimensions. In order to converge as quickly as possible to the solution, it is useful to confine the region of parameter space that needs to be searched. This can be conveniently done if the cost function is a sum of quasi-convex functions. For instance, suppose the cost is C2 (X) = i fi (X)2 , and the optimal value is denoted by Xopt . If a good initial estimate X0 is available, with C2 (X0 ) = δ 2 , then C2 (Xopt ) = fi (Xopt )2 ≤ C2 (X0 ) = δ 2 . i
This implies that each fi (Xopt ) ≤ δ, and so Xopt ∈ i Sδ (fi ), which is a convex set enclosing both X0 and Xopt . One can find a rectangle in parameter space that
Optimal Algorithms in Multiview Geometry
27
encloses this convex region, and begin the branch-and-bound algorithm starting with this rectangle. This general method was used in [1] to solve the L2 multiview triangulation problem and the uncalibrated camera resection (pose) problem. In that paper fractional programming (described later in section 8.2) was used to define the convex sub-envelope of the cost function, and hence provide a cost lower bound on each rectangle. The same branch-and-bound idea, but with a different bounding method, was described in [29] to provide a simpler solution to the triangulation problem. The triangulation problem for other geometric features, more specifically lines and conics, was solved in [17] using the same approach.
7 7.1
Branch-and-Bound in Rotation Space Essential Matrix
All the problems that we have so-far considered, solvable using quasi-convex optimization or SOCP, involved no rotations. If there are rotations, then the optimization problem is no longer quasi-convex. An example of this type of problem is estimation of the essential matrix from a set of matching points xi ↔ xi . A linear solution to this problem was given in 1981 by Longuet-Higgins [28]. From the Essential matrix E, that satisfies the defining equation xi Exi = 0 for all i, we can extract the relative orientation (rotation and translation) of the two cameras. To understand why this is not a quasi-convex optimization problem, we look at the minimal problem involving 5 point correspondences. It is well know that with 5 points, there are 10 solutions for the relative orientation. (Recent algorithms for solving the 5-point orientation are given in [31,26]). However, if there are many possible discrete solutions, then the problem can not be quasi-convex or convex – such problems have a unique solution. This is so whether we are seeking an L∞ or L2 solution, since the 5-point problem has an exact solution, and hence the cost is zero in either norm. Many algorithms for estimating the essential matrix have been given, without however any claim to optimality. Recently, an optimal solution, at least in L∞ norm was given in [10]. To solve the essential matrix problem optimally (at least in L∞ norm) we make the following observation. If the rotation of the two cameras were somehow known, then the problem would reduce to the one discussed in Fig 6, where the translation of the cameras can be estimated optimally (in L∞ norm) given the rotations. The residual cost of this optimal solution may be found, as a function of the assumed rotation. To solve for the relative pose (rotation and translation) of the two cameras, we merely have to consider all possible rotations, and select the one that yields the smallest residual. The trick is to do this without having to look at an infinite number of rotations. Fortunately, branch-and-bound provides a means of carrying out this search. A key to the success of branch-and-bound is that the optimal cost (residual) estimated for one value of the rotation constrains the optimal cost for “nearby”
28
R. Hartley and F. Kahl
rotations. This allows us to put a lower bound on the optimal cost associated with all rotations in a region of rotation space. The branch-and-bound algorithm carries out a search of rotation space (a 3-dimensional space). The translation is not included in the branch-and-bound search. Instead, for a given rotation, the optimal translation may be computed using SOCP, and hence factored out of the parameter-space search. A similar method of search over rotation space is used in [10] to solve the calibrated camera pose problem. This is the problem of finding the position and orientation of a camera given known 3D points and their corresponding image points. An earlier iterative algorithm that addresses this problem using L∞ norm is given in [47]. The algorithm appears to converge to the global optimum, but the author states that this is unproven. 7.2
General Structure and Motion
The method outlined in section 7.1 for solving the structure and motion problem for two views could in principle be extended to give an optimal solution (albeit in L∞ norm) to the complete structure and motion problem for any number of views. This would involve a search over the combined space of all rotations. For the two camera problem there is only one relative rotation, and hence the branchand-bound algorithm involves a search over a 3-dimensional parameter space. In the case of n cameras, however the parameter space is 3(n − 1)-dimensional. Had we but world enough and time a branch-and-bound search over the combined rotation parameter space would yield an optimal L∞ solution to structure and motion with any number of cameras and points. Unfortunately, in terms of space and time requirements, this algorithm would be vaster than empires and more slow[30]. 7.3
1D Retinal Vision
One-dimensional cameras have proven useful in several different applications, most prominently for autonomous guided vehicles (see Fig 8), but also in ordinary vision for analysing planar motion and the projection of lines. Previous results on one-dimensional vision are limited to classifying and solving of minimal cases, bundle adjustment for finding local minima to the structure and motion problem and linear algorithms based on algebraic cost functions. A method for finding the global minimum to the structure and motion problem using the max norm of reprojection errors is given in [2]. In contrast to the 2Dcase which uses SOCP, the optimal solution can be computed efficiently using simple linear programming techniques. It is assumed that neither the positions of the objects nor the positions and orientations of the cameras are known. However, it is assumed that the correspondence problem is solved, i.e., it is known which measured bearings correspond to the same object. The problem can formally be stated as follows. Given n bearings from m different positions, find the camera positions and 2D points
Optimal Algorithms in Multiview Geometry
29
2
reflector
1.5
1
0.5
0
−0.5 −2
−1.5
−1
−0.5
0
0.5
1
1.5
2
2.5
Fig. 8. Left: A laser guided vehicle. Middle: A laser scanner or angle meter. Right: Calculated structure and motion for the icehockey experiment.
in the plane, such that the reprojected solution has minimal residual errors. The norm for measuring the errors will be the L∞ norm. The basic idea of the optimization scheme is to first consider optimization with fixed camera orientations - which is a quasi-convex problem - and then use branch-and-bound over the space of possible orientations, similar to that of section 7.1. Hockey rink data. By combining optimal structure and motion with optimal resection and intersection it is possible to solve for arbitrarily many cameras and views. We illustrate this with the data from a real set of measurements performed at an ice-hockey rink. The set contains 70 images of 14 points. The result is shown in the right of Fig 8. 7.4
3D – 3D Alignment
A similar method of doing branch-and-bound in rotation space was used in [27] to find an optimal solution for the problem of aligning two sets of points in 3D with unknown pairing. The solution consists of a specified pairing of the two point sets, along with a rotation and translation to align the paired points. The algorithm relies on the fact that if the rotation is known, then the optimal pairing can be computed directly using the Hungarian algorithm [36]. This enables the problem to be addressed by a branch-and-bound search over rotations. The problem is solved for L2 norm in [27] using a bounding method based on Lipschitz bounds. Though the L∞ problem is not specifically addressed in the paper, it would probably also yield to a similar approach. 7.5
An Open Problem: Optimal Essential Matrix Estimation in L2 Norm
The question naturally arises of whether we can use similar techniques to section 7.1 to obtain the optimal L2 solution for the essential matrix. At present, we have no solution to this problem. Two essential steps are missing.
30
R. Hartley and F. Kahl
1. If the relative rotation R between the two cameras is given, can we estimate the optimal translation t. This is simple in L∞ norm using SOCP, or in fact linear programming. In L2 norm, a solution has been proposed in [15], but still it is iterative, and it is not clear that it is guaranteed to converge. For n points, this seems to be a harder problem than optimal n-view L2 triangulation (for which solutions have been recently given [1,29]). 2. If we can find the optimal residual for a given rotation, how does this constrain the solution for nearby rotations? Loose bounds may be given, but they may not be sufficiently tight to allow for efficient convergence of the branch-and-bound algorithm.
8 8.1
Other Methods for Minimizing L2 Norm Convex Relaxations and Semidefinite Programming
Another general approach for solving L2 problems was introduced in [19] based on convex relaxations (underestimators) and semidefinite programming. More specifically, the approach is based on a hierarchy of convex relaxations to solve non-convex optimization problems. Linear matrix inequalities (LMIs) are used to construct the convex relaxations. These relaxations generate a monotone sequence of lower bounds of the minimal value of the objective function and it is shown how one can detect whether the global optimum is attained at a given relaxation. Standard semidefinite programming software (like SeDuMi [44]) is extensively used for computing the bounds. The technique is applied to a number of classical vision problems: triangulation, camera pose, homography estimation and epipolar geometry estimation. Although good results are obtained, there is no guarantee of achieving the optimal solution and the sizes of the problem instances are small. 8.2
Fractional Programming
Yet another method was introduced in [1,35]. It was the first method to solve the n-view L2 triangulation problem with a guarantee of optimality. Other problem applications include camera resectioning (that is, uncalibrated camera pose), camera pose estimation and, homography estimation. In its most general form, fractional programming seeks to minimize/maximize the sum of fractions subject to convex constraints. Our interest from the point of view of multiview geometry, however, is specific to the minimization problem min x
p fi (x) i=1
gi (x)
subject to x ∈ D
where fi : Rn → R and gi : Rn → R are convex and concave functions, respectively, and the domain D ⊂ Rn is a convex compact set. This is because the residual of the projection of a 3D point into an image may be written in this form. Further, it is assumed that both fi and gi are positive with lower
Optimal Algorithms in Multiview Geometry
31
and upper bounds over D. Even with these restrictions the above problem is N P -complete [8], but practical and reliable estimation of the global optimum is still possible for many multiview problems through an iterative algorithm that solve an appropriate convex optimization problem at each step. The procedure is based on branch and bound. Perhaps the most important observation made in [1] is that many multiview geometry problems can be formulated as a sum of fractions where each fraction consists of a convex over a concave function. This has inspired a new more efficient ways of computing the L2 -minimum for n-view triangulation, see [29].
9
Applications
There have been various application papers that have involved this type of optimization methodology, though they can not be said to have found an optimal solution to the respective problems. In [38] SOCP has been used to solve the problem of tracking and modelling a deforming surface (such as a sheet of paper) from a single view. Results are shown in Fig 9.
Fig. 9. Modelling a deforming surface from a single view. Left: the input image, with automatically overlaid grid. Right: the computed surface model viewed from a new viewpoint. Image features provide cone constraints that constrain the corresponding 3D points to lie on or near the corresponding line-of-sight, namely a ray through the camera centre. Additional convex constraints on the geometry of the surface allow the shape to be determined unambiguously using SOCP.
In another application-inspired problem, SOCP has been applied (see [22]) to the odometry problem for a vehicle with several rigidly mounted cameras with almost non-overlapping fields of view. Although the algorithm in [22] is tested on laboratory data, it is motivated by its potential use with vehicles such as the one shown in Fig 10. Such vehicles are used for urban-mapping. Computation of individual essential matrices for each of the cameras reduces the computation of the translation of the vehicle to a multiple view triangulation problem, which is solved using SOCP.
32
R. Hartley and F. Kahl
Fig. 10. Camera and car mount used for urban mapping. Images from non-overlapping cameras on both sides of the car can be used to do odometry of the vehicle. An algorithm based on triangulation and SOCP is proposed in [22]. (The image is used with permission of the UNC Vision group).
10
Concluding Remarks
The application of new optimization methods to the problems of Multiview Geometry have led to the development of reliably and provably optimal solutions under different geometrically meaningful cost functions. At present these algorithms are not as fast as standard methods, such as bundle adjustment. Nevertheless the run times are not wildly impractical. Recent work on speeding up the optimization process is yielding much faster run times, and further progress is likely. Such optimization techniques are also being investigated in other areas of Computer Vision, such as discrete optimization. A representative paper is [24]. For 15 years or more, geometric computer vision has relied on a small repertoire of optimization methods, with Levenberg-Marquardt [37] being the most popular. The benefit of using new methods such as SOCP and other convex and quasi-convex optimization methods is being realised.
References 1. Agarwal, S., Chandraker, M.K., Kahl, F., Kriegman, D.J., Belongie, S.: Practical global optimization for multiview geometry. In: European Conf. Computer Vision, Graz, Austria, pp. 592–605 (2006) 2. ˚ Astr¨ om, K., Enqvist, O., Olsson, C., Kahl, F., Hartley, R.: An L∞ approach to structure and motion problems in 1d-vision. In: Int.Conf. Computer Vision, Rio de Janeiro, Brazil (2007) 3. Boyd, S., Vanderberghe, L.: Convex Optimization. Cambridge University Press, Cambridge (2004) 4. Byr¨ od, M., Josephson, K., ˚ Astr¨ om, K.: Improving numerical accuracy in gr¨ obner basis polynomial equation solvers. In: Int. Conf.Computer Vision, Rio de Janeiro, Brazil (2007) 5. Chandraker, M.K., Agarwal, S., Kriegman, D.J., Belongie, S.: Globally convergent algorithms for affine and metric upgrades in stratified autocalibration. In: Int. Conf. Computer Vision, Rio de Janeiro, Brazil (2007)
Optimal Algorithms in Multiview Geometry
33
6. Farenzena, M., Fusiello, A., Dovier, A.: Reconstruction with interval constraints propagation. In: Proc. Conf. Computer Vision and Pattern Recognition, New York City, USA, pp. 1185–1190 (2006) 7. Faugeras, O.D., Maybank, S.J.: Motion from point matches: Multiplicity of solutions. Int. Journal Computer Vision 4, 225–246 (1990) 8. Freund, R.W., Jarre, F.: Solving the sum-of-ratios problem by an interior-point method. J. Glob. Opt. 19(1), 83–102 (2001) 9. Hartley, R., de Agapito, L., Hayman, E., Reid, I.: Camera calibration and the search for infinity. In: Proc. 7th International Conference on Computer Vision, Kerkyra, Greece, September 1999, pp. 510–517 (1999) 10. Hartley, R., Kahl, F.: Global optimization through searching rotation space and optimal estimation of the essential matrix. Int. Conf. Computer Vision (2007) 11. Hartley, R., Schaffalitzky, F.: L∞ minimization in geometric reconstruction problems. In: Conf. Computer Vision and Pattern Recognition, Washington DC, USA, vol. I, pp. 504–509 (2004) 12. Hartley, R., Sturm, P.: Triangulation. Computer Vision and Image Understanding 68(2), 146–157 (1997) 13. Hartley, R.I., Zisserman, A.: Multiple View Geometry in Computer Vision, 2nd edn. Cambridge University Press, Cambridge (2004) 14. Horn, B.K.P.: Closed form solution of absolute orientation using unit quaternions. J. Opt. Soc. America 4(4), 629–642 (1987) 15. Horn, B.K.P.: Relative orientation. Int. Journal Computer Vision 4, 59–78 (1990) 16. Horn, B.K.P.: Relative orientation revisited. J. Opt. Soc. America 8(10), 1630–1638 (1991) 17. Josephson, K., Kahl, F.: Triangulation of points, lines and conics. In: Scandinavian Conf. on Image Analysis, Aalborg, Denmark (2007) 18. Kahl, F.: Multiple view geometry and the L∞ -norm. In: Int. Conf. Computer Vision, Beijing, China, pp. 1002–1009 (2005) 19. Kahl, F., Henrion, D.: Globally optimal estimates for geometric reconstruction problems. Int. Journal Computer Vision 74(1), 3–15 (2007) 20. Ke, Q., Kanade, T.: Quasiconvex optimization for robust geometric reconstruction. In: Int. Conf. Computer Vision, Beijing, China, pp. 986–993 (2005) 21. Ke, Q., Kanade, T.: Uncertainty models in quasiconvex optimization for geometric reconstruction. In: Conf. Computer Vision and Pattern Recognition, New York City, USA, pp. 1199–1205 (2006) 22. Kim, J.H., Hartley, R., Frahm, J.M., Pollefeys, M.: Visual odometry for nonoverlapping views using second-order cone programming. In: Asian Conf. Computer Vision (November 2007) 23. Koenderink, J.J., van Doorn, A.J.: Affine structure from motion. J. Opt. Soc. America 8(2), 377–385 (1991) 24. Kumar, P., Torr, P.H.S., Zisserman, A.: Solving markov random fields using second order cone programming relaxations. In: Conf. Computer Vision and Pattern Recognition, pp. 1045–1052 (2006) 25. Li, H.: A practical algorithm for L-infinity triangulation with outliers. In: CVPR, vol. 1, pp. 1–8. IEEE Computer Society, Los Alamitos (2007) 26. Li, H., Hartley, R.: Five-point motion estimation made easy. In: Int. Conf. Pattern Recognition, pp. 630–633 (August 2006) 27. Li, H., Hartley, R.: The 3D – 3D registration problem revisited. In: Int. Conf. Computer Vision (October 2007) 28. Longuet-Higgins, H.C.: A computer algorithm for reconstructing a scene from two projections. Nature 293, 133–135 (1981)
34
R. Hartley and F. Kahl
29. Lu, F., Hartley, R.: A fast optimal algorithm for l2 triangulation. In: Asian Conf. Computer Vision (November 2007) 30. Marvell, A.: To his coy mistress. circa (1650) 31. Nist´er, D.: An efficient solution to the five-point relative pose problem. IEEE Trans. Pattern Analysis and Machine Intelligence 26(6), 756–770 (2004) 32. Nist´er, D., Hartley, R., Stew´enius, H.: Using Galois theory to prove that structure from motion algorithms are optimal. In: Conf. Computer Vision and Pattern Recognition (June 2007) 33. Nist´er, D., Kahl, F., Stew´enius, H.: Structure from motion with missing data is N P -hard. In: Int. Conf. Computer Vision, Rio de Janeiro, Brazil (2007) 34. Olsson, C., Eriksson, A., Kahl, F.: Efficient optimization of L∞ -problems using pseudoconvexity. In: Int. Conf. Computer Vision, Rio de Janeiro, Brazil (2007) 35. Olsson, C., Kahl, F., Oskarsson, M.: Optimal estimation of perspective camera pose. In: Int. Conf. Pattern Recognition, Hong Kong, China, vol. II, pp. 5–8 (2006) 36. Papadimitriou, C., Steiglitz, K.: Combinatorial Optimization: Algorithms and Complexity. Prentice-Hall, Englewood Cliffs (1982) 37. Press, W., Flannery, B., Teukolsky, S., Vetterling, W.: Numerical Recipes in C. Cambridge University Press, Cambridge (1988) 38. Salzman, M., Hartley, R., Fua, P.: Convex optimization for deformable surface 3D tracking. In: Int. Conf. Computer Vision (October 2007) 39. Seo, Y., Hartley, R.: A fast method to minimize L∞ error norm for geometric vision problems. In: Int. Conf. Computer Vision (October 2007) 40. Seo, Y., Hartley, R.: Sequential L∞ norm minimization for triangulation. In: Asian Conf. Computer Vision (November 2007) 41. Sim, K., Hartley, R.: Recovering camera motion using the L∞ -norm. In: Conf. Computer Vision and Pattern Recognition, New York City, USA, pp. 1230–1237 (2006) 42. Sim, K., Hartley, R.: Removing outliers using the L∞ -norm. In: Conf. Computer Vision and Pattern Recognition, New York City, USA, pp. 485–492 (2006) 43. Stew´enius, H., Schaffalitzky, F., Nist´er, D.: How hard is three-view triangulation really? In: Int. Conf. Computer Vision, Beijing, China, pp. 686–693 (2005) 44. Sturm, J.F.: Using SeDuMi 1.02, a Matlab toolbox for optimization over symmetric cones. Optimization Methods and Software 11(12), 625–653 (1999) 45. Tomasi, C., Kanade, T.: Shape and motion from image streams under orthography: A factorization approach. Int. Journal Computer Vision 9(2), 137–154 (1992) 46. Triggs, W., McLauchlan, P.F., Hartley, R.I., Fitzgibbon, A.: Bundle adjustment for structure from motion. In: Vision Algorithms: Theory and Practice, pp. 298–372. Springer, Heidelberg (2000) 47. Zhang, X.: Pose estimation using L∞ . In: Image and Vision Computing New Zealand (2005)
Machine Vision in Early Days: Japan’s Pioneering Contributions Masakazu Ejiri R & D Consultant in Industrial Science, formerly at Central Research Laboratory, Hitachi, Ltd.
Abstract. The history of machine vision started in the mid-1960s by the efforts of Japanese industry researchers. A variety of prominent visionbased systems was made possible by creating and evolving real-time image processing techniques, and was applied to factory automation, office automation, and even to social automation during the 1970-2000 period. In this article, these historical attempts are briefly explained to promote understanding of the pioneering efforts that opened the door and formed the bases of today’s computer vision research. Keywords: Factory automation, office automation, social automation, real-time image processing, video image analysis, robotics, assembly, inspection.
1
Introduction
There is an old saying, “knowing the old brings you a new wisdom for tomorrow,” that originated with Confucius (a Chinese philosopher, 551BC-479BC). This is the basic idea underlying this article, and its purpose is to enlighten young researchers on old technologies rather than new ones. In the 1960s, one of the main concerns of researchers in the field of information science was the realization of intelligence by using a conventional computer, which had been used mainly for numerical computing. At that time, a hand-eye system was thought to be an excellent research tool to visualize intelligence and demonstrate its behavior. The hand-eye system was, by itself, soon recognized as an important research target, and it became known as the “intelligent robot.” One of the core technologies of the intelligent robot was, of course, vision, and people started to call this vision research area “computer vision.” However, the academic research on computer vision was apt to be stagnant. Its achievements stayed at the level of simulated tasks and could not surpass our expectations because of its intrinsic difficulty and the limitation of computing power in those days. On the other hand, practical vision technology was eagerly anticipated in industry, particularly in Japan, as one of the core technologies towards attaining flexible factory automation. Research was initiated in the mid-1960s, typically at our group of Hitachi’s Central Research Laboratory. In contrast to the word “computer vision,” we used the word “machine vision” for representing a more Y. Yagi et al. (Eds.): ACCV 2007, Part I, LNCS 4843, pp. 35–53, 2007. c Springer-Verlag Berlin Heidelberg 2007
36
M. Ejiri
pragmatic approach towards “useful” vision systems, because the use of computers was less essential in the pragmatic approach. Most of the leading companies followed this lead, and they all played an important role in the incubation and development of machine vision technology. Currently, computer vision is regarded as a fundamental and scientific approach to investigate the principles that underlie vision and how artificial vision can be best achieved, in contrast to the more pragmatic, needs-oriented machine vision approach. We have to note, however, that there is no difference in the ultimate goals of these two approaches. Though the road to machine vision was not smooth, Japanese companies fortunately achieved some key successes. In this article, we briefly introduce these pioneering vision applications and discuss the history of machine vision.
2
Prehistoric Machine Vision
Our first attempt at vision application, in 1964, was to automate the assembly process (i.e., wire-bonding process) of transistors. In this attempt, we used a very primitive optical sensor by combining a microscope and a rotating-drum type scanner with two slits on its surface. By detecting the reflection from the transistor surface with photo-multipliers, the position and orientation of transistor chips were determined with about a 95% success rate. However, this percentage was still too low to enable us to replace human workers; thus, our first attempt failed and was eventually abandoned after a two-year struggle. What we learned from this experience was the need for reliable artificial vision comparable to a human’s pattern recognition capability, which quickly captures the image first, and then reduces the information quantity drastically until the positional information is firmly determined. Our slit-type optical-scanning method inherently lacked the right quantity of captured information; thus, the recognition result was apt to be easily affected by reflective noises. In those days, microprocessors had not yet been developed and the available computers were still too expensive, bulky, and slow, particularly for image processing and pattern recognition. Moreover, memory chips were extremely costly, so the use of full-frame image memories was prohibitive. Though there was no indication that these processors would soon improve, we started seminal research on flexible machines in 1968. A generic intelligent machine conceived at that time was the one that consisted of three basic functions: understanding of a human’s instruction or intention to clarify the goal of the task, understanding of objects or environment to clarify the start of the task, and decision-making to find the optimum route between the start and the goal. Based on this conception, a prototype intelligent robot was developed in 1970. The configuration of this intelligent robot is shown in Fig.1. In this robot, the image of a trihedral plan drawing was captured by one of the cameras and was analyzed to clarify the goal assemblage as well as the components of the assemblage. Another camera looked at the real blocks scattered on a table, and found their positions and postures. From these blocks, the
Machine Vision in Early Days: Japan’s Pioneering Contributions
37
Fig. 1. Intelligent robot (1970)
computer recognized the component blocks needed to complete this assembly task, and made a plan to assemble these blocks. For this assembly planning, backward reasoning was used to find the route from the goal to the start, not from the start to goal. That is, the route was found by analyzing the disassembly task from the goal assemblage to each component. The assembly sequence was determined as the reverse of this disassembly sequence. Thus, the robot could assemble blocks into various forms by responding to the objectives presented macroscopically by an assembly drawing [2]. This research formed part of Japan’s national project on PIPS (Pattern Information Processing System), initiated in the following year.
3
Factory Applications
Our project on the prototype intelligent robot in 1970 revealed many basic problems underlying “flexible machines” and gave us useful insights into future applications of robotics. One significant problem we were confronted with was the robot’s extremely slow image-processing speed in understanding objects. Our next effort was therefore focused on developing high-speed dedicated hardware for image processing with the minimum use of memory, instead of using rather slow and expensive computers. One of the core ideas was to adaptively threshold the image signal into a binary form by responding to the signal behavior and to input it into a shift-register-type local memory that dynamically stored the latest pixel data of several horizontal scan-lines. This local-paralleltype configuration enabled us to simultaneously extract plural pixel data from a 2-D local area in synchronization with image scanning. By designing the logic circuit connected to this 2-D local area according to the envisaged purpose, the processing hardware could be adapted to many applications. One useful yet simple method using this local-parallel-type image processing was windowing. This method involved setting up a few particular window areas
38
M. Ejiri
Fig. 2. Bolting robot for piles/poles (1972)
in the image plane, and the pixels in the windows were selectively counted to find the background area and the object area occupying the windows. In 1972 a bolting robot applying windowing was developed in order to automate the molding process of concrete piles and poles [3]. It became the first application of machine vision to moving objects. Note that this paper [3] was published at a later time. This was likely to happen because publication was not the first priority for industry researchers. Another effective method based on local-parallel architecture was erosion/ dilation of patterns, which was executed by simple AND/OR logic on a 2-D local area. This method could detect defects in printed circuit boards (PCBs), and formed one of the bases of today’s morphological processing. This defectdetection machine in 1972 also became the first application of machine vision to the automation of visual inspection [4]. These two pioneering applications are illustrated in Figs. 2 and 3. Encouraged by the effectiveness of these machine-vision systems in actual production lines, we again started to develop a new assembly machine for transistors that was, this time, based fully on image processing. A multiple local pattern matching method was extensively studied for this purpose. In this method, each local pattern position in a transistor image was found by matching to a standard pattern. The distance and the angle between a pair of detected local pattern positions were sequentially checked to see if these local patterns were correctly detected. The electrode positions for wiring were then calculated from the coordinates of the first detected correct pair. By basing a local-parallel-type image processor on this matching, we finally developed fully automatic transistor assembly machines in 1973 [5]. This successful development was the result of a ten-year effort since our first failed attempt. The developed assembly machines were recognized as the world’s first image-based machines for fully automatic assembly of semiconductor devices. These machines and their configuration are shown in Fig. 4.
Machine Vision in Early Days: Japan’s Pioneering Contributions
39
Fig. 3. PCB inspection machine (1972)
Fig. 4. Transistor assembly machine (1973)
After this development, our efforts were then focused on expanding machinevision applications from transistors to other semiconductor devices, such as integrated circuits (ICs), hybrid ICs, and large-scale integrated circuits (LSIs). Consequently, the automatic assembly of all types of semiconductor devices was completed by 1977. With the export of this automatic assembly technology to a US manufacturer as a start, the technology gained widespread attention from semiconductor manufacturers worldwide and expanded quickly into industry. As a result, the semiconductor industry as a whole prospered by virtue of higher speed production of higher quality products with more uniform performance than had been achieved previously. Encouraged by the success of semiconductor assembly, our efforts were further broadened to other industrial applications in the mid-1970s to early 1980s. Examples of such applications during this period are a hose-connecting robot
40
M. Ejiri
Fig. 5. Wafer inspection machines (1984-1987)
for pressure testing in pump production lines, a reading machine of 2-D objectcode for an intra-factory physical distribution system, and a quality inspection machine for marks and characters printed on electronic parts [6][7][8]. Machines for inspecting photo-masks in semiconductor fabrication and CRT black-matrix fabrication were other examples [9] (by Toshiba) and [10] (by Hitachi). Machines for classifying medical tablets and capsules [11] (by Fuji Electric) and machines for classifying agricultural products and fish [12][13][14] (by Mitsubishi Electric) were also unique and epoch-making achievements in those days. These examples show that the key concept representing those years seemed to be the realization of a “productive society” through factory automation, and the objectives of machine vision were mainly position detection for assembly, shape detection for classification, and defect detection for inspection. In 1980, the PIPS project finished after a 10-year effort by Japanese industry. A variety of recognition systems were successfully prototyped for hand-written Kanji characters, graphics, drawings, documents, color pictures, speeches, and three-dimensional objects. One particular outcome among others was the development of high-speed, general-purpose image processors [15], which in turn served as the basis of subsequent research and development in industry. The most difficult but rewarding development in the mid-1980s was an inspection machine for detecting defects in semiconductor wafers [16]. It was estimated that even the world’s largest super-computer available at that time would require at least one month of computing for finishing the defect detection of an 8-inch single wafer. We therefore had to develop special hardware for lowering the processing time to less than 1 hour/wafer. The resulting hardware was a network of local-parallel-type image processors that use a “design pattern referring method,” shown in Fig. 5. In this machine, hardware-based knowledge processing, in which each processor was regarded as a combination of IF-part and THEN-part logical circuits, was first attempted [17].
Machine Vision in Early Days: Japan’s Pioneering Contributions
41
Meanwhile, the processing speed of microprocessors improved considerably since their appearance in the mid-1970s, and the memory capacity drastically increased without excessively increasing costs. These improvements facilitated the use of gray-scale images instead of binary ones, and dedicated LSI chips for image processing were developed in the mid-1980s [18]. These developments all contributed to achieving more reliable, microprocessor-based general-purpose machine vision systems with full-scale buffers for gray-level images. As a result, applications of machine vision soon expanded from circuit components, such as semiconductors and PCBs, to home-use electronic equipment, such as VCRs and color TVs. Currently, machine vision systems are found in various areas such as electronics, machinery, medicine, and food industries.
4
Office Applications
Besides the above-described machine-vision systems for factory applications, there was extensive research on character recognition in the area of office automation. For example, in the mid-1960s, a FORTRAN program reader was developed to replace key-punching tasks. Mail-sorting machines for post offices were developed in the late 1960s to automatically read handwritten postal codes (by Toshiba et al.). Another topical developmental effort started in 1974 for automatic classification of fingerprint patterns, and in 1982 the system was first put in use at a Japanese police office with great success, and later at US police offices (by NEC). Our first effort to apply machine-vision technology to areas other than factory automation was the automatic recognition of monetary bills in 1976. This recognition system was extremely successful in spurring the development of automated teller machines (ATMs) for banks (see Fig. 6). Due to the processing time limitation, the entire image of a bill was not captured, but by combining several partial images obtained from optical sensors with those from magnetic sensors, so-called sensor-fusion was first attempted and thus resulted in highaccuracy bill recognition with a less than 1/1015 theoretical error rate. Early ATM models for domestic use employed vertical safes, but in the later models, the horizontal safes were extensively used for increasing spatial efficiency and for facilitating use in Asian countries having a larger number of bill types. Our next attempt, in the early 1980s, was the efficient handling of a large amount of graphic data in the office [19]. The automatic digitization of paperbased engineering drawings and maps was first studied. The recognition of these drawings and maps was based on a vector representation technique, such as that shown in Fig. 7. The recognition was usually executed by spatially-paralleltype image processors, in which each processor was designated to a specific image area. Currently, geographic information systems (GIS) based on these digital maps gained popularity and are being used by many service companies and local governments to manage their electric power supply, gas supply, water supply facilities, and sewage service facilities (see Fig. 8). The use of digital maps was then extended to car navigation systems and more recently to various
42
M. Ejiri
Fig. 6. ATM: automated teller machines (1976-1995)
Fig. 7. Automatic digitizer for maps and engineering drawings (1982)
other information service systems via the Internet. Machine-vision technology contributed, mainly in the early developmental stage of these systems, to the digitization of original paper-based maps into electronic form until these digital maps began to be produced directly from measured data through computer-aided map production. Spatially divided parallel processing was also useful for large-scale images such as those from satellite data. One of our early attempts in this area was the recognition of wind vectors, back in 1972, by comparing two simulated satellite images with a 30-minute interval. This system formed a basis of weather forecasting using Japan’s first meteorological geo-stationary satellite “Himawari,” launched a few years later. Also, an algorithm for deriving a sea temperature contour map from infra-red satellite images was built for environmental study and fisheries.
Machine Vision in Early Days: Japan’s Pioneering Contributions
43
Fig. 8. GIS: geographic information systems(1986)
Research on document understanding also originated as part of machine-vision research in the mid-1980s [20]. During those years, electronic document editing and filing became popular owing to the progress in word-processing technology for over 4000 Kanji and Kana characters. The introduction of an electronic patent-application system in Japan in 1990 was an important stimulus for further research on office automation. We developed dedicated workstations and a parallel-disk-type distributed filing system for the use of patent examiners. This system enabled examiners to efficiently retrieve and display the images of past documents for comparison. The recognition of handwritten postal addresses was one of the most challenging topics in machine-vision applications. In 1992, a decision was made by a government committee (to which the author served as a member) to adopt a new 7-digit postal code system in Japan beginning in 1998. To this end, three companies (Hitachi, Toshiba and NEC) developed new automatic mail-sorting machines for post offices in 1997. An example of the new sorting machines is shown in Fig. 9. In those machines, hand-written/printed addresses in Kanji characters are fully read together with the 7-digit postal codes; both results are then matched for consistency; and the recognized full address is printed on each letter as a transparent barcode consisting of 20-digit data. The letters are then dispatched to other post offices for delivery. In a subsequent process, only these barcodes are read, and prior to home delivery the letters are arranged by the new sorting machine in such a way that the order of the letters corresponds to the house order on the delivery route. In these postal applications, the recognition of all types of printed fonts and hand-written Kanji characters was made possible by using a multi-microprocessor type image processing system. A mail image is sent to one of the unoccupied processors, and this designated processor analyzes the image. The address recognition by a designated single processor usually requires 1.0 to 2.5 seconds, depending on the complexity of the address image. As up to 32 microprocessors are used
44
M. Ejiri
Fig. 9. New mail sorting machine (1997)
in parallel for successively flowing letters, the equivalent recognition time of the whole system is less than 0.1 seconds/letter, producing a maximum processing speed of 50,000 letters per an hour. The office applications of vision technology described above show that the key concept representing those years seemed to be the realization of an “efficient society” through office automation, and the objectives of machine vision were mainly efficient handling of large-scale data and also high-precision, high-speed recognition and handling of paper-based information. Recent progress in network technology has also increased the importance of office automation. To secure the reliability of information and communication systems, a variety of advanced image processing technologies will be expected. These will include more effective and reliable compression, encryption, scrambling, and watermarking technologies of image data.
5
Social Applications
In recent years, applications to social automation have become increasingly important. Social automation here means “making the systems designed for social use more intelligent,” and it includes systems for traffic and for environmental use. The technologies used in these systems are, for example, surveillance, monitoring, flow control and security assurance. The earliest attempt at social automation was probably our elevator-eye project in 1977, in which we tried to implement machine vision in an elevator system in order to control the human traffic in large-scale buildings. The elevator hall on each floor was equipped with a camera to observe the hall, and a vision system to which these cameras were connected surveyed all floors in a time-sharing manner and estimated the number of persons waiting for an elevator. The vision system then designated an elevator cage to quickly serve the crowded floor [21]. The configuration of this system is shown in Fig. 10.
Machine Vision in Early Days: Japan’s Pioneering Contributions
45
Fig. 10. Elevator and other traffic applications (1977-1986)
In this elevator system, a robust change-finding algorithm based on edge vectors was used in order to cope with the change in the brightness of the surroundings. In this algorithm, the image plane was divided into several blocks, and the edge-vector distribution in each block was compared with that of the background image, which was updated automatically by new image data when no motion was observed and thus nobody was in the elevator hall. This system could minimize the average waiting time for the elevator. Though a few systems were put into use in Tokyo area in the early 1980s, there has not been enough market demand to continue to develop the system further. More promising applications of image recognition seemed to be for monitoring road traffic, where license plates, traffic jams, and illegally parked cars were identified so that traffic could be controlled smoothly and parking lots could be automatically allocated [22]. Charging tolls automatically at toll gates without stopping cars, by means of a wireless system with IC card, is now popular on highways as a result of the ITS (Intelligent Transport System) project. The system will be further improved if the machine vision can be effectively combined to quickly recognize other important information such as license plate numbers and even driver’s faces and other identities. A water-purity monitoring system using fish behavior [23] was in operation for at least 10 years at a river control center in a local city in Japan since the river water was accidentally polluted by toxicants. A schematic diagram of the system is shown in Fig. 11. The automatic observation of algae in water in sewage works was also studied. Volcanic lava flow was continuously monitored at the base of Mt. Fugendake in Nagasaki, Japan, during the eruption period in 1993. To optically send images from unmanned remote observation posts to the central control station, laser communication routes were planned by using 3-D undulation data derived from GIS digital contour maps. A GIS was also constructed to assist in restoration after the “Hanshin-Awaji” earthquake in Kobe, Japan, in 1995. Aerial photographs after the earthquake were analyzed by
46
M. Ejiri
Fig. 11. Environmental use (1990-1995)
matching them with digital 3-D urban maps containing additional information on the height of buildings. Buildings with damaged walls and roofs could thus be quickly detected and given top priority for restoration [24]. Intruder detection is also becoming important in the prevention of crimes and in dangerous areas such as those around high-voltage electric equipment. Railroad crossings can also be monitored intensively by comparing the vertical line data in an image with that in a background image updated automatically [25]. Arranging the image differences in this vertical window gives a spatiotemporal image of objects intruding onto the crossing. In almost all of these social applications, real-time color-image processing is becoming increasingly important for reliable detection and recognition. As mentioned before, the application of image processing to communications is increasingly promising as multimedia and network technologies improve. Humanmachine interfaces will be greatly improved if the machine is capable of recognizing every media used by humans. Human-to-human communication assisted by intelligent machines and networks is also expected. Machine vision will contribute to this communication in such fields as motion capturing, face recognition, facial expression understanding, gesture recognition, sign language understanding, and behavior understanding. In addition, applications of machine vision to the field of human welfare, medicine, and environment improvement will become increasingly important in the future. Examples of these applications are rehabilitation equipment, medical surgery assistance, and water purification in lakes. Thus, the key concept representing the future seems to be the realization of a calm society, in which all uneasiness will be relieved through networked social automation, and the important objectives of machine vision will typically be the realization of the following two functions: 24-hour/day “abnormality monitoring” via networks and reliable “personal identification” via networks.
Machine Vision in Early Days: Japan’s Pioneering Contributions
6
47
Key Technologies
In most of the future applications, dynamic image processing will be a key to success. There are various approaches already for analyzing incoming video images in real-time by using smaller-scale personal computers. One typical example is the “Mediachef” system, which automatically cuts video images into a set of scenes by finding significant changes between consecutive image frames [26]. The principle of the system is shown in Fig. 12. This is one of the essential technologies for video indexing and video-digest editing. To date, this technology has been put into use in the video inspection process in a broadcasting company so that subliminal advertising can be detected before the video is on the air.
Fig. 12. Key technologies: “Mediachef” for video indexing and editing (1990)
For the purpose of searching scenes, we developed a real-time video coding technique that uses an average color in each frame and represents its sequence by a “run” between frames. This method can compress 24-hour video signals into a memory capacity of only 2 MB. This video-coding technology can be applied to automatically detect the time of broadcast of a specific TV commercial by continuously monitoring TV signals by means of a compact personal computer. It therefore allows manufacturers to monitor their commercials being broadcast by an advertising company and, thus, provides evidence of a broadcast. The technology called “Cyber BUNRAKU,” in which human facial expressions are monitored by small infrared-sensitive reflectors put on a performer’s face, is also noteworthy. By combining the facial expressions thus obtained with the limb motions of a 19-jointed “Bunraku doll” (used in traditional Japanese theatrical performance), a 3-D character model in a computer can be animated in real-time to create video images [27], as shown in Fig. 13. This technology can create TV animation programs much faster than through traditional handdrawing methods. Another example of dynamic video analysis is “Tour into the picture (TIP)” technology. As shown in Fig. 14, a 2-D picture is scanned and interpreted into
48
M. Ejiri
Fig. 13. Key technologies: Cyber BUNRAKU(1996)
Fig. 14. Key technologies: Tour into the picture (1997)
three-dimensional data by manually fitting vanishing lines on the displayed picture. The picture can then be looked at from different angles and distances [28]. A motion video can thus be generated from a single picture and viewers can feel as if they were taking a walk in an ancient city when an old picture of the city is available. A real-time creation of panoramic pictures is also an important application of video-image processing [29]. A time series of each image frame from a video camera during panning and tilting is spatially connected in real-time into a single still picture (i.e. image mosaicing), as shown in Fig. 15. Similarly, by connecting all the image frames obtained during the zooming process, a high-resolution picture (having higher resolution in the inner areas) can be obtained as shown in Fig. 16.
Machine Vision in Early Days: Japan’s Pioneering Contributions
49
Fig. 15. Key technologies: Panoramic view by panning and tilting(1998)
Fig. 16. Key technologies: Panoramic view by zooming (1999)
As mentioned already, one important application of machine vision is personal identification in social use. Along these lines, there have been a few promising developments. These include the personal identification systems by means of fingerprint patterns [30] (by NEC, 1996), of iris patterns (by Oki Electric, 1997), of finger vein patterns (by Hitachi, 2000, see Fig. 17) and of palm vein patterns (by Fujitsu, 2002). These are now finding the wide use in security systems, including the application to automated teller machines (ATMs). We have given a few examples of real-time image processing technologies, which will be key technologies applicable to wide variety of systems in the future. The most difficult technical problem that social automation is likely to face, however, is how to make robust machine-vision systems that can be used day or night in all types of weather conditions. To cope with the wide changes in illumination, the development of a variable-sensitivity imaging device with a wide dynamic range is still a stimulating challenge. Artificial retina chips [31] (by Mitsubishi, 1998) and high-speed vision chips (by Fujitsu and University of Tokyo, 1999) are expected to play an important role along these lines.
50
M. Ejiri
Fig. 17. Key technologies: Biometrics based on finger vein patterns (2000)
7
Summaries
The history of machine vision and its applications in Japan was briefly reviewed by focusing on the efforts in industry, and is summarized roughly in Fig. 18 as a chronological form.
Fig. 18. History of machine vision research
Details of topics of industrial activities are listed on a year-to-year basis in Table 1, together with the various topics in other related fields for easier understanding of each period. The history is also summarized in Table. 2 as a list form characterizing each developmental stage. As indicated, we can see that, in addition to factory automation, office automation and social automation have been greatly advanced in those years by the evolution of machine-vision technology, owing to the progress of processor and
Machine Vision in Early Days: Japan’s Pioneering Contributions
51
Table 1. History of machine vision research (1961-2000)
memory technologies. However, it is also a fact that one of the most prominent contributions of machine vision technology was in the production of semiconductors. The semiconductor industry, and thus our human life, would not have been able to enjoy prosperity without machine vision technology. This article was prepared from the view point of the old saying; “knowing the old brings you a new wisdom for tomorrow” by Confucius. The author will be extremely pleased if this article is read widely by young researchers, as it would give them some insights to this field, and would encourage them to get into, and play a great role in, this seemingly simple but actually difficult research field.
52
M. Ejiri Table 2. History of machine vision research
The closing message from the author to young researchers is as follows: Lift your sights, raise your spirits, and get out into the world!
References 1. Ejiri, M., Miyatake, T., Sako, H., Nagasaka, A., Nagaya, S.: Evolution of realtime image processing in practical applications. In: Proc. IAPR MVA, Tokyo, pp. 177–186 (2000) 2. Ejiri, M., Uno, T., Yoda, H., Goto, T., Takeyasu, K.: A prototype intelligent robot that assembles objects from plan drawings. IEEE Trans. Comput. C-21(2), 161–170 (1972) 3. Uno, T., Ejiri, M., Tokunaga, T.: A method of real-time recognition of moving objects and its application. Pattern Recognition 8, 201–208 (1976) 4. Ejiri, M., Uno, T., Mese, M., Ikeda, S.: A process for detecting defects in complicated patterns, Comp. Graphics & Image Processing 2, 326–339 (1973) 5. Kashioka, S., Ejiri, M., Sakamoto, Y.: A transistor wire-bonding system utilizing multiple local pattern matching techniques. IEEE Trans. Syst. Man & Cybern. SMC-6(8), 562–570 (1976) 6. Ejiri, M.: Machine vision: A practical technology for advanced image processing. Gordon & Breach Sci. Pub, New York (1989) 7. Ejiri, M.: Recent image processing applications in industry. In: Proc. 9th SCIA, Uppsala, pp. 1–13 (1995) 8. Ejiri, M.: A key technology for flexible automation. In: Proc. of Japan-U.S.A. Symposium on Flexible Automation, Otsu, Japan, pp. 437–442 (1998) 9. Goto, N.: Toshiba Review 33, 6 (1978) (in Japanese) 10. Hara, Y., et al.: Automatic visual inspection of LSI photomasks. In: Proc. 5th ICPR (1980) 11. Haga, K., Nakamura, K., Sano, Y., Miyamori, N., Komuro, A.: Fuji Jiho 52(5), pp.294–298 (1979) (in Japanese) 12. Nakahara, S., Maeda, A., Nomura, Y.: Denshi Tokyo. IEEE Tokyo Section 18, 46–48 (1979)
Machine Vision in Early Days: Japan’s Pioneering Contributions
53
13. Nomura, Y., Ito, S., Naemura, M.: Mitsubishi Denki Giho 53(12), 899–903 (1979) (in Japanese) 14. Maeda, A., Shibayama, J.: Pattern measurement, ITEJ Technical Report, 3, 32 (in Japanese) (1980) 15. Mori, K., Kidode, M., Shinoda, H., Asada, H.: Design of local parallel pattern processor for image processing. In: AFIP Conf. Proc., vol. 47, pp. 1025–1031 (1978) 16. Yoda, H., Ohuchi, Y., Taniguchi, Y., Ejiri, M.: An automatic wafer inspection system using pipelined image processing techniques, IEEE Trans. Pattern Analysis & Machine Intelligence. PAMI-10 1 (1988) 17. Ejiri, M., Yoda, H., Sakou, H.: Knowledge-directed inspection for complex multilayered patterns. Machine Vision and Applications 2, 155–166 (1989) 18. Fukushima, T., Kobayashi, Y., Hirasawa, K., Bandoh, T., Ejiri, M.: Architecture of image signal processor, Trans. IEICE, J-66C 12, 959–966 (1983) 19. Ejiri, M., Kakumoto, S., Miyatake, T., Shimada, S., Matsushima, H.: Automatic recognition of engineering drawings and maps. In: Proc. Int. Conf. on Pattern Recognition, Montreal, Canada, pp. 1296–1305 (1984) 20. Ejiri, M.: Knowledge-based approaches to practical image processing. In: Proc. MIV-89, Inst. Ind. Sci, Univ. of Tokyo, pp. 1–8. Tokyo (1989) 21. Yoda, H., Motoike, J., Ejiri, M., Yuminaka, T.: A measurement method of the number of passengers using real-time TV image processing techniques, Trans. IEICE, J-69D 11, 1679–1686 (1986) 22. Takahashi, K., Kitamura, T., Takatoo, M., Kobayashi, Y., Satoh, Y.: Traffic flow measuring system by image processing. In: Proc. IAPR MVA, Tokyo, pp. 245–248 (1996) 23. Yahagi, H., Baba, K., Kosaka, H., Hara, N.: Fish image monitoring system for detecting acute toxicants in water. In: Proc. 5th IAWPRC, pp. 609–616 (1990) 24. Ogawa, Y., Kakumoto, S., Iwamura, K.: Extracting regional features from aerial images based on 3-D map matching, Trans. IEICE, D-II 6, 1242–1250 (1998) 25. Nagaya, S., Miyatake, T., Fujita, T., Itoh, W., Ueda, H.: Moving object detection by time-correlation-based background judgment. In: Li, S., Teoh, E.K., Mital, D., Wang, H. (eds.) Recent Developments in Computer Vision. LNCS, vol. 1035, pp. 717–721. Springer, Heidelberg (1996) 26. Nagasaka, A., Miyatake, T., Ueda, H.: Video retrieval method using a sequence of representative images in a scene. In: Proc. IAPR MVA, Kawasaki, pp. 79–82 (1994) 27. Arai, K., Sakamoto, H.: Real-time animation of the upper half of the body using a facial expression tracker and an articulated input device, Research Report 96CG-83, Information Processing Society of Japan (in Japanese), 96, 125, pp. 1–6 (1996) 28. Horry, Y., Anjyo, K., Arai, K.: Tour into the picture: Using a spidery mesh interface to make animation from a single image. In: Proc. ACM SIGGRAPH 1997, pp. 225– 232 (1997) 29. Nagasaka, A., Miyatake, T.: A real-time video mosaics using luminance-projection correlation, Trans. IEICE, J82-D-II 10, 1572–1580 (1999) 30. Kamei, T., Shinbata, H., Uchida, K., Sato, A., Mizoguchi, M., Temma, T.: Automated fingerprint classification, IEICE Technical Report, Pattern Recognition and Understanding, 95(470), 17–24 (in Japanese) (1996) 31. Ui, H., Arima, Y., Murao, F., Komori, S., Kyuma, K.: An artificial retina chip with pixel-wise self-adjusting intensity response, ITE Technical Report, 23(30), pp.29–33 (in Japanese) (1999)
Coarse-to-Fine Statistical Shape Model by Bayesian Inference Ran He, Stan Li, Zhen Lei, and ShengCai Liao Institute of Automation, Chinese Academy of Sciences, Beijing, China
[email protected]
Abstract. In this paper, we take a predefined geometry shape as a constraint for accurate shape alignment. A shape model is divided in two parts: fixed shape and active shape. The fixed shape is a user-predefined simple shape with only a few landmarks which can be easily and accurately located by machine or human. The active one is composed of many landmarks with complex shape contour. When searching an active shape, pose parameter is calculated by the fixed shape. Bayesian inference is introduced to make the whole shape more robust to local noise generated by the active shape, which leads to a compensation factor and a smooth factor for a coarse-to-fine shape search. This method provides a simple and stable means for online and offline shape analysis. Experiments on cheek and face contour demonstrate the effectiveness of our proposed approach. Keywords: Active shape model, Bayesian inference, statistical image analysis, segmentation.
1 Introduction Shape analysis is an important area in computer vision. A common task of shape analysis is to recover both pose parameters and low-dimensional representation of the underlying shape from an observed image. Applications of shape analysis spread from medical image processing, face recognition, object tracking and etc. After the pioneering work on active shape model (ASM) put forward by Cootes and Taylor [1,2], various shape models have been developed for shape analysis, which mainly focus on two parts: (1) statistic framework to estimate the shape and pose parameters and (2) optimal features to accurately model appearance around landmarks. For parameter estimation, Zhou, Gu, and Zhang [3] propose a Bayesian tangent shape model to estimate parameters more accurately by Bayesian inference. Liang et al. [4] adopt Markov network to find an optimal shape which is regularized by the PCA based shape prior through a constrained regularization algorithm. Li and Ito [5] use AdaBoosted histogram classifiers to model local appearances and optimize shape parameters. Thomax Brox et al. [6] integrated 3D shape knowledge into a variational model for pose estimation and image segmentation. For optimal features, van Ginneken et al. [7] propose a non-linear ASM with Optimal Features (OF-ASM), which allows distributions of multi-modal intensities and uses a k-nearest neighbors classifier for local textures classification. Federico Sukno et al. [8] further develop Y. Yagi et al. (Eds.): ACCV 2007, Part I, LNCS 4843, pp. 54–64, 2007. © Springer-Verlag Berlin Heidelberg 2007
Coarse-to-Fine Statistical Shape Model by Bayesian Inference
55
this non-linear appearance model, incorporating a reduced set of differential invariant features as local image descriptors. A Cascade structure containing multiple ASMs is introduced in [9] to make location of landmarks more accurate and robust. However, these methods will lose their efficiency when dealing with complicated geometry of shapes or large texture variations. Can we utilize some accurate information to simplify ASM algorithm and make shape parameters estimation more robust? For example, we can utilize face detection algorithm to detect the coordinates of eyes and mouth or manually label these coordinates when we want to find a facial contour for further analysis. In this paper, the problem of shape analysis is addressed from three aspects. Firstly, we present geometry constrained active shape model (GCASM) and divide it in two parts: fixed shape and active shape. The fixed shape is a user-predefined shape with only a few points and lines. Those points could be easily and accurately located by machine or human. The active one is a user's desired shape and is composed of many landmarks with a complex contour. It will be located automatically with the help of the fixed shape. Secondly, Bayesian inference is introduced to make parameter estimation more robust to local noise generated by the active shape, which leads to a compensation factor and a smooth factor to perform a coarse-to-fine shape search. Thirdly, optimal features are selected as local image descriptors. Since the pose parameters can be calculated by the fixed shape, classifiers are trained for each landmark without scarifying performance. The rest of the paper is organized as follows: In Section 2, we begin with a brief review of ASM. Section 3 describes our proposed algorithm and Bayesian inference. Experimental results are provided in Section 4. Finally, we draw the conclusions in Section 5.
2 Active Shape Models This section briefly reviews the ASM segmentation scheme. We follow the description and notation of [2]. An object is described by points, referred as landmark points. The landmark points are (manually) determined in a set of N training images. From these collections of landmark points, a point distribution model (PDM) [10] is constructed as follows. The landmark points (x1, y1, … , xn, yn) are stacked in shape vectors.
x = ( x1 , y1 ,..., xn , yn )T .
(1)
Principal component analysis (PCA) is applied to the shape vectors x by computing the mean shape, covariance and eigensystem of the covariance matrix.
x=
1 N
∑
N
x and S =
i =1 i
N N ( xi − x )( xi − x )T . ∑ i =1 N −1
(2)
The eigenvectors corresponding to the k largest eigenvalues λ j are retained in a matrix Φ = (φ1 | φ2 | ⋅ ⋅ ⋅ | φk ) . A shape can now be approximated by x ≈ x + Φb .
(3)
56
R. He et al.
Where b is a vector of k elements containing the shape parameters, computed by b = ΦT ( x − x ) .
(4)
When fitting the model to a set of points, the values of b are constrained to lie within a range | b j |≤ ±c λ j .
(5)
where c usually has a value between two and three. Before PCA is applied, the shapes can be aligned by translating, rotating and scaling so as to minimize the sum of squared distances between the landmark points. We can express the initial estimate x of a shape as a scaled, rotated and translated version of original shape
x = M ( s,θ )[ x] + t .
(6)
Where M ( s,θ ) and t are pose parameters (See [1] for details). Procrustes analysis [11] and EM algorithm [3] are often used to estimate the pose parameters and align the shapes. This transformation and its inverse are applied both before and after projection of the shape model. The alignment procedure makes the shape model independent of the size, position, and orientation of the objects.
3 Coarse-to-Fine Statistical Shape Model 3.1 Geometry Constrained Statistical Shape Model
To make use of the user-predefined information, we extend PDM to two parts: active shape and fixed shape. The active shape is a collection of landmarks to describe an object in the basic PDM. It is composed of many points with a complex contour. The fixed shape is a predefined simple shape accurately marked by user or machine. It is composed of several connected lines between these points which can be easily and accurately marked by machine or human. Considering there are tremendous points in a line, we present a line with several equidistant points. Thus the extended PDM is constructed as follows. The landmarks (x1, y1, … , xm, ym) are stacked in active shape vectors, and landmarks (xm+1, ym+1, … , xn, yn) are stacked in fixed shape vectors. x = ( x1 , y1 ,..., xm , ym , xm +1 , ym +1 ,..., xn , yn )T .
(7)
As in PDM, a shape can now be approximated by x ≈ x + Φb .
(8)
When aligning shapes during training, the pose parameters of a shape (scaling, rotation and translation) are estimated by the fixed shape. An obvious reason is that the fixed shape is simpler and more accurate than the active one. Taking cheek contour as an example, the active shape is composed of landmarks in a cheek contour and the fixed shape is composed of 13 landmarks derived from three manual labeled points: left eye center, right eye center and mouth center. Five
Coarse-to-Fine Statistical Shape Model by Bayesian Inference
57
landmarks are added equidistantly between two eyes center to represent horizontal connected line. And five landmarks are inserted equidistantly in the vertical line passing the mouth center and perpendicular to the horizontal line. (See left graph of Fig.1 for details) During training, two shapes are aligned according to the points between two eyes only. Each item of b reflects a specific variation along the corresponding principle component (PC) axis. Shape variation along first three PCs is shown in right graph of Fig.1. The interpretation of these PCs is straight forward. The first PC describes left-right head rotations. The second PC accounts for face variation in vertical direction: long or short. And the third one explains a human face fat or thin.
Fig. 1. The fixed shape and shapes reconstructed by the first three PCs. The thirteen white circles in left image are points of the fixed shape. In right image, the middle one in each row is the mean shape.
3.2 Bayesian Inference
When directly calculating shape parameter b by formula (4), there is an offset between the reconstructed fixed shape and the given fixed shape. But the fixed shape is supposed to be accurate. This noise comes from reconstruction error of the active shape. Inspired by paper [3], we associate PCA with a probabilistic explanation. An isotropic Gaussian noise item is added to both fixed and active shape; thereby we can compute the poster of model parameters. The model can be written as: y = x + Φb + ε .
(9)
y − x − Φb = ε .
(10)
Where the shape parameter b is a n-dimensional vector distributed as multivariate Gaussian N (0, Λ ) and Λ = diag (λ1 ,..., λk ) . ε denotes an isotropic noise on the whole shape. It is a n-dimensional random vector which is independent with b and distributes as
p(ε ) ~ exp{− || ε ||2 / 2( ρ ) 2 } .
(11)
ρ = ∑ i =1α i || yiold − yi ||2 .
(12)
n
Where yold is the shape estimated in the last iteration and y is an observed shape in the current iteration. ai is classification confidence related to a classifier used in locating a
58
R. He et al.
landmark. When ai is 0, which implies that classifier can perfectly predict shape’s boundary; when ai is 1, which means classifier fails to predict the boundary. Combing (10) and (11) we obtain the likelihood of model parameters: 1 P(b | y ) = constP ( y | b) P (b) ~ exp( − [( y − x − Φb)T ( y − x − Φb) / ρ + bT Λ −1b]) 2
Let
(13)
∂ (ln P (b | y )) = 0 , we get: ∂b
b j = (λ j \ (λ j + ρ ))φ Tj ( y − x ) .
(14)
Combining (4), we obtain: b j = (λ j \ (λ j + ρ ))b j .
(15)
It is obvious that value of bj will become smaller after updating of (15) ( ρ ≥ 0 ). This updating will slow down search speed. Hence, a compensation factor p1 is introduced to make shape variation along eigenvectors corresponding to large eigenvalues more aggressive (see formula 18). If p1 is equal to (λmax + ρ ) / λmax , we get b j = ((λmax + ρ ) \ λmax ) × (λ j \ (λ j + ρ )) × b j .
(16)
Formula (16) shows that the parameter bj corresponding to a larger eigenvalue will receive a small punishment. And the parameter bj corresponding to a small eigenvalue will become smaller after updating. Moreover, we expect a smooth shape contour and neglect details in the first several iterations. A smooth factor p2 (see formula 18) is introduced to further punish the parameter bj. It is noticed that ρ is smaller than the largest eigenvalue and will become smaller. The p2 regularizes the parameters by enlarging the punishment. As in Fig.2, the reconstructed shape’s contour by Bayesian inference is smoother than the one by PCA in regions pointed by the black arrows. Although the PCA reconstruction can remove some noise, the reconstructed shape is still unstable when the image is noisy. Formula (18) makes the parameter estimation more robust to local noise.
Fig. 2. Shapes reconstructed from PCA and Bayesian Inference. Left shape is mean shape after desired movements; middle shape is reconstructed by PCA; right shape is reconstructed by Bayesian Inference. The black arrows highlight the regions to be compared.
Coarse-to-Fine Statistical Shape Model by Bayesian Inference
59
3.3 Optimal Features
Recently, optimal features are applied in ASMs and have drawn more and more attentions. [5, 6, 7] Experimental results show that optimal features can make shape segmentation more accurate. But a main drawback of optimal features method is that it takes ASMs more time to find the desired landmarks due to extract optimal features in each iteration. An efficient speed-up strategy is to select a subset of the available features for all landmarks. [6, 7] It is clear that textures around different landmarks are different. It is impossible for a single subset of optimal features to describe various textures around all the landmarks. In GCASM, the pose parameters of scale, rotation and translation can be calculated by the fixed shape. All landmarks can be categorized into several groups, for each of which we select the same discriminate features. When search a shape, the image is divided into several areas according to the categories. For each area, the same optimal features are extracted to determine movement. Optimal features are features reported in both paper [6] and [7]. Fig.3 shows classification results for each landmark. The Mean classification accuracy is 76.67%. We can learn about that landmarks near jaw and two ears have low classification accuracy, and the landmarks near cheek have high classification accuracy. Considering this classification error, we introduce Bayesian Inference and ai of formula (12) to make shape estimation more robust.
Fig. 3. Classification results for each cheek landmark. Classification accuracy stands for a classifier’s ability to classify whether a point near the landmark is in or outside of the shape. The points around the indices of 4 and 22 are close to ears and the points around the index of 13 are close to jaw.
3.4 Coarse-to-Fine Shape Search
During image search, main differences between GCASM and ASM lie in twofold. One is that since the pose parameters of GCASM have been calculated by the fixed shape, we needn’t to think about the pose variation during iterative updating procedure. The other is that the fixed shape is predefined accurately in GCASM. After reconstruction from the shape parameters, the noise will make the reconstructed fixed shape leave away from the given fixed shape. Because the fixed shape is supposed to be accurate, it should be realigned to the initial points. The iterative updating procedure of GCASM and ASM are shown in Fig.4. We use formula (17) to calculate shape parameter b=[b1,…,bk]T and normalize b by formula (18).
60
R. He et al.
b j = ΦTj ( y − x ) .
(17)
b j = ( p1λ j /(λ j + p2 ρ ))b j
(18)
Where 1 ≤ p1 ≤ (λmax + p2 ρ ) / λmax , p2 ≥ 1 . We call the parameter p1 compensation factor which makes shape variation in a more aggressive way. The parameter p2 is a smooth factor which gives a penalty to the shape parameter when shape has a large variation. The compensation factor and smoother factor give more emphasis on shape parameters corresponding to large eigenvalues. This can adjust a shape along major PCs and neglect shape’s local detail in initial several iterations. When the algorithm converges ( ρ → 0 ), p1λ j /(λ j + p2 ρ ) is equal to 1. Hence, the compensation factor and smoother factor lead a coarse-to-fine shape searching. Here, we simply set p1 = (λmax + p2 ρ ) / λmax , α i = 1 and p2 = 4 . Obviously, the formula (18) can also be used in ASMs to normalize the shape parameter.
Fig. 4. Updating rules of ASM and GCASM. The left block diagram is the basic ASM’s updating rule and the right block diagram is GCASM updating rule.
4 Experiments In this section, our proposed method is tested on two experiments: cheek contour search and facial contour search. A total of 100 face images are randomly taken from the XM2VTS face database. [12] Each image is aligned by coordinates of two eyes. The average distance between two eyes is 80 pixels. Three points of the fixed shape including two eyes and mouth center are manually labeled. The fixed shape takes a shape of letter ‘T’. Hamarneh’s ASM source code [13] is taken as the standard ASM without modification. Optimal features are collected from features reported in both paper [7] and [8]. The number of optimal features is reduced by sequential feature
Coarse-to-Fine Statistical Shape Model by Bayesian Inference
61
selection [14]. In this work, all the points near the landmarks are classified by linear regression to predict whether they lie in or out of a shape. 4.1 Experiments on Cheek Contour
A designed task to directly search a cheek contour without eyes, brow, mouth, and nose is presented to validate our method. A total of 25 cheek landmarks are labeled manually on each image. The PCA thresholds are set to 99% for every ASMs. The fixed shape is composed of points between two eyes and mouth. As in Fig.3, it is difficult to locate points around landmarks near ears and jaw. When a contour shape is simple and textures around landmarks are complex, the whole shape will be dragged off from the right position if there are several inaccurate points. It is clear that the cheek shape can be accurately located with the help of the fixed shape.
Fig. 5. Comparison of different algorithms’ cheek searching results: Shapes in first column are results of ASM searching; Shapes in second column are results of simple OF-ASM; Shapes in third column are results of the basic GCASM; Shapes in fourth column are results of GCASM with optimal features; Shapes in fifth column are results of GCASM with optimal features and Bayesian inference
As in Fig.5, first two columns are the searching results of ASM and OF-ASM. It is clear that the searching results miss desired position because of local noise. Several inaccurate landmarks will drag the shape from desired position. It also illustrates that optimal features can model contour appearance more accurately. As illustrated in the last three columns in Fig.5, searching results are well trapped in a local area when the fixed shape is introduced. Because the fixed shape is accurate without noise, reconstructed shape will fall into a local area around the fixed shape even if some landmarks are inaccurate. Every landmark will find a local best matched point instead of a global one. Comparing the third and fourth column, we can learn about that optimal features can locate landmarks more accurately. But optimal
62
R. He et al.
features couldn’t keep local contour detail very well. There is still some noise in searching results. Looking at the fifth column of Fig.5, it is clear that borders of the shapes become smoother. The Bayesian inference can further improve the accuracy. 4.2 Experiments on Facial Contour
A total of 96 face landmarks are labeled manually on each image. The PCA thresholds are set to 95% for every ASMs. Three landmarks are inserted into two eyes to present horizontal connected line. And three landmarks are inserted between mouth and horizontal line to present the vertical line. For the sake of simplicity, optimal features don’t used in this subsection. The results are shown in table 1. Table 1. Comparison results of traditional ASM and our method without optimal features
ASM Our algorithm Improvement
Face 7.74 4.68 39.5%
F.S.O 6.45 4.41 31.6%
Cheek Contour 11.4 5.47 52.0%
Where F.S.O. means five sense organs. Location error is measured in pixel. It is clear that our algorithm is much more accurate than ASM.
Fig. 6. Comparison results of ASM and GCASM with Bayesian inference. The first row is ASM results, and the second row is our results.
Fig.6 shows a set of searching results of basic ASM and GCASM with Bayesian inference. In the case, there are wrinkles and shadings on the facial contour or other facial sub-parts. It is clear that our method can recover the shape from local noise. A direct reason is that the shape variation is restricted in a local area when combining accurate information in ASM. The Bayesian inference holds the whole shape and smoothes the shape border.
Coarse-to-Fine Statistical Shape Model by Bayesian Inference
63
5 Conclusion This work focuses on an interesting topic how to combine some accurate informationgiven by user or machine to further improve shape alignment accuracy. The PDM is extended by adding a fixed shape which is generated from given information. After PCA reconstruction, local noise of the active shape will make the whole shape unsmooth. Hence Bayesian inference is proposed to further normalize parameters of the extended PDM. Both compensate factor and smooth factor lead a coarse-to-fine shape adjustment. Comparisons of our algorithm and the ASM algorithms demonstrate the effectiveness and efficiency.
Acknowledgements This work was supported by the following funding resources: National Natural Science Foundation Project #60518002, National Science and Technology Supporting Platform Project #2006BAK08B06, National 863 Program Projects #2006AA01Z192 and #2006AA01Z193, Chinese Academy of Sciences 100 people project, and the Authen-Metric Collaboration Foundation.
References 1. Cootes, T.F., Taylor, C.J., Cooper, D., Graham, J.: Active shape models-Their training and application. Comput. Vis. Image Understanding 61(1), 38–59 (1995) 2. Cootes, T.F., Taylor, C.J.: Statistical models of appearance for computer vision, Wolfson Image Anal. Unit, Univ. Manchester, Manchester, U.K., Tech. Rep (1999) 3. Zhou, Y., Gu, L., Zhang, H.-J.: Bayesian tangent shape model: Estimating shape and pose parameters via Bayesian inference. In: IEEE Conf. on Computer Vision and Pattern Recognition, Madison, WI (June 2003) 4. Liang, L., Wen, F., Xu, Y.Q., Tang, X., Shum, H.Y.: Accurate Face Alignment using Shape Constrained Markov Network. In: Proc. CVPR (2006) 5. Li, Y.Z., Ito, W.: Shape parameter optimization for Adaboosted active shape model. In: ICCV, pp. 259–265 (2005) 6. Brox, T., Rosenhahn, B., Weickert, J.: Three-Dimensional Shape Knowledge for Joint Image Segmentation and Pose Estimation. In: Kropatsch, W.G., Sablatnig, R., Hanbury, A. (eds.) Pattern Recognition. LNCS, vol. 3663, pp. 109–116. Springer, Heidelberg (2005) 7. Ginneken, B.V., Frangi, A.F., Staal, J.J., ter Har Romeny, B.M., Viergever, M.A.: Active shape model segmentation with optimal features. IEEE Transactions on Medical Imaging 21(8), 924–933 (2002) 8. Sukno, F., Ordas, S., Butakoff, C., Cruz, S., Frangi, A.F.: Active shape models with invariant optimal features IOF-ASMs. In: Kanade, T., Jain, A., Ratha, N.K. (eds.) AVBPA 2005. LNCS, vol. 3546, pp. 365–375. Springer, Heidelberg (2005) 9. Zhang, S., Wu1, L.F., Wang, Y.: Cascade MR-ASM for Locating Facial Feature Points. The 2nd International Conference on Biometrics (2007) 10. Dryden, I., Mardia, K.V.: The Statistical Analysis of Shape. Wiley, London, U.K (1998)
64
R. He et al.
11. Goodall, C.: Procrustes methods in the statistical analysis of shapes. J.Roy. Statist. 53(2), 285–339 (1991) 12. Messer, K., Matas, J., Kittler, J., Luettin, J., Maitre, G.: XM2VTSDB: The extended M2VTS database. In: Proc. AVBPA, pp. 72–77 (1999) 13. Hamarneh, G.: Active Shape Models with Multi-resolution, http://www.cs.sfu.ca/~hamarneh/ software/asm/index.html 14. Kudo, M., Sklansky, J.: Comparison of algorithms that select features for pattern classiers. Pattern Recognition, 25–41 (2000)
Efficient Texture Representation Using Multi-scale Regions Horst Wildenauer1 , Branislav Miˇcuˇs´ık1,2, and Markus Vincze1 Automation and Control Institute Institute of Computer Aided Automation, PRIP Group, Vienna University of Technology, Austria 1
2
Abstract. This paper introduces an efficient way of representing textures using connected regions which are formed by coherent multi-scale over-segmentations. We show that the recently introduced covariancebased similarity measure, initially applied on rectangular windows, can be used with our newly devised, irregular structure-coherent patches; increasing the discriminative power and consistency of the texture representation. Furthermore, by treating texture in multiple scales, we allow for an implicit encoding of the spatial and statistical texture properties which are persistent across scale. The meaningfulness and efficiency of the covariance based texture representation is verified utilizing a simple binary segmentation method based on min-cut. Our experiments show that the proposed method, despite the low dimensional representation in use, is able to effectively discriminate textures and that its performance compares favorably with the state of the art.
1
Introduction
Textures and structured patterns are important cues towards image understanding, pattern classification and object recognition. The analysis of texture properties and their mathematical and statistical representation is attracting the interest of researchers since many years; with the primary goal of finding low dimensional and expressive representations that allow for reliable handling and classification of texture patterns. Texture representations, which have been successfully applied to image segmentation tasks, include steerable filter responses [1], color changes in a pixel’s neighborhood [2], covariance matrices of gradients, color, and pixel coordinates [3], Gaussian Mixture Models (GMM) computed from color channels [4,5], color histograms [6], or multi-scale densities [7,8]. Since textures ”live” at several scales, a scale-dependent discriminative treatment should be aimed for. In this paper, we explore the possibility to refine coarse texture segmentation by matching textures between adjacent scales, taking
The research has been supported by the Austrian Science Foundation (FWF) under the grant S9101, and the European Union projects MOVEMENT (IST-2003-511670), Robots@home (IST-045350), and MUSCLE (FP6-507752).
Y. Yagi et al. (Eds.): ACCV 2007, Part I, LNCS 4843, pp. 65–74, 2007. c Springer-Verlag Berlin Heidelberg 2007
66
H. Wildenauer, B. Miˇcuˇs´ık, and M. Vincze
(a)
(c)
(b)
(d)
(e)
Fig. 1. Segmentation results using the min-cut algorithm [12] with different texture representations. (a) Input image with user specified foreground and background markers. (b) The proposed multi-scale texture representation. (c) Color Histograms [6]. (d) GrabCut [4] using GMMs. (e) Color changes in the pixel neighbourhoods [2].
advantage of spatial and statistical properties which persist across scale. We show that texture segments can be efficiently treated in a multi-scale hierarchy similarly to [8], however building on superpixels. In our approach, textures are represented by covariance matrices, for which an effective similarity measure based on the symmetric generalized eigenproblem was introduced in [9]. In contrast to the rectangular windows used in [3], covariance matrices are computed from irregular structure-coherent patches, found at different scales. In order to allow for an efficient image-partitioning into scalecoherent regions we devised a novel superpixel method, utilizing watershed segmentations imposed by extrema of an image’s mean curvature. However, the suggested framework of multi-scale texture representation is generally applicable for other superpixel methods, such as [10,11], depending on accuracy and time complexity constraints imposed by the application domain. We verify the feasibility and meaningfulness of the multi-scale covariance based texture representation by a binary segmentation method based on the min-cut algorithm [12]. Figure 1 shows an example of how different types of texture descriptors influence the min-cut segmentation of a particularly challenging image, consisting of textured regions with highly similar color characteristics. The remainder of the paper is organized as follows. We present the details of the proposed method in Section 2. Section 3 reports experimental results and compares them to the results obtained using state-of-the-art methods. The paper is concluded with a discussion in Section 4.
Efficient Texture Representation Using Multi-scale Regions
2 2.1
67
Our Approach Superpixels
Probably one of the most commonly used blob detectors is based on the properties of the Laplacian of Gaussians (LoG) or its approximation, the Difference of Gaussians (DoG) [13]. Given a Scale-Space representation L(t) obtained by repeatedly convolving an input image by Gaussians of increasing sizes t, the shape of the intensity surface around a point x at scale t can be described using the Hessian matrix L (x, t) Lxy (x, t) . (1) H(x, t) = xx Lxy (x, t) Lyy (x, t) The LoG corresponds to the trace of the Hessian: ∇2 L(x, t) = Lxx (x, t) + Lyy (x, t),
(2)
and equals the mean intensity curvature multiplied by two. The LoG’s computation results in strong √ negative or positive responses for bright and dark blob-like structures of size t respectively. Using this, the position and characteristic scale of blobs can by found by detecting Scale-Space extrema of scale normalized LoG responses [14]. In our approach, we do not directly search for blob positions and scales, but rather use spatial response extrema as starting points for a watershed-based oversegmentation of an image’s mean curvature surface. Specifically, we proceed as follows: √ 1. Computation of LoG responses at scales of t = 2m/3 , with m = 1 . . . M , where M denotes a predefined number of scales. I.e., we calculate 3 scale levels per Scale-Space octave. 2. Watershed-segmentation: (a) Detection of spatial response extrema at all scales. Extrema with low contrast, i.e. those with a minimum absolute difference to adjacent pixels smaller than a predefined threshold, are discarded. (b) At each scale, segment the image into regions assigned to positive or negative mean curvature. This is achieved by applying the watershed to the negative absolute Laplacian −|∇2 L(x, t)| using the seeds from (a). The majority of watersheds thus obtained follow the zero-crossings of the Laplacian; i.e., the edges where the mean curvature of the intensity surface changes its sign. Though, for irregularly shaped blobs, which exhibit significant variations in mean curvature, usually several seed-points are detected. This results in an over-segmentation of regions with otherwise consistent curvature signs. Figure 2 shows a direct comparison of the superpixels produced by our method at a single scale and the normalized-cut based superpixels suggested in [10]. Another method for image over-segmentation, which is partially favoured for its speed, utilizes the Minimum Spanning Tree [11]. However, for larger superpixels, which are needed to stably compute the covariance-based descriptor
68
H. Wildenauer, B. Miˇcuˇs´ık, and M. Vincze
Fig. 2. Left: Superpixels obtained by the proposed method. Right: Superpixels obtained by the method of Ren et al. [10].
on, the regions obtained by this method are highly irregular and often do not align well with object boundaries. Figure 3 shows the effect of using different superpixels in conjunction with the method proposed in this paper. As one can see, our method gives acceptable results compared to the normalized-cut based approach, which needs more than 100 times longer to compute the segmentation. The outlined approach is similar in spirit to the watershed segmentation of principal curvature images proposed by Deng et al. [15]. In their approach, the first principle curvature image (i.e., the image of the larger eigenvalue of the Hessian matrix) is thresholded near zero and either the positive, or negative remainder is flooded starting from the resulting zero-valued basins. Hence, as opposed to our method, the watersheds follow the ridges of the image’s principal curvature surface. In experiments we found that this approach was not suitable for our purposes since it tends to under-segment images, aggressively merging regions with same-signed principal curvature. 2.2
Covariance-Based Texture Similarity
Recently, Tuzel et al. [3] have introduced region covariance matrices as potent, low-dimensional image patch descriptors, suitable for object recognition and texture classification. One of the authors’ main contributions was the introduction of an integral-image like preprocessing stage, allowing for the computation of covariances from image features of an arbitrarily sized rectangular window in constant time. However, in the presented work covariances are directly obtained from irregularly shaped superpixels, the aforementioned fast covariance computation is not applicable. We proceed to give a brief description of the covariance-based texture descriptor in use. The sample covariance matrix of feature vectors collected inside a superpixel is give by: M=
N 1 (z n − μ)(z n − μ) , N − 1 n=1
(3)
Efficient Texture Representation Using Multi-scale Regions
69
Fig. 3. Effect of superpixels on the final image segmentation. From left to right: Superpixels through a color-based Minimum Spanning Tree [11]. The proposed approach. Superpixels based on normalized-cuts using combined color and textured gradient [10].
where μ denotes the sample mean, and {z n }n=1...N are the d-dimensional feature vectors extracted at N pixel positions. In our approach, these feature vectors are composed of the values of the RGB color channels R, G, and B and the absolute values of the first derivatives of the Intensity I at the n-th pixel ∂I ∂I (4) z n = Rn , Gn , Bn , , . ∂x ∂y The resulting 5×5 covariance matrix gives a very compact texture representation with the additional advantage of exhibiting a certain insensitivity to illumination changes. And, as will be shown experimentally, offers sufficient discriminative power for the segmentation task described in the remainder of the paper. To measure the similarity ρ(M i , M j ) of two covariance matrices M i and M j we utilize the distance metric initially proposed by F¨ orstner [9]: d ρ(M i , M j ) = ln λ2k (M i , M j ), (5) k=1
where the {λk }k=1...d are the eigenvalues obtained by solving the generalized eigenvalue problem (6) M i ek = λk M j ek , k = 1 . . . d with ek = 0 denoting the generalized eigenvectors. The cost for computing ρ is in the order of O(d3 ) flops which, due to the low dimensionality of the representation, leads to speed advantages compared to histogram matching methods. For a detailed discussion among the topic, other choices of feature combinations as well as the useful properties of region covariances see [3]. 2.3
Foreground and Background Codebooks
From the covariance-based descriptors proposed in Subsection 2.2 we compute representative codebooks for foreground and background regions. These are used later on to drive the image segmentation.
70
H. Wildenauer, B. Miˇcuˇs´ık, and M. Vincze
Sink
Source
Fig. 4. Two-layer MRF with superpixels detected at two different scales. To avoid clutter not all superpixels are connectd to sink/source nodes.
For the foreground and background codebook, we require user-specified markB ers, as shown in Figure 1(a), computing the covariance matrices M F i and M i from all points under the marker region. Usually, the background contains more textures and cluttered areas requiring more seeds to be establish. Moreover, in applications like object detection or recognition, the background can significantly vary across images while objects of interest usually remain quite consistent in appearance. To somewhat alleviate the burden of manually selecting many seeds, we propose to avoid the need of background markers by following a simple strategy: We take a rim at the boundary of the image and feed all superpixels under the rim into a hierarchical clustering method with a predefined stopping distance threshold, with the distance between superpixels given by Equation 5. After clustering we take the K most occupied clusters and compute the mean covariance matrix for each cluster out of all covariance matrices belonging to the cluster. For efficiency reasons, we do not calculate the mean covariance matrix by polling over all participating feature vectors, but use the method described in [16,3], which has its roots in formulations of symmetric positive definite matrices lying on connected Riemannian manifolds. Using this procedure, we arrive at the background codebook matrices M B i . Of course, the applicability of this ad-hoc technique is limited to cases where the object of interest touches the boundary, or when the rim is not representative enough. However, in most cases the approach lead to background codebooks with sufficient explanatory power for a successful segmentation. 2.4
Multi-scale Graph-Cut
In order to verify the validity of the covariance-based texture representation, taking into account the superpixel behaviour across different scales we adopted a binary segmentation method based on the min-cut algorithm [12]. Suppose that the image at a certain scale t is represented by a graph Gt = Vt , Et , where Vt is a set of all vertices representing superpixels, and Et is a set of all intrascale edges connecting spatially adjacent vertices. To capture the Scale-Space behaviour we connect the graphs by interscale edges forming a set of edges S. We form the entire graph G = V, E, S consisting of the union of
Efficient Texture Representation Using Multi-scale Regions
71
all vertices Vt , and all intrascale Et and interscale edges S. For more clarity, the resulting graph structure is depicted in Figure 4. The binary segmentation of the graph G is achieved by finding the minimumcut [12], minimizing the Gibbs energy, Edata (xi , M i ) + λ δ(xi , xj ) Esm im (M i , M j )+ E(x) = i∈V
(i,j)∈E
+γ
δ(xi , xj ) Esm
sc (M i , M j ),
(7)
(i,j)∈C
where x = [x0 , x1 , . . .] corresponds to a vector with label xi for each vertex. We concentrate on a bi-layer segmentation where the label xi is either 0 (background) or 1 (foreground). M i corresponds to the measurement in the i-th graph vertex, i.e., to a covariance matrix for a given superpixel. The weight constants λ, γ control the influence of the image (intrascale), and interscale smoothness terms respectively; δ denotes the Kronecker delta. The data term. describes how likely the superpixel is foreground or background. The data term for the foreground is defined as l(M i , F ) , (8) l(M i , F ) + l(M i , B)
where l(Mi , F ) = exp − mink=1...|F | ρ(Mi , MkF )/(2σ12 ) stands for the foreground likelihood of the superpixel i. M F k denotes the k-th covariance matrix from a foreground codebook set F , and σ1 is an experimentally determined parameter. As the derivation of the background terms and likelihoods follows analogously, we will omit its description. Edata (xi = 1, M i ) =
The smoothness term. describes how strongly neighborhood pixels are bound together. There are two types of the smoothness terms, see Equation (7), one for intrascale neighborhoods, Esm im , one for interscale neighborhoods, Esm sc . The intrascale smoothness term using α blending is defined as Esm im (Mi , Mj ) = α exp − ρ(M i , M j )/(2σ22 ) +
2 2 + (1 − α) exp − l(M i , F ) − l(M j , F ) /(2σ3 ) , (9) where σ2 and σ3 are pre-defined parameters. The interscale smoothness term is only defined for edges between two vertices from neighboring scales when the corresponding superpixels share at least one same image pixel. The weight on the edge between superpixels i and j from consecutive scales is set to Esm sc (Mi , Mj ) = β
area(i ∩ j) + area(i)
2 2 + (1 − β) exp − l(M i , F ) − l(M j , F ) /(2σ3 ) . (10)
72
H. Wildenauer, B. Miˇcuˇs´ık, and M. Vincze
Fig. 5. Importance of inter-scale graph edges. From left to right: Only one lower scale used. Only one higher scale used. Three consecutive scales used.
The second term in both Equations (9), (10) increases the dependency of smoothness terms on the foreground likelihood, making it more robust as originally suggested by [8]. However, we rely on this term only partially through the interpolation parameteres α, β, since a full dependency on the likelihood often resulted in compact, but otherwise incomplete segmentations. Figure 5 shows how the use of multiple scales and inter-scale edges improves the segmentation compared to segmentation performed separately for given scales.
3
Experimental Results
We performed segmentation tests on images from the Berkeley dataset1 . We compare the result to the recent approach proposed by Micusik&Pajdla [2]. Their method looks at color changes in the pixel neighbourhood, yielding superior results on textured images compared to other methods. For both methods the same manually established foreground and background markers were used. To guarantee a fair comparison, the automatic background codebook creation proposed in Section 2.3 was omitted. We present some results where our proposed method performs superior or comparable to [2]. These images typically contain textures with similar colors and are, as stated in [2], the most crucial for their texture descriptor. One must realize that covariance based texture description cannot cope reliably with homogenoeus color regions, see the missing roof of the hut in Figure 6. This should be kept in mind, and use such a descriptor complementary with some color features. Overall, as experiments show, the newly proposed technique performs very well on textures. The advantage over methods, e.g. [6,4,2], is computational efficienty. Moreover, using more accurate superpixels, e.g. [10], improve the accuracy of the result for the price of higher time consumption.
4
Summary and Conclusion
We present an efficient way of representing textures using connected regions, formed by coherent multi-scale over-segmentations. We show the favourable 1
http://www.cs.berkeley.edu/projects/vision/grouping/segbench
Efficient Texture Representation Using Multi-scale Regions
(a)
(b)
73
(c)
Fig. 6. Segmentation comparison. (a) Input image with user marked seeds. (b) The method from [2]. (c) Our approach.
74
H. Wildenauer, B. Miˇcuˇs´ık, and M. Vincze
performance on segmentation of textures images. However, our primary goal is not to segment images accurately, but to demonstrate the feasibility of the covariance matrix based descriptor used in a multi-scale hierarchy built on superpixels. The method is aimed at a further use in recognition and an image understanding systems where so accurate segmentation is not required.
References 1. Malik, J., Belongie, S., Leung, T., Shi, J.: Contour and texture analysis for image segmentation. IJCV 43(1), 7–27 (2001) 2. Miˇcuˇs´ık, B., Pajdla, T.: Multi-label image segmentation via max-sum solver. In: Proc. CVPR (2007) 3. Tuzel, O., Porikli, F., Meer, P.: Region covariance: A fast descriptor for detection and classification. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3952, pp. 589–600. Springer, Heidelberg (2006) 4. Rother, C., Kolmogorov, V., Blake, A.: ”grabcut”: interactive foreground extraction using iterated graph cuts. In: Proc. ACM SIGGRAPH, pp. 309–314. ACM Press, New York (2004) 5. Kolmogorov, V., Criminisi, A., Blake, A., Cross, G., Rother, C.: Probabilistic fusion of stereo with color and contrast for bi-layer segmentation. PAMI 28(9), 1480–1492 (2006) 6. Boykov, Y., Jolly, M.P.: Interactive graph cuts for optimal boundary & region segmentation of objects in N-D images. In: Proc. ICCV, pp. 105–112 (2001) 7. Hadjidemetriou, E., Grossberg, M., Nayar, K.S.: Multiresolution histograms and their use for recognition. PAMI 26(7), 831–847 (2004) 8. Turek, W., Freedman, D.: Multiscale modeling and constraints for max-flow/mincut problems in computer vision. In: Proc. CVPR Workshop, vol. 180 (2006) 9. F¨ orstner, W., Moonen, B.: A metric for covariance matrices. Technical report, Dpt. of Geodesy and Geoinformatics, Stuttgart University (1999) 10. Ren, X., Malik, J.: Learning a classification model for segmentation (2003) 11. Felzenszwalb, P., Huttenlocher, D.: Efficient graph-based image segmentation. IJCV 59(2), 167–181 (2004) 12. Boykov, Y., Kolmogorov, V.: An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision. PAMI 26(9), 1124–1137 (2004) 13. Lowe, D.G.: Distinctive image features from scale-invariant keypoints. IJCV 60(2), 91–110 (2004) 14. Lindeberg, T.: Scale-Space Theory in Computer Vison. Kluwer Academic Publishers, Dordrecht (1994) 15. Deng, H., Zhang, W., Diettrich, T., Shapiro, L.: Principal curvature-based region detector for object recognition. In: Proc. CVPR (2007) 16. Pennec, X., Fillard, P., Ayache, N.: A riemannian framework for tensor computing. International Journal of Computer Vision 66(1), 41–66 (2006)
Comparing Timoshenko Beam to Energy Beam for Fitting Noisy Data Ili´c Slobodan Deutsche Telekom Laboratories Berlin University of Technology Ernst-Reuter Platz 7, 14199 Berlin, Germany
[email protected]
Abstract. In this paper we develop highly flexible Timoshenko beam model for tracking large deformations in noisy data. We demonstrate that by neglecting some physical properties of Timoshenko beam, classical energy beam can be derived. The comparison of these two models in terms of their robustness and precision against noisy data is given. We demonstrate that Timoshenko beam model is more robust and precise for tracking large deformations in the presence of clutter and partial occlusions. The experiments using both synthetic and real image data are performed. In synthetic images we fit both models to noisy data and use Monte Carlo simulation to analyze their performance. In real images we track deformations of the pole vault, the rat whiskers and the car antenna.
1 Introduction In this paper, we develop true physical 2D Timoshenko beam model and use it for tracking large deformations in noisy image data. Timoshenko beam relays on shear deformation to account for non-linearities. We derive from it physically based energy beam, by neglecting shear deformation. The models which closely approximate real physics we call true physical models, in this case Timoshenko beam, while the models which are designed to retain some physical properties we call physically based models, in this case energy beam. Physically based models introduced almost twenty years ago [1,2,3,4] have demonstrated their effectiveness in the Computer Vision problems. However, they typically rely on simplifying assumptions to yield easy to minimize energy functions and ignore the complex non-linearities that are inherent to large deformations present in highly flexible structures. To justify the use of complex true physical models over simplified physically based models we compare Timoshenko beam to energy beam. Both models were fitted to noisy synthetic data and real images in presence of clutter and partial occlusions. We demonstrate that using fully non-linear Timoshenko beam model, which approximates the physical behavior more closely, yields to more robust and precise fitting to noisy data and tracking of large deformations in the presence of clutter and partial occlusions.
The rat whiskers images shown in this paper were obtained at Harvard University’s School of Engineering and Applied Sciences by Robert A. Jenks.
Y. Yagi et al. (Eds.): ACCV 2007, Part I, LNCS 4843, pp. 75–84, 2007. c Springer-Verlag Berlin Heidelberg 2007
76
I. Slobodan
The fitting algorithm used for both beam representations is guided by the image forces. Since the image forces are proportional to the square distance of the model to the image data, they are usually not sufficient to deform Timoshenko beam model of known material properties, so that it fits the data immediately. The image forces only move the model toward the image observations. To fit the model to the data we repeat Gauss-Newton minimization in several steps. We stop when the distance of the model to the data of two consecutive Gauss-Newton minimizations is smaller then some given precision. We use Levenberg-Marquardt optimizer to fit quadratic energy function of the energy beam. In this case image forces are sufficient to deform the beam in a single minimization because the model energy, being only the approximation of the real model strain energy, does not impose realistic physical restrictions to the beam deformations. In the reminder of the paper we give a brief overview of the know physically based techniques, then introduce non-linear Timoshenko planar beam model, derive energy beam from it, then describe our fitting algorithm and finally present the results.
2 Related Work Recovering model deformations from images requires deformable models to constrain the search space and make the problem tractable. In recent decade deformable models were exploited in Computer Graphics [5], Medical Imaging [6] and Computer Vision [7]. There are several important categories of physically based models: mass-spring models [8], finite element models (FEM) [9,10], snake like models [1,3,4] and models obtained from FEM, to reduce number of dofs, by modal analysis [11,2,12,13]. In this paper we are particularly interested in physical models, especially those based on FEM. FEM are known to be precise and to produce physically valid deformations. However, because of their mathematical complexity the FEM were mainly developed for small linear deformations [14] where the model stiffness matrix is constant. In case of large deformations the stiffness matrix and the applied forces become the function of the displacement. Such non-linear FEM were used by [15] to recover material parameters from images of highly elastic shell like models. The model deformation was measured form 3D model scan and then given to the Finite Element Analysis (FEA) software. By contract we develop non-linear beam equations and recover the model deformations automatically through optimization. In computer vision physically based models, based on the continuous energy function were extensively used. The original ones [1] were 2D and have been shown to be effective for 2D deformable surface registration [16]. They were soon adapted for 3D surface modeling purposes by using deformable superquadrics [3,4], triangulated surfaces [2], or thin-plate splines [6]. In this framework, modeling generic structures often requires many degrees of freedom that are coupled by regularization terms. In practice, this coupling implicitly reduces the number of degrees of freedom, which makes these models robust to noise and is one of the reasons for their immense popularity. In this paper we compare complex true physical 2D Timoshenko beam model to 2D elastic energy beam in terms of their robustness against noisy data. We reveal, in spite of their complexity, the real benefits of true physical models.
Comparing Timoshenko Beam to Energy Beam for Fitting Noisy Data
77
3 Plane Timoshenko Beam Model The beam represents the most common structural component in civil and mechanical engineering. A beam is a rod-like — meaning that one dimension is considerably larger then the other two — structural element that resists transversal loading applied between its supports. Timoshenko beam, we develop, assumes geometrically large deformations under small strains with the linearly elastic materials. Timoshenko beam theory [17] accounts for nonlinear effect, such as shear assuming that the cross-section does not remains normal to the deformed longitudinal axis. The beam is divided to a number of finite elements. The beam deformation is defined by 3 dof per node being the axial displacement uX (X), the transverse displacement uY (X) and the cross-section rotation θ(X), where X is longitudinal coordinate in the reference configuration as shown in Fig 1. The undeformed initial configuration is referred as reference and the deformed one as current configuration. The parameters describing beam geometry and the material properties are A0 cross-sectional area, L0 element length in reference configuration, L element length in deformed configuration, I0 second moment of inertia, E Young modulus of elasticity, and G is shear modulus. The material remains linearly elastic. The beam rotation is defined by the angle ψ also equal to the rotation of the cross-section. The angle γ¯ is a shear angle for which the cross section deviates from its normal position defined by the angle ψ. The total rotation of a beam cross section becomes θ = ψ − γ¯ which is exactly one of dofs defined above. To describe beam kinematics we consider the motion of the point P0 (X, Y ) in reference configuration to the point P (x, y) in the current configuration. We keep the assumption that the cross section dimensions do not change and that the shear rotation
(a)
(b)
Fig. 1. (a) Plane Timoshenko beam kinematics notation. (b)Synthetic example of fitting the plane beam, initially aligned along the x-axis , to the synthetic image data shown as magenta dots. The intermediate steps shown in blue are the output of number of repeated Gauss-Newton optimizations driven by the image forces.
78
I. Slobodan
is small γ¯ 1 so that cosγ ≈ 1. The Lagrangian representation of the motion relating points P0 (X, Y ) and P (x, y) is then given by x = X +uX −Y sin θ, y = uY +Y cos θ. The displacement of any point on the beam element can be than represented by a vec tor w = uX (X) uY (X) θ(X) . In the FEM formulation for 2-node C 0 element it is natuaral to express the displacements and rotation functions of w as linear 2 2 2combination of node displacements uX = i=1 Ni uXi , vX = i=1 Ni vXi , θX = i=1 Ni θXi or in matrix form w = Nu where N1 = 12 (1 − ξ), N2 = 12 (1 + ξ) are the linear element shape functions. The strain is a measure of the change of the object shape, in this case the length, before and after the deformation caused by some applied load. The stress is the internal distribution of force per unit area that balances and reacts to external loads applied to a body. We have three different stress components per beam element: e axial strain measuring the beam relative extension, γ shear strain measuring the relative angular change between any two lines in a body before and after the deformation and κ measuring the curvature change. They can be computed from the deformation gradient of motion F =
∂x ∂x ∂X ∂Y ∂y ∂y ∂X ∂Y 1 T 2 (F F
. Green-Lagrange(GL) strain tensor describing the model strain
becomes e = − I). After the derivation the only nonzero elements are axial strain eXX and the shear strain 2eXY = eXY + eY X . Under small strain assumption we can finally express strain vector as: eXX e−Yκ e1 = = (1) e= e2 2eXY γ where three strain quantities introduced above are e axial strain, γ shear strainand κ curvature.These can be collected in the generalized strain vector hT = e γ κ . Because of the assumed linear variations in X of uX (X), uY (Y ) and θ(X), e and γ depend on θ and κ is constant over the element depending only on rotation angles at element end nodes. e and γ can be expressed in a geometrically invariant form: e=
L L L L cos γ¯ − 1 = cos (θ − ψ) − 1, γ = sin γ¯ = sin (θ − ψ) L0 L0 L0 L0
(2)
These geometrically invariant strain quantities can be used for the beam in arbitrary reference configuration. The variations of δe, δγ and δκ with respect to the nodal displacement variations are required for derivations of strain-displacement relation δh = Bδu. To form strain-displacement matrix B we take partial derivatives of e, γ and κ with respect to the node displacements and collect them into matrix B: ⎤ ⎡ cω sω L0 N2 γ −cω −sω L0 N1 γ (3) B = ⎣ sω −cω −L0 N1 (1 + e) −sω cω −L0 N2 (1 + e)⎦ 0 0 −1 0 0 1 where ω = θ + ψ, cω = cos ω and sω = sin ω. We introduce pre-stress resultants N 0 , V 0 and M 0 which define the axial forces, transverse shear forces and bending moments respectively, in the reference configuration. We also define the stress resultants in the current configuration using linear elastic 0 0 equation to be N = N 0 + EA0 e, V = V + GA 0 γ and M = M + EI0 κ, and collect them into stress-resultant vector z = N V M .
Comparing Timoshenko Beam to Energy Beam for Fitting Noisy Data
79
The internal model strain energy along the beam, under zero pre-stress resultants N 0 = V 0 = M 0 = 0, can be expressed as the length integral: 1 1 1 zT hdX = EA0 e2 dX + GA0 γ 2 dX + EI0 κ2 dX (4) U= 2 2 2 L0 L0 L0 L0 where L0 is the beam length in reference configuration. The internal force vector can be obtained by taking the fist variation of the strain energy with respect to the nodal displacement: ∂U = BT (u)zdX. (5) p= ∂u L0 This expression we evaluate by reduced Gauss integration in order to eliminate shear locking which overstiff the model deformation making a shear energy to dominate. In addition we use residual bending flexibility (RBF) correction and replace GA0 for shear energy of Eq. 4 by 12EI0 /L20 . Finally, the first variation of the internal force defines the tangent stiffness matrix: ∂B ∂z ∂p = + z)dX = (KM + KG ). (BT (6) KT = ∂u ∂u ∂u L0 where KM is material stiffness and KG is geometric stiffness. The material stiffness is constant and identical to the linear stiffness matrix of the C 1 Euler-Bernoulli beam. The geometric stiffness comes the variation of B while stress resultants are kept fixed and caries the beam nonlinearity responsible for large geometric deformations.
4 Energy Beam Model The model energy of the energy beam can be derived directly from the Timoshenko beam strain energy of Eq. 4. Let us neglect the shear deformation by putting shear anγ − 1 ≈ LL0 − 1, gle to be zero γ¯ . The strain quantities of Eq. 2 become e = LL0 cos¯ γ = LL0 sin γ¯ ≈ 0. In this way the shear energy is eliminated and only the axial energy and bending energy are left. Also, since shear is eliminated the rotational dof θ(X) disappears, and only displacements in X and Y directions are taken to form new displacement vector w = [uX uY ]. Since we deal with discrete beam its energy can be expressed as: U=
1 ws 2
(
(i,j)∈1..n
vi − vj 1 − 1)2 + wb L0 2
2vj − vi − vk
2
(7)
(i,j,k)∈1..n
where i, j are pairs of element nodes, and i, j, k are triplets of element nodes necessary to define curvature at the j t h beam node. Derived energy can be considered as physically based since it directly comes from the realistic physical beam model. The weight coefficients ws and wb can be considered proportional to the Young modulus of elasticity E. However, we will show that, in practice, they significantly change behavior of the fitting algorithm.
80
I. Slobodan
5 Model Fitting The general approach in mechanical simulations is that some external load f is applied to the model and the displacement u is computed. This can be done through energy minimization, where the total potential energy Π of the system is computed as the sum of the system internal strain energy U and the energy caused by the external load P . The minimum of the energy in respect to the displacement u can be found by derivation: ∂U ∂P ∂Π = + ⇒ r(u) = p(u) + f = 0 (8) ∂u ∂u ∂u The r(u) is a force residual, p(u) is internal force of Eq. 5 and f is the external load. This is a nonlinear system of equations and is usually solved by using Newton-Raphson method. It is incremental method and at each iteration we solve for the displacement update du by solving linear system KT du = −(f + p(u)). In classical mechanical simulation the external forces f are a priori given. In our case we do not know them. To compute model displacements we solve Eq. 8, using image T forces. We create a vector of image observations F(u) = d1 (u) d2 (u) . . . dN (u) where di (u) are distances of the image observations from the beam segments. We use the edges in the image, obtained using Canny edge detector, to be our observations. In practice we sample every beam segment and then search for the multiple observations in the direction of the beam normal. The external image energy becomes PI = 12 FT (u)F(u). The images forces are obtained as a derivative of the energy in respect to the displacement fI = ∇FT (u)F(u). The force residual of Eq. 8 becomes: r(u) = p(u) + fI (u) = 0. We derive the displacement increment by developing the residual in a Taylor series around the current displacement u as follows: r(u + du) = p(u + du) + fI (u + du) = 0 ∇FT F + (∇2 FT F + ∇FT ∇F)du + p(u) + KT du = 0 (KT + ∇2 FT F + ∇FT ∇F)du = −(∇FT F + p(u)) ≈ (KT + ∇FT ∇F)du = −(∇FT F + p(u))
(9)
we obtain Gauss-Newton optimization step. We neglect the second order term ∇2 FT F of Eq. 9. To make it more robust we use Tukey robust estimator ρ of [18]. This is simply done by weighting the residuals di (u) of the image observation vector F(u) at each Newton iteration: Each di (u) is replaced by di (u) = wi di (u) such that: (di )2 (u) = (wi )2 d2i (u), therefore the weight is chosen to be: wi = ρ(di (u))1/2 /di (u). We then create a weighting matrix W = diag(. . . , wi , . . .). We then solve in each step: (KT + ∇FT W∇F)du = −(∇FT WF + p(u))
(10)
By solving the Eq. 9 we compute the displacement caused by the image forces. Since the image forces are proportional to the square distance of the model to the image edge points, they are not sufficient to deform the model so that it fits the data. They only move the model toward the image observations. To obtain the exact fit of the model to the data we repeatedly fit the model to the data performing Gauss-Newton method in several steps. We stop when the distance of the model to the data of two consecutive Gauss-Newton minimizations becomes smaller then some given precision. The
Comparing Timoshenko Beam to Energy Beam for Fitting Noisy Data
81
optimization algorithm is illustrated on the synthetic example of Fig. 1(b). We obtain the total displacement uT as a sum of all intermediate displacements. For fitting energy beam we slightly modify Eq. 10 by adding λI to the left side of Eq. 10 such that the we obtain Levenberg-Marquart optimizer. In practice it turned out to be more suitable for optimizing energy function of Eq. 7. The computational complexity of our method is quadratic and corresponds to the complexity of the Gauss-Newton minimization.
6 Results We fit the Timoshenko beam and energy beam to synthetic data and compare their performance in presence of different amount of added noise. We then run our experiments on real images in three different cases: the deformation of a car antenna, pole vault and the deformation of the rat whiskers. 6.1 Fitting Synthetic Data We generate the synthetic data clouds around two given ground truth positions of the deformed beams depicted in Fig. 1(b) by adding certain amount of Gaussian noise around them. The amount of noise is controlled by the variance σ ∈ {0.01, 0.1, 0.5, 1.0, 2.0}. We perform a Monte-Carlo simulation such that for each value of σ we fit, in a number of trials, Timoshenko and energy beam to randomly generated data clouds. The number of trials is 100 for every value of σ. We measure mean square error of the fitting result to the ground truth position of the beam. Error measures from ground trought for different values of Young modulus
Error measures from ground trought for different energy weights
0.25 E=1E2 E=1E3 E=1E4 E=1E5 E=1E6
0.06
2.5 ws=1E3,wb=1E1 2
ws=1E1,wb=1E3 ws=1E4,wb=1E2 ws=1E4,wb=1E2
1.5
ws=1E4,wb=1E4
1
0.05
0.04
0.03
0.02
0.15
0.1
0.05
0.5
0.01
0
0.2
0.4
0.6
0.8 1 1.2 1.4 Noise standard derivation σ
1.6
1.8
0
2
0
0.5
1 1.5 Noise standard derivation σ
2
0
2.5
Average mean square error
ws=1E4,wb=1E2 ws=1E4,wb=1E4
3
2
0.8 1 1.2 1.4 Noise standard derivation σ
1.6
1.8
2
Physics Beam Energy Beam
0.16
0.15
0.1
0.05
1
0.6
0.18
Average mean square error
0.2
ws=1E4,wb=1E2 4
0.4
0.2 E=1E2 E=1E3 E=1E4 E=1E5 E=1E6 E=1E7
ws=1E3,wb=1E1 ws=1E1,wb=1E3
0.2
Error in respect to the ground trought vs. different noise levels
0.25 ws=1E0,wb=1E0
5
0
Error measures from ground trought for different values of Young modulus
Error measures from ground trought for different energy weights 6
Average mean square error
Physics Beam Energy Beam 0.2 Average mean square error
Average mean square error
Average mean square error
ws=1E0,wb=1E0
0
Error in respect to the ground trought vs. different noise levels
0.07
3
0.14 0.12 0.1 0.08 0.06 0.04 0.02
0
0
0.2
0.4
0.6
0.8 1 1.2 1.4 Noise standard derivation σ
(a)
1.6
1.8
2
0
0
0.2
0.4
0.6
0.8 1 1.2 1.4 Noise standard derivation σ
(b)
1.6
1.8
2
0
0
0.2
0.4
0.6
0.8 1 1.2 1.4 Noise standard derivation σ
1.6
1.8
2
(c)
Fig. 2. Mean square error measured for number of fittings using Monte Carlo simulation from the ground truth in respect to the different amount of noise. (a) Energy beam fitting errors for different values of energy weights. (b) Timoshenko beam fitting errors for different values of Young modulus of elasticity. (c) Comparison of fitting errors for energy beam shown in red, and Timoshenko beam shown in blue.
82
I. Slobodan
(a)
(b)
(c)
(d)
Fig. 3. Failiour examples using energy beam with different energy weighting coefficients. (a) For wb = 104 , ws = 102 stays smooth but changes its length producing failiour in 5th frame. (b) For wb = 102 , ws = 104 tries to retains its length but no the smoothness producing failiour in 3rd frame. (c,d) The rat whisker fails in the 10th frame because of the occluded ear, and the energy coefficients are also wb = 103 , ws = 10 and ws = 102 , wb = 10 respectively.
Fig. 4. Tracking the car antenna using Temoshenko beam. Selected frames from the tracking sequence with the recovered model shown in white.
Fig. 5. Tracking the pole in a pole vault using Timoshenko beam. Because of the moving camera the image frames are warped to one reference frame using robustly estimated homography. Selected frames from the tracking sequence with the recovered model shown in yellow.
Initially we perform fittings for different values of energy weights ws and wb for energy beam and different values of Young modulus E for Timoshenko beam as shown in Fig. 2(a,b) respectively. The error differs for the different values of the energy weights. We take those values for which the error is minimal and refit the beams to the noisy data with different values of σ. Usually a good balance between stretching and bending energy is required for reasonable fitting performance of the energy beam. For Timoshenko
Comparing Timoshenko Beam to Energy Beam for Fitting Noisy Data
83
Fig. 6. Tracking the deformation of the rat whisker using Timoshenko beam. Selected frames from the tracking sequence with the recovered model are shown in white.
beam the small values of Young modulus ranging from 102 to 103 are unrealistic for materials with the small strains to which the Timoshenko theory applies. It means that small values of Young modulus are suitable for elastic materials with the large strains, while big values of Young modulus are suitable for elastic materials which have small strains, i.e tend to retain their length but can have large rotations. For that reasons we obtained the best fitting performance for values of E being 105 and 106 . The errors in respect to the ground truth for both beams in two synthetic examples are shown in Fig. 2(c). Timoshenko beam retains the same error measure with the increase of the amount of noise, while the error for the energy beam increases with the increase of the amount of added noise. This indicates that Timoshenko beam is more robust when fitted to noisy data. The same is proven bellow during tracking in real images. 6.2 Real Images In real images we chose to track highly flexible structures such as a car antenna, pole vault and the deformation of the rat whiskers. The car antenna example of Fig. 4 has simple background and both Timoshenko and energy beam track it with no problems as can be seen in supplementary videos. More complex pole vault and rat whiskers were successfully tracked using Timoshenko beam while they failed when energy beam was used. The failiour examples are depicted in Fig. 3. The selected frames form successful tracking using Timoshenko beam are depicted in Fig. 5 and Fig 6. In all examples the initialization was done manually in the first frame and then the frame to frame fitting was done. The energy beam has tendance to attach to the strong edges regardless the combination of the energy weights as depicted by Fig. 3, while the Timoshenko beam overcomes this problem because its naturally imposed physical constrains implicitly contained in the model description.
7 Conclusion In this paper we investigated true physical Timoshenko beam model to track large non-linear deformations in images. We compared it to physically based energy beam
84
I. Slobodan
approach which uses simplifying physical assumptions to create the model energy, similar to most physically based models used in computer vision. These approaches ignore the complex non-linearities that are inherent to large deformations. We discovered that using Timoshenko beam, which approximates the physical behavior more closely, contributed to robust fitting to noisy data and efficient tracking of large deformations in the presence of clutter and partial occlusions.
References 1. Kass, M., Witkin, A., Terzopoulos, D.: Snakes: Active Contour Models. International Journal of Computer Vision 1(4), 321–331 (1988) 2. Cohen, L., Cohen, I.: Deformable models for 3-d medical images using finite elements and balloons. In: Conference on Computer Vision and Pattern Recognition, pp. 592–598 (1992) 3. Terzopoulos, D., Metaxas, D.: Dynamic 3D models with local and global deformations: Deformable superquadrics. IEEE Transactions on Pattern Analysis and Machine Intelligence 13, 703–714 (1991) 4. Metaxas, D.T.D.: Constrained deformable superquadrics and nonrigid motion tracking. IEEE Transactions on Pattern Analysis and Machine Intelligence 15(6), 580–591 (1993) 5. Gibson, S., Mirtich, B.: A survey of deformable modeling in computer graphics. Technical report, Mitsubishi Electric Research Lab, Cambridge, MA (1997) 6. McInerney, T., Terzopoulos, D.: Deformable models in medical images analysis: a survey. Medical Image Analysis 1(2), 91–108 (1996) 7. Metaxas, D.: Physics-Based Deformable Models: Applications to Computer Vision, Graphics, and Medical Imaging. Kluwer Academic Publishers, Dordrecht (1996) 8. Lee, Y., Terzopoulos, D., Walters, K.: Realistic modeling for facial animation. In: SIGGRAPH 1995. Proceedings of the 22nd annual conference on Computer graphics and interactive techniques, pp. 55–62. ACM Press, New York (1995) 9. Essa, I., Sclaroff, S., Pentland, A.: Physically-based modeling for graphics and vision. In: Martin, R. (ed.) Directions in Geometric Computing. Information Geometers, U.K (1993) 10. Sclaroff, S., Pentland, A.P.: Physically-based combinations of views: Representing rigid and nonrigid motion. Technical Report 1994-016 (1994) 11. Pentland, A.: Automatic extraction of deformable part models. International Journal of Computer Vision 4(2), 107–126 (1990) 12. Delingette, H., Hebert, M., Ikeuchi, K.: Deformable surfaces: A free-form shape representation. SPIE Geometric Methods in Computer Vision 1570, 21–30 (1991) 13. Nastar, C., Ayache, N.: Frequency-based nonrigid motion analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence 18(11) (1996) 14. O’Brien, J.F., Cook, P.R., Essl, G.: Synthesizing sounds from physically based motion. In: Fiume, E. (ed.) SIGGRAPH 2001. Computer Graphics Proceedings, pp. 529–536 (2001) 15. Tsap, L., Goldgof, D., Sarkar, S.: Fusion of physically-based registration and deformation modeling for nonrigid motion analysis (2001) 16. Bartoli, A., Zisserman, A.: Direct Estimation of Non-Rigid Registration. In: British Machine Vision Conference, Kingston, UK (2004) 17. Timoshenko, S., MacCullogh, G.: Elements of Strength in Materials, 3rd edn., van Nostrand. New York (1949) 18. Lepetit, V., Fua, P.: Monocular model-based 3d tracking of rigid objects: A survey. Foundations and Trends in Computer Graphics and Vision 1(1), 1–89 (2005)
A Family of Quadratic Snakes for Road Extraction Ramesh Marikhu1 , Matthew N. Dailey2 , Stanislav Makhanov3 , and Kiyoshi Honda4 Information and Communication Technologies, Asian Institute of Technology Computer Science and Information Management, Asian Institute of Technology 3 Sirindhorn International Institute of Technology, Thammasat University 4 Remote Sensing and GIS, Asian Institute of Technology
1 2
Abstract. The geographic information system industry would benefit from flexible automated systems capable of extracting linear structures from satellite imagery. Quadratic snakes allow global interactions between points along a contour, and are well suited to segmentation of linear structures such as roads. However, a single quadratic snake is unable to extract disconnected road networks and enclosed regions. We propose to use a family of cooperating snakes, which are able to split, merge, and disappear as necessary. We also propose a preprocessing method based on oriented filtering, thresholding, Canny edge detection, and Gradient Vector Flow (GVF) energy. We evaluate the performance of the method in terms of precision and recall in comparison to ground truth data. The family of cooperating snakes consistently outperforms a single snake in a variety of road extraction tasks, and our method for obtaining the GVF is more suitable for road extraction tasks than standard methods.
1
Introduction
The geographic information system industry would benefit from flexible automated systems capable of extracting linear structures and regions of interest from satellite imagery. In particular, automated road extraction would boost the productivity of technicians enormously. This is because road networks are among the most important landmarks for mapping, and manual marking and extraction of road networks is an extremely slow and laborious process. Despite years of research and significant progress in the computer vision and image processing communities (see, for example, [1,2] and Fortier et al.’s survey [3]), the methods available thus far have still not attained the speed and accuracy necessary for practical application in GIS tools. Among the most promising techniques for extraction of complex objects like roads are active contours or snakes, originally introduced by Kass et al. [4]. Since the seminal work of Kass and colleagues, techniques based on active contours have been applied to many object extraction tasks [5] including road extraction [6]. Rochery et al. have recently proposed higher-order active contours, in particular quadratic snakes, which hold a great deal of promise for extraction of linear Y. Yagi et al. (Eds.): ACCV 2007, Part I, LNCS 4843, pp. 85–94, 2007. c Springer-Verlag Berlin Heidelberg 2007
86
R. Marikhu et al.
structures like roads [7]. The idea is to use a quadratic formulation of the contour’s geometric energy to encourage anti-parallel tangents on opposite sides of a road and parallel tangents along the same side of a road. These priors increase the final contour’s robustness to partial occlusions and decrease the likelihood of false detections in regions not shaped like roads. In this paper, we propose two heuristic modifications to Rochery et al.’s quadratic snakes, to address limitations of a single quadratic snake and to accelerate convergence to a solution. First, we introduce the use of a family of quadratic snakes that are able to split, merge, and disappear as necessary. Second, we introduce an improved formulation of the image energy combining Rochery et al.’s oriented filtering technique [7] with thresholding, Canny edge detection, and Xu and Prince’s Gradient Vector Flow (GVF) [8]. The modified GVF field created using the proposed method is very effective at encouraging the quadratic snake to snap to the boundaries of linear structures. We demonstrate the effectiveness of the family of snakes and the modified GVF field in a series of experiments with real satellite images, and we provide precision and recall measurements in comparison with ground truth data. The results are an encouraging step towards the ultimate goal of robust, fully automated road extraction from satellite imagery. As a last contribution, we have developed a complete GUI environment for satellite image manipulation and quadratic snake evolution, based on the Matlab platform. The system is freely available as open source software [9].
2 2.1
Experimental Methods Quadratic Snake Model
Here we provide a brief overview of the quadratic snake proposed by Rochery et al. [7]. An active contour or snake is parametrically defined as T γ(p) = x(p) y(p) ,
(1)
T where p is the curvilinear abscissa of the contour and the vector x(p) y(p) defines the Cartesian coordinates of the point γ(p). We assume the image domain Ω to be a bounded subset of R2 . The energy functional for Rochery et al.’s quadratic snake is given by Es (γ) = Eg (γ) + λEi (γ),
(2)
where Eg (γ) is the geometric energy and Ei (γ) is the image energy of the contour γ. λ is a free parameter determining the relative importance of the two terms. The geometric energy functional is defined as β tγ (p) · tγ (p ) Ψ (γ(p) − γ(p )) dp dp , (3) Eg (γ) = L(γ) + αA(γ) − 2
A Family of Quadratic Snakes for Road Extraction
87
where L(γ) is the length of γ in the Euclidean metric over Ω, A(γ) is the area enclosed by γ, tγ (p) is the unit-length tangent to γ at point p, and Ψ (z), given the distance z between two points on the contour, is used to weight the interaction between those two points (see below). α and β are constants weighting the relative importance of each term. Clearly, for positive β, Eg (γ) is minimized by contours with short length and parallel tangents. If α is positive, contours with small enclosed area are favored; if it is negative, contours with large enclosed area are favored. The interation function Ψ (z) is a smooth function expressing the radius of the region in which parallel tangents should be encouraged and anti-parallel tangents should be discouraged. Ψ (z) incorporates two constants: d, the expected road width, and , the expected variability in road width. During snake evolution, weighting by Ψ (z) in Equation 3 discourages two points with anti-parallel tangents (the opposite sides of a putative road) from coming closer than distance d from each other. The image energy functional Ei (γ) is defined as Ei (γ) =
nγ (p) · ∇I(γ(p)) dp − tγ (p) · tγ (p ) ∇I(γ(p)) · ∇I(γ(p )) Ψ (γ(p) − γ(p )) dp dp , (4)
where I : Ω → [0, 255] is the image and ∇I(γ(p)) denotes the 2D gradient of I evaluated at γ(p). The first linear term favors anti-parallel normal and gradient vectors, encouraging counterclockwise snakes to shrink around or clockwise snakes to expand to enclose dark regions surrounded by light roads.1 The quadratic term favors nearby point pairs with two different configurations, one with parallel tangents and parallel gradients and the other with anti-parallel tangents and anti-parallel gradients. After solving the Euler-Lagrange equations for minimizing the energy functional Es (γ) (Equation 2), Rochery et al. obtain the update equation nγ (p) ·
+ 2λ
1
∂Es (p) = −κγ (p) − α − λ∇I(γ(p))2 + G(γ(p)) ∂γ + β r (γ(p), γ(p )) · nγ (p ) Ψ (γ(p) − γ(p )) dp
r (γ(p), γ(p )) · nγ (p ) (∇I(γ(p)) · ∇I(γ(p ))) Ψ (γ(p) − γ(p )) dp + 2λ ∇I(γ(p )) · (∇∇I(γ(p)) × nγ (p )) Ψ (γ(p) − γ(p )) dp , (5)
For dark roads in light background, we negate all the terms involving image, including G(γ(p)) in Equation 5. In the rest of the paper, we assume light roads on a dark background.
88
R. Marikhu et al.
where κγ (p) is the curvature of γ at γ(p) and G(γ(p)) is the “specific energy,” γ(p)−γ(p ) evaluated at point γ(p) (Section 2.2). r (γ(p), γ(p )) = γ(p)−γ(p ) is the unit vector pointing from γ(p) towards γ(p ). ∇∇I(γ(p)) is the Hessian of I evaluated at γ(p). α, β, and λ are free parameters that need to be determined experimentally. d and are specified a priori according to the desired road width. Following Rochery et al., we normally initialize our quadratic snakes with a rounded rectangle covering the entire image. 2.2
Oriented Filtering
We use Rochery’s oriented filtering method [10] to enhance linear edges in our satellite imagery. The input image is first convolved with oriented derivativeof-Gaussian filters at various orientations. Then the minimum (most negative) filter response over the orientations is run through a ramp function equal to 1 for low filter values and −1 for high filter values. The thresholds are user-specified. An example is shown in Fig. 1(b). 2.3
GVF Energy
Rather than using the oriented filtering specific image energy G(x) from Section 2.2 for snake evolution directly, we propose to combine the oriented filtering approach with Xu and Prince’s Gradient Vector Flow (GVF) method [8]. T The GVF is a vector field V GVF (x) = u(x) v(x) minimizing the energy functional GVF E(V )= μ(u2x (x) + u2y (x) + vx2 (x) + vy2 (x)) (6) Ω 2 2 ˜ ˜ V (x) − ∇I(x) dx, + ∇I(x) ∂u ∂v ∂v ˜ where ux = ∂u ∂x , uy = ∂y , vx = ∂x , vy = ∂y , and I is a preprocessed version of image I, typically an edge image of some kind. The first term inside the integral encourages a smooth vector field whereas the second term encourages fidelity to ˜ μ is a free parameter controlling the relative importance of the two terms. ∇I. Xu and Prince [8] experimented with several different methods for obtaining ˜ We propose to perform Canny edge detection on G (the result of oriented ∇I. filtering and thresholding, introduced in Section 2.2) to obtain a binary image I˜ for GVF, then to use the resulting GVF V GVF as an additional image energy for quadratic snake evolution. The binary Canny image is ideal because it only includes information about road-like edges that have survived sharpening by oriented filters. The GVF field is ideal because during quadratic snake evolution, it points toward road-like edges, pushing the snake in the right direction from a long distance away. This speeds evolution and makes it easier to find suitable parameters to obtain fast convergence. Fig. 1 compares our method to alternative GVF formulations based on oriented filtering or Canny edge detection alone.
A Family of Quadratic Snakes for Road Extraction
89
Fig. 1. Comparison of GVF methods. (a) Input image. (b) G(x) obtained from oriented filtering on I(x). (c) Image obtained from G(x) using threshold 0. (d) Canny edge detection on (c), used as I˜ for GVF. (e-f) Zoomed views of GVFs in region delineated ˜ (f) in (d). (e) Result of using the magnitude of the gradient ∇(Gσ ∗ I) to obtain I. ˜ Result of using Canny edge detection alone to obtain I. (g) GVF energy obtained using our proposed edge image. This field pushes most consistently toward the true road boundaries.
2.4
Family of Quadratic Snakes
A single quadratic snake is unable to extract enclosed regions and multiple disconnected networks in an image. We address this limitation by introducing a family of cooperating snakes that are able to split, merge, and disappear as necessary. In our formulation, due to the curvature term κγ (p) and the area constant α in Equation 5, specifying the points on γ in a counterclockwise direction creates a shrinking snake and specifying the points on γ in a clockwise direction creates a growing snake. An enclosed region (loop or a grid cell) can be extracted effectively by initializing two snakes, one shrinking snake covering the whole road network and another growing snake inside the enclosed region. On the one hand, our method is heuristic and dependent on somewhat intelligent user initialization, but it is much simpler than level set methods for the same problem [7], and, assuming a constant number of splits and merges per iteration, it does not increase the asymptotic complexity of the quadradic snake’s evolution. Splitting a Snake. We split a snake into two snakes whenever two of its arms are squeezed too close together, i.e. when the distance between two snake points is less than dsplit and those two points are at least k snake points from each other in both directions of traversal around the contour. dsplit should be less than 2η, where η is the maximum step size.
90
R. Marikhu et al.
Merging Two Snakes. Two snakes are merged when they have high curvature points within a distance dmerge of each other, the two snakes’ order of traversal (clockwise or counterclockwise) is the same, and the tangents at the two high curvature points are nearly antiparallel. High curvature points are those with where κmax is the maximum curvature for any point on γ. High κγ (p) > 0.6κmax γ γ curvature points are taken to ensure merging only occurs if two snakes have the semi-circular tip of their arms facing each other. Filtering out the low curvature points necessitates computing angle between the tangents at two points only for the high curvature points. When these conditions are fulfilled, the two snakes are merged by deleting the high curvature points and joining the snakes into a single snake while preserving the direction of traversal for the combined snake. Deleting a Snake. A snake γ is deleted if it has low compactness ( 4πA(γ) L(γ)2 ) and delete . a perimeter less than L 2.5
Experimental Design
We analyze extraction results on different types of road networks using the single quadratic snake proposed by Rochery et al. [7] and the proposed family of cooperating snakes. The default convergence criterion is when the minimum Es (γ) has not improved for some number of iterations. Experiments have been performed to analyze the extraction of tree-structured road networks and those with loops, grids and disconnected networks. We then analyze the effectiveness of GVF energy obtained from the proposed edge image in Experiment 4. For all the experiments, we digitize the images manually to obtain the ground truth data necessary to compute precision and recall.
3
Results
We have obtained several parameters emperically. For splitting a snake, dsplit should be less than d. k to be chosen depending on how far the two splitting points should be to ensure that the snakes formed after splitting have at least k points. In order to ensure that merging of snakes takes place only among the arms with the semi-circular tips facing each other, the tangents at the high curvature points are checked for antiparallel threshold of 130π/180.. The compactness should be greater than 0.2 to ensure that linear structured contours are not deleted. 3.1
Experiment 1: Simple (Tree-Structured) Road Networks
A single quadratic snake is well suited for tree-structured road networks as the snake will not need to change its topology during evolution (Figure 2). A family of snakes enable faster and better road extraction as non-road regions are eliminated using splitting and deletion of snakes.
A Family of Quadratic Snakes for Road Extraction
91
Fig. 2. Evolution of quadratic snake on roads with tree structure. Each column displays an image with initial contour in red and the extracted road network below it.
Fig. 3. Evolution of quadratic snake on roads with loops and disconnected networks. Each column displays an image with initial contour in red and the extracted road network below it.
92
3.2
R. Marikhu et al.
Experiment 2: Road Networks with Single Loop and Multiple Disconnected Networks
The family of quadratic snakes are able to extract disconnected networks with high accuracy (Figure 3) but are not able to extract enclosed regions automatically as the snakes are not able to develop holes inside it in the form of growing snakes. 3.3
Experiment 3: Complex Road Networks
A road network is considered complex if it has multiple disconnected networks and enclosed regions and large number of branches. With the appropriate user initialization (Figure 4), the snakes are able to extract the road networks with high accuracy and in less time. 3.4
Experiment 4: GVF Energy to Enable Faster Evolution
The Gradient Vector Flow Field [8] boosts the evolution process as we can see from the number of iterations required for each evolution in Experiment 4 with and without the use of GVF energy. From the evolution in the fifth column, we see that the snake was able to extract the network with greater detail. Also, from the evolution in the last column, we see that it is necessary for the quadratic image energy to enable robust extraction and thus the GVF weight and λ need to be balanced appropriately.
Fig. 4. Evolution of quadratic snake on roads with enclosed regions. Each column displays an image with initial contour in green and the extracted road network below it.
A Family of Quadratic Snakes for Road Extraction
93
Fig. 5. Evolution of quadratic snake on roads with enclosed regions. Each column displays an image with initial contour in green and the extracted road network below it.
4
Discussion and Conclusion
In Experiment 1, we found that the our modified quadratic snake is able to move into concavities to extract entire tree-structured road networks with very high accuracy. Experiment 2 showed that the family of quadratic snakes is effective at handling changes in topology during evolution, enabling better extraction of road networks. Currently, loops cannot be extracted automatically. We demonstrated the difficulty in extracting complex road networks with multiple loops and grids in Experiment 3. However, user initialization of a family of contours enable extraction of multiple closed regions and help the snake to avoid road-like regions. The level set framework could be used to handle change in topology enabling effective extraction of enclosed regions. Rochery et al. [10] evolved the contour using the level set methods introduced by Osher and Sethian. However, our method is faster, conceptually simpler, and a direct extension of Kass et al.’s computational approach. In Experiment 4, we found that faster and robust extraction is achieved using oriented filtering and GVF energy along with image energy of the quadratic snakes. Our proposed edge image obtained from oriented filtering is effective for computing GVF energy to enhance the process of extraction. We also found that our method for obtaining the GVF outperforms standard methods. Finally, we have developed a complete GUI environment for satellite image manipulation and quadratic snake evolution, based on the Matlab platform. The system is freely available as open source software [9].
94
R. Marikhu et al.
Future work will focus on possibilities to automate the extraction of enclosed regions. Digital elevation models could be integrated with image energy for increased accuracy.
Acknowledgments This research was supported by Thailand Research Fund grant MRG4780209 to MND. RM was supported by a graduate fellowship from the Nepal High Level Commission for Information Technology.
References 1. Fischler, M., Tenenbaum, J., Wolf, H.: Detection of roads and linear structures in low-resolution aerial imagery using a multisource knowledge integration technique. Computer Graphics and Image Processing 15, 201–223 (1981) 2. Geman, D., Jedynak, B.: An active testing model for tracking roads in satellite images. IEEE Transactions on Pattern Analysis and Machine Intelligence 18(1), 1–14 (1996) 3. Fortier, A., Ziou, D., Armenakis, C., Wang, S.: Survey of work on road extraction in aerial and satellite images. Technical Report 241, Universit´e de Sherbrooke, Quebec, Canada (1999) 4. Kass, M., Witkin, A., Terzopoulos, D.: Snakes: Active contour models. International Journal of Computer Vision 1(4), 321–331 (1987) 5. Cohen, L.D., Cohen, I.: Finite-element methods for active contour models and balloons for 2-D and 3-D images. IEEE Transactions on Pattern Analysis and Machine Intelligence 15(11), 131–147 (1993) 6. Laptev, I., Mayer, H., Lindeberg, T., Eckstein, W., Steger, C., Baumgartner, A.: Automatic extraction of roads from aerial images based on scale space and snakes. Machine Vision and Applications 12(1), 23–31 (2000) 7. Rochery, M., Jermyn, I.H., Zerubia, J.: Higher order active contours. International Journal of Computer Vision 69(1), 27–42 (2006) 8. Xu, C., Prince, J.L.: Gradient Vector Flow: A new external force for snakes. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 66–71 (1997) 9. Marikhu, R.: A GUI environment for road extraction with quadratic snakes Matlab software (2007), available at http://www.cs.ait.ac.th/∼mdailey/snakes. 10. Rochery, M.: Contours actifs d’order sup´erieur et leur application ` a la d´etection de lin´eiques dans des images de t´el´ed´etection. PhD thesis, Universit´e de Nice, Sophia Antipolis — UFR Sciences (2005)
Multiperspective Distortion Correction Using Collineations Yuanyuan Ding and Jingyi Yu Department of Computer and Information Sciences University of Delaware Newark, DE 19716, USA {ding,yu}@eecis.udel.edu
Abstract. We present a new framework for correcting multiperspective distortions using collineations. A collineation describes the transformation between the images of a camera due to changes in sampling and image plane selection. We show that image distortions in many previous models of cameras can be effectively reduced via proper collineations. To correct distortions in a specific multiperspective camera, we develop an interactive system that allows users to select feature rays from the camera and position them at the desirable pixels. Our system then computes the optimal collineation to match the projections of these rays with the corresponding pixels. Experiments demonstrate that our system robustly corrects complex distortions without acquiring the scene geometry, and the resulting images appear nearly undistorted.
1
Introduction
A perspective image represents the spatial relationships of objects in a scene as they would appear from a single viewpoint. Recent developments have suggested that alternative multiperspective camera models [5,16] can combine what is seen from several viewpoints into a single image. These cameras provide potentially advantageous imaging systems for understanding the structure of observed scenes. However, they also exhibit multiperspective distortions such as the curving of lines, apparent stretching and shrinking, and duplicated projections of a single point [12,14]. In this paper, we present a new framework for correcting multiperspective distortions using collineations. A collineation describes the transformation between the images of a camera due to changes in sampling and image plane selection. We show that image distortions in many previous cameras can be effectively reduced via proper collineations. To correct distortions in a specific multiperspective camera, we develop an interactive system that allows users to select feature rays from the camera and position them at the desirable pixels. Our system then computes the optimal collineation to match the projections of these rays with the corresponding pixels. Compared with classical distortion correction methods [12,2,11], our approach does not require prior knowledge on scene geometry and it can handle highly Y. Yagi et al. (Eds.): ACCV 2007, Part I, LNCS 4843, pp. 95–105, 2007. c Springer-Verlag Berlin Heidelberg 2007
96
Y. Ding and J. Yu
complex distortions. We demonstrate the effectiveness of our technique on various synthetic and real multiperspective images, including the General Linear Cameras [14], catadioptric mirrors, and reflected images from arbitrary mirror surfaces. Experiments show that our method is robust and reliable, thus the resulting images appear nearly undistorted.
2
Previous Work
In recent years, there has been a growing interest in designing multiperspective cameras which capture rays from different viewpoints in space. These multiperspective cameras include pushbroom cameras [5], which collect rays along parallel planes from points swept along a linear trajectory, the cross-slit cameras [8,16], which collect all rays passing through two lines, and the oblique cameras [7], in which each pair of rays are oblique. The recently proposed General Linear Cameras (GLC) uniformly model these multiperspective cameras as 2D linear manifolds of rays (Fig. 1). GLCs produce easily interpretable images, which are also amenable to stereo analysis [9]. However, these images exhibit multiperspective distortions [14]. In computer vision, image-warping has been commonly used to reduce distortions. Image-warping computes an explicit pixel-to-pixel mapping to warp the original image onto a nearly perspective image. For cameras that roughly maintain a single viewpoint [6], simple parametric functions are sufficient to eliminate perspective, radial, and tangential distortions [2,3]. However, for complex imaging systems, especially those exhibiting severe caustic distortions [12], the warping function is difficult to model and may not have a closed-form solution. Image-based rendering algorithms have also been proposed to reduce image distortions [10,4]. There, the focus has been to estimate the scene structure from a single or multiple images. Swaminathan and Nayar [13] have shown that simple geometry proxies, such as the plane, sphere, and cylinder, are often sufficient to reduce caustic distortions on catadioptric mirrors, provided that the prior on scene structure is known. We present a third approach based on multiperspective collineations. A collineation describes the transformation between the images of a camera due to changes in sampling and image plane selection. For many multiperspective cameras such as the pushbroom [5] and the cross-slit [8], collineations can be uniformly modeled using the recently proposed General Linear Cameras (GLC) [15]. 2.1
GLC Collineation
In the GLC framework, every ray is parameterized by its intersections with the two parallel planes, where [u, v] is the intersection with the first and [s, t] the second, as shown in Fig. 1(a). This parametrization is often called a two-plane parametrization (2PP) [4,15]. We can reparameterize each ray by substituting σ = s − u and τ = t − v. In this paper, we will use this [σ, τ, u, v] parametrization to simplify our analysis. We also assume the default uv plane is at z = 0 and st plane at z = 1. Thus [σ, τ, 1] represents the direction of the ray.
Multiperspective Distortion Correction Using Collineations
97
r3 r2 t
(s2, t2)
(s3, t3)
(u2, v2)
s
r3
r1
(u3, v2) z
(s1, t1)
r3
r2
v
u
(u1, v1)
r2
r1
α⋅ r 1 + β ⋅ r 2 + (1 −α−β ) ⋅ r 3
(a)
Π1
r1
Π2
Π3
Π4
C
(b)
(c)
(d)
(e)
Fig. 1. General Linear Camera Models. (a) A GLC collects radiance along all possible affine combination of three rays. The rays are parameterized by their intersections with two parallel planes. The GLC model unifies many previous cameras, including the pinhole (b), the orthographic (c), the pushbroom (d), and the cross-slit (e).
A GLC is defined as the affine combination of three rays parameterized under 2PP: r = α[σ1 , τ1 , u1 , v1 ] + β[σ2 , τ2 , u2 , v2 ] + (1 − α − β)[σ3 , τ3 , u3 , v3 ], ∀α, β
(1)
Many well-known multiperspective cameras, such as pushbroom, cross-slit, linear oblique cameras are GLCs as shown in Fig. 1. If we assume uv is the image plane, we can further choose three special rays with [u, v] coordinates [0, 0], [1, 0], and [0, 1] to form a canonical GLC as: r[σ, τ, u, v] = (1 − α − β) · [σ1 , τ1 , 0, 0]+α · [σ2 , τ2 , 1, 0] + β · [σ3 , τ3 , 0, 1]
(2)
It is easy to see that α = u, β = v, and σ and τ are linear functions in u and v. Therefore, under the canonical form, every pixel [u, v] maps to a ray r(u, v) in the GLC. A GLC collineation maps every ray r(u, v) to a pixel [i, j] on the image plane Π[p, ˙ d1 , d2 ], where p˙ specifies the origin and d1 , and d2 specify the two spanning directions of Π. For every ray r[σ, τ, u, v], we can intersect r with Π to compute [i, j]: [u, v, 0] + λ[σ, τ, 1] = p˙ + id1 + jd2
(3)
Solving for i, j, and λ gives:
where
i=
y x z x (τ dz2 −dy 2 )(u−px )+(d2 −σd2 )(v−py )−(σd2 −τ d2 )pz γ
j=
y z z x x (dy 1 −τ d1 )(u−px )+(σd1 −d1 )(v−py )−(τ d1 −σd1 )pz γ
x x d1 d2 −σ y y γ = d1 d2 −τ dz1 dz2 −1
(4)
(5)
For a canonical GLC, since σ and τ are both linear functions in u and v, γ must be linear in u and v. Therefore, we can rewrite i and j as:
98
Y. Ding and J. Yu
i= j=
a1 u2 +b1 uv+c1 v 2 +d1 u+e1 v+f1 a3 u+b3 v+c3 2
(6)
2
a2 u +b2 uv+c2 v +d2 u+e2 v+f2 a3 u+b3 v+c3
˜ Π (u, v) of a GLC from the uv image plane to a new Thus, the collineation Col image plane Π is a quadratic rational function. Fig. 2 shows the images of a GLC under different collineations. It implies that image distortions may be reduced using a proper collineation. Π2
Π1 GLC
GLC
(a)
(b)
(c)
(d)
Fig. 2. The image of a cross-slit GLC (d) under collineation (c) appear much less distorted than the image (b) of the same camera under collineation (a)
3
Correct Distortions in GLCs
Given a specific GLC, our goal is to find the optimal collineation to minimize its distortions. Similar to previous approaches [12,11], we assume the rays captured by the camera are known. We have developed an interactive system to allow users to design their ideal undistorted images. Our system supports two modes. In the first mode, the user can select feature rays from the camera and position them at desirable pixels in the target images. In the second mode, the user can simply provide a reference perspective image. Our system then automatically matches the features points. Finally, the optimal collineation is estimated to fit the projections of the feature rays with the target pixels. 3.1
Interactive Distortion Correction
Given a canonical GLC, the user can first select n feature rays (blue crosses in Fig. 3(a)) from the source camera and then position them at desirable pixels (red crosses in Fig. 3(b)) on the target image. Denote [uk , vk ] as the uv coordinate of each selected ray rk in the camera and [ik , jk ] as the desired pixel coordinate ˙ d1 , d2 ] that of rk on the target image, we want to find the collineation Π[p, maps [u, v] as close to [i, j] as possible. We formalize it as a least squares fitting problem: n ˜ Π (uk , vk ) − [ik , jk ]||2 min ||Col (7) Π
k=1
Since each collineation Π[p, ˙ d1 , d2 ] has 9 variables, we need a minimal number of five ray-pixel pairs. This is not surprising because four pairs uniquely
Multiperspective Distortion Correction Using Collineations
99
determine a projective transformation, a degenerate collineation in the case of perspective cameras. Recall that the GLC collineations are quadratic rational functions. Thus, finding the optimal Π in Equation (7) requires using non-linear optimizations. To solve this problem, we use the Levenberg-Marquardt method. A common issue with the Levenberg-Marquardt method, however, is that the resulting optimum depends on the initial condition. To avoid getting trapped in a local minimum, we choose a near optimal initial condition by sampling different spanning directions of Π. We rewrite the spanning directions as: di = ηi · [cos(φi )cos(θi ), cos(φi )sin(θi ), sin(φi )],
i = 1, 2
(8)
We sample several θ1 , θ2 , φ1 , and φ2 and find the corresponding p, ˙ η1 , and η2 as the initial conditions. Finally, we choose the one with the minimum error. This preconditioned optimization robustly approximates a near optimal collineation that significantly reduces distortions as shown in Fig. 3(b).
(a)
(b)
Fig. 3. Interactive Distortion Correction. (a) The user selects feature rays (blue crosses) and positions them at desirable pixels (red crosses). (b) shows the new image under the optimal collineation. The distortions are significantly reduced. The green crosses illustrate the final projections of the feature rays.
3.2
Automatic Distortion Correction
We also present a simple algorithm to automatically reduce distortions. Our method consists of two steps. First, the user provides a target perspective image that captures the same scene. Next, we automatically select the matched features between the source camera and the target image and compute the optimal collineation by minimizing Equation (7). Recall that a GLC captures rays from different viewpoints in space and hence, its image may appear very different from a perspective image. To match the feature points, we use Scale Invariant Feature Transform (SIFT) to preprocess the two images. SIFT robustly handles image distortion and generates transformation-invariant features. We then perform global matching to find the potential matching pairs. Finally, we prune the outliers by using RANSAC with the homography model. To tolerate parallax, we use a loose inlier threshold of 20 pixels. In Fig. 4, we show our automatic distortion correction results on various GLCs including the pushbroom, the cross-slit, and the pencil cameras. The user inputs
100
Y. Ding and J. Yu Original Pushbroom
Original Cross-slit
Original Pencil
Target
(b) Corrected Pushbroom
(c) Corrected Cross-slit
(d) Corrected Pencil
(a)
(e)
(f)
(g)
Fig. 4. Automatic Distortion Correction. (a) Perspective reference image; (b),(c), and(d) are distorted images captured from pushbroom camera, cross-slit camera, and pencil camera. (e), (f), and (g) are the distortion corrected results of (b), (c), and (d) using the automatic algorithm.
a perspective image (Fig. 4(a)) and the corrected GLC images appear nearly undistorted using the optimal collineations (bottom row of Fig. 4).
4
Correcting Distortions on Catadioptric Mirrors
Next, we show how to correct multiperspective distortions on catadioptric mirrors. Conventional catadioptric mirrors place a pinhole camera at the focus of a hyperbolic or parabolic surface to synthesize a different pinhole camera with a wider field of view [6]. When the camera moves off the focus, the reflection images exhibit complex caustic distortions that are generally difficult to correct [12]. We apply a similar algorithm using multiperspective collineations. Our method is based on the observation that, given any arbitrary multiperspective imaging system that captures smoothly varying set of rays, we can map the rays onto a 2D ray manifold in the 4D ray space. The characteristics of this imaging system, such as its projection, collineation, and image distortions can be analyzed by the 2-D tangent ray planes, i.e., the GLCs [14]. This implies that a patch on an arbitrary multiperspective image can be locally approximated as a GLC. We first generalize the GLC collineation to arbitrary multiperspective imaging systems. Notice that not all rays in these systems can be parameterized as [σ, τ, u, v] (e.g., some rays may lie parallel to the parametrization plane). Thus, we use the origin o˙ and the direction l to represent each ray r. ˙ l] to a pixel [i, j] as: The collineation Π[p, ˙ d1 , d2 ] maps r[o, [ox , oy , oz ] + λ[lx , ly , lz ] = p˙ + id1 + jd2
(9)
Solving for i, j in Equation (9) gives: i=
x x z x x z y y x y y x z z (ly dz2 −lz dy 2 )(o −p )+(l d2 −l d2 )(o −p )+(l d2 −l d2 )(o −p ) γ∗
j=
y z x x x z z x y y y x x y z z (lz dy 1 −l d1 )(o −p )+(l d1 −l d1 )(o −p )+(l d1 −l d1 )(o −p ) γ∗
(10)
Multiperspective Distortion Correction Using Collineations
(a)
101
(b) (e)
(f) (c)
(d)
(g)
Fig. 5. Selecting different feature rays ((a) and (c)) produces different distortion correction results ((b) and (d)). (f) shows the automatic feature matching between a region (blue rectangle) on the spherical mirror and a perspective image. (g) is the final distortion corrected image. The holes are caused by the under-sampling of rays.
where
x x x d1 d2 −l ∗ γ = dy1 dy2 −ly dz1 dz2 −lz
(11)
˜ Π (o, ˙ l). We abbreviate Equation (10) as [i, j] = Col The user then selects n feature rays from the catadioptric mirror and positions them at target pixels [ik , jk ], k = 1 . . . n. Alternatively, they can provide a target perspective image (Fig. 5(f)) and our system will automatically establish feature correspondences using the SIFT-RANSAC algorithm. We then use the Levenberg-Marquardt method (equation (7)) with sampled initial conditions to ˜ Π. find the optimal collineation Col In the case of catadioptric mirrors, if the selected patch is too large, the resulting image may depend on which rays-pixel pairs are selected. In the kitchen scene example (Fig. 5(a)), selecting the rays from the right side of the spherical mirror produces different results than selecting the rays from the middle part, although distortions are reduced in both cases. This is because the rays inside the patch cannot be approximated as a single GLC model.
5
Results
We have experimented our system on various multiperspective images. We modify the PovRay [18] ray tracer to generate both GLC images and reflected images on catadioptric mirrors. Fig. 3 shows an image of a cross-slit camera in
102
Y. Ding and J. Yu
(b)
(c)
(a)
(d)
(e)
Fig. 6. Correcting distortions on a spherical mirror. The user selects separate regions on the sphere (a) to get (b) and (d). (c) and (e) are the resulting images by matching the selected features (blue) and target pixels (red) in (b) and (d) using collineations.
which the two slits form an acute angle. The user then selects feature rays (blue) from the GLC image and positions them at desirable pixels (red). Our system estimates the optimal collineation and re-renders the image under this collineation as shown in Fig. 3(b). The distortions in the resulting image are significantly reduced. Next, we apply our algorithm to correct reflection distortions on a spherical mirror shown in Fig. 6. It has been shown [14] that more severe distortions occur near the boundary of the mirror than at the center. Our algorithm robustly corrects both distortions in the center region and near the boundary. In particular, our method is able to correct the highly curved silhouettes of the refrigerator (Fig. 6(d)). The resulting images are rendered by intersecting the rays inside the patch with the collineation plane, thus, containing holes due to the undersampling of rays. Our algorithm can further correct highly complex distortions on arbitrary mirror surfaces. In Fig. 7, we render a reflective horse model of 48, 000 triangles at two different poses. Our system robustly corrects various distortions such as stretching, shrinking, and duplicated projections of scene points in the reflected image, and the resulting images appear nearly undistorted. We have also experimented our automatic correction algorithm on both the GLC models and catadioptric mirrors. In Fig. 4, the user inputs a target perspective image 4(a) and our system automatically matches the feature points between the GLC and the target image. Even though the ray structures in the GLCs are significantly different from a pinhole camera, the corrected GLC images appear close to perspective. In Fig. 5(f), a perspective image of a kitchen scene is used to automatically correct distortions on a spherical mirror. This
Multiperspective Distortion Correction Using Collineations
(a)
(b)
(c)
(d)
(e)
(f)
103
Fig. 7. Correcting complex distortions on a horse model. We render a reflective horse model under two different poses (a) and (d) and then select regions (b) and (e). (c) and (f) are the resulting images by matching the selected features (blue) and target pixels (red) in (b) and (e) using collineations.
(a)
(b)
(c)
(d)
Fig. 8. Correcting reflection distortions. (a) and (c) are two captured reflected images on a mirror sphere. Our algorithm not only reduces multiperspective distortions but also synthesizes strong perspective effects (b) and (d).
implies that our collineation framework has the potential for benefiting automatic catadioptric calibrations. Finally, we have applied our algorithm on real reflected images of a mirror sphere in a deep scene. We position the viewing camera far away from the sphere so that it can be approximated as an orthographic camera. We then calculate the corresponding reflected ray for each pixel and use our collineation algorithm to correct the distortions. Our system not only reduces multiperspective distortions but also synthesizes strong perspective effects as shown in Fig. 8.
104
6
Y. Ding and J. Yu
Discussions and Conclusion
We have presented a new framework for correcting multiperspective distortions using collineations. We have shown that image distortions in many previous cameras can be effectively reduced via proper collineations. To find the optimal collineation for a specific multiperspective camera, we have developed an interactive system that allows users to select feature rays from the camera and position them at the desirable pixels. Our system then computes the optimal collineation to match the projections of these rays with the corresponding pixels. Experiments demonstrate that our system robustly corrects complex distortions without acquiring the scene geometry, and the resulting images appear nearly undistorted.
(a)
(b)
(c)
(d)
Fig. 9. Comparing collineations with the projective transformation. The user selects feature rays (blue) and target pixels (red). (c) is the result using the optimal collineation. (d) is the result using the optimal projective transformation.
It is important to note that a collineation computes the mapping from a ray to a pixel whereas image warping computes the mapping from a pixel to a pixel. One limitation of using collineations is that we cannot compute the inverse mapping from pixels to rays. Therefore, if the rays in the source camera are undersampled, e.g., in the case of a fixed-resolution image of the catadioptric mirrors, the collineation algorithm produces images with holes. As for future work, we plan to explore using image-based rendering algorithms such as the push-pull method [4] to fill in the holes in the ray space. We have also compared our collineation method with the classical projective transformations. In Fig. 9, we select the same set of feature points (rays) from a reflected image on the horse model. Fig. 9(c) computes the optimal projective transformation and Fig. 9(d) computes the optimal collineation, both using the Levenberg-Marquardt method for fitting the feature points. The optimal collineation result is much less distorted and is highly consistent with the pinhole image while the projective transformation result remains distorted. This is because multiperspective collineation describes a much broader class of warping functions than the projective transformation.
Acknowledgement This work has been supported by the National Science Foundation under grant NSF-MSPA-MCS-0625931.
Multiperspective Distortion Correction Using Collineations
105
References 1. Chahl, J., Srinivasan, M.: Reflective surfaces for panoramic imaging. Applied Optics 37(8), 8275–8285 (1997) 2. Chen, S.E.: QuickTime VR – An Image-Based Approach to Virtual Environment Navigation. Computer Graphcs 29, 29–38 (1995) 3. Derrien, S., Konolige, K.: Approximating a single viewpoint in panoramic imaging devices. International Conference on Robotics and Automation, 3932–3939 (2000) 4. Gortler, S.J., Grzeszczuk, R., Szeliski, R., Cohen, M.F.: ’The Lumigraph. SIGGRAPH 1996, 43–54 (1996) 5. Gupta, R., Hartley, R.I.: Linear Pushbroom Cameras. IEEE Trans. Pattern Analysis and Machine Intelligence 19(9), 963–975 (1997) 6. Nayar, S.K.: Catadioptric Omnidirectional Cameras. In: Proc. CVPR, pp. 482–488 (1997) 7. Pajdla, T.: Stereo with Oblique Cameras. Int’l J. Computer Vision 47(1/2/3), 161–170 (2002) 8. Pajdla, T.: Geometry of Two-Slit Camera, Research Report CTU–CMP–2002–02, March (2002) 9. Seitz, S., Kim, J.: The Space of All Stereo Images. In: Proc. ICCV, pp. 26–33 (July 2001) 10. Shum, H., He, L.: Rendering with concentric mosaics. Computer Graphcs 33, 299– 306 (1999) 11. Stein, G.P.: Lens distortion calibration using point correspondences. In: Proc. CVPR, pp. 143–148 ( June 1997) 12. Swaminathan, R., Grossberg, M.D., Nayar, S.K.: Caustics of Catadioptric Cameras. In: Proc. ICCV, pp. 2–9 (2001) 13. Swaminathan, R., Grossberg, M.D., Nayar, S.K.: A Perspective on Distortions. In: Proc. IEEE Computer Vision and Pattern Recognition, Wisconsin (June 2003) 14. Yu, J., McMillan, L.: Multiperspective Projection and Collineation. In: Sebe, N., Lew, M.S., Huang, T.S. (eds.) Computer Vision in Human-Computer Interaction. LNCS, vol. 3766, Springer, Heidelberg (2005) 15. Yu, J., McMillan, L.: Modelling Reflections via Multiperspective Imaging. In: Proc. IEEE Computer Vision and Pattern Recognition, San Diego (June 2005) 16. Zomet, A., Feldman, D., Peleg, S., Weinshall, D.: Mosaicing New Views: The Crossed-Slits Projection. IEEE Trans. on PAMI, 741–754 (2003) 17. Zorin, D., Barr, A.H.: Correction of Geometric Perceptual Distortions in Pictures. Computer Graphics 29, 257–264 (1995) 18. POV-Ray: The Persistence of Vision Raytracer, http://www.povray.org/
Camera Calibration from Silhouettes Under Incomplete Circular Motion with a Constant Interval Angle Po-Hao Huang and Shang-Hong Lai Department of Computer Science, National Tsing Hua University, Hsinchu, Taiwan {even,lai}@cs.nthu.edu.tw
Abstract. In this paper, we propose an algorithm for camera calibration from silhouettes under circular motion with an unknown constant interval angle. Unlike previous silhouette-based methods based on surface of revolution, the proposed algorithm can be applied to sparse and incomplete image sequences. Under the assumption of circular motion with a constant interval angle, epipoles of successive image pairs remain constant and can be determined from silhouettes. A pair of epipoles formed by a certain interval angle can provide a constraint on the angle and focal length. With more pairs of epipoles recovered, the focal length can be determined from the one that most satisfies the constraints and determine the interval angle concurrently. The rest of camera parameters can be recovered from image invariants. Finally, the estimated parameters are optimized by minimizing the epipolar tangency constraints. Experimental results on both synthetic and real images are shown to demonstrate its performance. Keywords: Circular Motion, Camera Calibration, Shape Reconstruction.
1 Introduction Reconstructing 3D model from image sequences has been studied for decades [1]. In real applications, for instance 3D object digitization in digital museum, modeling from circular motion sequences is a practical and widely used approach in computer vision and computer graphic communities. Numerous methods, which focus on circular motion, have been proposed and they can be classified into two camps; namely, the feature-based [2,3,4,5] and silhouette-based [6,7,8,9] approaches. In the feature-based approaches, Fitzgibbon et al. [2] proposed a method that makes use of the fundamental matrices and trifocal tensors to uniquely determine the rotation angles and determine the reconstruction up to a two-parameter family. Jiang et al. [3,4] further developed a method that avoids the computation of multiview tensors to recover the circular motion geometry by either fitting conics to tracked points in at least five images or computing a plane homography from minimally two points in four images. Cao et al. [5] aimed at the problem of varying focal lengths under circular motion with a constant but unknown rotation angle. However, it is difficult to establish accurate feature correspondences from the image sequences for objects of texture-less, semi-transparent, or reflective materials, such as jade. Instead of feature correspondences, the silhouette-based approach Y. Yagi et al. (Eds.): ACCV 2007, Part I, LNCS 4843, pp. 106–115, 2007. © Springer-Verlag Berlin Heidelberg 2007
Camera Calibration from Silhouettes Under Incomplete Circular Motion
107
integrates the object contours to recover the 3D geometry. In [6], Mendonca and Cipolla addressed the problem of estimating the epipolar geometry from apparent contours under circular motion. Under the assumption of constant rotation angle, there are only two common epipoles of successive image pairs needed to be determined. The relation between the epipolar tangencies and the image of rotation axis is used to define a cost function. Nevertheless, the initialization of the epipole positions can influence the final result and make the algorithm converge to a local minimum. In [7], Mendonca et al. exploited the symmetry properties of the surface of revolution (SoR) swept out by the rotating object to obtain an initial guess of image invariants, followed by several one-dimensional searching steps to recover the epipolar geometry. Zhang et al. [8] further extended this method to achieve auto-calibration. The rotation angle is estimated from three views which sometimes results in inaccurate estimation. In [9], they formulated the circular motion as 1D camera geometry to achieve more robust motion estimation. Most of the silhouette-based methods are based on the SoR to obtain an initial guess of image invariants, thus making them infeasible when the image sequence is sparse (interval angle larger than 20 degree [7]) or incomplete. In this paper, we propose an algorithm for camera calibration from silhouettes of an object under circular motion with a sparse and incomplete sequence. In our approach, we first use the same cost function, as proposed in [6], to determine the epipoles of successive image pairs from silhouettes. Thus, constant interval angle is the main assumption of our algorithm. In addition, we propose a method for initializing the positions of epipoles, which is important in practice. A pair of epipoles formed by a certain interval angle can provide a constraint on the angle and focal length. With more pairs of epipoles recovered, the focal length can therefore be determined as the one that best satisfies these constraints and the angle is determined concurrently. After obtaining the camera intrinsic parameters, the rotation matrix about camera center can be recovered from the image invariants up to two sign ambiguities which can be further resolved by making the sign of back-projection epipoles consistent with the camera coordinate. Finally, the epipolar tangency constraints for all pairs of views are minimized to refine the camera parameters by using all determined parameters as an initial guess in the nonlinear optimization process. The remainder of this paper is organized as follows. Section 2 describes the image invariants under circular motion. Section 3 describes the epipolar tangency constraints and explains how to extract epipoles from contours. The estimation of camera parameters is described in section 4. Experimental results on both synthetic and real data are given in section 5. Finally, we conclude this paper in section 6.
2 Image Invariants Under Circular Motion The geometry of circular motion can be illustrated in Fig. 1(a). A camera C rotates about an axis Ls, and its track forms a circle on a plane Πh that is perpendicular to Ls and intersects at the circle center Xs. Without loss of generality, we assume the world coordinate system be centered at Xs with Y-axis along Ls and C be placed on the negative Z-axis. If the camera parameters are kept constant under circular motion, the
108
P.-H. Huang and S.-H. Lai
Ls Πi
ls
Z Πh C
Xs
Πi
X
lh
xs
vx
vy
Y (a)
(b)
Fig. 1. (a) The geometry of circular motion. (b) The image invariants under circular motion.
image Πi of C will contain invariant entities of the geometry as shown in Fig. 1(b). Line lh (ls) is the projection of Πh(Ls). The three points, vx, vy, and xs, are the vanishing points of X-axis, Y-axis and Z-axis, respectively. Similar description of the image invariants can also be found in [2,3,4,8]. Assume the camera intrinsic parameters and the rotation matrix about camera center, which will be referred as camera pose in the rest of this paper, be denoted as K and R, respectively. The camera projection matrix P can be written as:
P = KR [ R y (θ ) | − C ] .
(1)
where R= [r1 r2 r3], Ry(θ) is the rotation matrix about Ls with angle θ, and C=[0 0 –t]T. In mathematical expression, the three points can be written as:
[v x
vy
xs ] ~ KR = K [r1
r2
r3 ] .
(2)
where the symbol “~” denotes the equivalence relation in the homogenous coordinate.
3 Epipolar Geometry from Silhouettes Epipoles can be obtained by computing the null vectors of the fundamental matrix when feature correspondences between two views are available. However, from only silhouettes, it takes more efforts to determine the epipoles. In this section, the relationship between the epipoles and the silhouettes are discussed. 3.1 Constraints on Epipoles and Silhouettes Under Circular Motion In two-view geometry, frontier point is the intersection of contour generators and its projection will be located at the epipolar tangency of the object contour as shown in Fig. 2(a). Hence, the tangent points (lines) induced by epipoles can be regarded as corresponding points (epipolar lines). In addition, as mentioned in [6], under circular motion, the intersection of the corresponding epipolar lines will lie on ls when two views are put in the same image, as shown in Fig. 2(b). This property provides constraints on epipoles and silhouettes. In [6], the cost function is defined as the distance between the intersections of corresponding epipolar
Camera Calibration from Silhouettes Under Incomplete Circular Motion
ls
frontier point contour generator
l1 t1 x2
lh e2
e1 contour C2 C1
e1
e2
contour C1
t2
109
l2 lh
x1 e2
e1
contour C1
contour C2
C2
(a)
(b)
(c)
Fig. 2. (a) Frontier point and the epipolar tangencies. (b) Epipolar tangencies and ls under circular motion (c) Epipolar tangency constraints.
lines and ls. In general, a pair of views has two epipoles with four unknowns but provides only two constraints (intersections), which are not enough to unique determine the answer. Therefore, they assume the interval angle of adjacent views is kept constant, thus reducing the number of epipoles needed to be estimated to only two with four unknowns. Given the epipoles, ls can be determined by line fitting the intersections. In their method, with appropriate initialization of epipoles, the cost function is iteratively minimized to determine the epipoles. (see [6] for details). 3.2 Initialization of Epipoles In [6], they only showed experiments on synthetic data. In practice, it is crucial to obtain good initial positions of epipoles. In our algorithm, the assumption of constant interval angle is also adopted to reduce the unknowns. When taking an image sequence under the turn-table environment, the camera pose is usually close to the form R=Rz(0)Ry(0)Rx(θx)=Rx(θx), therefore the harmonic homography derived in [7] can be reduced to bilateral symmetry as follows: ⎡− 1 0 2u0 ⎤ W = I − 2Kr r K = ⎢⎢ 0 1 0 ⎥⎥ . ⎢⎣ 0 0 1 ⎥⎦ T 11
−1
(3)
where r1=[1 0 0]T and u0 is the x-coordinate of optical center. Assume the optical center coincide with the image center, we can obtain a roughly harmonic homography from (3). Using this harmonic homography, the initial position of epipoles can be obtained from the epipole estimation step as described in [7]. In fact, given contours C1 and C2, and a harmonic homography W, the corresponding epipole e1 (e2) can be directly located from the bi-tangent lines of the contours WC1 and C2 (C1 and WC2) without performing several one-dimensional searching steps as in [7]. Note that here WC means the contour C is transformed by W. 3.3 Epipolar Tangency Constraints In the silhouette-based approach, the most common energy function to measure the model is the epipolar tangency constraints, which can be illustrated in Fig. 2(c). In
110
P.-H. Huang and S.-H. Lai
Fig. 2(c), a pair of contours is put on the same image. The epipole e1(e2) is the projection of C1 (C2) onto the camera C2 (C1), and x1 (x2) is the tangent point induced by epipole e2 (e1) with tangent line t1 (t2). As mentioned in section 3.1, the tangent points x1 and x2 are considered as the correspondence points. The dashed line l1 (l2) is the corresponding epipolar line of x2 (x1). Ideally, l1 (l2) should be the same as t1 (t2). Assume the projection matrix of camera C1 (C2) be denoted as P1 (P2). The error associated with the epipolar tangency constraints can be written as:
err(x1 , x2 , P1 , P2 ) = d (x1 , l1 ) + d ( x2 , l2 ) .
(4)
where the function d(.,.) gives the Euclidean distance from a point to a line, + l1 = P1 P2+ x2 × e2 , l2 = P2 P1+ x1 × e1 , and P is the pseudo-inverse of the projection matrix. Given a set of silhouettes S and its corresponding projection matrices P, the overall cost function can be written as:
Cost (P, S ) =
∑ err (x , x , P , P ) .
∑
(Si ,S j )∈S p ( xa , xb )∈Tpi , j
a
b
i
(5)
j
where the set Sp contains all contour pairs that are used in the cost function, and Tpi,j is the set of tangent points induced by epipoles (of Pi and Pj) with contours (Si and Sj).
4 Camera Calibration In the previous section, the method to extract epipoles from silhouettes under circular motion with a constant interval angle is presented. In this section, we describe how to compute the camera parameters from epipoles. For simplification, the camera is assumed to be zero skew, unit aspect ratio and principle point at the image center, which is a reasonable assumption for current cameras. 4.1 Recovery of Focal Length and Interval Angle The geometry of circular motion under a constant interval angle can be illustrated in Fig. 3(a). In Fig. 3(a), Xs is the circle center, cameras distribute on the circle with a constant interval angle θ. With a certain interval angle, a pair of determined epipoles Xs θ C1
θ θ
θ
optical center
Πh
e2 C5
e1 e5
e4
C4
C2 C3 (a)
Πi
C3 (b)
Fig. 3. (a) Circular motion with a constant interval angle. (b) The image of one camera.
Camera Calibration from Silhouettes Under Incomplete Circular Motion
111
can provide a constraint on the angle and focal length. For instance, for the image of C3 as shown in Fig. 3(b), epipoles formed by the angle θ are e2 and e4. Therefore, we can derive a relationship of the angle and focal length with epipoles as follows:
θ = π − ang(K −1e2 , K −1e4 ) .
(6)
where the function ang(.,.) gives the angle between two vectors. In equation (6), we have one constraint but two unknowns, which are the angle θ and the focal length, the constraint is not enough to determine the unknowns. Recall that, the image sequence is taken under a constant interval angle. Take Fig. 3(a) for example, C1-C2-C3-C4-C5 is an image sequence with a constant interval angle θ. Also, C1-C3-C5 is an image sequence with a constant interval angle 2θ, which can provide another constraint as following:
(
)
2θ = π − ang K −1e1 , K −1e5 .
(7)
Two pairs of epipoles are sufficient to determine the unknowns. With more pairs of epipoles formed by different interval angles to be recovered, more constraints similar to equation (6) and (7) can be applied to precisely determine the focal length. In our implementation, a linear search on the angle θ is performed to find the one that best satisfies these constraints. For instance, given an angle θ, the focal length can be determined by solving a quadratic equation from equation (6). Substituting the estimated focal length into right-hand side of equation (7), we have the difference between left-hand side and right-hand side of equation (7). Therefore, the interval angle and focal length are determined as the one that best satisfies these constraints. 4.2 Recovery of Image Invariants From the extracted epipoles, lh can be computed by line fitting these epipoles, and also ls can be determined concurrently in the epipole extraction stage. Then, xs is the intersection of lh and ls. After camera intrinsic parameter K is obtained, vx can be computed from the pole-polar relationship [1], i.e. vx~KKTls. 4.3 Recovery of Camera Pose From equation (2), with the camera intrinsic parameters and image invariants known, the camera pose R can be computed up to two sign ambiguities as follows:
( ) . = β × norm (K x ) , β = ±1
r1 = α × norm K −1v x , α = ±1 r3
−1
s
(8)
r2 = r3 × r1
where the function norm(.) normalizes a vector to unit norm. Notice that, the sign of rotation axis has no difference for projection to the image coordinates, but back-projection of image points will lead to a sign ambiguity. This ambiguity can be resolved by back-projecting the epipole, which is obtained from image, and checking the sign with the corresponding camera position which is transformed to the camera coordinate system from the world coordinate system by using the determined camera pose R. Because the camera position in the world
112
P.-H. Huang and S.-H. Lai
coordinate system is irrelevant to camera pose R, we still can recover the camera position in the presence of rotation ambiguity. Furthermore, the Gram-Schmidt process is applied to obtain the orthogonal basis. 4.4 Summarization of the Proposed Algorithm INPUT: n object contours, from S1 to Sn, under circular motion with an unknown constant interval angle. OUTPUT: Camera parameters and 3D model. 1. Choose a frame interval Δv, the contours Sv and Sv+Δv are considered as contour pair for determining the two epipoles formed by the interval, where v=1...(n-Δv). 2. Initialize the two common epipoles by the method described in section 3.2. 3. Extract epipoles with a nonlinear minimization step as described in section 3.1. 4. Choose different frame intervals and repeat step 1-3 to extract more epipoles. 5. Use the epipoles extracted from step 1-4 as initial guesses and perform the nonlinear minimization step again to uniquely determine the ls. 6. Recover camera parameters as mentioned in section 4. 7. Set the projection matrices according to equation (1) as initial guesses. 8. Minimize the overall epipolar tangency errors as described in section 3.3. 9. Generate the 3D model using the image-based visual hull technique [10].
5 Experimental Results In this section, we show some experimental results on both synthetic and real data sets of applying the proposed silhouette-based algorithm to reconstruct 3D object models from sparse and incomplete image sequences under circular motion.
(a)
(b)
Fig. 4. (a) Experimental images for reconstructing the bunny model. (b) Intersections of corresponding epipolar lines and the estimated rotation axis ls before and after minimization. Table 1. Accuracy of the recovered camera parameters error avg. std.
Δe (%) 0.565 0.270
Δθx (o) 0.301 0.315
Δθy (o) 0.046 0.038
Δθz (o) 0.080 0.059
Δθi (o) 0.279 0.244
Δf (%) 1.822 1.484
Camera Calibration from Silhouettes Under Incomplete Circular Motion
113
5.1 Synthetic Data In this part, we used the Stanford bunny model to randomly generate 100 synthetic data sets to test the algorithm. Each set contain 12 images of size 800x600 pixels with interval angle, θi = 30o, which means each sequence is sparsely located and the methods based on SoR will fail. Example images of one set are depicted in Fig. 4(a). The generated range of focal length, f, is 1500~5000 pixels, three angles, θx, θy, and θz, of camera pose are within -10o~-50o, -5o~5o, and -5o~5o, respectively. Two different frame intervals, 2 and 3, are chosen to extract the epipoles. The comparison of the recovered parameters and the ground truth are listed in Table 1. In Table 1, the error of angles is in degrees. The error of focal length is expressed as percentage, which is the difference divided by the ground truth. The error of the epipoles is expressed as the difference divided by the length of ground truth to the image center. The experimental results show that the proposed algorithm can provide a good initial guess for the camera parameter optimization. Fig. 4(b) shows an example result of the epipole extraction stage before and after iteratively minimization. The dashed line is the initial ls according to ‘x’ points, which are the intersections induced by the initial epipoles as described in Fig. 2(b). The solid line is the estimated ls after minimization and the intersections (‘o’ points) are close to ls. The obtained ls is very close to the ground truth, expressed by dashed-dot line, as shown in the enlarged figure. 5.2 Real Data In the experiments on real data, two image sequences are used. Example images are shown in Fig. 5. Fig. 5(top) is the Oxford dinosaur sequence, which contains 36 images of size 720x576 pixels. Fig. 5(down) is a sequence of jadeite that contains 36 images of size 2000x1303 pixels and it is very difficult to establish feature correspondences for this kind of material. In both sequences, only the silhouette information is used for reconstruction. Different views of the reconstructed models are shown in Fig. 6 and Fig. 7, respectively. After overall optimization, the RMS errors of the recovered interval angles in two sequences are 0.192o and 0.247o, respectively. In addition, when only the first 18 images of the sequence used, which means the image sequence is incomplete, the estimated results are similar. Due to space limitation, we cannot give the details of the experimental results.
Fig. 5. Example images of (top) the Oxford dinosaur sequence and (down) the jadeite sequence
114
P.-H. Huang and S.-H. Lai
Fig. 6. Different views of the reconstructed Oxford dinosaur model
Fig. 7. Different views of the reconstructed jadeite model
6 Conclusion In this paper, we propose a novel silhouette-based algorithm for camera calibration and 3D reconstruction from sparse and incomplete image sequences of objects under circular motion with an unknown but constant interval angle. Different from previous silhouette-based methods, the proposed algorithm does not require either dense image sequences or the assumption of known camera intrinsic parameters in advance. Under the assumption of constant interval angle, the epipoles of successive images are kept constant and can be determined from silhouettes by a nonlinear optimization process. With more pairs of epipoles recovered from silhouettes, constraints on the interval angle and focal length can be provided to determine the camera parameters. Experimental results on synthetic and real data sets are presented to demonstrate the performance of the proposed algorithm. Acknowledgments. This work was supported by National Science Council, Taiwan, under the grant NSC 95-2221-E-007-224.
References 1. Hartley, R., Zisserman, A.: Multiple View Geometry in Computer Vision. Cambridge University Press, Cambridge (2000) 2. Fitzgibbon, A.W., Cross, G., Zisserman, A.: Automatic 3D Model Construction for TurnTable Sequences. In: Proceedings of SMILE Workshop on 3D Structure from Multiple Images of Large-Scale Environments, pp. 155–170 (1998)
Camera Calibration from Silhouettes Under Incomplete Circular Motion
115
3. Jiang, G., Tsui, H.T., Quan, L., Zisserman, A.: Single Axis Geometry by Fitting Conics. IEEE Transactions on Pattern Analysis and Machine Intelligence 25, 1343–1348 (2002) 4. Jiang, G., Quan, L., Tsui, H.T.: Circular Motion Geometry Using Minimal Data. IEEE Transactions on Pattern Analysis and Machine Intelligence 26, 721–731 (2004) 5. Cao, X., Xiao, J., Foroosh, H., Shah, M.: Self-calibration from Turn-table Sequences in Presence of Zoom and Focus. Computer Vision and Image Understanding 102, 227–237 (2006) 6. Mendonca, P.R.S., Cipolla, R.: Estimation of Epipolar Geometry from Apparent Contours: Affine and Circular Motion Cases. In: Proceedings of Computer Vision and Pattern Recognition, pp. 9–14 (1999) 7. Mendonca, P.R.S., Wong, K.-Y.K., Cipolla, R.: Epipolar Geometry from Profiles under Circular Motion. IEEE Transactions on Pattern Analysis and Machine Intelligence 23, 604–616 (2001) 8. Zhang, H., Zhang, G., Wong, K.-Y.K.: Auto-Calibration and Motion Recovery from Silhouettes for Turntable Sequences. In: Proceedings of British Machine Vision Conference, pp. 79–88 (2005) 9. Zhang, G., Zhang, H., Wong, K.-Y.K.: 1D Camera Geometry and Its Application to Circular Motion Estimation. In: Proceedings of British Machine Vision Conference, pp. 67–76 (2006) 10. Matusik, W., Buehler, C., Raskar, R., Gortler, S.J., McMillan, L.: Image-Based Visual Hulls. In: Proceedings of SIGGRAPH, pp. 369–374 (2000)
Mirror Localization for Catadioptric Imaging System by Observing Parallel Light Pairs Ryusuke Sagawa, Nobuya Aoki, and Yasushi Yagi Institute of Scientific and Industrial Research, Osaka University, 8-1 Mihogaoka, Ibaraki-shi, Osaka, 567-0047, Japan {sagawa,aoki,yagi}@am.sanken.osaka-u.ac.jp
Abstract. This paper describes a method of mirror localization to calibrate a catadioptric imaging system. While the calibration of a catadioptric system includes the estimation of various parameters, we focus on the localization of the mirror. The proposed method estimates the position of the mirror by observing pairs of parallel lights, which are projected from various directions. Although some earlier methods for calibrating catadioptric systems assume that the system is single viewpoint, which is a strong restriction on the position and shape of the mirror, our method does not restrict the position and shape of the mirror. Since the constraint used by the proposed method is that the relative angle of two parallel lights is constant with respect to the rigid transformation of the imaging system, we can omit both the translation and rotation between the camera and calibration objects from the parameters to be estimated. Therefore, the estimation of the mirror position by the proposed method is independent of the extrinsic parameters of a camera. We compute the error between the model of the mirror and the measurements, and then estimate the position of the mirror by minimizing this error. We test our method using both simulation and real experiments, and evaluate the accuracy thereof.
1 Introduction For various applications, e.g. robot navigation, surveillance and virtual reality, a special field of view is desirable to accomplish the task. For example, omnidirectional imaging systems [1,2,3] are widely used in various applications. One of the main methods to obtain a special field of view, is to construct a catadioptric imaging system, which observes rays reflected by mirrors. By using various shapes of mirrors, different fields of view are easily obtained. There are two types of catadioptric imaging systems; central and noncentral. The former has a single effective viewpoint, and the latter has multiple ones. Though central catadioptric systems have an advantage in that the image can be transformed to a perspective projection image, they have strong restrictions on the shape and position of the mirror. For example, it is necessary to use a telecentric camera and a parabolic mirror whose axis is parallel to the axis of the camera. Thus, misconfiguration can be the reason that a catadioptric system is not a central one. To obtain more flexible fields of view, several noncentral systems [4,5,6,7,8] have been proposed for various purposes. For geometric analysis with catadioptric systems, it is necessary to calibrate both camera and mirror parameters. Several methods of calibration have been proposed for Y. Yagi et al. (Eds.): ACCV 2007, Part I, LNCS 4843, pp. 116–126, 2007. c Springer-Verlag Berlin Heidelberg 2007
Mirror Localization for Catadioptric Imaging System
117
central catadioptric systems. Geyer and Daniilidis [9] have used three lines to estimate the focal length, mirror center, etc. Ying and Hu [10] have used lines and spheres to calibrate the parameters. Mei and Rives [11] have used a planar marker to calibrate the parameters, which is based on the calibration of a perspective camera [12]. However, since these methods assume that the system has a single viewpoint, they cannot be applied to noncentral systems. On the other hand, several methods have also been proposed to calibrate noncentral imaging systems. Aliaga [13] has estimated the parameters of a catadioptric system with a perspective camera and a parabolic mirror using known 3D points. Strelow et al. [14] have estimated the position of a misaligned mirror using known 3D points. Micus´ık and Pajdla [15] have fitted an ellipse to the contour of the mirror and calibrated a noncentral camera by approximating it to a central camera. Mashita et al. [16] have used the boundary of a hyperboloidal mirror to estimate the position of a misaligned mirror. However, all of these methods are restricted to omnidirectional catadioptric systems. There are also some approaches for calibrating more general imaging systems. Swaminathan et al. [17] computed the parameters of noncentral catadioptric systems by estimating a caustic surface from known camera motion and the point correspondences of unknown scene points. Grossberg and Nayar [18] proposed a general imaging model and computed the ray direction for each pixel using two planes. Sturm and Ramalingam [19] calibrated the camera of a general imaging model by using unknown camera motion and a known object. Since these methods estimate both the internal and external parameters of the system, the error of measurement affects the estimated result of all of the parameters. In this paper, we focus on the localization of the mirror in the calibration of catadioptric systems. Assumptions of the other parameters are as follows: – The intrinsic parameters, such as the focal length and principal point of a camera, are known. – The shape of the mirror is known. The only remaining parameters to be estimated are the translation and rotation of the mirror with respect to the camera. If we calibrate the parameters of an imaging system by observing some markers, it is necessary to estimate the extrinsic parameters, such as rotation and translation, with respect to the marker. If we include these parameters as parameters to be estimated, the calibration results are affected by them. We proposed a method to localize a mirror by observing a parallel light [20] that estimates the mirror parameters independently of the extrinsic parameters. Since the translation between a marker and a camera is omitted from the estimation, this method can reduce the number of parameters. The method however, needs a rotation table to observe a parallel light from various directions. Instead of using a rotation table, the method proposed in this paper observes pairs of parallel lights as calibration markers. We can therefore, omit both rotation and translation from the estimation and reduce the number of parameters that are affected by the measurement error in the calibration. We describe the geometry of projection of two parallel lights in Section 2. Next, we propose an algorithm for mirror localization using pairs of parallel lights in Section 3. We test our method in Section 4 and finally summarize this paper in Section 5.
118
R. Sagawa, N. Aoki, and Y. Yagi Mirror surface x’
Parallel light
x
Mirror surface
x1
v
O
x2’
v1
v2’
v1’
NR,t (x) Image plane
x1’
x2
Image plane
m Camera
m1
m2 O
Fig. 1. Projecting a parallel light onto a catadioptric imaging system
m’1
v2 m’2
O’
Fig. 2. Projecting a pair of parallel lights with two different camera positions and orientations
2 Projecting a Pair of Parallel Lights onto a Catadioptric Imaging System In this section, we first explain the projection of a parallel light, which depends only on the rotation of a camera. Next, we describe the projection of a pair of parallel lights and the constraint on the relative angle between them. 2.1 Projecting a Parallel Light First, we explain the projection of a parallel light. Figure 1 shows the projection of a parallel light onto a catadioptric system. Since a parallel light is not a single ray, but a bunch of parallel rays, such as sunlight, it illuminates the whole catadioptric system. v is the vector of the incident parallel light. m is the vector at the point onto which the light is projected. m is computed as follows: ˆ, m = K −1 p
(1)
ˆ = (px , py , 1) is the point onto which the light is projected in the homogeneous where p image coordinate system. K is a 3×3 matrix that represents the intrinsic parameters of the camera. Although the incident light is reflected at every point on the mirror surface where the mirror is illuminated, the reflected light must go through the origin of the camera to be observed. Since the angle of the incident light is the same as that of the reflected light, the camera only observes the ray reflected at a point x. Therefore, the equation of projection becomes −v =
m m + 2(NR,t (x) · )N (x), m m R,t
(2)
where NR,t (x) is the normal vector of the mirror surface at the point x. R and t are the rotation and translation, respectively, of the mirror relative to the camera. 2.2 Projecting a Pair of Parallel Lights Since the direction of the incident parallel light is invariant even if it is observed from different camera positions, the direction of the light relative to the camera depends only
Mirror Localization for Catadioptric Imaging System
119
on the orientation of the camera. Now, if we observe two parallel lights simultaneously, the relative angle between these parallel lights does not change irrespective of the camera orientation. Figure 2 shows a situation, in which a pair of parallel lights is projected onto a catadioptric system, and which has two different camera positions and orientations. The relative position of the mirror is fixed to the camera. The two parallel lights are reflected at the points x1 , x2 , x2 and x1 , respectively. The reflected rays are projected onto the points m1 , m2 , m2 and m1 in the image plane, respectively. Since the relative angle between the pair of parallel lights is invariant, we obtain the following constraint: (3) v 1 · v 2 = v 1 · v 2 , where v 1 and v 2 are represented in a different camera coordinate system from v 1 and v 2 , which are computed by (2).
3 Mirror Localization Using Pairs of Parallel Lights This section describes an algorithm to estimate mirror position by observing pairs of parallel lights. 3.1 Estimating Mirror Position by Minimizing Relative Angle Error By using the constraint (3), we estimate the mirror position by minimizing the following cost function: v i1 · v i2 − cos αi 2 , (4) E1 = i
where i is the number of the pair and αi is the angle of the i-th pair. If we do not know the angle between the parallel lights, we can use E2 =
v i1 · v i2 − v j1 · v j1 2 .
(5)
i=j
The parameters of these cost functions are R and t, which are the rotation and translation, respectively, of the mirror relative to the camera. Since minimizing (4) or (5) is a nonlinear minimization problem, we estimate R, t and RC by a nonlinear minimization method, such as the Levenberg-Marquardt algorithm. Our algorithm can then be described as follows: 1. 2. 3. 4. 5. 6. 7.
Set initial parameters of R and t. Compute the intersecting point x for each image point m. Compute the normal vector NR,t (x) for each intersecting point x. Compute the incident vector v for each intersecting point x. Compute the cost function (4) or (5). Update R and t by a nonlinear minimization method. Repeat steps 2-6 until convergence.
120
R. Sagawa, N. Aoki, and Y. Yagi
Concave parabolic mirror Collimator 1
Collimator 2
Light source
Pinhole
Fig. 3. Two collimators generate a pair of parallel lights. Each collimator consists of a light source, a pinhole and a concave parabolic mirror.
In the current implementation, the initial parameters are given by user. We set them so that the every image point m has the intersecting point x. As described in Section 3.2, computing the intersecting points is high cost if a mirror surface is represented by a mesh model. Therefore, we describe a GPU-based method for steps 2-4 to directly compute the incident vectors to reduce the computational time. For updating the parameters, we numerically compute the derivatives required in the Levenberg-Marquardt algorithm. To keep so that every image point has the intersecting point, if an image point has no intersecting point, we penalize it with a large value instead of computing (4) or (5). 3.2 Computing the Incident Vector The important step in this algorithm is the computation of the incident vector v, for which there are two methods. The first of these computes x by solving a system of equations. If the mirror surface is represented as a parametric surface, x is obtained by simultaneously solving the equations of the viewing ray and the mirror surface, because the intersecting point x is on both the viewing ray and the mirror surface. Once x is computed, the normal vector NR,t (x) is obtained by the cross product of two tangential vectors of the mirror surface at x, and then the incident vector v is computed by (2). However, it is high cost to solve the simultaneous equations if the mirror surface is an intricate shape or non-parametric surface. If a mirror surface is represented as a mesh model, it is necessary to search the intersecting point for each image point by solving the equations for each facet of the model. To accommodate any mirror shape, the second method computes x by projecting the mirror shape onto the image plane of the camera with R, t and the intrinsic parameter K. Since this operation is equivalent to rendering the mirror shape onto the image plane, it can be executed easily using computer graphics techniques if the mirror shape is approximated by a mesh model. Furthermore, if we use recent graphics techniques, the incident vector v is computed directly by the rendering process. The source code to compute v for every pixel is shown in Appendix A. 3.3 Generating a Pair of Parallel Lights Our proposed method requires observation of parallel lights. A parallel light can be viewed by adopting one of the following two approaches: – Use a feature point of a distant marker. – Generate a collimated light.
Mirror Localization for Catadioptric Imaging System
121
In the former approach, a small translation of camera motion can be ignored because it is much smaller than the distance to the marker. Thus, the ray vector from the feature point is invariant even if the camera moves. The issue of this approach is a lens focus problem. When the focus setting of the camera is not at infinite focus, the image is obtained with a minimum aperture and long shutter time to avoid a blurred image. Instead of using distant points to obtain two parallel lights, vanishing points can be used. Some methods [21,22,23] was proposed for the calibration of a perspective camera. In the latter approach, a parallel light is generated by a collimator. A simple method is to use a concave parabolic mirror and a point-light source. Figure 3 shows an example of such a system. By placing pinholes in front of the light sources, they become point-light sources. Since pinholes are placed at the focus of the parabolic mirrors, the reflected rays are parallel. The illuminated area is indicated in yellow in the figure. The advantage of this approach is that a small and precise system can be constructed although optical apparatus is required.
4 Experiments 4.1 Estimating Accuracy by Simulation We first evaluate the accuracy of our method by simulation. In this simulation, we estimate the position of a parabolic mirror relative to a perspective camera. The intrinsic parameter K of the perspective camera is represented as ⎞ ⎛ f 0 cx (6) K = ⎝ 0 f cy ⎠ . 00 1 1 The shape of the mirror is represented as z = 2h (x2 + y 2 ), where h is the radius of a paraboloid. In this experiment, the image size is 512×512 pixels and f = 900, cx = cy = 255 and h = 9.0. The ground truths of the rotation and translation of the mirror are R = I and t = (0, 0, 50), respectively. We tested two relative angles between two incident parallel lights, namely 30 and 90 degrees. 24 pairs of the incident lights are used by rotating the camera and mirror around the y- and z-axes. We estimate R and t by adding noise to the position of the input points. The added Gaussian noise has standard deviations of 0, 0.1, 0.5, and 1.0 pixels. As for E1 , since the relative angle α between the two points has to be given, we add noise to α, which has standard deviations of 0, 0.1, and 0.5 degrees. To evaluate the accuracy of the estimated parameters, we compute the root-mean-square (RMS) errors between the input points and the reprojection of the incident lights. Figure 4 shows the RMS errors of E1 and E2 . It is clear that the results obtained with the relative angle equal to 90 degrees are better than those for 30 degrees. A reason for this may be that the constraint is weaker when the relative angle is smaller and the projected points are close to each other. The error depends mainly on the noise of the input points, as the effect of the noise of the relative angle is small. Since the accuracy of E2 is similar to that of E1 , we can apply our method even if we do not know the relative angle. Next, we evaluate the error if the intrinsic parameter K is different from the ground truth. Figure 5 shows the RMS errors of E1 with varying values of f and cx . The
122
R. Sagawa, N. Aoki, and Y. Yagi 4
3.5 °
f σ =0.0
E 30 σ =0.0
α
α
1
3.5
°
E1 30 σα=0.1
x
E 90 σ =0.0 α
1
2.5
α
°
E1 90 σα=0.5
RMS Error (pixels)
RMS Error (pixels)
1
°
E2 30 2
α
cx σα=0.1
°
E 90° σ =0.1 2.5
α
c σ =0.0
α
1
3
f σ =0.1
3
E 30° σ =0.5
°
E2 90
1.5
2
1.5
1 1
0.5
0.5
0 0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 −50
−40
−30
Standard deviation of noise (pixels)
−20 −10 0 10 20 Difference from Ground Truth (pixels)
30
40
50
Fig. 4. The RMS errors with respect to the Fig. 5. The RMS errors with respect to the ernoise of image points ror of the intrinsic parameters
Side2
Side3
Mirrors
Camera
Side4
Side1
Center
Fig. 6. Compound parabolic mirrors attached Fig. 7. An example image from compound parabolic mirrors to a camera
other parameters are fixed to the ground truth. The horizontal axis means the difference between the ground truth and f or cx . The results show that the error from reprojecting the incident lights is significantly affected by cx , while the effect of f is small. This shows that the principal point (cx , cy ) must be computed accurately before minimizing E1 and that the error of f is more acceptable than that of the principal point. 4.2 Localizing Mirrors from Real Images In the next experiment, we compute the mirror positions of a catadioptric system with compound parabolic mirrors [24,25] as shown in Figure 6. Figure 7 shows an example of an image obtained from such a system. Our system has 7 parabolic mirrors and a perspective camera, PointGrey Scorpion, which has 1600 × 1200 pixels and about 22.6◦ field of view. The distortion of the lens is calibrated by the method described in [26], and the intrinsic parameters of the camera are already calibrated. With this setup, the catadioptric system is not single viewpoint. The radii h of a center mirror and the side mirrors are 9.0mm and 4.5mm, respectively. The diameter and height of the center mirror are 25.76mm and 9.0mm, respectively, and the diameter and height of the side mirrors are 13.0mm and 4.5mm, respectively. The diameters of the center and side mirrors projected onto the image are 840 and 450 pixels, respectively.
Mirror Localization for Catadioptric Imaging System Side2
Side1
123
Side3
Side4
Center
Fig. 8. A distant point used as a parallel light Fig. 9. The mirror positions estimated by the source proposed method
Table 1. The RMS errors of (7) are computed using the estimated mirror positions Mirror Number of Pairs RMS Error (pixels) Center 78 0.84 Side1 21 0.87 Side2 45 1.05 Side3 45 1.16 Side4 21 0.59
To localize the mirrors from real images, we experimented with different ways of acquiring parallel lights, namely distant markers and collimated lights. In the first case, we chose points on a distant object in the image. Figure 8 shows the chosen point, which is a point on a building that is about 260 meters away from the camera. We rotated the catadioptric system and obtained 78 pairs of parallel lights. The relative angles of the pairs of parallel lights vary between 15 degrees and 170 degrees. We estimated the positions of the center and four side mirrors independently. Figure 9 shows the estimated mirror positions by rendering the mirror shapes from the viewpoint of the camera. Since we do not know the ground truth of the mirror position and the incident light vectors, we estimate the accuracy of the estimated parameters by the following criterion. If the observed points of a pair of parallel lights are p1 and p2 , and the corresponding incident vectors, as computed by (2), are v 1 and v 2 , respectively, (7) min p2 − q 2 subject to v q · v 1 = cos α, q where v q is the incident vector corresponding to an image point q. This criterion computes the errors in pixels. Table 1 shows the estimated results. Since some of the lights are occluded by the other mirrors, the number of lights used for calibration varies for each mirror. The error is computed by the RMS of (7). Since the position of a feature point is considered to have 0.5 pixel error, the error computed by using the estimated position of the mirrors is appropriate. Next, we tested our method by observing collimated lights generated by the system shown in Figure 3. The relative angle of the two collimated lights is 87.97 degrees.
124
R. Sagawa, N. Aoki, and Y. Yagi struct VS_OUT { float4 Pos : POSITION; float3 Tex : TEXCOORD0; }; VS_OUT VS(float4 Pos : POSITION, float4 Nor : NORMAL) { VS_OUT Out = (VS_OUT)0; float3 tmpPos, tmpNor, v; float a; tmpPos = normalize(mul(Pos, T)); tmpNor = mul(Nor, R); a = dot(-tmpPos, tmpNor); v = tmpPos + 2 * a * tmpNor; Out.Pos = mul(Pos, KT); Out.Tex = normalize(v); return Out; } float4 PS(VS_OUT In) : COLOR { float4 Col = 0; Col.rgb = In.Tex.xyz; return Col; }
Fig. 10. Top: an example of the acquired Fig. 11. The source code for computing the incident image. Bottom: the image of two collimated vector in HLSL lights after turning off the room light.
We acquired 60 pairs of parallel lights. Figure 10 shows an example of an image, onto which two collimated lights are projected. In this experiment, we estimated the position of the center mirror. The RMS error of (7) is 0.35 pixels, which is smaller than that obtained using distant markers. This shows that the accuracy of the estimated results is improved by using the collimated lights.
5 Conclusion This paper describes a method of mirror localization to calibrate a catadioptric imaging system. In it, we focused on the localization of the mirror. By observing pairs of parallel lights, our method utilizes the constraint that the relative angle of two parallel lights is invariant with respect to the translation and rotation of the imaging system. Since the translation and rotation between a camera and the calibration objects are omitted from the parameters, the only parameter to be estimated is the rigid transformation of the mirror. Our method estimates the rigid transformation by minimizing the error between the model of the mirror and the measurements. Since our method makes no assumptions about the mirror shape or its position, the proposed method can be applied to noncentral systems. If we compute the incident light vector by projecting the mirror shape onto an image, our method is able to accommodate any mirror shape. Finally, to validate the accuracy of our method, we tested our method in a simulation and in real experiments. For future work, we plan to apply the proposed method to various shapes of mirrors using the collimated lights and analyzing the best settings for the parallel lights.
Mirror Localization for Catadioptric Imaging System
125
References 1. Ishiguro, H., Yamamoto, M., Tsuji, S.: Omni-directional stereo. IEEE Transactions on Pattern Analysis and Machine Intelligence 14(2), 257–262 (1992) 2. Yamazawa, K., Yagi, Y., Yachida, M.: Obstacle detection with omnidirectional image sensor hyperomni vision. In: IEEE The International Conference on Robotics and Automation, Nagoya, pp. 1062–1067. IEEE Computer Society Press, Los Alamitos (1995) 3. Nayar, S.: Catadioptric omnidirectional camera. In: Proc. IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 482–488. IEEE Computer Society Press, Los Alamitos (1997) 4. Gaspar, J., Decco, C., Okamoto Jr., J., Santos-Victor, J.: Constant resolution omnidirectional cameras. In: Proc. The Third Workshop on Omnidirectional Vision, pp. 27–34 (2002) 5. Hicks, R., Perline, R.: Equi-areal catadioptric sensors. In: Proc. The Third Workshop on Omnidirectional Vision, pp. 13–18 (2002) 6. Swaminathan, R., Nayar, S., Grossberg, M.: Designing Mirrors for Catadioptric Systems that Minimize Image Errors. In: Fifth Workshop on Omnidirectional Vision (2004) 7. Kondo, K., Yagi, Y., Yachida, M.: Non-isotropic omnidirectional imaging system for an autonomous mobile robot. In: Proc. 2005 IEEE International Conference on Robotics and Automation, Barcelona, Spain, IEEE Computer Society Press, Los Alamitos (2005) 8. Kojima, Y., Sagawa, R., Echigo, T., Yagi, Y.: Calibration and performance evaluation of omnidirectional sensor with compound spherical mirrors. In: Proc. The 6th Workshop on Omnidirectional Vision, Camera Networks and Non-classical cameras (2005) 9. Geyer, C., Daniilidis, K.: Paracatadioptric camera calibration. IEEE Transactions on Pattern Analysis and Machine Intelligence 24(5), 687–695 (2002) 10. Ying, X., Hu, Z.: Catadioptric camera calibration using geometric invariants. IEEE Transactions on Pattern Analysis and Machine Intelligence 26(10), 1260–1271 (2004) 11. Mei, C., Rives, P.: Single view point omnidirectional camera calibration from planar grids. In: Proc. 2007 IEEE International Conference on Robotics and Automation, Rome, Italy, pp. 3945–3950. IEEE Computer Society Press, Los Alamitos (2007) 12. Hartley, R.I., Zisserman, A.: Multiple View Geometry in Computer Vision, 2nd edn. Cambridge University Press, Cambridge (2004) 13. Aliaga, D.: Accurate catadioptric calibration for realtime pose estimation of room-size environments. In: Proc. IEEE International Conference on Computer Vision, vol. 1, pp. 127–134. IEEE Computer Society Press, Los Alamitos (2001) 14. Strelow, D., Mishler, J., Koes, D., Singh, S.: Precise omnidirectional camera calibration. In: Proc. of IEEE Conference on Computer Vision and Pattern Recognition, vol. 1, pp. 689–694. IEEE Computer Society Press, Los Alamitos (2001) 15. Micus´ık, B., Pajdla, T.: Autocalibration and 3d reconstruction with non-central catadioptric cameras. In: Proc. IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Washington US, vol. 1, pp. 58–65. IEEE Computer Society Press, Los Alamitos (2004) 16. Mashita, T., Iwai, Y., Yachida, M.: Calibration method for misaligned catadioptric camera. In: Proc. The Sixth Workshop on Omnidirectional Vision (2005) 17. Swaminathan, R., Grossberg, M., Nayar, S.: Caustics of catadioptric camera. In: Proc. IEEE International Conference on Computer Vision, vol. 2, pp. 2–9. IEEE Computer Society Press, Los Alamitos (2001) 18. Grossberg, M., Nayar, S.: The raxel imaging model and ray-based calibration. International Journal on Computer Vision 61(2), 119–137 (2005) 19. Sturm, P., Ramalingam, S.: A generic camera calibration concept. In: Proc. European Conference on Computer Vision, Prague, Czech, vol. 2, pp. 1–13 (2004)
126
R. Sagawa, N. Aoki, and Y. Yagi
20. Sagawa, R., Aoki, N., Mukaigawa, Y., Echigo, T., Yagi, Y.: Mirror localization for a catadioptric imaging system by projecting parallel lights. In: Proc. IEEE International Conference on Robotics and Automation, Rome, Italy, pp. 3957–3962. IEEE Computer Society Press, Los Alamitos (2007) 21. Caprile, B., Torre, V.: Using vanishing points for camera calibration. International Journal of Computer Vision 4(2), 127–140 (1990) 22. Daniilidis, K., Ernst, J.: Active intrinsic calibration using vanishing points. Pattern Recognition Letters 17(11), 1179–1189 (1996) 23. Guillemaut, J., Aguado, A., Illingworth, J.: Using points at infinity for parameter decoupling in camera calibration. IEEE Transactions on Pattern Analysis and Machine Intelligence 27(2), 265–270 (2005) 24. Mouaddib, E., Sagawa, R., Echigo, T., Yagi, Y.: Two or more mirrors for the omnidirectional stereovision? In: Proc. of The second IEEE-EURASIP International Symposium on Control, Communications, and Signal Processing, Marrakech, Morocco, IEEE Computer Society Press, Los Alamitos (2006) 25. Sagawa, R., Kurita, N., Echigo, T., Yagi, Y.: Compound catadioptric stereo sensor for omnidirectional object detection. In: Proc. IEEE/RSJ International Conference on Intelligent Robots and Systems, Sendai, Japan, vol. 2, pp. 2612–2617 (2004) 26. Sagawa, R., Takatsuji, M., Echigo, T., Yagi, Y.: Calibration of lens distortion by structuredlight scanning. In: Proc. 2005 IEEE/RSJ International Conference on Intelligent Robots and Systems, Edmonton, Canada, pp. 1349–1354 (2005)
A Source Code for Rendering Incident Vectors The reflected vector for each pixel is computed using the source code in Figure 11. It is written in High-Level Shader Language (HLSL) and executed by graphics hardware. The shape of a mirror is represented by a mesh model that consists of vertices and triangles. The inputs of the vertex shader (VS) are the positions of vertices of the mirror (Pos) and the normal vectors of the vertices (Nor). R, T and KT are constant matrices given by a main program. R is the rotation matrix of the mirror, and T = [R|t], where t is the translation vector of the mirror. KT is the projection matrix computed as KT = K[R|t], where K is the intrinsic matrix of the camera. The reflected vector v is computed for each vertex. Since it is interpolated by the rasterizer of the graphics hardware, the pixel shader (PS) outputs the reflected vector for each pixel.
Calibrating Pan-Tilt Cameras with Telephoto Lenses Xinyu Huang, Jizhou Gao, and Ruigang Yang Graphics and Vision Technology Lab (GRAVITY) Center for Visualization and Virtual Environments University of Kentucky, USA {xhuan4,jgao5,ryang}@cs.uky.edu http://www.vis.uky.edu/∼gravity
Abstract. Pan-tilt cameras are widely used in surveillance networks. These cameras are often equipped with telephoto lenses to capture objects at a distance. Such a camera makes full-metric calibration more difficult since the projection with a telephoto lens is close to orthographic. This paper discusses the problems caused by pan-tilt cameras with long focal length and presents a method to improve the calibration accuracy. Experiments show that our method reduces the re-projection errors by an order of magnitude compared to popular homographybased approaches.
1 Introduction A surveillance system usually consists of several inexpensive wide fields of view (WFOV) fixed cameras and pan-tilt-zoom (PTZ) cameras. The WFOV cameras are often used to provide an overall view of the scene while a few zoom cameras are controlled by pan-tilt unit (PTU) to capture close-up views of the subject of interest. The control of PTZ camera is typically done manually using a joystick. However, in order to automate this process, calibration of the entire camera network is necessary. One of our driving applications is to capture and identify subjects using biometric features such as iris and face over a long range. A high-resolution camera with a narrow field of view (NFOV) and a telephoto lens is used to capture the rich details of biometric patterns. For example, a typical iris image should have 100 to 140 pixels in iris radius to obtain a good iris recognition performance [1]. That means, in order to capture the iris image over three meters using a video camera (640×480), we have to use a 450mm lens assuming sensor size is 4.8 × 3.6 mm. If we want to capture both eyes (e.g., the entire face) at once, then the face image resolution could be as high as 5413×4060 pixels–well beyond the typical resolution of a video camera. In order to provide adequate coverage over a practical working volume, PTZ cameras have to be used. The simplest way to localize the region of interest (ROI) is to pan and tilt the PZT camera iteratively until the region is approximately in the center of field of view [2]. This is time-consuming and only suitable for still objects. However, if the PTZ cameras are fully calibrated, including the axes of rotation, the ROI can be localized rapidly with a single pan and tilt operation. In this paper we discuss the degeneracy caused by cameras with telephoto lenses and develop a method to calibrate such a system with significantly improved accuracy. The remaining of this paper is organized as the following. We first briefly overview the related work in section 2. In section 3, we describe our system and a calibration Y. Yagi et al. (Eds.): ACCV 2007, Part I, LNCS 4843, pp. 127–137, 2007. c Springer-Verlag Berlin Heidelberg 2007
128
X. Huang, J. Gao, and R. Yang
method for long focal length cameras. Section 4 contains experimental results. Finally, a summary is given in section 5. We also present in the appendix a simple method to calculate the pan and tilt angle when the camera coordinate is not aligned with the pan-tilt coordinate.
2 Related Work It is generally considered that camera calibration reached its maturity in the late 90’s. A lot of works have been done in this area. In the photogrammetry community, a calibration object with known and accurate geometry is required. With markers of known 3D positions, camera calibration can be done efficiently and accurately (e.g., [3], [4]). In computer vision, a planar pattern such as a checkerboard pattern is often used to avoid the requirement of a 3D calibration object with a good precision (e.g., [5], [6]). These methods estimate intrinsic and extrinsic parameters including radial distortions from homographies between the planar pattern at different positions and the image plane. Self-calibration estimates fixed or varying intrinsic parameters without the knowledge of special calibration objects and with unknown camera motions (e.g., [7], [8]). Furthermore, self-calibration can compute a metric reconstruction from an image sequence. Besides the projective camera model, the affine camera model in which camera center lies on the plane at infinity is proposed in ([9], [10]). Quan presents a self-calibration method for an affine camera in [11]. However, the affine camera model should not be used when many feature points at different depths [12]. For the calibrations of PTZ cameras, Hartley proposed a self-calibration method for stationary cameras with purely rotations in [13]. Agapito extended this method in [14] to deal with varying intrinsic parameters of a camera. Sinha and Pollefeys proposed a method for calibrating pan-tilt-zoom cameras in outdoor environments in [15]. Their method determines intrinsic parameters over the full range of zoom settings. These methods above approximate PTZ cameras as rotating cameras without translations since the translations are very small compared to the distance of scene points. Furthermore, These methods are based on computing absolute conic from a set of inter-image homographies. In [16], Wang and Kang present an error analysis of intrinsic parameters caused by translation. They suggest self-calibrate using distant scenes, larger rotation angles, and more different homographies in order to reduce effects from camera translation. The work most similar to ours is proposed in [17,18]. In their papers, Davis proposed a general pan-tilt model in which the pan and tilt axes are arbitrary axes in 3D space. They used a bright LED to create a virtual 3D calibration object and a Kalman filter tracking system to solve the synchronization between different cameras. However, they did not discuss the calibration problems caused by telephoto lens. Furthermore, their method cannot be easily applied to digital still cameras with which it will be tedious to capture hundreds or even thousands of frames.
3 Method In this section, we first describe the purpose of our system briefly. Then, we discuss the calibration for long focal length camera in details.
Calibrating Pan-Tilt Cameras with Telephoto Lenses
129
3.1 System Description The goal of our system is to capture face or iris images over a long range with a resolution high enough for biometric recognitions. As shown in Fig. 1, a prototype of our system consists of two stereo cameras and a NFOV high resolution (6M pixels) still camera. The typical focal length for the pan-tilt camera is 300mm, while previous papers dealing with pan-tilt cameras have reported the use of lenses between 1mm to 70mm. When a person walks into the working area of the stereo cameras, facial features are detected in each frame and their 3D positions can be easily determined by triangulation. The pan-tilt camera is steered so that the face is in the center of the observed image. A high-resolution image with enough biometric details then can be captured. Since the field of view of pan-tilt camera is only about 8.2 degrees, the ROI (e.g., the eye or entire face) is likely to be out of the field of view if the calibration is not accurate enough.
1) Pan-tilt Camera (Nikon 300mm)
6
3
2) Laser Pointer 3) Stereo Camera (4mm)
2 5 1
4) Pan Axis 5) Tilt Axis 6) Flash
3 4
Fig. 1. System setup with two WFOV cameras and a pan-tilt camera
3.2 Calibration In [5], given one homography H = [h1 , h2 , h3 ] between a planar pattern at one position and the image plane, two constraints on the absolute conic ω can be formulated as in Eq.(1). hT1 ωh2 = 0 hT1 ωh1 = hT2 ωh2
(1)
By imaging the planar pattern n times at different orientations, a linear system Ac = 0 is formed, where A is a 2n × 6 matrix from the observed homographies and c represents ω as a 6 × 1 vector. Once c is solved, intrinsic matrix K can be solved by Cholesky factorization since ω = (KK T )−1 . Equivalently, one could rotate the camera instead of moving a planar pattern. This is the key idea in self-calibration of pan-tilt cameras ( [15], [13]). First, inter-image homographies are computed robustly. Second, the absolute conic ω is estimated by a
130
X. Huang, J. Gao, and R. Yang
linear system ω = (H i )−T ω(H i )−1 , where H i is the homography between each view i and a reference view. Then, Cholesky decomposition of ω is applied to compute the intrinsic matrix K. Furthermore, a Maximum Likelihood Estimation (MLE) refinement could be applied using the above close-form solution as the initial guesses. However, the difference is small between the close-form solution and that from MLE refinement [12]. As mentioned in [5], the second homography will not provide any new constraints if it is parallel to the first homography. In order to avoid this degeneracy and generate a over-determined system, the planar pattern has to be imaged many times with different orientations. This is also true for the self-calibration of rotating cameras. If conditions are near singular, the matrix A formed from the observed homographies will be illconditioned, making the solution inaccurate. Generally, the degeneracy is easy to avoid when the focal length is short. For example, we only need to change the orientation for each position of planar pattern. However, this is not true for long focal length cameras. When the focal length increases and the filed-of-view decreases, the camera’s projection becomes less projective and more orthographic. The observed homographies contain a lot of depth ambiguities that make the matrix A ill-conditioned and solution is very sensitive to a small perturbation. If the projection is purely orthographic, then observed homographies can not provide any depth information no matter where we put the planar pattern or how we rotate the camera. In summary, traditional calibration methods based on observed homographies are in theory not accurate for long focal length camera. We will also demonstrate this point with real data in the experiments section. X
x1 O
x
Stereo Cameras
( px , p y )
World
x2
R, T PTU
R*
Camera (pan=0, tilt=0)
Fig. 2. Pan-tilt camera model
The best way to calibrate a long focal length camera is to create 2D-3D correspondences directly. One could use a 3D calibration object but this approach is not only costly, but also un-practical given the large working volume we would like to cover. In our system 3D feature points are triangulated by stereo cameras, therefore it will not induce any ambiguities caused by the methods based on observed homographies. With a set of known 2D and 3D features, we can estimate intrinsic parameters and the relative
Calibrating Pan-Tilt Cameras with Telephoto Lenses
131
transformation between the camera and the pan-tilt unit. The pan-tilt model is shown in Fig. 2 and is written as x = KR∗−1 Rtilt Rpan R∗ [R|t]X
(2)
where K is intrinsic parameters, R and t are extrinsic parameters of pan-tilt camera at reference view that is pan = 0 and tilt = 0 in our setting. X and x are 3D and 2D feature points. Rpan and Rtilt are rotation matrices around pan and tilt axes. R∗ is the rotation matrix between coordinates of the camera and the pan-tilt unit. We did not consider the pan and tilt axes as two arbitrary axes in 3D space as in [17] since translation between the two coordinates are very small (usually only a few millimeters in our setting) and a full-scale simulation shows that adding the translational offset yield little accuracy improvement. Based on the pan-tilt model in Eq.(2), we could estimate the complete set of parameters using MLE to minimize the re-projected geometric distances. This is given by the following functional: argminR∗ ,K,R,t
n m
xij − x ˆij (K, R∗ , Rpan , Rtilt , R, t, Xij )2
(3)
i=1 j=1
The method of acquiring of calibration data in [17] is not applicable in our system because that our pan-tilt camera is not a video camera that could capture a video sequence of LED points. Typically a commodity video camera does not support both long focal length and high-resolution image. Here we propose another practical method to acquire calibration data from a still camera. We attach a laser pointer close enough to the pan-tilt camera as shown in Fig.1. The laser’s reflection on scene surfaces generates a 3D point that can be easily tracked. The laser pointer rotates with the pan-tilt camera simultaneously so that its laser dot can be observed by the pan-tilt camera at most of pan and tilt settings. In our set-up, we mount the laser pointer on the tilt axis. A white board is placed at several positions between the near plane and the far plane within the working area of two wide-view fixed cameras. For each pan and tilt step, three images are captured by the pan-tilt camera and two fixed cameras respectively. A 3D point is created by triangulation from the two fixed cameras. The white board does not need to be very large since we can always move it around during the data acquisition process so that 3D points cover the entire working area. The calibration method is summarized as Algorithm 1. In order to compare our method with methods based on observed homographies, we formulate a calibration framework similar to the methods in [15] and [12]. The algorithm is summarized in Algorithm 2. An alternative of step 4 in Algorithm 2 is to build a linear system ω = (H i )−T ω(H i )−1 and solve ω. Intrinsic matrix K is solved by Cholesky decomposition ω = KK T . However, this closed-form solution often fails since infinite homography is hard to estimate with narrow fields of view. After calibration step, R∗ , intrinsic and extrinsic parameters of three cameras are known. Hence, We can solve the pan and tilt angles easily (see Appendix A for the details) for almost arbitrary 3D points triangulated by stereo cameras.
132
X. Huang, J. Gao, and R. Yang
Algorithm 1. Our calibration method for a pan-tilt camera with a long focal length Input: observed laser point images by three cameras. Output: intrinsic matrix K extrinsic parameters R, t, and rotation matrix R∗ between coordinates of camera and PTU. 1. Calibrate the stereo cameras and reference view of the pan-tilt camera using [19]. 2. Rectify stereo images such that epipolar lines are parallel with the y-axis (optional). 3. Capture laser points on a 3D plane for three cameras at each pan and tilt setting in the working area. 4. Based on blob detection and epipolar constraint, find two laser points in the stereo cameras. Generate 3D points by triangulation of two laser points. 5. Plane fitting for each plane position using RANSAC. 6. Remove outliers of 3D points based on the fitted 3D plane. 7. Estimate R∗ , K, R, t by minimizing Eq.(3).
Algorithm 2. Calibration method for a pan-tilt camera with a long focal length based on homographies Input: images captured at each pan and tilt setting. Output: intrinsic matrix K and rotation matrix R∗ between coordinates of camera and PTU. 1. Detect features based on Scale-invariant feature transform (SIFT) in [20] and find correspondences between neighboring images. 2. Robust homography estimation using RANSAC. 3. Compute homography between each image and reference view (pan = 0, tilt = 0). 4. Estimate K using Calibration Toolbox [19]. 5. Estimate R∗ and refine intrinsic matrix K by minimizing argminR∗ ,K
n m
xij − KR∗−1 Rtilt Rpan R∗ K −1 xiref 2
(4)
i=1 j=1
where xij and xiref are ith feature point at jth and reference view respectively.
4 Experiments Here we present experimental results from two fixed cameras (Dragonfly2 DR2-HICOL with resolution 1024 × 768) and a pan-tilt still camera (Nikon D70 with resolution 3008 × 2000). First, we compare the calibration accuracy with short and long focal length lenses using traditional homograph-based method. Then, we demonstrate that our calibration method significantly improves accuracy for telephoto lenses. In order to validate the calibration accuracy, we generate about 500 3D testing points that are randomly distributed cover the whole working area following step 2 to step 4 in Algorithm 1, i.e., tracking and triangulating the laser dot. The testing points are different from the points used for calibration. First we present the calibration results of the still camera with a short (18mm) and a long focal length (300mm) lenses. For simplicity, we assume the coordinate systems
Calibrating Pan-Tilt Cameras with Telephoto Lenses
133
Table 1. The comparison between short and long focal length cameras. α and β are focal length. μ0 and ν0 are principal point. The uncertainties of principal point for 300mm camera cannot be estimated by Calibration Toolbox [19]. focal length α β μ0 ν0 RMS (in pixels) 300mm 40869.2 ± 1750.2 41081.7 ± 1735.1 1503.5 ± ∗ 999.5 ± ∗ 3.85 18mm 2331.2 ± 9.1 2339.7 ± 9.1 1550.8 ± 12.1 997.9 ± 14.4 2.11
of pan-tilt camera and pan-tilt unit are aligned perfectly. This means R∗ is an identity matrix. We use the Calibration Toolbox [19] to do the calibration for the reference view of pan-tilt camera and stereo cameras. In order to reduce the ambiguities caused by the long focal length, we capture over 40 checkerboard patterns at different orientations for pan-tilt camera. Table 1 shows results of the intrinsic matrix K and RMS of calibration data. The uncertainties of the focal length with the 300mm lens is about 10 times larger than that with a 18mm lens although the RMS of calibration data for both cases are similar. Fig.3 shows distributions of re-projection errors for the 500 testing points with 18mm and 300mm cameras. From this figure, we find that calibration is quite accurate for short focal length camera even that we assume R∗ is an identity matrix. Given the high resolution image, the relative errors from 18mm and 300mm cameras are about 1.3% and 30% respectively. This is computed as the ratio of the mean pixel error and the image width. Furthermore, many of the test points are out of field of view of the 300mm camera. Focal Length: 18mm
0
10
20
30
40
50
Focal Length: 300mm
60
Error in Pixels, Mean: 37.9 Variance: 265.9
70
0
500
1000
1500
2000
Error in Pixels, Mean:898.6, Variance: 2.5206e+05
Fig. 3. Distributions of re-projection error (in pixels) based on 500 testing data for 18mm and 300mm pan-tilt cameras
We then recalibrate the 300mm case with methods outlined in Algorithm 1 and 2, both of which include the estimation of R∗ . About 600 3D points are sampled for calibration over the working area in Algorithm 1. We pre-calibrate the reference view for the pan-tilt camera as the initial guess. After calibration, we validate the accuracy with 500 3D points. Fig. 4 shows the distributions of re-projection errors from the two different methods. Our method is about 25 times better than the homography-based one. The
134
X. Huang, J. Gao, and R. Yang Calibration Based on Algorithm 1
0
20
40
60
80
100
120
Error in Pixels, Mean: 34.6, Variance: 424.3
Calibration Based on Algorithm 2
140
0
500
1000
1500
2000
Error in Pixels, Mean: 880.0 Variance: 2.2946e+05
Fig. 4. Distributions of re-projection error based on 500 testing data (in pixels) for Algorithm 1 and 2 Table 2. The comparison between Algorithm 1 and 2. α and β are focal length. μ0 and ν0 are principal point. θx and θy are rotation angles between pan-tilt camera and pan-tilt unit. Algorithm α β μ0 ν0 θx θy 1 40320.9 39507.7 1506.6 997.7 −0.14 −1.61 2 39883.3 40374.6 1567.5 1271.5 1.99 1.41
relative errors from Algorithm 1 and 2 are about 1.2% and 29% respectively. It should be noted that R∗ can not be estimated accurately from observed homographies. Hence, the percentage error from Algorithm 2 remains very large. In fact, the improvement over assuming an identity R∗ is little. Table 2 shows the results for intrinsic matrix K, θx , and θy after MLE refinement. Here we decompose R∗ into two rotation matrices. One is the rotation around x axis for θx degree, and the other is the rotation around y axis for θy degree.
5 Conclusion This paper shows that calibration methods based on observed homographies are not suitable for cameras with telephoto (long-focal-length) lenses. This is caused by the ambiguities induced by the near-orthographic projection. We develop a method to calibrate a pan-tilt camera with long focal length in a surveillance network. In stead of using a large precisely-manufactured calibration object, our key idea is to use fixed stereo cameras to create a large collection of 3D calibration points. Using these 3D points allows full metric calibration over a large area. Experimental results show that the re-projection relative error is reduced from 30% to 1.2% with our method. In future work, we plan to extend our calibration method to auto-zoom cameras and build a complete surveillance system that can adjust zoom settings automatically by estimating the object’s size.
Calibrating Pan-Tilt Cameras with Telephoto Lenses
135
References 1. Daugman, J.: How Iris Recognition Works. In: ICIP (2002) 2. Guo, G., Jones, M., Beardsley, P.: A System for Automatic Iris Capturing. In: MERL TR2005-044 (2005) 3. Tsai, R.Y.: A Versatile Camera Calibration Technique for High-accuracy 3D Machine Vision Metrology Using Off-The-Shelf TV Cameras and Lenses. IEEE Journal of Robotics and Automation 4(3), 323–344 (1987) 4. Faugeras, O.: Three-Dimensional Computer Vision: a Geometric Viewpoint. MIT Press, Cambridge (1993) 5. Zhang, Z.: A Flexible New Technique for Camera Calibration. PAMI 22, 1330–1334 (2000) 6. Heikkila, J., Silven, O.: A Four-Step Camera Calibration Procedure with Implicit Image Correction. In: Proceedings of CVPR, pp. 1106–1112 (1997) 7. Pollefeys, M., Koch, R., Gool, L.V.: Self-Calibration and Metric Reconstruction in spite of Varying and Unknown Internal Camera Parameters. In: Proceedings of ICCV, pp. 90–95 (1997) 8. Pollefeys, M.: Self-Calibration and metric 3D reconstruction from uncalibrated image sequences. PhD thesis, K.U.Leuven (1999) 9. Mundy, J., Zisserman, A.: Geometric Invariance in Computer Vision. MIT Press, Cambridge (1992) 10. Aloimonos, J.Y.: Perspective Approximations. Image and Vision Computing 8, 177–192 (1990) 11. Quan, L.: Self-Calibration of an Affine Camera from Multiple Views. International Journal of Computer Vision 19(1), 93–105 (1996) 12. Hartley, R.I., Zisserman, A.: Multiple View Geometry. Cambridge University Press, Cambridge (2000) 13. Hartley, R.I.: Self-Calibration of Stationary Cameras. International Journal of Computer Vision 1(22), 5–23 (1997) 14. de Agapito, L., Hayman, E., Reid, I.: Self-Calibration of a Rotating Camera with Varying Intrinsic Parameters. In: BMVC (1998) 15. Sinha, N., Pollefeys, M.: Towards Calibrating a Pan-Tilt-Zoom Camera Network. In: Pajdla, T., Matas, J(G.) (eds.) ECCV 2004. LNCS, vol. 3021, Springer, Heidelberg (2004) 16. Wang, L., Kang, S.B.: Error Analysis of Pure Rotation-Based Self-Calibration. PAMI 2(26), 275–280 (2004) 17. Davis, J., Chen, X.: Calibrating pan-tilt cameras in wide-area surveillance networks. In: Proceedings of ICCV, vol. 1, pp. 144–150 (2003) 18. Chen, X., Davis, J.: Wide Area Camera Calibration Using Virtual Calibration Objects. In: Proceedings of CVPR (2000) 19. Bouguet, J.Y.: Camera Calibration Toolbox for Matlab. http://www.vision.caltech.edu/bouguetj/calib doc/ 20. Lowe, D.G.: Distinctive Image Features from Scale-Invariant Keypoints. International Journal of Computer Vision 20, 91–110 (2003)
Appendix A: Solving Pan and Tilt Angles Here we discuss how to solve the pan and tilt angles so that the projection of an arbitrary point X in the 3D space is in the center of the image plane. We assume there is a rotation between the pan-tilt coordinate system and the camera’s. Because of the dependency of the pan-tilt unit, that is, the tilt axis depends on the pan axis, the solution is not as
136
X. Huang, J. Gao, and R. Yang
simple as it appears. In order to address this problem, we back project the image center ˜ 2 . The center of projection and point X form another line L ˜ 1 . After the to a line L ˜ ˜ calibration steps described in Section 3, L1 and L2 are transformed into L1 and L2 in the coordinate system of the pan-tilt unit. Hence, the problem is simplified to panning around y-axis and tilting around x-axis to make L1 and L2 coincident or as close as possible to each other. If L1 and L2 are represented by the Pl¨uker matrices, one method to compute the transformation of an arbitrary 3D line to another line by performing only rotations around x and y axes could be a minimization of the following functional, argminRx ,Ry ,λ L1 − λL2 2
(5)
where λ is a scalar, L2 is a 6 × 1 vector of the Pl¨uker coordinates of L2 , and L1 is the 6 × 1 Pl¨uker coordinates of the multiplication of (Ry Rx )L1 (Ry Rx )T , where Rx and Ry are rotation matrices around the x and y axes. Y
-2
( x2 , y 2 , z 2 )
C2 ( a 2 , b2 , c 2 )
L2
-1
(a1 , b1 , c1 )
O
L1
M2
M1
X
( x1 , y1 , z1 )
Z
C1
Fig. 5. Solve pan and tilt angles from L1 to L2
However, the problem can be further simplified because L1 and L2 are intersected in the origin of the pan-tilt unit in our model. As shown in Fig. 5, we want to pan and tilt line L1 to coincide with another line L2 . Assuming both of the two lines have unit lengths, the tilt angles are first computed by Eq. (6). y1 y2 ) − arctan ( ) z1 r y1 y2 ) ϕ2 = arctan ( ) − arctan ( z1 −r r = y12 + z12 − y22
ϕ1 = arctan (
(6)
If (y12 + z12 − y22 ) is less than 0, two conics C1 and C2 are not intersected that means no exact solution exists. However, it almost never happens in practice since the rotation
Calibrating Pan-Tilt Cameras with Telephoto Lenses
137
between the pan-tilt unit and the camera is small. After tilting, (x1 , y1 , z1 ) is rotated to (a1 , b1 , c1 ) or (a2 , b2 , c2 ). Then the pan angles are computed by Eq. (7). z2 c1 ) − arctan ( ) x2 a1 z2 c2 ϑ2 = arctan ( ) − arctan ( ) x2 a2
ϑ1 = arctan (
(7)
Hence, two solutions, (ϕ1 , ϑ1 ) and (ϕ2 , ϑ2 ), are obtained. We choose the minimum rotation angles as the final solution.
Camera Calibration Using Principal-Axes Aligned Conics Xianghua Ying and Hongbin Zha National Laboratory on Machine Perception Peking University, Beijing, 100871 P.R. China {xhying,zha}@cis.pku.edu.cn
Abstract. The projective geometric properties of two principal-axes aligned (PAA) conics in a model plane are investigated in this paper by utilized the generalized eigenvalue decomposition (GED). We demonstrate that one constraint on the image of the absolute conic (IAC) can be obtained from a single image of two PAA conics even if their parameters are unknown. And if the eccentricity of one of the two conics is given, two constraints on the IAC can be obtained. An important merit of the algorithm using PAA is that it can be employed to avoid the ambiguities when estimating extrinsic parameters in the calibration algorithms using concentric circles. We evaluate the characteristics and robustness of the proposed algorithm in experiments with synthetic and real data. Keywords: Camera calibration, Generalized eigenvalue decomposition, Principal-axes aligned conics, Image of the absolute conic.
1 Introduction Conic is one of the most important image features like point and line in computer vision. The motivation to study the geometry of conics arises from the facts that conics have more geometric information, and can be more robustly and more exactly extracted from images than points and lines. In addition, conics are very easy to be produced and identified than general algebraic curves, though general algebraic curves may have more geometric information. Unlike a large number of researches have been developed on points and lines, there are just several algorithms proposed based on conics for pose estimation [2][10], structure recovery [11][15][7][17][13], object recognition [8] [14][5], and camera calibration [18][19][3]. Forsyth et al. [2] discovered the projective invariants for pairs of conics then developed an algorithm to determine the relative pose of a scene plane from two conic correspondences. However the algorithm requires solving quartics and has no closed form solutions. Ma [10] developed an analytical method based on conic correspondences for motion estimation and pose determination from stereo images. Quan [14] discovered two polynomial constraints from corresponding conics in two uncalibrated perspective images and applied them to object recognition. Weiss [18] demonstrated that two conics are sufficient for calibration under the affine projection and derived a nonlinear calibration algorithm. Kahl and Heyden [7] proposed an algorithm for epipolar geometry estimation from conic correspondences. They found that one conic correspondence gives two independent constraints on the fundamental matrix and a method to Y. Yagi et al. (Eds.): ACCV 2007, Part I, LNCS 4843, pp. 138–148, 2007. © Springer-Verlag Berlin Heidelberg 2007
Camera Calibration Using Principal-Axes Aligned Conics
139
estimate the fundamental matrix from at least four corresponding conics was presented. Sugimoto [17] proposed a linear algorithm for solving the homography from conic correspondences, but it requires at least seven correspondences. Mudigonda et al. [13] shown that two conic correspondences are enough for solving the homography but requires solutions of polynomial equations. The closest works to that proposed here are [19] and [3]. Yang et al. [19] presented a linear approach for camera calibration from concentric conics on a model plane. They showed that 2 constraints could be obtained from a single image of these concentric conics. However, it requires at least three concentric conics, and the equations of all these conics must be given in advance. Gurdjos et al. [3] utilized the projective and Euclidean properties of confocal conics to perform camera calibration. These properties are that the line conic consisted of the images of the circular points should belong to the conic range of these confocal conics. Two constraints on the IAC can be obtained from a single image of the confocal conics. Gurdjos et al. [3] claimed that the important reason to propose confocal conics for camera calibration is that there exist ambiguities in the calibration methods using concentric circles [9][6] when recovering the extrinsic parameters of the camera, and the algorithms using the confocal conics can avoid such ambiguities. In this paper, we discover a novel useful pattern, PAA. And the properties of two arbitrary PAA conics with unknown or known eccentricities are deeply investigated and discussed in this paper.
2 Basic Principles 2.1 Pinhole Camera Model Let X = [X Y Z 1]T be a world point and ~ x = [u v 1]T be its image point, both in the homogeneous coordinates, they satisfy:
μ~x = PX ,
(1)
where P is a 3× 4 projection matrix describing the perspective projection process. μ is an unknown scale factor. The projection matrix can be decomposed as:
P = K [R | t ] ,
(2)
where
⎡ fx K = ⎢⎢ 0 ⎢⎣ 0
s fy 0
u0 ⎤ v0 ⎥⎥ . 1 ⎥⎦
(3)
Here the matrix K is the matrix of the intrinsic parameters, and (R, t ) denote a rigid transformation which indicate the orientation and position of the camera with respect to the world coordinate system.
140
X. Ying and H. Zha
2.2 Homography Between the Model Plane and Its Image Without loss of generality, we assume the model plane is on Z = 0 of the world coordinate system. Let us denote the i th column of the rotation matrix R by ri . From (1) and (2), we have,
μ~x = K [r1 r2 We denote x = [X homography H :
⎡X ⎤ ⎢Y ⎥ t ]⎢ ⎥ = K [r1 r2 ⎢0⎥ ⎢ ⎥ ⎣1⎦
r3
⎡X ⎤ t ]⎢⎢ Y ⎥⎥ . ⎢⎣ 1 ⎥⎦
(4)
Y 1]T , then a model point x and its image ~ x is related by a 2D
μ~x = Hx ,
(5)
where H = K [r1 r2
t] .
(6)
Obviously, H is defined up to a scale factor. 2.3 Standard Forms for Conics All conics are projectively equivalent under the projective transformation [16]. This means that any conic can be converted into any anther conic by some projective transformations. A conic is an ellipse (including circle), a parabola or a hyperbola, respectively, if and only if its intersection with the line at infinity on the projective plane consists of 2 imaginary points, 2 repeated real points or 2 real points, respectively. In cases of central conics (ellipses and hyperbolas), by moving the coordinate origin to the center and choosing the directions of the coordinate axes coincident with the socalled principal axes (axes of symmetry) of the conic, we can obtain that the equation in standard form for an ellipse is X 2 a 2 + Y 2 b 2 = 1 , where a 2 ≥ b 2 , and the equation in standard form for a hyperbola is X 2 a 2 − Y 2 b 2 = 1 . These equations can be written in a simpler form: AX 2 + BY 2 + C = 0 ,
(7)
and rewritten in matrix form, we obtain, xT Ax = 0 ,
(8)
⎡A ⎤ ⎢ ⎥. A=⎢ B ⎥ ⎢⎣ C ⎥⎦
(9)
where
Camera Calibration Using Principal-Axes Aligned Conics
141
For a parabola, let the unique axis of symmetry of the parabola coincident with the X-axis, and let the Y-axis pass through the vertex of the parabola, then the equation of the parabola is brought into the form: Y 2 = 2 pX ,
(10)
xT Bx = 0 ,
(11)
− p⎤ ⎡ ⎥. B = ⎢⎢ 1 ⎥ ⎢⎣− p ⎥⎦
(12)
or
where
Equation (12) can be rewritten in a homogenous form: ⎡ B = ⎢⎢ ⎢⎣ E
D
E⎤ ⎥. ⎥ ⎥⎦
(13)
2.4 Equations for the Images of Conics in Standard Form Given the homography H between the model plane and its image, from (5) and (8), we can obtain the image of a central conic in standard form satisfies: ~ ~ x T A~ x=0, (14) where ~ A = H −T AH −1 .
Similarly, the image of a parabola in standard form satisfies, ~ ~ x T B~ x =0,
(15)
(16)
where ~ B = H −T BH −1 .
(17)
3 Properties of PAA Conics 3.1 Properties of Two Conics Via the GED Conics are still conics under an arbitrary 2D projective transformation [16]. An interesting property of two conics is that the GED of the two conics is projectively invariant [12]. This property is interpreted in details as follows: Given two point ~ ~ conic pairs ( A1 , A 2 ) and ( A1 , A 2 ) , they are related by a plane homography H , i.e., ~ A i ~ H −T A i H −1 , i = 1,2 . If x is the generalized eigenvector of ( A1 , A 2 ) i.e.,
142
X. Ying and H. Zha
~ ~ A1x = λA 2 x , then ~ x = Hx must be the generalized eigenvector of ( A1 , A 2 ) , i.e., ~ ~~ A1~ x = λ A 2~ x . In general, there are 3 generalized eigenvectors for two 3 × 3 matrixes. Therefore, for a point conic pair, we may obtain three points (i.e., the three generalized eigenvectors of a point conic pair), which are projectively invariant under the 2D projective transformation in the projective plane. Similarly, for a line conic pair, we may obtain three lines (i.e., the three generalized eigenvectors of a line conic pair), which are projectively invariant under the 2D projective transformation in the projective plane.
3.2 Properties of Two PAA Central Conics Two PAA central conics (point conics) in standard form are: ⎡ A1 A1 = ⎢⎢ ⎢⎣
B1
⎤ ⎡ A2 ⎥, A =⎢ 2 ⎥ ⎢ ⎢⎣ C1 ⎥⎦
B2
⎤ ⎥. ⎥ C 2 ⎥⎦
(18)
The GED of the two conics is:
A1x = λA 2 x .
(19)
It is not difficult to find that, the generalized eigenvalues and the generalized eigenvectors of A1 and A 2 are as follows:
⎡1 ⎤ ⎡0 ⎤ ⎡0 ⎤ B1 C1 A1 ⎢ ⎥ ⎢ ⎥ λ1 = , x 1 = ⎢0 ⎥ , λ 2 = , x 2 = ⎢1⎥ , λ3 = , x 3 = ⎢⎢0⎥⎥ , A2 B2 C2 ⎢⎣0⎥⎦ ⎢⎣0⎥⎦ ⎢⎣1⎥⎦
(20)
where x1 is the directional vector in the X-axis, x 2 is the directional vector in the Y-axis, and x 3 is the homogeneous coordinates of the common center of the two central conics. From the projective geometric properties of two point conics via the GED as presented in Section 3.1, we obtain,
Proposition 1. From the images of two PAA central conics, we can obtain the image of the directional vector in the X-axis, the image of the directional vector in the Y-axis, and the image of the common center of the two central conics via the GED. 3.3 Properties and Ambiguities in Concentric Circles Two concentric circles in standard form are: ⎡ A1 A1 = ⎢⎢ ⎢⎣
A1
⎤ ⎡ A2 ⎥, A =⎢ 2 ⎥ ⎢ ⎢⎣ C1 ⎥⎦
A2
⎤ ⎥. ⎥ C 2 ⎥⎦
(21)
It is not difficult to find that, the generalized eigenvalues and the generalized eigenvectors of A1 and A 2 are as follows:
Camera Calibration Using Principal-Axes Aligned Conics
143
⎡1 ⎤ ⎡0 ⎤ ⎡1⎤ ⎡0 ⎤ ⎡0 ⎤ C1 A1 ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ λ1 = λ2 = , x1 = ρ1 ⎢0⎥ + μ1 ⎢1⎥ , x 2 = ρ 2 ⎢0⎥ + μ 2 ⎢1⎥ , λ3 = , x 3 = ⎢⎢0⎥⎥ , (22) A2 C2 ⎢⎣0⎥⎦ ⎢⎣0⎥⎦ ⎢⎣0⎥⎦ ⎢⎣0⎥⎦ ⎢⎣1⎥⎦ where ρ1 , μ1 , ρ 2 , μ 2 are four real constants which are only required to satisfy that x1 ≠ x 2 up to a scale factor. This means ρ1 , μ1 , ρ 2 , μ 2 cannot be determined uniquely. There are infinitely many solutions for ρ1 , μ1 , ρ 2 , μ 2 , thus infinitely many solutions for x1 and x 2 . x1 and x 2 are two points at infinity, and x 3 is the homogeneous coordinates of the common center of the two central conics. The ambiguities in x1 and x 2 can be comprehended from the facts that we cannot establish a unique XY coordinate system from the two concentric circles on the model plane because there exists a degree of freedom in the 2D rotation around the common center. However for two general central PAA conics, it is very easy to establish a XY coordinate system in the supporting plane without any ambiguities because we can choose the coordinate axes coincident with the principal axes of two PAA conics.
Proposition 2. From the images of two concentric circles, we can obtain the image of the common center, and the image of the line at infinity of the supporting plane via the GED.
4 Calibration 4.1 Dual Conic of the Absolute Points from Conics in Standard Form The eccentricity e is one of the very important parameters in a conic. If e = 0 , the conic is a circle. If 0 < e < 1 , the conic section is an ellipse. If e = 1 , it is a parabola. If e > 1 , it is a hyperbola. The equation in standard form for an ellipse is: 2 2 2 2 2 2 X 2 a 2 + Y 2 b 2 = 1 , then e = c a , where c = a − b , thus, b = (1 − e )a . ThereT fore, we can obtain that the line at infinity l∞ = (0,0,1) of the supporting plane intersects the ellipse at two imaginary points: 1 ⎡ ⎤ ⎡ 1 ⎤ ⎢ ⎢ 2 ⎥ I E = ⎢ 1 − e i ⎥ , J E = ⎢ − 1 − e 2 i ⎥⎥ . ⎢⎣ ⎥⎦ ⎢⎣ 0 ⎥⎦ 0
(23)
The equation in standard form for a hyperbola is X 2 a 2 − Y 2 b 2 = 1 , then e = c a , where c 2 = a 2 + b 2 , thus, b 2 = (e 2 − 1)a 2 . Therefore, we can obtain that the line at infinity l∞ = (0,0,1)T of the supporting plane intersects the hyperbola at two real points: 1 ⎡ ⎤ ⎡ 1 ⎤ I H = ⎢⎢ e 2 − 1 ⎥⎥ , J H = ⎢⎢ − e 2 − 1 ⎥⎥ . ⎢⎣ ⎥⎦ ⎢⎣ 0 ⎥⎦ 0
(24)
The equation in standard form for a parabola is Y 2 = 2 pX , it is not difficult to obtain that the line at infinity l∞ = (0,0,1)T of the supporting plane intersects the parabola
144
X. Ying and H. Zha
at two repeated real points, or say that the line at infinity is tangent to the parabola at one real point:
⎡1 ⎤ I P = J P = ⎢⎢0⎥⎥ . ⎢⎣0⎥⎦
(25)
From discussions above, we obtain:
Definition 1. The line at infinity intersects a conic in standard form at two points, which are called the absolute points of a conic in standard form: 1 ⎡ 1 ⎤ ⎡ ⎤ ⎢ ⎥ ⎢ 2 I A = ⎢ e − 1 ⎥ , J A = ⎢− e 2 − 1 ⎥⎥ . ⎢⎣ 0 ⎥⎦ ⎢⎣ ⎥⎦ 0
(26)
For a circle ( e = 0 ), the two absolute points is the well-known circular points, I = [1 i 0]T and J = [1 − i 0]T .
Definition 2. The conic C*∞ = I A J TA + J AITA
(27)
is the conic dual to the absolute points. The conic C*∞ is a degenerate (rank 2 or 1) line conic, which consists of the two absolute points. In a Euclidean coordinate system it is given by ⎡1 C*∞ = I A J TA + J A I TA = ⎢⎢ 1 − e 2 ⎢⎣
⎤ ⎥. ⎥ 0⎥⎦
(28)
The conic C*∞ is fixed under scale and translation transformation. The reasons are as follows: Under the point transformation ~ x = Hx , where H is a scale and translation transformation, one can easily verify that, ~ C*∞ = HC*∞ H T = C*∞ . (29) The converse is also true, and we have,
Proposition 3. The dual conic C*∞ is fixed under the projective transformation H if and only if H is a scale and translation transformation. For circles, C*∞ is fixed not only under scale and translation transformation, but also fixed under rotation transformation [4].
4.2 Calibration from Unknown PAA Central Conics Given the images of two PAA central conics, from Proposition 1, we can determine the images of the directional vectors in the X-axis and Y-axis, then denote them as ~ x1
Camera Calibration Using Principal-Axes Aligned Conics
145
and ~ x 2 , respectively. From [4] we know, the vanishing points of lines with perpendicular directions satisfy: ~ x1T ω~ x2 = 0 ,
(30)
where ω = K −T K −1 is the IAC [4]. Therefore, we have:
Proposition 4. From a single image of two PAA conics, if the parameters of the two conics are both unknown, one constraint can be obtained on the IAC. Given 5 images taken in general positions, we can linearly recover the IAC ω . The intrinsic parameter matrix K can be obtained by the Cholesky factorization of the IAC ω . After the intrinsic parameters are known, it is not difficult to obtain the images of the circular points for each image by intersecting the image of the line at infinity and the IAC ω . From the images of the circular points, the image of the common center, and the images of the directional vectors in the X-axis and Y-axis, we can obtain the extrinsic parameters without ambiguities [4].
4.3 Calibration from Eccentricity-Known PAA Central Conics Assume that the eccentricity of one of the PAA central conics is known, from Proposition 2, we can determine the image of the line at infinity from the images of the two PAA conics. Then we can obtain the images of the absolute points of the conic with known eccentricity by intersecting the image of the line at infinity and the image of ~ this conic. Thus we can obtain the image of the conic dual to the absolute points, C*∞ . Actually, a suitable rectifying homography may be obtained directly from the identi~ fied C*∞ in an image using the eigenvalue decomposition, and after some manipulation, we can obtain, ⎡1 ~* C ∞ = U ⎢⎢ 1 − e 2 ⎢⎣
⎤ ⎥ UT . ⎥ 0⎥⎦
(31)
The rectifying projectivity is H = U up to a scale and translation transformation.
Proposition 5. Once the dual conic C*∞ is identified on the projective plane then projective distortion may be rectified up to a scale and translation transformation. After performing the rectification, we can translate the image so that the coordinate origin is coincident with the common center. Thus we obtain the 2D homography between the supporting plane and its image while the coordinate system in the supporting plane is established whose axes are coincident with the principal axes of the central PAA conics. Let us denote H = [h1 h 2 h 3 ] , from (6), we have,
H = [h1 h 2
h 3 ] = K [r1 r2
t] .
(32)
146
X. Ying and H. Zha
Using the fact that r1 and r2 are orthonormal, we have [20],
h1T K −T K −1h 2 = 0 , i.e., h1T ωh 2 = 0 , h1T K −T K −1h1 = h T2 K −T K −1h 2 , i.e., h1T ωh1 = h T2 ωh 2 .
(33) (34)
These are 2 constraints on the intrinsic parameters from one homography. If the eccentricities of two PAA central conics are both known, we can obtain a least squares solution for the homography. From discussions above, we have,
Proposition 6. From a single image of two PAA conics, if the eccentricity of one of the two conics is known, two constraints can be obtained on the IAC. Given 3 images taken in general positions, we can obtain the IAC ω . The intrinsic parameter matrix K can be obtained by the Cholesky factorization of the IAC ω . Once the intrinsic parameter matrix K is obtained, the extrinsic parameters for each image can be recovered without ambiguity as proposed in [20].
5 Experiments We perform a number of experiments, both simulated and real, to test our algorithms with respect to noise sensitivity. Due to lack of space, the simulated experimental results are not shown here. In order to demonstrate the performance of our algorithm, we capture an image sequence of 209 real images, with resolution 800 × 600 , to perform augmented reality. Edges were extracted using Canny’s edge detector and the ellipses were obtained using a least squares ellipse fitting algorithm [3]. Some augmented realities examples are shown in Fig. 1 to illustrate the calibration results.
Fig. 1. Some augmented realities results
6 Conclusion A very deep investigation in the projective geometric properties of the principal-axes aligned conics is given in this paper. These properties are obtained by utilizing the generalized eigenvalue decomposition of two PAA conics. We define the absolute
Camera Calibration Using Principal-Axes Aligned Conics
147
points of a conic in standard form, which is analogy of the circular points of a circle. Furthermore, we define the dual conic consisted of the two absolute points, which is analogy of the dual conic consisted of the circular points. By using the dual conic consisted of the two absolute points, we propose a linear algorithm to obtain the extrinsic parameters of the camera. We also discovered a novel example of the PAA conics, which is consisted of a circle and a conic concentric with each other while the parameters of the circle and the conic are both unknown, and two constraints on the IAC can be obtained from a single image of this pattern. Due to lack of space, these are not discussed in this paper. To explore more novel patterns containing conics is our ongoing work.
Acknowledgements This work was supported in part by the NKBRPC 973 Grant No. 2006CB303100, the NNSFC Grant No. 60605010, the NHTRDP 863 Grant No. 2006AA01Z302, and the Key grant Project of Chinese Ministry of Education No. 103001.
References 1. Fitzgibbon, A.W., Pilu, M., Fisher, R.B.: Direct least squares fitting of ellipses. IEEE Trans. Pattern Analysis and Machine Intelligence 21(5), 476–480 (1999) 2. Forsyth, D., Mundy, J.L., Zisserman, A., Coelho, C., Heller, A., Rothwell, C.: Invariant descriptors for 3-D object recognition and pose. IEEE Trans. Pattern Analysis and Machine Intelligence 13(10), 971–991 (1991) 3. Gurdjos, P., Kim, J.-S., Kweon, I.-S.: Euclidean Structure from Confocal Conics: Theory and Application to Camera Calibration. In: Proc. IEEE. Conf. Computer Vision and Pattern Recognition, vol. 1, pp. 1214–1222. IEEE Computer Society Press, Los Alamitos (2006) 4. Hartley, R., Zisserman, A.: Multiple View Geometry in computer vision, 2nd edn. Cambridge University Press, Cambridge, UK (2003) 5. Heisterkamp, D., Bhattacharya, P.: Invariants of families of coplanar conics and their applications to object recognition. Journal of Mathematical Imaging and Vision 7(3), 253– 267 (1997) 6. Jiang, G., Quan, L.: Detection of Concentric Circles for Camera Calibration. In: Proc. Int’l. Conf. Computer Vision, pp. 333–340 (2005) 7. Kahl, F., Heyden, A.: Using conic correspondence in two images to estimate the epipolar geometry. In: Proc. Int’l. Conf. Computer Vision, pp. 761–766 (1998) 8. Kanatani, K., Liu, W.: 3D Interpretation of Conics and Orthogonality. Computer Vision and Image Understanding 58(3), 286–301 (1993) 9. Kim, J.-S., Gurdjos, P., Kweon, I.-S.: Geometric and Algebraic Constraints of Projected Concentric Circles and Their Applications to Camera Calibration. IEEE Trans. Pattern Analysis and Machine Intelligence 27(4), 637–642 (2005) 10. Ma, S.: Conics-Based Stereo, Motion Estimation, and Pose Determination. Int’l J. Computer Vision 10(1), 7–25 (1993) 11. Ma, S., Si, S., Chen, Z.: Quadric curve based stereo. In: Proc. of The 11th Int’l. Conf. Pattern Recognition, vol. 1, pp. 1–4 (1992)
148
X. Ying and H. Zha
12. Mundy, J.L., Zisserman, A. (eds.): Geometric Invariance in Computer Vision. MIT Press, Cambridge (1992) 13. Mudigonda, P., Jawahar, C.V., Narayanan, P.J.: Geometric structure computation from conics. In: Proc. Indian Conf. Computer Vison, Graphics and Image Processing (ICVGIP), pp. 9–14 (2004) 14. Quan, L.: Algebraic and geometric invariant of a pair of noncoplanar conics in space. Journal of Mathematical Imaging and Vision 5(3), 263–267 (1995) 15. Quan, L.: Conic reconstruction and correspondence from two views. IEEE Transactions on Pattern Analysis and Machine Intelligence 18(2), 151–160 (1996) 16. Semple, J.G., Kneebone, G.T.: Algebraic Projective Geometry. Oxford University Press, Oxford (1952) 17. Sugimoto, A.: A linear algorithm for computing the homography from conics in correspondence. Journal of Mathematical Imaging and Vision 13, 115–130 (2000) 18. Weiss, I.: 3-D curve reconstruction from uncalibrated cameras. In: Proc. of Int’l. Conf. Pattern Recognition, vol. 1, pp. 323–327 (1996) 19. Yang, C., Sun, F., Hu, Z.: Planar Conic Based Camera Calibration. In: Proc. of Int’l. Conf. Pattern Recognition, vol. 1, pp. 555–558 (2000) 20. Zhang, Z.: A flexible new technique for camera calibration. IEEE Transactions on Pattern Analysis and Machine Intelligence 22(11), 1330–1334 (2000)
3D Intrusion Detection System with Uncalibrated Multiple Cameras Satoshi Kawabata, Shinsaku Hiura, and Kosuke Sato Graduate School of Engineering Science, Osaka University, Japan
[email protected], {shinsaku,sato}@sys.es.osaka-u.ac.jp
Abstract. In this paper, we propose a practical intrusion detection system using uncalibrated multiple cameras. Our algorithm combines the contour based multi-planar visual hull method and a projective reconstruction method. To set up the detection system, no advance knowledge or calibration is necessary. A user can specify points in the scene directly with a simple colored marker, and the system automatically generates a restricted area as the convex hull of all specified points. To detect an intrusion, the system computes intersections of an object and each sensitive plane, which is the boundary of the restricted area, by projecting an object silhouette from each image to the sensitive plane using 2D homography. When an object exceeds one sensitive plane, the projected silhouettes from all cameras must have some common regions. Therefore, the system can detect intrusion by any object with an arbitrary shape without reconstruction of the 3D shape of the object.
1
Introduction
In this paper, we propose a practical system for detecting 3D volumetric intrusion in a predefined restricted area using uncalibrated multiple cameras. Intrusion detection techniques (e.g., person–machine collision prevention, offlimits area observation, etc.) are important for establishing safe, secure societies and environments. Today, equipment which detects the blocking of a light beam, referred to as a light curtain, are widely used for this purpose. Although the light curtain is useful to achieve very safe environments which were previously considered dangerous, it is excessive for widespread applications. For example, the light curtain method requires us to set equipment at both sides of a rectangle for detection, which leads to higher cost, limited shape of the detection plane and set-up difficulty. In the meantime, surveillance cameras have been installed into many various environments; however, the scenes observed by these cameras are used only for recording or visual observation by distant human observers, and they are merely used to warn a person in a dangerous situation or to immediately halt a dangerous machine. There are many computer enhancements that recognize events in a scene [1], but it is difficult to completely detect dangerous situations, including unexpected phenomena. Furthermore, we do not have sufficient knowledge and methodologies to use the recognition result from these systems to ensure safety. Therefore, our proposed system simply detects an intrusion Y. Yagi et al. (Eds.): ACCV 2007, Part I, LNCS 4843, pp. 149–158, 2007. c Springer-Verlag Berlin Heidelberg 2007
150
S. Kawabata, S. Hiura, and K. Sato
in a specific area in 3D space using multiple cameras. We believe this system will help establish a safe and secure society. As mentioned above, flexibility and ease in setting up the equipment and detection region are important factors to the cost and practical use. However, there are two problems in image based intrusion detection: one is the necessity of the complex and nuisance calibration for a multiple camera system, and the other is the intuitiveness for defining a restricted area. Thus, we propose a method to complete the calibration and the restricted area definition simultaneously by simply moving a colored marker in front of the cameras.
2
Characterization and Simplification of the Intrusion Detection Problem
In the last decade of computer vision, there have been many studies to measure or recognize a scene taken by cameras in an environment. In particular, methods to extract or track a moving object in an image have been investigated with great effort and have rapidly progressed. In most of this research, the region of an object can be detected without consideration of the actual 3D shape. Therefore, although these techniques may be used for rough intrusion detection, they cannot handle detailed motion and deformation, such as whether a person is reaching for a dangerous machine or an object of value. On the other hand, there has been other research to reconstruct the whole shape of a target object from images taken by multiple cameras. Using this method, it is possible to detect the intrusion of an object in a scene by computing the overlapping region of the restricted area and target object. This approach is not reasonable because the reconstruction computation generally needs huge CPU and memory resources, and, as described later, the approach involves unnecessary processes to detect an intrusion. In addition, it is not easy for users to set up such a system because the cameras must be calibrated precisely. Thus, we resolve these issues by considering two characteristics of the intrusion detection problem. The first is the projective invariance of the observed space in intrusion detection. The state of intrusion, that is, the existence of an overlapping region of a restricted area and object, is invariant if the entire scene is projectively transformed. Hence, we can use weak calibration, instead of full calibration, to detect an intrusion. Furthermore, setting the restricted area can be done simultaneously with the calibration, because the relationship between the area and cameras can also be represented in a projective space. Although the whole shape of an intruding object has projective indefiniteness, it doesn’t affect the detection of intrusion. The second characteristic is that a restricted area is always a closed region. Consequently, we do not have to check the total volume of a restricted area; it is sufficient to observe only the boundary of the restricted area. This manner of thinking is one of the standard approaches for ensuring safety, and is also adopted by the abovementioned light curtain. Our system detects an intrusion by projecting the silhouette on each camera image onto the boundary plane, then
3D Intrusion Detection System with Uncalibrated Multiple Cameras
151
computing the common region of all the silhouettes. This common region on the boundary plane is equivalent to the intersection of the reconstructed shape of an object by the visual hull method and the shape of the boundary plane. The remainder of this paper is organized as follows. In the next section, the principle of our approach is described. We explain our approach in more detail in Section 3. In Section 4, we derive the simultaneous initialization (calibration and restricted area setting). We describe an experiment of intrusion detection in Section 5. In Section 6 we present our conclusion.
3 3.1
Detection of an Intruding Object The Visual Hull Method
To decide if an object exists in a specific area, the 3D shape of the object in the scene must be obtained. We adopt the visual hull method for shape reconstruction. In the visual hull method, the shape of an object can be reconstructed by computing the intersection of all cones, which are defined by a set of rays through the viewpoint and one point on the edge of the silhouette on an image plane. This method has the advantage that the texture of an object does not affect the reconstructed shape, because there is no necessity to search the corresponding points between images. However, this method tends to reconstruct a shape larger than the real shape, particularly with concave surfaces. Also, an invisible area from any of the cameras can also make it impossible to measure the shape. Although this is a common problem for image-based surveillance, our approach is always safe because the proposed system handles the invisible area as a part of the object. Although the visual hull method has great merit for intrusion detection, it needs large computational resources for the set operation in 3D space. Therefore, it is difficult to construct an intrusion detection system that is reasonable and works in real time. 3.2
Section Shape Reconstruction on a Sensitive Plane
As mentioned above, it is sufficient to observe only sensitive planes, the boundary of a restricted area, for intrusion detection. Accordingly, only the shape of the intersection region on a sensitive plane is reconstructed by homography based volume intersection [2]. In this case, the common region of projected silhouettes on the plane is equivalent to the intersection of the visual hull and the plane. Therefore, when an object exceeds a sensitive plane, the common region appears on the plane (Fig. 1). In this way, the 3D volumetric intrusion detection problem is reduced to efficient processes of inter-plane projection and common region computation in 2D space. 3.3
Vector Representation of the Silhouette Boundary
The visual hull method only uses information of the boundary of a silhouette. Therefore, the amount of data can be decreased by replacing the boundary with
152
S. Kawabata, S. Hiura, and K. Sato
(a) Non-intruding
(b) Intruding
Fig. 1. Intrusion detection based on the existence of an intersection
Fig. 2. Vector representation of silhouette contours
vector representation by tracking the edge of the silhouette in an image (Fig. 2). In the vector representation, the projection between planes is achieved by transforming a few vertices on the edge. It is easy for the common region computation to decide whether each vertex is inside or outside the other contour. With this representation, we are able to reduce the computational costs for the transformation and common region calculation, and it is not necessary to adjust the resolution of the sensitive plane to compute the common region with sufficient preciseness. In a distributed vision system, it is possible to reduce the amount of communication data because many camera-connected nodes extract silhouette contours and one host gathers the silhouette data and computes the common region. 3.4
Procedure of the Proposed System
For summarization, intrusion detection on the boundary is realized by the following steps: 1. 2. 3. 4. 5. 6.
Defining sensitive planes. Extracting the silhouette of a target object. Generating the vector representation from the silhouette. Projecting each silhouette vector onto sensitive planes. Computing the common region. Deciding the intrusion.
In the next section, we discuss step 1.
4
Construction of a Restricted Area
Using the following relationship, the silhouette of an object on an image plane can be transformed onto a sensitive plane. Let x(∈ 2 ) be the coordinate of a
3D Intrusion Detection System with Uncalibrated Multiple Cameras
153
㪭㫀㪼㫎㫇㫆㫀㫅㫋
㪠㫄㪸㪾㪼㩷㫇㫃㪸㫅㪼
㪪㪼㫅㫊㫀㫋㫀㫍㪼㩷㫇㫃㪸㫅㪼
Fig. 3. Homography between two planes
point on a sensitive plane. The corresponding point on the image plane can be calculated, as follows: ˜, μ˜ x = H x ⎡ ⎤ h11 h12 h13 H = ⎣h21 h22 h23 ⎦ h31 h32 h33
(1) (2)
˜ is the notation of homogeneous coordinates of x. Matrix H is referred where x to as a homography matrix, which has only 8 DOF for the scale invariant. From Eq. (2), the homography matrix can be determined by more than four pairs of corresponding points which are specified by a user. However, this method is a burden to users, who must set up the system in proportion to the product of the number of cameras and the number of sensitive planes. Also, it is not easy for users to define an arbitrarily restricted area without a reference object. Therefore, in the next section, we introduce a more convenient method for setting a sensitive plane. 4.1
Relation of the Homography Matrix and Projection Matrix
Instead of specifying the points on an image from a camera view, it is easy to place a small marker in the real observed space so that we obtain the corresponding points using cameras. However, in this case, it is difficult to point out the four points on a plane in real 3D space. Therefore, we consider the method in which users input enough ‘inner’ points of the restricted area so that the system automatically generates a set of sensitive planes which cover all the input points. Now, when we know the projection matrix P , which translates a coordinate in a scene onto an image plane, the relationship between X, a point in 3D space, and x, a point on an image plane, is given by ˜ λ˜ x = P X.
(3)
Likewise, as shown in Fig. 4, a point on the plane Π in 3D space is projected onto the image plane as follows.
154
S. Kawabata, S. Hiura, and K. Sato
㪊㪄㪛㩷㫇㫃㪸㫅㪼 㪭㫀㪼㫎㫇㫆㫀㫅㫋 㪠㫄㪸㪾㪼㩷㫇㫃㪸㫅㪼 Fig. 4. A plane in 3D space projected onto the image plane
˜ 0) λ˜ x = P (α˜ e1 + β˜ e2 + π ⎡ ⎤ α ˜1 e ˜2 π ˜ 0 ⎣β ⎦ =P e 1
(4) (5)
where e1 , e2 are bases of Π in 3D, and π 0 , (α, β) are the origin and parameter of Π, respectively. From Eq. (5), we can compute the homography matrix between an arbitrary plane in 3D and the image plane by ˜1 e ˜2 π ˜0 . H=P e (6) Therefore, when we know the projection matrices of the cameras and are given three or more points on a plane in 3D, it is possible to define the plane as a sensitive plane, except in a singular case (e.g., all points are on a line.). For example, the three adjacent points X 0 , X 1 , X 2 make one plane: ⎧ ⎨ e1 := X 1 − X 0 , e2 := X 2 − X 0 , (7) ⎩ π0 := X 0 . As mentioned above, a set of homogeneous matrices can be automatically generated from each given camera projection matrix and the vertices of the sensitive planes in 3D space. However, in our problem, we assume both the camera parameters and 3D points are unknown. Therefore, we have to calculate both by the projective reconstruction technique [3] using the given corresponding points between cameras. 4.2
Generation of Sensitive Planes from Reconstructed Inner Points
Now we have the projection matrices and many reconstructed 3D points which reside in the restricted area, so we have to determine enough pairs of 3D points as the vertices of the sensitive planes. We compute the convex hull, which handles all the input points for generating sensitive planes. The system defines a restricted
3D Intrusion Detection System with Uncalibrated Multiple Cameras
155
5GPUKVKXG2NCPG5GVWR +PRWVVKPIRQKPVUD[COCTMGT 2TQLGEVKXGTGEQPUVTWEVKQP %QPXGZJWNNECNEWNCVKQP )GPGTCVKQPQHUGPUKVKXGRNCPGU
+PVTWUKQP&GVGEVKQP 5KNJQWGVVGGZVTCEVKQP 5KNJQWGVVGXGEVQTK\CVKQP 2TQLGEVKQPQPVQUGPUKVKXGRNCPGU %QOOQPTGIKQPEQORWVCVKQP
Fig. 5. Points and their convex hull Fig. 6. Flow chart of the proposed system (2D case)
area as the boundary of the convex hull computed using qhull [4] (Fig. 5). The reconstructed points, except on the boundary, are removed because they do not make a sensitive plane.
5
Experiment
We implemented the proposed intrusion detection method in a multiple-camera system. From the users’ view, the system has two phases: one is setting the sensitive planes and the other is executing intrusion detection (see Fig. 6). Since the latter phase is completely automated, users need only to input corresponding points with a simple marker. Therefore, any complicated technical process, such as calibration of the multiple camera system, is already managed for setting the actual sensitive plane. In this experiment, we confirm the proposed method of sensitive plane generation and intrusion detection in projective space. The system consists of three cameras (SONY DFW-VL500) and a PC (Dual Intel Xeon @ 3.6 [GHz] w/ HT). We set the cameras at an appropriate position so that each camera can observe the whole region to detect an intrusion (Fig. 7). 5.1
Input of Sensitive Plane Using a Colored Marker
We use a simple red colored marker to input corresponding points among all image planes. First, the user specifies the color of the marker by clicking on the area of the marker, then the system computes the mean and the variance of the area. According to the Mahalanobis distance between an input color at each pixel and the reference color, the system extracts similar pixels by thresholding the distance. For noise reduction, the center of gravity of the largest region is
156
S. Kawabata, S. Hiura, and K. Sato
Fig. 7. Cameras and observed Fig. 8. Setting of restricted area (top: camera space view, bottom: extracted marker position)
Fig. 9. Inputted points and generated convex hull
calculated as the marker position (Fig. 8). The user in a real scene moves the marker position to set up the restricted area. Fig. 9 shows an example of the sensitive planes generated from inputted points. In this case, 16 sensitive planes are generated from 10 of 12 inputted points, and remaining two points of them are removed because they are inside of the convex hull. 5.2
Intrusion Detection
In this experiment, we input eight points on the vertices of a hexahedron. Fig. 10 depicts the generated set of sensitive planes from the input points. In this case, 12 planes are generated by the proposed method. The result of the intrusion detection is shown in Fig. 11. In our implementation, we use a statistical background subtraction method [5] to extract a silhouette of the object from an image. The silhouette is transformed into vector representation by tracking the edge and projected onto each sensitive
3D Intrusion Detection System with Uncalibrated Multiple Cameras
157
Fig. 10. Generated sensitive planes
Fig. 11. Detection result (top: intrusion of a leg, bottom: intrusion of a wrist, reaching for the object)
plane. Then, the system computes the common region on each sensitive plane. In the figure, the leg or wrist of the intruder is detected on the boundary of the restricted area. Although one can see some false positive extraction areas of the silhouette (e.g., the shadow cast in the image of the top row, third column), our method has a robustness against such noise because of the common region computation of all extracted silhouettes.
158
6
S. Kawabata, S. Hiura, and K. Sato
Conclusion
In this paper, we introduce an intrusion detection system for an arbitrary 3D volumetric restricted area using uncalibrated multiple cameras. Although our algorithm is based on the visual hull method, the whole shape of intruding object does not need to be reconstructed; instead, the system can efficiently detect an intrusion by perspective projections in 2D space. In general, an intricate calibration process for a distributed camera system has been necessary, but the proposed system automatically calibrates the cameras when users input corresponding points through the restricted region setting. Furthermore, the user does not need any previous knowledge about cameras because of the projective reconstruction. Also, any combination of cameras having varying intrinsic camera parameters can be used. Therefore, non-expert users can intuitively operate the proposed system for intrusion detection by only setting the cameras in place.
References 1. Collins, R.T., Lipton, A.J., Kanade, T., Fujiyoshi, H., Duggins, D., Tsin, Y., Tolliver, D., Enomoto, N., Hasegawa, O., Burt, P., et al.: A system for video surveillance and monitoring (VSAM project final report). Technical report, CMU Technical Report CMU-RI-TR-00 (2000) 2. Wada, T., Wu, X., Tokai, S., Matsuyama, T.: Homography Based Parallel Volume Intersection: Toward Real-Time Volume Reconstruction Using Active Cameras. In: Proc. Computer Architectures for Machine Perception, pp. 331–339 (2000) 3. Mahamud, S., Hebert, M.: Iterative projective reconstruction from multiple views. In: Proc. CVPR, vol. 2, pp. 430–437 (2000) 4. Barber, C.B., Dobkin, D.P., Huhdanpaa, H.: The quickhull algorithm for convex hulls. ACM Trans. Mathematical Software (TOMS) 22(4), 469–483 (1996), http://www.qhull.org 5. Horprasert, T., Harwood, D., Davis, L.S.: A statistical approach for real-time robust background subtraction and shadow detection. In: ICCV 1999, pp. 1–19 (1999)
Non-parametric Background and Shadow Modeling for Object Detection Tatsuya Tanaka1 , Atsushi Shimada1, Daisaku Arita1,2 , and Rin-ichiro Taniguchi1 1
Department of Intelligent Systems, Kyushu University, 744, Motooka, Nishi-ku, Fukuoka 819–0395 Japan 2 Institute of Systems & Information Technologies/KYUSHU 2–1–22, Momochihama, Sawara-ku, Fukuoka 814–0001 Japan
Abstract. We propose a fast algorithm to estimate background models using Parzen density estimation in non-stationary scenes. Each pixel has a probability density which approximates pixel values observed in a video sequence. It is important to estimate a probability density function fast and accurately. In our approach, the probability density function is partially updated within the range of the window function based on the observed pixel value. The model adapts quickly to changes in the scene and foreground objects can be robustly detected. In addition, applying our approach to cast-shadow modeling, we can detect moving cast shadows. Several experiments show the effectiveness of our approach.
1 Introduction Background subtraction technique has been traditionally applied to detection of objects in image. Without prior information about the objects, we can get object regions by subtracting a background image from an observed image. However, when simple background subtraction technique is applied to video-based surveillance which usually captures outdoor scenes, it often detects not only objects but also a lot of noise regions. This is because it is quite sensitive to small illumination changes caused by moving clouds, swaying tree leaves, etc. There are many approaches to handle these background changes [1,2,3,4]. Han et al. proposed a background estimation method, in which mixture-of-Gaussians is used to approximate background model, and the number of Gaussians is variable in each pixel. Their method can handle variations in lighting since a Gaussian is inserted or deleted according to the illuminant condition. However, it takes a long time to estimate background model. There are also several approaches to estimate background model in shorter time [5,6]. For example, Stauffer et al. proposed a fast estimation method to avoid a costly matrix inversion by ignoring covariance components of multi-dimensional Gaussians [6]. However, the number of Gaussians is constant in their background model. When recently observed pixel values frequently change, a constant number of Gaussians is not always enough to estimate the background model accurately, and it is very difficult to determine the appropriate number of Gaussians in advance. Shimada et al proposed a fast method in which the number of Gaussians are changed dynamically to adapt to the change of the lighting condition [7]. However, in principle, Gaussian Mixture Model (GMM) can not make a well-suited background model and can not detect foreground objects accurately when the intensity Y. Yagi et al. (Eds.): ACCV 2007, Part I, LNCS 4843, pp. 159–168, 2007. c Springer-Verlag Berlin Heidelberg 2007
160
T. Tanaka et al.
of the background changes frequently. Especially when the intensity distribution of the background is very wide, it is not easy to represent the distribution with a set of Gaussians. In addition, if the number of Gaussians is increased, the computation time to estimate the background model is also increased. Thus, GMM is not powerful enough to represent the various changes of the lighting condition. To solve the problem, Elgammal et al employed non-parametric representation of the background intensity distribution, and estimated the distribution by Parzen density estimation [2]. However, in their approach, the computation cost of the estimation is quite high, and it is not easy to apply it to real-time processing. Another problem of background subtraction is that detected foreground regions generally include not only objects to be detected but their cast shadows since the shadow intensity differs from that of the modeled background. This misclassification of shadow regions as foreground objects can cause various unwanted behavior such as object shape distortion and object merging, affecting surveillance capability like target counting and identification. To obtain better segmentation quality, detection algorithms must correctly separate foreground objects from the shadows they cast. Then, various approachs has been proposed [8,9,10,11,12]. Martel-Brisson et al. proposed a shadow detection method [12], in which detection of moving cast shadows is incorporated into a background subtraction algorithm. However, they use GMM to model background and shadow, and the aforementioned problem of GMM remains. In this paper, we propose a fast algorithm to estimate non-parametric probability distribution based on Parzen density estimation, which is applied to background modeling. Also, applying our approach to chast-shadow modeling the shadow models, we can detect moving cast shadow. Several experiments show its effectiveness, i.e., its accuracy and computation efficiency.
2 Background Estimation by Parzen Density Estimation 2.1 Basic Algorithm At first, we describe basic background model estimation and object detection process. The backgrounmd model is established to represent recent pixel information of an input image sequence, reflecting the change of intensity, or pixel-value, distribution as quickly as possible. We consider values of a particular pixel (x, y) over time as a “pixel process”, which is a time series of pixel values, e.g. scalars for gray values and vectors for color images. Each pixel is judged to be a foreground pixel or a background pixel by observing the pixel process. In Parzen density estimation, or the kernel density estimation, the probability density function (PDF) of a pixel value is estimated referring to the latest pixel process, and, here, we assume that a pixel process consists of the latest N pixel values. Let X be a pixel value observed at pixel (x, y), and {X 1 , · · · , X N } be the latest pixel process. The PDF of the pixel value is estimated with the kernel estimator K as follows P (X) =
N 1 K(X − X i ) N i=1
(1)
Non-parametric Background and Shadow Modeling for Object Detection
161
Usually a Gaussian distribution function N (0, Σ) is adopted for the estimator K 1 . In this case the equation (1) is reduced to as follows: N 1 1 1 −1 T exp − (X − X i ) Σ (X − X i ) P (X) = N i=1 (2π) d2 |Σ| 12 2
(2)
where d is the dimension of the distribution (for example, d = 3 in color image pixels). To reduce the computation cost, the covariance matrix in equation (2) is often approximated as follows: ⎞ ⎛ 2 σ1 0 · · · 0 ⎜ . ⎟ ⎜ 0 σ22 . . . .. ⎟ ⎟ (3) Σ=⎜ ⎟ ⎜ .. . . . . ⎝ . . . 0⎠ 0 · · · 0 σd2 . This means that each dimension of the distribution is independent from one another. By this approximation, equation (2) is reduced into the following.
2 N d 1 1 1 (X − Xi )j P (X) = exp − (4) N i=1 j=1 (2πσj2 ) 12 2 σj2 This approximation might make the density estimation error a little bigger, but the computation is considerably reduced. The detailed algorithm of background model construction and foreground object detection is summarized as follows: 1. When a new pixel value X N +1 is observed, P (X N +1 ), the probability that X N +1 occurs is estimated by equation (4). 2. If P (X N +1 ) is greater than a given threshold, the pixel is judged to be a background pixel. Otherwise, it is judged to be a foreground pixel. 3. The newly observed pixel value X N +1 is kept in the “pixel process,” while the oldest pixel value X 1 is removed from the pixel process. Applying the above calculation to every pixel, the background model is generated and distinction between a background pixel and a foreground pixel is accomplished. 2.2 Fast Algorithm When we estimate the generation probability of pixel value X in every frame using equation (4) and estimate the background model, its computation cost becomes quite large. To reduce the computation, Elgammal et al. computed the kernel, K, for all possible (X − X i ) in advance, which is stored in a look-up table. However, in their method, computation cost of N -times addition in the kernel K() in equation (1) is not small, which makes the computation time for background estimation large. To solve this problem, we have developed a fast estimation scheme of the PDF as follows. 1
Here, Σ works as the smoothing parameter.
162
T. Tanaka et al.
P( X )
Pt ( X ) = Pt −1 ( X ) +
1 ⎛ | X − X N +1 | ⎞ 1 ⎛ | X − X1 | ⎞ Ȁ⎜ Ȁ⎜ ⎟− ⎟ d Nh d ⎝ h h ⎠ ⎠ Nh ⎝
Update within the range of the window function
h=5
Background
K(u)
d=1
1 h
Threshold
u
䈅䈅䈅 − h
2
0
h 2
Fig. 1. Kernel function of our algorithm
Pixel Value
Observed value
Oldest data
Fig. 2. Update of background model
At first, we use a kernel with rectangular shape, or hypercube, instead of Gaussian distribution function. For example, in 1-dimensional case, the kernel is represented as follows (see Figure 1). 1 if − 12 ≤ u ≤ h2 K(u) = h (5) 0 otherwise where h is a parameter representing the width of the kernel,i.e., some smoothing parameter [13]. Using this kernel, equation (1) is represented as follows: N |X − X i | 1 1 ψ (6) P (X) = N i=1 hd h where, |X − X i | means the chess-board distance in d-dimensional space, and ψ(u) is calculated by the following formula. 1 if u ≤ 12 ψ(u) = (7) 0 otherwise When an observed pixel value is inside of the kernel located at X, ψ(u) is 1; otherwise ψ(u) is 0. Thus, we estimate the PDF based on equation (6), and P (X) is calculated by enumerating pixels in the latest pixel process whose values are inside of the kernel located at X. However, if we calculate the PDF, in a naive way, by enumerating pixels in the latest pixel process whose values are inside of the kernel located at X, the computational time is proportional to N . Instead, we propose a fast algorithm to compute the PDF, whose computation cost does not depend on N . In background modeling we estimate P(X) referring to the latest pixel process consisting of pixel values of the latest N frames. Let us suppose that at time t we have a new pixel value X N +1 , and that we estimate an updated PDF P t (X) referring to the new X N +1 . Basically, the essence of PDF estimation is accumulation of the kernel
Non-parametric Background and Shadow Modeling for Object Detection
163
estimator, and, when a new value, X N +1 , is acquired the kernel estimator corresponding to X N +1 should be accumulated. At the same time, the oldest one, i.e., the kernel estimator at N frames earlier, should be discarded, since the length of the pixel process is constant, N . This idea leads to reduction of the PDF computation into the following incremental computation: 1 1 |X − X N +1 | |X − X 1 | )− ) (8) ψ( ψ( N hd h N hd h where Pt−1 is the PDF estimated at the previous frame. The above equation means that the PDF when a new pixel value is observed can be acquired by: Pt (X) = Pt−1 (X) +
– increasing the probabilities of pixel values which are inside of the kernel located at the new pixel value X N +1 by N1hd – decreasing those which are inside of the kernel located at the oldest pixel value, a pixel value at N frames earlier, X 1 by N1hd . In other words, the new PDF is acquired by local operation of the previous PDF, assuming the latest N pixel values are stored in the memory, which achieves quite fast computation of PDF estimation. Figure 2 illustrates how the PDF, or the background model, is modified.
3 Cast-Shadow Modeling by Parzen Density Estimation In this section, we propose a method to detect moving cast shadows in a background subtraction algorithm. We have developed a cast shadow detection method, which is based on a similar idea to [12], and it is based on the observation that a shadow cast on a surface will equally attenuate the values of three components of its YUV color. We first estimate this attenuation ratio referring to the component Y , and, then, we examine whether both U and V components are reduced by a similar ratio. More specifically, if color vector X represents the shadow cast on a surface whose background color vector is B, we have XY αmin < αY < 1 with αY = BY min{|X U |, |X V |} > αY − X U < Λ BU αY − X V < Λ BV
(9) (10) (11) (12)
where B means a pixel value of the highest probability. αmin is a threshold on maximum luminance reduction. This threshold is important when the U and V components of a color are small, in which case any dark pixel value would be labeled as a shadow for a light color surface. is a threshold for minimum value of the U and V components. If either X U or X V does not satisfy the equation (10), we use only equation (9). Λ represents the tolerable chromaticity fluctuation around the surface value B U ,
164
T. Tanaka et al.
B V . If these conditions are satisfied, the pixel value is regarded “pseudo-shadow”, and the shadow model is updated with the procedure which is similar to those which are expressed in section 2.2 making use of the pixel value. The detailed algorithm of shadow model construction and shadow detection is summarized as follows: 1. Background subtraction is done with the dynamic background model described in section 2.2. 2. If the pixel is labeled as foreground, PS (X N +1 ), the probability that X N +1 belongs to a cast shadow is estimated. If PS (X N +1 ) is greater than a given threshold, the pixel is judged to be a shadow pixel. Otherwise, it is judged to be an object pixel. However, in the shadow model, there is a possibility that the number of “pseudoshadow” pixel value which is necessary to approximate the shadow model is not enough, because the shadow model is updated only when the observed pixel is regarded as “pseudo-shadow”. Therefore in such pixel, equation(9)‘(12) are used for shadow detection. 3. When the observed pixel value satisfies the equation (9)‘(12), the pixel value is regarded as “pseudo-shadow”, the shadow model is updated by a similar way expressed in section 2.2.
4 Experiment 4.1 Experiment 1: Experiment on the Dynamic Background Model In our experiment verifying the effectiveness of the proposed method, we have used data set of PETS2001 2 after the image resolution is reduced into 320 × 240 pixels. The data set includes images in which people and cars are passing through streets, tree leaves are flickering, i.e., the illumination condition are varying rapidly. Using this data set, we have compared the proposed method with Adaptive GMM (GMM in which the number of Gaussians is adaptively changed) [7], and Elgammal’s method based on Parzen density estimation [2]. In this experiment, supposing that R, G, B components of pixel values are independent of one another, we estimate a one-dimensional PDF of each component. Then, we have judged a pixel as a foreground pixel when at least the probability of one component, either R, G or B, is below a given threshold. For the evaluation of computation speed, we have used a PC with a Pentium IV 3.3GHz and 2.5GB memory. Next, we have evaluated computation time to process one image frame. For the proposed algorithm, we have used h = 5, N = 500, Figure 3 shows comparison between the proposed method and the adaptive GMM method, where the horizontal axis is the frame-sequence number and where the vertical axes are the processing speed (left) and the average of number of Gaussians assigned to each pixel. In this experiment, after the 2500th frame, the number of Gaussians increases, where people and cars, i.e., foreground objects begin to appear in the scene. 2
Benchmark data of International Workshop on Performance Evaluation of Tracking and Surveillance. From ftp://pets.rdg.ac.uk/PETS2001/ available.
Non-parametric Background and Shadow Modeling for Object Detection Gaussian Mixture Model
2.5
Processing time (msec)
100 80
2
60
1.5
40
1
20
0.5
0
Processing time (msec)
Traditional approach
3 Number of Gaussians
Proposed method Number of Gaussians 120
0 0
1000
2000
Frame
3000
Accuracy (%)
Fig. 3. Processing time of adaptive GMM and average number of Gaussians
Proposed method
350 300 250 200 150 100 50 0 100
4000
165
200
300 400 Number of sample data
500
Fig. 4. The number of samples, or N , and require processing time
100 90 80 70 60 50 40 30 20 10 0
Recall Precission
Propose method
Gaussian Mixture Model
Traditional approach
Fig. 5. Recall and Precision
In the adaptive GMM method, the number of Gaussians is increased so that changes of pixel values are properly represented in GMM. However, when the number of Gaussians increases, the computation time also increases. On the other hand, the computation time of the proposed method does not change depending on the scene, which shows that the real-time characteristic, i.e., invariance of processing speed, of the proposed method is much better than the adaptive GMM method. Next, figure 4 shows comparison between the proposed method and Elgammal’s method based on Parzen density estimation. In the Elgammal’s method, the computation time is almost proportional to the length of the pixel process in which the PDF is estimated, and, from the viewpoint of real-time processing, we can not use long image sequence to estimate the PDF. For example, when we use a standard PC environment, like our experiment, only up to 200 frames can be used for the PDF estimation in the Elgammal’s method. On the other hand, in our method, when we estimate the PDF, we just update it in the local region, i.e., in the kernel located at the oldest pixel value and in the kernel located at the newly observed pixel value, and the computation cost does not depend on the length of the pixel process at all. Finally, to evaluate the object detection accuracy, we examine precision and recall rates of object detection. Precision and recall are respectively defined as follows: precision =
recall =
# correctly detected objects # of detected objects
# of correctly detected objects # of objects which should be detected
(13)
(14)
166
T. Tanaka et al.
(a) Input image
(b) Background image (c) Proposed method
(d) GMM method
Fig. 6. Object detection by the proposed method and GMM-based method
When we apply our proposed method and Elgammal’s method, we set N = 500. In addition, we set h = 5 in our method. Figure 5 shows precision and recall when the data set is processed by the proposed method, adaptive GMM method, and Elgammal’s method, where the vertical axis means recall and precision rate. This shows that the proposed method outperform the adaptive GMM method. Also, it is shown that the proposed method gives almost the same performance as Elgammal’s method, although, in the proposed method, we use a simple kernel function, i.e., rectangular function shown in Figure 5. We have achieved a recall rate with 94.38% and a precision rate with 88.89%. Figure 6 shows results of object detection by the proposed method. Figure 6(a) is an input image frame. Figure 6(b) is acquired background model when that input image frame is acquired, which shows a pixel value having the highest probability at each pixel. Figure 6(c) shows detected objects. Figure 6(d) shows object detection result acquired by the adaptive GMM method. Comparing these two result, the proposed method exhibits very good result. 4.2 Experiment 2: Experimtnt on the Dynamic Shadow Model We took indoor scenes where people were walking on the floor. Those images include shadows with various darkness which is cast from the pedestrian. The size of image is 320 × 240 and each pixel had a 24-bit RGB value. We have compared the proposed method with Adaptive Gaussian Mixture Shadow Model(GMSM) [12]. Furthermore, respectively, the dynamic background model which uses Parzen density estimation and which uses GMM are used to object detection. Figure 7 shows results of shadow detection by the proposed method. Figure 7(a) is an input image frame. Figure 7(b) shows shadow detection result acquired by the
Non-parametric Background and Shadow Modeling for Object Detection
(a) Input image
(b) Proposed method
167
(c) GMSM method
Fig. 7. Shadow detection by the proposed method and GMSM-based method
proposed method. Figure 7(c) shows shadow detection result acquired by the GMSM method. The red colored pixels represent pixels judged to be a shadow pixels. In Figure 7(b), the green colored pixels represent pixels judged to be shadow pixels just according to equation (9)‘(12), where they can not be examined by the probabilistic model, because the number of pseudo-shadow pixels is not enough to estimate the probability distribution of the shadow pixel value. Comparing these two results with each other, the proposed method exhibits a good result. In addition, the computation time of the proposed method is superior to that of GMSM, i.e., the former is 88 msec per image frame while the latter is 97 msec.
5 Conclusion In this paper, we have proposed a fast computation method to estimate non-parametric background model using Parzen density estimation. We estimate the PDF of background pixel value at each pixel position. In general, to estimate the PDF at every image frame, a pixel value sequence of the latest N frames, or a pixel process, should be referred to. In our method, using a simple kernel function, the PDF can be estimated from the PDF at the previous frame using local operations on the PDF. This much improves the computation cost of the PDF estimation. Comparison of our method with GMM-based method and Elgammal’s method based on Parzen density estimation shows that our method has the following merits: small computation cost, real-time characteristic (invariance of computation speed), object detection accuracy.
168
T. Tanaka et al.
In addition, applying our approach to shadow modeling, we can construct shadow model and detect moving cast shadows correctly. Comparison of our method with GMSM-based method shows its effectiveness, i.e., its accuracy and computation speed. Future works are summarized as follows: – Reduction of memory space. – Precision improvement of shadow detection.
References 1. Han, B., Comaniciu, D., Davis, L.: Sequential Kernel Density Approximation through Mode Propagation: Applications to Background Modeling. In: Asian Conference on Computer Vision 2004, pp. 818–823 (2004) 2. Elgammal, A., Duraiswami, R., Harwood, D., Davis, L.S.: Background and Foreground Modeling Using Nonparametric Kernel Density Estimation for Visual Surveillance. In: Proceedings of the IEEE, vol. 90, pp. 1151–1163 (2002) 3. Toyama, K., Krumm, J., Brumitt, B., Meyers, B.: Wallflower: Principle and Practice of Background Maintenance. In: International Conference on Computer Vision, pp. 255–261 (1999) 4. Harville, M.: A Framework for High-Level Feedback to Adaptive, Per-Pixel, Mixture-ofGaussian Background Models. In: the 7th European Conference on Computer Vision, vol. III, pp. 543–560 (2002) 5. Lee, D.-S.: Online Adaptive Gaussian Mixture Learning for Video Applications. In: Pajdla, T., Matas, J(G.) (eds.) ECCV 2004. LNCS, vol. 3021, pp. 105–116. Springer, Heidelberg (2004) 6. Stauffer, C., Grimson, W.E.L.: Adaptive background mixture models for real-time tracking. Computer Vision and Pattern Recognition 2, 246–252 (1999) 7. Shimada, A., Arita, D., Taniguchi, R.i.: Dynamic Control of Adaptive Mixture-of-Gaussians Background Model. In: Proceedings of IEEE International Conference on Advanced Video and Signal Based Surveillance 2006 (2006) 8. Salvador, E., Cavallaro, A., Ebrahimi, T.: SHADOW IDENTIFICATION AND CLASSIFICATION USING INVARIANT COLOR MODELS. In: Proc. of IEEE International Conference on Acoustics, vol. 3, pp. 1545–1548 (2001) 9. Cucchiara, R., Grana, C., Piccardi, M., Prati, A., Sirotti, S.: Improving Shadow Suppression in Moving Object Detection with HSV Color Information. In: IEEE Intelligent Transportation Systems Conference Proceedings, pp. 334–339 (2001) 10. Schreer, O., Feldmann, I., Golz, U., Kauff, P.: FAST AND ROBUST SHADOW DETECTION IN VIDEOCONFERENCE APPLICATION. 4th IEEE Intern. Symposium on Video Proces. and Multimedia Comm, 371–375 (2002) 11. Bevilacqua, A.: Effective Shadow Detection in Traffic Monitoring Applications. In: The 11th International Conference in Central Europe on Computer Graphics, Visualization and Computer Vision (2003) 12. Martel-Brisson, N., Zaccarin, A.: Moving Cast Shadow Detection from a Gaussian Mixture Shadow Model. IEEE Computer Society International Conference on Computer Vision and Pattern Recognition (2005) 13. Parzen, E.: On the estimation of a probability density function and mode. The Annals of Mathematical Statistics 33(3), 1065–1076 (1962)
Road Sign Detection Using Eigen Color Luo-Wei Tsai1, Yun-Jung Tseng1, Jun-Wei Hsieh2, Kuo-Chin Fan1, and Jiun-Jie Li1 1
Department of CSIE, National Central University Jung-Da Rd., Chung-Li 320, Taiwan
[email protected] 2 Department of E. E., Yuan Ze University 135 Yuan-Tung Road, Chung-Li 320, Taiwan
[email protected]
Abstract. This paper presents a novel color-based method to detect road signs directly from videos. A road sign usually has specific colors and high contrast to its background. Traditional color-based approaches need to train different color detectors for detecting road signs if their colors are different. This paper presents a novel color model derived from Karhunen-Loeve(KL) transform to detect road sign color pixels from the background. The proposed color transform model is invariant to different perspective effects and occlusions. Furthermore, only one color model is needed to detect various road signs. After transformation into the proposed color space, a RBF (Radial Basis Function) network is trained for finding all possible road sign candidates. Then, a verification process is applied to these candidates according to their edge maps. Due to the filtering effect and discriminative ability of the proposed color model, different road signs can be very efficiently detected from videos. Experiment results have proved that the proposed method is robust, accurate, and powerful in road sign detection.
1 Introduction Traffic sign detection is an important and essential task in a driver support system. The texts on road signs carry much useful information like limited speed, guided direction, and current traffic situation for helping the drivers drive safely and comfortably. However, it is very challenging to detect road signs directly from still images or videos due to the large changes of environmental conditions. In addition, when the camera is moving, the perspective effects will make a road sign have different sizes, shapes, contrast changes, or motion blurs. Moreover, sometimes it will be occluded with some natural objects like trees. To tackle the above problems, there have been many works [1]-[9]proposed for automatic road sign detection and recognition. Since a road sign usually has a high-contrast color and regular shape, these approaches can be categorized into color-based or shape-based ones. For the color-based approach, in [1], Escalera et al. used a color threshoding technique to separate road sign regions from the background in the RGB color domain. In addition to the RGB space, other color spaces like YIQ and HSV are also good for road sign detection. For example, in [2], Kehtarnavaz and Ahmad used a discriminant analysis on the YIQ color space for Y. Yagi et al. (Eds.): ACCV 2007, Part I, LNCS 4843, pp. 169–179, 2007. © Springer-Verlag Berlin Heidelberg 2007
170
L.-W. Tsai et al.
detecting desired road signs from the background. Since road signs have different colors (like red, blue, or green) for showing different warning or direction messages, different color detectors should be designed for tackling the above color variations. In addition to color, shape is another important feature for detecting road signs. In [7], Barnes and Zelinsky adopted the fast radial symmetry detector to detect possible road sign candidates and then to verify them using a correlation technique. Wu et al. [6] used the corner feature and a vertical plane criterion to cluster image data for finding possible road sign candidates. Blancard [9] used an edge linking technique and the contour feature to locate all possible road sign candidates and then verified them according to their perimeters and curvature features. Usually, different shapes of road signs represent different warning functions. Different shape detectors should be designed and make the detection process become very time-consumed. Therefore, there are some hybrid methods proposed for road sign detection. For example, Bahlmann et al. [8] used a color representation, integral features, and the AdaBoost algorithm for training a stronger classifier such that a real-time traffic sign detector can be achieved. Furthermore, Fang et al. [3] used fuzzy neural networks and gradient feature to locate and track road signs. The major disadvantage of the shape-based approach is that a road sign has large shape variations when the camera is moving. This paper presents a novel hybrid method to detect road signs from videos using eigen color and shape feature. First of all, this paper proposes a novel eigen color model for searching possible road signs candidates from videos. The model can make road sign colors be more compact and thus sufficiently concentrated on a smaller area. It is learned by observing how the road sign colors change in static images under different lighting conditions and cluttered backgrounds. It is global and doesn’t need to be re-estimated for any new road signs or new input images. Without prior knowledge of surface reflectance, weather condition, and view geometry is used in the training phase, the model still performs very efficiently to locate road sign pixels from the background. Even though road signs have different colors, only one single model is needed. After the transformation, the RBF network is used for finding the best hyper-plane to separate the road sign color from the background. Then, a verification engine is built to verify these candidates using their edge maps. The engine records appearance characteristic of road signs and has good discriminative properties to verify the correctness of each candidate. In this system, the eigen color model can filter out most of background pixels in advance. Only few candidates need to be further checked. In addition, no matter what color the road sign is, only one eigen color model is needed for color classification. Due to the filtering effect and discriminative abilities of the proposed method, different road signs can be effectively detected from videos. Experiment results have proved the superiority of the proposed method in road sign detection.
2 Eigen Color Detection A road sign usually has a specific color which is high contrast to the background. The color information can be used to narrow down the searching area for finding the road signs more efficiently. For example, in Fig. 1(a), the road sign has a specific “green” color. Then, we can use a green color detector to filter out all non-green color objects.
Road Sign Detection Using Eigen Color
(a)
171
(b)
Fig. 1. Green color detection in HIS color space. (a) Original image. (b) Result of green color detection.
However, after simple green color classification, there are many non-road-sign objects (with green color) to be detected as shown in Fig. 1(b). Precise color modeling method is necessary for road sign detection. In addition, different road signs have different specific colors (green, red, or blue). In contract to most previous system which designed different “specific” color detectors, this paper presents a single eigen color model to detect all kinds of road signs. 2.1 Eigen Color Detection Using Dimension Reduction Our idea is to design a single eigen-color transform model for detecting road sign pixels from the background. At first, we collect a lot of road sign images from various highways, roads, and nature images under different lighting and weather conditions. Fig. 2(a) shows parts of our training samples. Assume that there are N training images. Through a statistic analysis, we can get the covariance matrix ∑ of the color
(a)
(b)
Fig. 2. Parts of training samples. (a) Road sign images. (b) Non-road sign images.
distributions of R, G, and B channels from these N images. Using the Karhunen-Loeve(KL) transform, the eigenvectors and eigenvalues of ∑ can be further obtained and represented as ei and λi , respectively, for i = 1, 2, and 3. Then, three new color features Ci can be formed and defined, respectively,
172
L.-W. Tsai et al.
Ci = eir R + eig G + eib B for i =1, 2, and 3,
(1)
where ei = ( eir , eig , eib ) . The color feature C1 with the largest eigenvalue is the one used for color-to-gray transform, i.e.,
1 1 1 C1 = R + G + B . 3 3 3
(2)
Other two color features C2 and C3 are orthogonal to C1 and have the following forms: C2 =
2(R - B ) R +B-2G and C3 = . 5 5
(3)
In [10], Healey used the similar idea for image segmentation and pointed out that the colors of homogeneous dielectric surfaces will move close along the axis directed by Eq.(2), i.e., (1/3, 1/3, 1/3). In other words, if we try to project all the road sign colors to a plane which is perpendicular to the axis pointed by C1 , the road sign colors will concentrate around a small area. The above principal component analysis (PCA) gives us an inspiration to analyze road signs so that a new color model can be found.
(a)
(b)
Fig. 3. Eigen color re-projection. (a) Original image. (b) Result of projection on eigen color map.
This paper defines the plane ( C2 , C3 ) as a new color space (u, v). Then, given an input image, we first use Eq.(3) to project all color pixels on the (u, v) space. Then, the problem becomes a 2-class separation problem, which tries to find a best decision boundary from the (u, v) space such that all road sign color pixels can be well separated from non-road sign ones. Fig. 3(b) shows the projection result of road sign pixels and non-road sign pixels. The green and red regions denote the results of re-projection of green and red road signs, respectively. The blue region is the result of background. We also re-project the tree region and green road signs (shown in Fig. 3(a)) on the (u, v) space. Although these two regions are both “green”, they can be easily separated on the (u, v) space if a proper classifier is designed for finding the best separation boundary. In what follows, road sign pixels are fed into the RBF network for this classification task.
Road Sign Detection Using Eigen Color
173
2.2 Eigen Color Pixel Classification Using Radial Basis Function Network
A RBF network’s structure is similar to multilayer perceptrons. The RBF network we used includes an input layer, one hidden layer, and an output layer. Each hidden neuron is associated with a kernel function. The most commonly used kernel function (also called an activation function) is Gaussian. The output units is approximated as a linear combination of a set of kernel functions, i.e., R
ψ i ( x ) = ∑ wijϕ j ( x ) , for i=1, …, C, j =1
where wij is the connection weight between the jth hidden neuron and ith output layer neuron, and C the number of outputs. The output of the radial basis function is limited to the interval (0, 1) by a sigmoid function:
Fi ( x ) =
1 . 1 + exp(-ψ i ( x ))
When training the RBF network, we use the back-propagation rule to adjust the output connection weights, the mean vector, and the variance vectors of the hidden layer. The parameters wij of the RBF networks are computed by the gradient descent method such that the cost function is minimized: E=
1 N
N
C
∑∑ ( y (x ) − F ( x )) k =1 i =1
i
k
i
k
2
,
where N is the number of inputs and yi ( xk ) the ith output associated with the input sample xk from the training set. Then, if a pixel belongs to the road sign class, it will be labeled to 1; otherwise, 0. When training, all pixels in the (R, G, B) domain are first transformed to the (u, v) domain using Eq. (3).
3 Candidate Verification After color segmentation, different road sign candidates can be extracted. For verifying these candidates, we use road sign’s shape to filter out impossible candidates. The verification process is a coarse-to-fine scheme to gradually remove impossible candidates. At the coarse stage, two criteria are first used to roughly eliminate a large number of impossible candidates. The first criterion requires the dimension of a road sign R being large enough. The second criterion requires the road sign having enough edge pixels: ER / AreaR < 0.02 , where ER and Area R are the number of edge pixels and the area of R, respectively.
174
L.-W. Tsai et al.
(a)
(b)
(c)
Fig. 4. Result of distance transform. (a) Original Image. (b) Edge map. (c) Distance transform of (b).
After the coarse verification, a fine verification procedure is further applied to verifying each candidate using its shape. Assume that BR is a set of boundary pixels extracted from R. Then, the distance transform of a pixel p in R is defined as
DTR ( p ) = min d ( p, q) , q∈BR
(4)
where d ( p, q) is the Euclidian distance between p and q. In order to enhance the strength of distance changes, Eq.(4) is further modified as follows DT R ( p ) = min d ( p, q) × exp(κ d ( p, q)) , q∈BR
(5)
where κ = 0.1 . Fig. 4 shows the result of the distance transform. (a) is an image R of road sign and (b) is its edge map. Fig. 4(c) is the result of the distance transform of Fig. 4(b). If we scan all pixels of R in a row major order, a set FR of contour features can be represented as a vector, i.e.,
FR = [ DT R ( p0 ),...., DT R ( pi ),....] ,
(6)
where all pi belong to R and i is the scanning index. In addition to the outer contour, a road sign usually contains many text patterns. To verify a road sign candidate more accurately, its outer shape is more important than its inner text patterns. For reflecting this fact, a new weight wi which increases according to the distance between the pixel pi and the original O is included. Assume that O is the central of R and ri is the distance between pi and O, and the circumcircle of R has the radius z. Then, the weight wi is defined by: ⎧ exp(- | ri - z |2 ), if ri ≤ z; wi = ⎨ otherwise. ⎩0,
(7)
Then, Eq.(6) can be rewritten as follows: FR = [ w0 DT R ( p0 ),...., wi DT R ( pi ),....] .
(8)
This paper assumes that there are only three types of road signs, i.e., circle, triangle, and rectangle needed for verification. For each type Ri , a set of training samples is
Road Sign Detection Using Eigen Color
175
collected in advance for capturing shape characteristics. If there are N i templates in Ri , we can calculate its mean μi and variance Σi of FR from all samples in Ri . Then, given a road sign candidate H, the similarity between H and Ri can be measured by this equation: __
__
S ( H , Ri ) = exp(−( FH − ui ) ∑ i−1 ( FH − ui ) t ), ,
(9)
where t means the transpose of a vector.
4 Experimental Results To examine the performances of our proposed method, several video sequences on high way and roads were adapted. The sequences were captured under different road and weather conditions (like sunny, cloudy). The camera was embedded in the front position of the car and its optical axis is not required being perpendicular to the road sign. The frame rate of our system is over 20 fps. Fig. 5 shows the results of road sign color detection using the proposed method. For comparisons, the color thresholding technique [1] was also implemented. Fig. 5 (a) is the original image and Fig. 5 (b) is the result using the color thresholding technique. There were many false region detected for road sign detection in Fig. 5 (b). Fig. 5 (c) is the result of eigen color classification. It is noticed that only one eigen color model was used to detect all the desired road signs even though their colors were different. Compared with the thresholding technique, our proposed scheme has a much lower false detection rate. A lower false detection rate means that less computation time
(a)
(b)
(c)
Fig. 5. Result of color classification. (a) Original image. (b) Detection result of color thresholding [1]. (c) Eigen color classification.
176
L.-W. Tsai et al.
needed for candidate verification. In addition, the color thresholding technique needs several scanning passes to detect road signs if they have different colors. Thus, our method can perform much efficiently than traditional color-based approaches.
Fig. 6. Detection results of rectangular road sign
Fig. 7. Detection result when a skewed road sign or a low-quality frame was handled
Fig. 6 shows the detection results when rectangular road signs were handled. Even though the tree regions have similar color to the road signs, our method still worked very well to detect the desired road signs. Fig. 7 shows the detection results when skewed road signs or a low-quality video frame were handled. No matter how skewed and what color the road sign is, our proposed method performed well to detect it from the background.
Fig. 8. Detection results of circular road signs
Fig. 8 shows the detection results when the circular road signs were captured under different lighting conditions. The conditions included low lighting, skewed shape, or multiple signs. However, our method still worked well to detect all these circular road signs. Furthermore, we also used our scheme to detect triangular road signs. Fig. 9 shows the detection results when triangular roads were handled. No matter what types or colors of road signs were handled, our proposed method worked very successfully to detect them.
Road Sign Detection Using Eigen Color
177
Fig. 9. Detection results of triangular road signs
Fig. 10. Road sign detection in a video sequence under a sunny day. (a), (b), and (c): Consecutive detection results of a road sign from a video.
The next set of experiments was used to demonstrate the performances of our method to detect road signs under different weather conditions in video sequences. Fig. 10 shows a series of detection results when consecutive video frames under a sunny day were handled. In Fig. 10 (a) and (b), a smaller and darker road sign was detected. Then, its size gradually became larger. Fig. 10 (c) shows the detection result of a larger road sign. Clearly, even though the road sign had different size changes, all its variations were successfully detected using our proposed method. Fig. 11 shows the detection result when a series of road signs captured under a cloudy day were handled. In Fig. 11 (a), a very smaller road sign was detected. Its color was also darker. In Fig. 11(b), (c), and (d), its size gradually became larger. No matter how size of the road sign changes, it still can be well detected using our proposed method. Experiment results have proved the superiority of our proposed method in real time road sign detection.
178
L.-W. Tsai et al.
Fig. 11. Road sign detection in a video sequence under a cloudy day
5 Conclusion This paper presents a novel eigen color model for road sign detection. With this model, different road sign candidates can be quickly located no matter what colors they have. The model is global and doesn’t need to be re-estimated. Even though the road signs are lighted under different illuminations, the model still works very well to identify them from the background. After that, a coarse-to-fine verification scheme is applied to effectively identify all candidates according to their edge maps. Since most impossible candidates have been filtered in advance, desired road signs can be located very quickly. Experimental results have proved the superiority of our proposed method in real time road sign detection.
References [1] Escalera, A.D.L., et al.: Road Traffic Sign Detection and Classification. IEEE Transaction on Industrial Electronics 44(6), 848–859 (1997) [2] Kehtarnavaz, N., Ahmad, A.: Traffic sign recognition in noisy outdoor scenes. In: Proceedings of Intelligent Vehicles 1995 Symposium, pp. 460–465 (September 1995) [3] Fang, C.-Y., Chen, S.-W., Fuh, C.-S.: Road-sign detection and tracking. IEEE Transactions on Vehicular Technology 52(5), 1329–1341 (2003) [4] Chen, X., Yang, J., Zhang, J., Waibel, A.: Automatic detection and recognition of signs from natural scenes. IEEE Transactions on Image Processing 13(1), 87–99 (2004) [5] Loy, G., Barnes, N.: Fast shaped-based road sign detection for a Driver Assistance System. In: IROS 2004 (2004)
Road Sign Detection Using Eigen Color
179
[6] Wu, W., Chen, X., Yang, J.: Detection of Text on Road Signs From Video. IEEE Transactions on ITS 6(4), 378–390 (2005) [7] Barnes, N., Zelinsky, A.: Real-time radial symmetry for speed sign detection. In: Proc. IEEE Intelligent Vehicles Symposium, Italy, pp. 566–571 (June 2004) [8] Bahlmann, C., et al.: A system for traffic sign detection, tracking, and recognition using color, shape, and motion information. In: Proceedings of IEEE Intelligent Vehicles Symposium, pp. 255–260 (June 2005) [9] de Saint Blancard, M.: Road Sign Recognition: A Study of Vision-based Decision Making for Road Environment Recognition, ch. 7. Springer, Heidelberg (1991) [10] Healey, G.: Segmenting Images Using Normalized Color. IEEE Transactions on Systems, Man, and Cybernetics 22(1), 64–73 (1992)
Localized Content-Based Image Retrieval Using Semi-Supervised Multiple Instance Learning Dan Zhang1 , Zhenwei Shi2 , Yangqiu Song1 , and Changshui Zhang1 State Key Laboratory on Intelligent Technology and Systems Tsinghua National Laboratory for Information Science and Technology (TNList), Department of Automation,Tsinghua University, Beijing 100084, China Image Processing Center, School of Astronautics, Beijing University of Aeronautics and Astronautics, Beijing 100083, P.R. China
[email protected],
[email protected],
[email protected],
[email protected]
1
2
Abstract. In this paper, we propose a Semi-Supervised MultipleInstance Learning (SSMIL) algorithm, and apply it to Localized ContentBased Image Retrieval(LCBIR), where the goal is to rank all the images in the database, according to the object that users want to retrieve. SSMIL treats LCBIR as a Semi-Supervised Problem and utilize the unlabeled pictures to help improve the retrieval performance. The comparison result of SSMIL with several state-of-art algorithms is promising.
1
Introduction
Much work has been done in applying Multiple Instance Learning (MIL) to Localized Content-Based Image Retrieval (LCBIR). One main reason is that, in LCBIR, what a user wants to retrieve is often an object in a picture, rather than the whole picture itself. Therefore, in order to tell the retrieval system what he really wants, the user often has to provide several pictures with the desired object on it, as well as several pictures without this object, either directly or through relevance feedback. Then, each picture with the desired object is treated as a positive bag, while the other query pictures will be considered as negative ones. Furthermore, after using image segmentation techniques to divide the images into small patches, each patch represents an instance. In this way, the problem of image retrieval can be converted to an MIL one. The notion of Multi-Instance Learning was first introduced by Dietterich et al. [1] to deal with the drug activity prediction. A collection of different shapes of the same molecule is called a bag, while its different shapes represent different instances. A bag is labeled positive if and only if at least one of its instances is positive; otherwise, this bag is negative. This basic idea was extended by several following works. Maron et al. [2] proposed another MIL algorithm - Diverse Density (DD). They tried to find a target in the feature space that resembled positive instance most, and this target was called a concept point. Then they applied this
The work was supported by the National Science Foundation of China (60475001, 60605002).
Y. Yagi et al. (Eds.): ACCV 2007, Part I, LNCS 4843, pp. 180–188, 2007. c Springer-Verlag Berlin Heidelberg 2007
Localized Content-Based Image Retrieval Using SSMIL
181
method to solve the task of natural scene classification [3]. Zhang and Goldman [6] combined Expectation Maximization with DD together and developed an algorithm EM-DD, which was much more efficent than the previous DD algorithm, to search for the desired concept. They extended their idea in [7] and made some modifications to ensemble the different concept points returned by EM-DD with different initial values. This is reasonable, since the desired object can not be described by only one concept point. Andrew et al. [10] used a SVM based method to solve the MI problem. Then, they developed an efficient algorithm based on linear programming boosting [11]. Y. Chen et al. [4] combined EM-DD and SVM, and devised DD-SVM. Recently, P.M. Cheung et al.[9] give a regularized framework to solve this problem. Z. H. Zhou etc. [15], also initiate some research on Multiple-Instance Multiple-Label problem and apply it to scene classification. All the above works assume that each negative bag should not contain any positive instance. But there may exist exceptions. After the image segmentation, the desired object may be divided into several different patches. The pictures without this object may also contain a few particular patches that are similar to that of the object and should not be retrieved. So, negative bags may also contain positive instances, if we consider each patch as an instance. Based on this assumption, Y. Chen et al. [5] recently devised a new algorithm called MultipleInstance Learning via Embedded Instance Selection (MILES) to solve multiple instance problems. So far, some developments of MIL have been reviewed. When it comes to LCBIR, one natural problem is that users are often unwilling to provide so many labeled pictures, and therefore the inadequate number of labeled pictures poses a great challenge to the existing MIL algorithms. Semi-Supervised algorithms are just devised to handle the situation when the labeled information is inadequate. Some typical semi-supervised algorithms include Semi-Supervised SVM [13], Transductive SVM [12], graph-based semi-supervised learning [14], etc. How to convert a standard MIL problem to a Semi-Supervised one has received some notices. Recently, R. Rahmani and S. Goldman combined a modified version of DD and graph-based semi-supervised algorithms together, and put forward the first graph-based Semi-Supervised MIL algorithm - MISSL[8]. They adopted an energy function to describe the likelihood of an instance being the concept points, and redefined the weights between different bags. In this paper, we propose a new algorithm - Semi-Supervised Multiple-Instance Learning (SSMIL) to solve the Semi-Supervised MIL problem, and the result is promising. Our paper is outlined as follows: in Section 2, the motivation of our algorithm will be introduced. In Section 3, we will give the proposed algorithm. In Section 4 the experimental results will be presented. In the end, a conclusion is given in Section 5.
2
Motivation
A bag can be mapped into a feature space determined by the instances in all the labeled bags. To be more precise, a bag B is embedded in this feature space as follows [5]:
182
D. Zhang et al.
m(B) = [s(x1 , B), s(x2 , B), · · · , s(xn , B)]T
(1)
Here, s(xk , B) = maxt exp(− ||btσ−2x || ). σ is a predefined scaling parameter. xk is the kth instance among all the n instances in the labeled bags and bt denotes the tth instance in the bag B. Then, the whole labeled set can be mapped to such a matrix: k
+ − − [m+ 1 , · · · , m l+ , m 1 , · · · , m l− ] + − =⎡ [m(B+ ), · · · , m(B− 1 ), · · · , m(Bl+ ), m(B1 l− )] + − ⎤ 1 1 s(x , B1 ) · · · s(x , Bl− ) − ⎥ 2 ⎢ s(x2 , B+ 1 ) · · · s(x , Bl− ) ⎥ ⎢ =⎢ ⎥ . .. .. .. ⎣ ⎦ . .
(2)
− n s(xn , B+ 1 ) · · · s(x , Bl− )
+ − − B+ 1 , . . . , Bl+ denote the bags labeled positive, while B1 , . . . , Bl− refer to the k negatively labeled bags. Each column represents a bag. If x is near some positive bags and far from some negative ones, the corresponding dimension is useful for discrimination. In MILES[5], a 1-norm SVM is trained to select features and get their corresponding weights from this feature space as follows :
min λ
w,b,ξ,η
n k=1
l
−
+
|wk | + C1
i=1
ξi + C2
l
ηj
j=1
+ s.t. (wT m+ i + b) + ξi ≥ 1, i = 1, . . . , l , − − (wT m− j + b) + ηj ≥ 1, j = 1, . . . , l ,
ξi , ηj ≥ 0, i = 1, . . . , l+ , j = 1, . . . , l−
(3)
Here, C1 and C2 reflect the loss penalty imposed on the misclassification of positive and negative bags, respectively. λ is a regularizer parameter, which controls the trade-off between the complexity of the classifier and the hinge loss. It can be seen that this formulation does not restrict all the instances in negative bags to be negative. Since the 1-norm SVM is utilized, a sparse solution can be obtained, i.e. in this solution, only a few wk in Eq. (3) are nonzero. Hence, MILES finds the most important instances in the labeled bags and their corresponding weights. MILES gives an impressive result on several data sets and has shown its advantages over several other methods, such as DD-SVM [4], MI-SVM [10]and k-means SVM [16], both in accuracy and speed. However, the image retrieval task is itself a Semi-Supervised problem - with only a few labeled pictures searching in a tremendous database. The utilization of the unlabeled pictures may actually improve the retrieval performance.
Localized Content-Based Image Retrieval Using SSMIL
3 3.1
183
Semi-Supervised Multiple Instance Learning (SSMIL) The Formulation of Semi-Supervised Multiple Instance Learning
In this section, we give the formulation for Semi-Supervised Multiple Instance Learning. Our aim is to maximize margins not only on the labeled but the unlabeled bags. A straightforward way is to map both the labeled and unlabeled bags into the feature space determined by all the labeled bags, using Eq. (2). Then, we try to solve such an optimization problem: minw,b,ξ,η,ζ λ
n
l
−
+
|wk | + C1
ξi + C2
l
ηj + C3
i=1 j=1 k=1 + s.t. (wT m+ i + b) + ξi ≥ 1, i = 1, · · · , l T − − −(w mi + b) + ηj ≥ 1, j = 1, · · · , l yu∗ (wT mu + b) + ζu ≥ 1, u = 1, · · · , |U | ξi , ηj , ζu ≥ 0, i = 1, · · · , l+ , j = 1, · · · , l− ,
|U|
ζu
u=1
(4)
u = 1, · · · , |U |
The difference between Eq. (3) and Eq. (4) is the appended penalty term imposed on the unlabeled data. C3 is the penalty parameter that controls the effect of unlabeled data, and yu∗ is the label assigned to the uth unlabeled bag during the training phase. 3.2
The Up-Speed of Semi-Supervised Multiple Instance Learning (UP-SSMIL)
Directly solving the optimization problem (4) is too time-consuming, because, in Eq. (4), all the unlabeled pictures are required to be mapped into the feature space determined by all the instances in the labeled bags and most of the time will be spent on the feature mapping step(Eq. (2)). In this paper, we try to up-speed this process and propose UP-SSMIL. After each labeled bag is mapped into the feature space by Eq. (2), all the unlabeled bags can also be mapped into this feature space according to Eq. (1). As mentioned in Section 2, one norm SVM can find the most important features, i.e. predominant instances in training bags. Hence, the dimension for each bag can be greatly reduced, with the irrelevant features being discarded. So, we propose using MILES as the first step to select the most important instances and mapping each bag B in both the labeled and unlabeled set into the space determined by these instances as follows: m(B) = [s(z1 , B), s(z2 , B), · · · , s(zv , B)]T
(5)
Here, z k is the kth selected instance and v denotes the total number of the selected instances. This is a supervised step. Then, we intend to use the unlabeled bags to improve the performance by optimize the feature weights of the selected
184
D. Zhang et al. Table 1. UP-SSMIL Algorithm
1. Feature Mapping 1: Map each labeled bag (into the feature space determined by the instances in the labeled bags, using Eq.(2). 2. MILES Training: Use 1-norm SVM to train a classifier, utilizing only the training bags. Then, each feature in the feature space determined by the training instances is assigned a weight, i.e. wk in Eq. (3). The regularizer in this step is denoted as λ1 . 3. Feature Selecting: Select the features with nonzero weights. 4. Feature Mapping 2: Map all the unlabeled and labeled bags into the feature space determined by the features selected from the previous step, i.e. the selected instances, using Eq. (5). 5. TSVM Training: Taking into account both the re-mapped labeled and unlabeled bags, use TSVM to train a classifier. The regularizer in TSVM is denoted as λ2 . 6. Classifying: Use this classifier to rank the unlabeled bags.
features. A Transductive Support Vector Machine (TSVM) [12] algorithm is employed to learn these weights. The whole UP-SSMIL algorithm can be depicted in Table 1. In this algorithm, TSVM is a 2-norm Semi-Supervised SVM. The reason why 1-norm Semi-Supervised SVM is not employed is that, after the feature selection step, the selected features are most relevant to the final solution. However, 1-norm Semi-Supervised SVM favors the sparsity of w. Therefore, it is not used here.
4
Experiments
We test our method on SIVAL, which is obtained at www.cs.wustl.edu/∼sg/ multi-inst-data/. Some sample images are shown in Fig. (1). In this database, each image is pre-segmented into around 30 patches. Color, texture and
(a) SpriteCan
(b) WD40Can
Fig. 1. Some sample images in SIVAL dataset
Localized Content-Based Image Retrieval Using SSMIL
185
Table 2. Average AUC values with 95% confidence intervals, with 8 randomly selected positive and 8 randomly selected negative pictures
FabricSoftenerBox CheckeredScarf FeltFlowerRug WD40Can CockCan GreenTeaBox AjaxOrange DirtyRunningShoe CandleWithHolder SpriteCan JulisPot GoldMedal DirtyWorkGlove CardBoardBox SmileyFaceDoll BlueScrunge DataMiningBook TranslucentBowl StripedNoteBook Banana GlazedWoodPot Apple RapBook WoodRollingPin LargeSpoon Average
UP-SSMIL MISSL 97.2±0.7 97.7±0.3 95.5±0.5 88.9±0.7 94.6±0.8 90.5±1.1 90.5±1.3 93.9±0.9 93.4±0.8 93.3±0.9 90.9±1.9 80.4±3.5 90.1±1.7 90.0±2.1 87.2±1.3 78.2±1.6 85.4±1.7 84.5±0.8 84.8±1.1 81.2±1.5 82.1±2.9 68.0±5.2 80.9±3.0 83.4±2.7 81.9±1.7 73.8±3.4 81.1±2.3 69.6±2.5 80.7±1.8 80.7±2.0 76.7±2.6 76.8±5.2 76.6±1.9 77.3±4.3 76.3±2.0 63.2±5.2 75.1±2.6 70.2±2.9 69.2±3.0 62.4±4.3 68.6±2.8 51.5±3.3 67.8±2.7 51.1±4.4 64.9±2.8 61.3±2.8 64.1±2.1 51.6±2.6 58.6±1.9 50.2±2.1 80.6 74.8
MILES 96.8±0.9 95.1±0.8 94.1±0.8 86.9±3.0 91.8±1.3 89.4±3.1 88.4±2.8 85.6±2.1 83.4±2.3 82.1±2.8 78.8±3.5 76.1±3.9 80.4±2.2 78.4±3.0 77.7±2.8 73.2±2.8 74.0±2.3 74.0±3.1 73.2±2.5 66.4±3.4 69.0±3.0 64.7±2.8 64.6±2.3 63.5±2.0 57.7±2.1 78.6
Accio! 86.6±2.9 90.8±1.5 86.9±1.6 82.0±2.4 81.5±3.4 87.3±2.9 77.0±3.4 83.7±1.9 68.8±2.3 71.9±2.4 79.2±2.6 77.7±2.6 65.3±1.5 67.9±2.2 77.4±3.2 69.5±3.3 74.7±3.3 77.5±2.3 70.2±3.1 65.9±3.2 72.7±2.2 63.4±3.3 62.8±1.7 66.7±1.7 57.6±2.3 74.6
Accio!+EM 44.4±1.1 58.1±4.4 51.1±24.8 50.3±3.0 48.5±24.6 46.8±3.5 43.6±2.4 75.4±19.8 57.9±3.0 59.2±22.1 51.2±24.5 42.1±3.6 57.8±2.9 57.8±2.9 48.0±25.8 36.3±2.5 37.7±4.9 47.4±25.9 43.5±3.1 43.6±3.8 51.0±2.8 43.4±2.7 57.6±4.8 52.5±23.9 51.2±2.5 50.3
neighborhood features have already been extracted for each patch, and form a set of 30-dimension feature vectors. In our experiments, these features are normalized to be exactly in the range from 0 to 1, and the scaling parameter σ is chosen to be 0.5. Treat each picture as a bag, and each patch in this picture as an instance in this bag. The source code of MILES is obtained from [17], and TSVM is obtained from [18]. During each trial, 8 positive pictures are randomly selected from one category, and other 8 negative pictures are randomly selected as background pictures from the other 24 categories. The retrieval speed of UP-SSMIL is pretty fast. In my computer, for each round, UP-SSMIL takes only 25 seconds while SSMIL takes around 30 minutes. For convenience, only the results of UP-SSMIL are reported here. We will demonstrate below that it achieves the best performance on SIVAL database. In UP-SSMIL’s Training step in Table 1 and MILES (see Eq. (3)), λ1 is set to 0.2, C1 and C2 are set to 0.5. In UP-SSMIL’s TSVM Training step in Table 1 (for a detailed description of the parameters, see the reference of SVMlin [18]),
186
D. Zhang et al. SpriteCan |U|=1500−|L|
SpriteCan |L|=16
0.92 0.86
0.9
0.84
0.88 AUC value
AUC value
0.82
0.86 0.84
0.8 0.78
MILES
0.76
UP−SSMIL
0.74
0.82
MILES
0.8 0.78
UP−SSMIL
0.72
20
30 40 50 60 Number of labeled pictures (|L|)
70
0.7
80
200
400 600 800 Number of unlabeled pictures (|U|)
(a)
(b)
WD40can |U|=1500−|L|
WD40Can |L|=16
0.96
0.92
0.95
0.91 0.9
0.94
0.89 AUC value
AUC value
0.93 0.92 0.91 0.9
0.88 0.87 0.86 0.85
0.89 0.88 0.87
1000
20
MILES
0.84
UP−SSMIL
0.83
30 40 50 60 Number of labeled pictures (|L|)
(c)
70
80
0.82
MILES UP−SSMIL 200
400 600 800 Number of unlabeled pictures (|U|)
1000
(d)
Fig. 2. The comparison result between UP-SSMIL and MILES
λ2 is set to 0.1 The positive class fraction of unlabeled data is set to 0.01. The other parameters in SVMlin are all set to their default values. In the image retrieval, the ROC curve is a good measure of the performance. So, the area under ROC curve - AUC value is used here to measure the performance. All the results reported here are averaged over 30 independent runs, with a 95% confidence interval being calculated. The final comparison result is shown in Table 2. From this table, it can be seen that, compared with MISSL, among all the 25 categories, UP-SSMIL performs better than MISSL for most categories, with only a few categories worse than MISSL. This may be due to two reasons. For one thing, MISSL uses inadequate number of pictures to learn the likelihood for each instance being positive and the “steepness factor” in MISSL is relatively hard to determine. These may lead to an inaccurate energy function. For another, on the graph level, MISSL uses just one vertex to represent all the negative training vertexes, and assumes the weights connecting from this vertex to all the unlabeled vertexes to be the same, which will result in some inaccuracy as well. Furthermore, after the pre-calculation of the distances between different instances, MISSL takes 30-100 seconds to get a retrieval result, while UP-SSMIL takes no more than 30 seconds without the need to calculate these distances. This
Localized Content-Based Image Retrieval Using SSMIL
187
is quite understandable, In the first Feature Mapping Step in Table 1, UP-SSMIL only need to calculate the distances within the training bags. Since the number of query images is so small, this calculation burden is relatively light. Then, after the features being selected, the unlabeled bags only need to be mapped into the space determined by these few selected features. In our experiments, this dimension can be reduced to around 10. So, the calculation cost of the second Feature Mapping step in Table 1 is very low. With the dimensions being greatly reduced, TSVM gets the solution relatively fast. Compared with other supervised methods, such as MILES, Accio [7] and Accio+EM [7]. The performance of UP-SSMIL is also quite promising. Some comparisons result with its supervised opponent–MILES are provided in Fig. 2. We illustrate how the learning curve will change when both the number of labeled pictures(|L|) and the number of unlabeled pictures(|U |) vary. It can be seen that the performance of UP-SSMIL always outperforms its supervised opponent.
5
Conclusion
In this paper, we propose a semi-supervised SVM framework of Multiple Instance algorithm - SSMIL. It uses the unlabeled pictures to help improve the performance. Then, UP-SSMIL is presented to accelerate the retrieval speed. In the end, we demonstrate on SIVAL database its superior performances.
References 1. Dietterich, T.G., Lathrop, R.H., Lozano-P¨erez, T.: Solving the Multiple Instance Problem with Axis-Parallel Rectangles. Artificial Inteligence 1446, 1–8 (1998) 2. Maron, O., Lozano-P¨erez, T.: A Framework for Multiple-Instance Learning. Advances in Neural Information Processing System 10, 570–576 (1998) 3. Maron, O., Ratan, A.L.: Multiple-Instance Learning for Natural Scene Classification. In: Proc. 15th Int’l. Conf. Machine Learning, pp. 341–349 (1998) 4. Chen, Y., Wang, J.Z.: Image Categorization by Learning and Reasoning with Regions. J. Machine Learning Research 5, 913–939 (2004) 5. Chen, Y., Bi, J., Wang, J.Z.: MILES: Multiple-Instance Learning via Embedded Instance Selection. IEEE Transatctions on Pattern Analysis and Machine Intelligence 28(12) (2006) 6. Zhang, Q., Goldman, S.: EM-DD: An improved Multiple-Instance Learning. In: Advances in Neural Information Processing System, vol. 14, pp. 1073–1080 (2002) 7. Rahmani, R., Goldman, S., Zhang, H., et al.: Localized Content-Based Image Retrieval. In: Proceedings of ACM Workshop on Multimedia Image Retrieval, ACM Press, New York (2005) 8. Rahmani, R., Goldman, S.: MISSL: Multiple-Instance Semi-Supervised Learning. In: Proc. 23th Int’l. Conf. Machine Learning, pp. 705–712 (2006) 9. Cheung, P.-M., Kwok, J.T.: A Regularization Framework for Multiple-Instance Learning. In: ICML (2006) 10. Andrews, S., Tsochantaridis, I., Hofmann, T.: Support Vector Machines for Multiple-Instance Learning. In: Advances in Neural Information Processing System, vol. 15, pp. 561–568 (2003)
188
D. Zhang et al.
11. Andrews, S., Hofmann, T.: Multiple Instance Learning via Disjunctive Programming Boosting. In: Advances in Neural Information Processing System, vol. 16, pp. 65–72 (2004) 12. Joachims, T.: Transductive Inference for Text Classification using Support Vector Machine. In: Proc. 16th Int’l. Conf. Machine Learning, pp. 200–209 (1999) 13. Bennett, K.P., Demiriz, A.: Semi-supervised sup- port vector machines. In: Advances in Neural Information Processing System, vol. 11, pp. 368–374 (1999) 14. Zhu, X.: Semi-supervised learning literature survey, in Technical Report 1530, Department of Computer Sci- ences, University of Wisconsin at Madison (2006) 15. Zhou, Z.H., Zhang, M.L.: Multi-Instance Multi-Label Learning with Application to Scene Classification. In: Advances in Neural Information Processing System (2006) 16. Csurka, G., Bray, C., Dance, C., Fan, L.: Visual Categorization with Bags of Keypoints. In: Pajdla, T., Matas, J(G.) (eds.) ECCV 2004. LNCS, vol. 3021, pp. 59–74. Springer, Heidelberg (2004) 17. http://john.cs.olemiss.edu/∼ ychen/MILES.html 18. http://people.cs.uchicago.edu/∼ vikass/svmlin.html
Object Detection Combining Recognition and Segmentation Liming Wang1 , Jianbo Shi2 , Gang Song2 , and I-fan Shen1 1
2
Fudan University,Shanghai,PRC,200433 {wanglm,yfshen}@fudan.edu.cn University of Pennsylvania, 3330 Walnut Street, Philadelphia, PA 19104
[email protected],
[email protected]
Abstract. We develop an object detection method combining top-down recognition with bottom-up image segmentation. There are two main steps in this method: a hypothesis generation step and a verification step. In the top-down hypothesis generation step, we design an improved Shape Context feature, which is more robust to object deformation and background clutter. The improved Shape Context is used to generate a set of hypotheses of object locations and figureground masks, which have high recall and low precision rate. In the verification step, we first compute a set of feasible segmentations that are consistent with top-down object hypotheses, then we propose a False Positive Pruning(FPP) procedure to prune out false positives. We exploit the fact that false positive regions typically do not align with any feasible image segmentation. Experiments show that this simple framework is capable of achieving both high recall and high precision with only a few positive training examples and that this method can be generalized to many object classes.
1 Introduction Object detection is an important, yet challenging vision task. It is a critical part in many applications such as image search, image auto-annotation and scene understanding; however it is still an open problem due to the complexity of object classes and images. Current approaches [1,2,3,4,5,6,7,8,9,10] to object detection can be categorized by top-down, bottom-up or combination of the two. Top-down approaches [2,11,12] often include a training stage to obtain class-specific model features or to define object configurations. Hypotheses are found by matching models to the image features. Bottomup approaches start from low-level or mid-level image features, i.e. edges or segments [5,8,9,10]. These methods build up hypotheses from such features, extend them by construction rules and then evaluate by certain cost functions. The third category of approaches combining top-down and bottom-up methods have become prevalent because they take advantage of both aspects. Although top-down approaches can quickly drive attention to promising hypotheses, they are prone to produce many false positives when features are locally extracted and matched. Features within the same hypothesis may not be consistent with respect to low-level image segmentation. On the other hand, bottom-up approaches try to keep consistency in low level image segmentation, but usually need much more efforts in searching and grouping. Y. Yagi et al. (Eds.): ACCV 2007, Part I, LNCS 4843, pp. 189–199, 2007. c Springer-Verlag Berlin Heidelberg 2007
190
L. Wang et al.
Input Image Feature 1 Codebook Building
2 Matching &Voting Codebook Entries
Hypotheses
Image Segmentation
3 Verification
Detection
(Re−Evaluation)
Results
Fig. 1. Method overview. Our method has three parts (shaded rectangles). Codebook building (cyan) is the training stage, which generates codebook entries containing improved SC features and object masks. Top-down recognition (blue) generates multiple hypotheses via improved SC matching and voting in the input image. The verification part (pink) aims to verify these top-down hypotheses using bottom-up segmentation. Round-corner rectangles are processes and ordinary rectangles are input/output data.
Wisely combining these two can avoid exhaustive searching and grouping while maintaining consistency in object hypotheses. For example, Borenstein et al. enforce continuity along segmentation boundaries to align matched patches [2]. Levin et al. take into account both bottom-up and top-down cues simultaneously in the framework of CRF [3]. Our detection method falls into this last category of combining top-down recognition and bottom-up segmentation, with two major improvements over existing approaches. First, we design a new improved Shape Context (SC) for the top-down recognition. Our improved SC is more robust to small deformation of object shapes and background clutter. Second, by utilizing bottom-up segmentation, we introduce a novel False Positive Pruning (FPP) method to improve detection precision. Our framework can be generalized to many other object classes because we pose no specific constraints on any object class. The overall structure of the paper is organized as follows. Sec. 2 provides an overview to our framework. Sec.3 describes the improved SCs and the top-down hypothesis generation. Sec.4 describes our FPP method combining image segmentation to verify hypotheses. Experiment results are shown in Sec.5, followed by discussion and conclusion in Sec.6.
2 Method Overview Our method contains three major parts: codebook building, top-down recognition using matching and voting, and hypothesis verification, as depicted in Fig.1. The object models are learned by building a codebook of local features. We extract improved SC as local image features and record the geometrical information together with object figure-ground masks. The improved SC is designed to be robust to shape variances and background clutters. For rigid objects and objects with slight articulation, our experiments show that only a few training examples suffice to encode local shape information of objects. We generate recognition hypotheses by matching local image SC features to the codebook and use SC features to vote for object centers. A similar top-down voting scheme is described in the work of [4], which uses SIFT point features for pedestrian
Object Detection Combining Recognition and Segmentation r 1 2 3 1 2 3
(a)
(b)
θ θ 1
5
(c)
9
θ
θ
(d)
r 1 2 3 1 2 3
191
θ 1
5
(e)
9
θ
Fig. 2. Angular Blur. (a) and (b) are different bin responses of two similar contours. (c) are their histograms. (d) enlarges angular span θ to θ , letting bins be overlapped in angular direction. (e) are the responses on the overlapped bins, where the histograms are more similar.
detection. The voting result might include many false positives due to small context of local SC features. Therefore, we combine top-down recognition with bottom-up segmentation in the verification stage to improve the detection precision. We propose a new False Positive Pruning (FPP) approach to prune out many false hypotheses generated from top-down recognition. The intuition of this approach is that many false positives are generated due to local mismatches. These local features usually do not have segmentation consistency, meaning that pixels in the same segment should belong to the same object. True positives are often composed of several connected segments while false positives tend to break large segments into pieces.
3 Top-Down Recognition In the training stage of top-down recognition, we build up a codebook of improved SC features from training images. For a test image, improved SC features are extracted and matched to codebook entries. A voting scheme then generates object hypotheses from the matching results. 3.1 Codebook Building For each object class, we select a few images as training examples. Object masks are manually segmented and only edge map inside the mask is counted in shape context histogram to prune out edges due to background clutter. The Codebook Entries (CE) are a repository of example features: CE = {cei }. Each codebook entry cei = (ui , δi , mi , wi ) records the feature for a point i in labelled objects of the training images. Here ui is the shape context vector for point i. δi is the position of point i relative to the object center. mi is a binary mask of figure-ground segmentation for the patch centered at point i. wi is the weight mask computed on mi , which will be introduced later. 3.2 Improved Shape Context The idea of Shape Context (SC) was first proposed by Belongie et al. [13]. The basic definition of SC is a local histogram of edge points in a radius-angle polar grid. Following works [14,15] improve its distinctive power by considering different edge orientations. Besides SC, other local image features such as wavelets, SIFT and HOG have been used in keypoint based detection approaches [4,12].
192
L. Wang et al.
Suppose there are nr (radial) by nθ (angular) bins and the edge map E is divided into E1 , . . . , Eo by o orientations (similar to [15]), for a point at p, its SC is defined as u = {h1 , . . . , ho }, where →
hi (k) = #{q = p : q ∈ Ei , pq∈ bin(k)}, k = 1, 2, ..., nr nθ
(1)
Angular Blur. A common problem for the shape context is that when dense bins are used or contours are close to the bin boundaries, similar contours have very different histograms (Fig.2-(c)). This leads to a large distance for two similar shapes if L2 -norm or χ2 distance function is used. EMD [16] alleviates this by solving a transportation problem; but it is computationally much more expensive. The way we overcome this problem is to overlap spans of adjacent angular bins: bin(k) ∩ bin(k + 1) = ∅ (Fig.2-(d)). This amounts to blurring the original histogram along the angular direction. We call such an extension Angular Blur. One edge point in the overlapped regions are counted in both of the adjacent bins. So the two contours close to the original bin boundary will have similar histograms for the overlapping bins(Fig.2-(e)). With angular blur, even simple L2 -norm can tolerate slight shape deformation. It improves the basic SC without the expensive computation of EMD. Mask Function on Shape Context. In real images, objects SCs always contain background clutter. This is a common problem for matching local features. Unlike learning methods [1,12] which use a large number of labeled examples to train a classifier, we propose to use a mask function to focus only on the parts inside object while ignoring background in matching. For ce = (u, δ, m, w) and a SC feature f in the test image, each bin of f is masked by figure-ground patch mask m of ce to remove the background clutter. Formally, we compute the weight w for bin k and distance function with mask as: w(k) = Area(bin(k) ∩ m)/Area(bin(k)), k = 1, 2, ..., nr nθ Dm (ce, f ) = D(u, w · v) = ||u − w · v||
2
(2) (3)
where (·) is the element-wise product. D can be any distance function computing the dissimilarity between histograms (We simply use L2 -norm). Figure 3 gives an example for the advantage of using mask function. 3.3 Hypothesis Generation The goal of hypothesis generation is to predict possible object locations as well as to estimate the figure-ground segmentation for each hypothesis. Our hypothesis generation is based on a voting scheme similar to [4]. Each SC feature is compared with every codebook entry and makes a prediction of the possible object center. The matching scores are accumulated over the whole image and the predictions with the maximum scores are the possible object centers. Given a set of detected features {fi } at location {li }, we define the probability of matching codebook entry cek to fi as P (cek |li ) ∝ exp(−Dm (cek , fi )). Given the match of cek to fi , the probability of an object o with
Object Detection Combining Recognition and Segmentation
b1
a1
A
B
input feature 0.2 0.1 0
v
193
b2
a2
u
0.2 0.1 0 0.2 0.1 0
(a)
0
50 weighted feature
100
0
50 matched codebook entry
100
0
50
100
(b)
Fig. 3. Distance function with mask. In (a), a feature point v has the edge map of a1 around it. Using object mask b1 , it succeeds to find a good match to u in B (object model patch), whose edge map is b2 . a2 is the object mask b1 over a1 . Only the edge points falling into the mask area are counted for SC. In (b), histograms of a1 , a2 and b2 are shown. With the mask function, a2 is much closer to b2 , thus got well matched.
center located at c is defined as P (o, c|cek , li ) ∝ exp(−||c + δk − li ||2 ). Now the probability of the hypothesis of object o with center c is computed as: P (o, c) = P (o, c|cek , li )P (cek |li )P (li ) (4) i,k
P (o, c) gives a voting map V of different locations c for the object class o. Extracting local maxima in V gives a set of hypotheses {Hj } = {(oj , cj )}. Furthermore, figure-ground segmentation for each Hj can be estimated by backtracing the matching results. For those fi giving the correct prediction, the patch mask m in the codebook is “pasted” to the corresponding image location as the figure-ground segmentation. Formally, for a point p in image at location pl , we define P (p = f ig|cek , li ) as the probability of point p belonging to the foreground when the feature at location −→
li is matched to the codebook cek : P (p = f ig|cek , li ) ∝ exp(−||pl − li ||)mk (pl li ). And we assume that P (cek , li |Hj ) ∝ P (oj , cj |cek , li ) and P (fi |cek ) ∝ P (cek |fi ). The figure-ground probability for hypothesis Hj is estimated as −→ P (p = f ig|Hj ) ∝ exp(−||pl − li ||)mk (pl li )P (fi |cek )P (cek , li |Hj) (5) k
Eq. (5) gives the estimation of top-down segmentation. The whole process of top-down recognition is shown in Fig. 4. The binary top-down segmentation (F, B) of figure(F ) and background (B) is the obtained by thresholding P (p = f ig|Hj ).
4 Verification: Combining Recognition and Segmentation From our experiments, the top-down recognition using voting scheme will produce many False Positives (FPs). In this section, we propose a two-step procedure of False Positive Pruning (FPP) to prune out FPs. In the first step we refine the top-down hypothesis mask by checking its consistency with bottom-up segmention. Second the final score on the refined mask is recomputed by considering spatial constraints.
194
L. Wang et al. f1 f2
Hj
(a)
(b)
(c)
(d)
(e)
Fig. 4. Top-down recognition. (a) An input image; (b) A matched point feature votes for 3 possible positions; (c) The vote map V . (d) The hypothesis Hj traces back find its voters {fi }. (d) Each fi predicts the figure-ground configration using Eq. (5).
Combining Bottom-up Segmentation. The basic idea for local feature voting is to make global decision by the consensus of local predictions. However, these incorrect local predictions using a small context can accumulate and confuse the global decision. For example, in pedestrian detection, two trunks will probably be locally taken as human legs and produce a human hypothesis (in Fig. 5-(a)); another case is the silhouettes from two standing-by pedestrians.
A O1
A O B C D
E
O O2
D
(a)
E
O3
(b)
(c)
(d)
Fig. 5. Combining bottom-up segmentation. FPs tend to spread out as multiple regions from different objects. In example of (a). an object O consists of five parts (A, B, C, D, E). (A ∈ O1 , D ∈ O2 , E ∈ O3 ) are matched to (A, D, E) because locally they are similar. The hypothesis of O = (A , D , E ) is generated. (b) shows boundaries of a FP (in green) and a TP (in red) in a real image. (c) is the layered view of the TP in (b). The top layer is the top-down segmentation, which forms a force (red arrows) to pull the mask out from the image. The bottom layer is the background force (green arrows). The middle layer is the top-down segmentation (we threshold it to binary mask) over the segmentation results.(d) is the case for the FP.
In pedestrian detection, the top-down figure-ground segmentation masks of the FPs usually look similar to a pedestrian. However we notice that such top-down mask is not consistent with the bottom-up segmentation for most FPs. The bottom-up segments share bigger contextual information than the local features in the top-down recognition and are homogenous in the sense of low-level image feature. The pixels in the same segment should belong to the same object. Imagine that the top-down hypothesis mask(F, B) tries to pull the object F out of the whole image. TPs generally consists of several well-separated segments from the background so that they are easy to be pulled
Object Detection Combining Recognition and Segmentation
195
out (Fig. 5-(c)). However FPs often contain only part of the segments. In the example of tree trunks, only part of the tree trunk is recognized as foreground while the whole tree trunk forms one bottom-up segment. This makes pulling out FPs more difficult because they have to break the homogenous segments (Fig. 5-(d)). Based on these observations we combine the bottom-up segmentation to update the top-down figure-ground mask. Incorrect local predictions are removed from the mask if they are not consistent with the bottom-up segmentation. We give each bottom-up cut to propose segment Si a binary label. Unlike the work in [17] which uses graph Area(Si F ) the optimized hypothesis mask, we simply define the ratio Area(S B) as a criteria to i assign Si to F or B. We try further segmentation when such assignment is uncertain to avoid the case of under-segmentation in a large area. The Normalized Cut (NCut) cost [18] is used to determine if such further segmentation is reasonable. The procedure to refine hypothesis mask is formulated as follows: Input: top-down mask (F, B) and bottom-up segments {Si , i = 1, . . . , N }. Output: refined object mask (F, B). Set i = 0. 1) If i > N ,exit; else, i = i + 1. Area(Si F ) 2) If Λ = Area(S B) > κup , then F = F ∪ Si ,goto 1); i elseif Λ < κdown, then F = F − (F ∩ Si ), goto 1). Otherwise, go to 3). 3) Segment Si to (Si1 , Si2 ). If ζ = NCut(Si ) > Υup , F = F − (F ∩ Si ), goto 1); else SN +1 = Si1 ,SN +2 = Si2 , S = S ∪ {SN +1 , SN +2 }, N = N + 2, goto 1). Re-evaluation. There are two advantages with the updated masks. The first is that we can recompute more accurate local features by masking out the background edges. The second is that the shapes of updated FPs masks will change much more than those of TPs, because FPs are usually generated by locally similar parts of other objects, which will probably be taken away through the above process. We require TPs must have voters from all the different locations around the hypothesis center. This will eliminates those TPs with less region support or with certain partial matching score. The final score is the summation of the average scores over the different spatial bins in the mask. The shape of the spatial bins are predefined. For pedestrians we use the radius-angle polar ellipse bins; for other objects we use rectangular grid bins. For each hypothesis, SC features are re-computed over the masked edge map by F and feature fi is only allowed to be matched to cek in the same bin location. For each bin j, we P (cek |fi ) compute an average matching score Ej = #(ce , where both cek and fi come k ,fi ) from bin j. The final score of this hypothesis is defined as:
E=
j
Ej ,where Ej =
Ej , if Ej > α; −α , if Ej = 0 and #{cek , cek ∈ bin(j)} > 0.
(6)
The term α is used to penalize the bins which have no matching with the codebook. This decreases the scores of FPs with only part of true objects, i.e. bike hypothesis with one wheel. Experiments show that our FPP procedure can prune out FPs effectively.
196
L. Wang et al.
5 Results Our experiments test different object classes including pedestrian, bike, human riding bike, umbrella and car (Table. 1). These pictures were taken from scenes around campus and urban streets. Objects in the images are roughly at the same scale. For pedestrians, the range of the heights is from 186 to 390 pixels. Table 1. Dataset for detection task #Object Pedestrian Bike Human on bike Umbrella Car Training 15 3 2 4 4 Testing 345 67 19 16 60
For our evaluation criteria, a hypothesis whose center falls into an ellipse region around ground truth center is classified as true positive. The radii for ellipse are typically chosen as 20% of the mean width / height of the objects. Multiple detections for one ground truth object are only counted once. Angular Blur and Mask Function Evaluation. We compare the detection algorithm on images w/ and w/o Angular Blur (AB) or mask function. The PR curves are plotted in Fig.6. For pedestrian and umbrella detection, it is very clear that adding Angular Blur and mask function can improve the detection results. For other object classes, AB+Mask outperforms at high-precision/low-recall part of the curve, but gets no significant improvement at high-recall/low-precision part. The reason is that AB+Mask can improve the cases where objects have deformation and complex background clutter. For bikes,
1
1
1
0.8
0.8
0.8
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0
0
0.2
0.4
0.6
0.8
1
(a) Pedestrian
1
0
0
0.2
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0
0
0.6
0.8
1
0
0
(b) Bike
1
0.8
0.4
0.2
0.4
0.6
Angular Blur+Mask Function w/o Angular Blur w/o Mask Function w/ FPP
0
0.2
0.4
0.6
(c) Umbrella
0.8
1
0.8
(c) Human on bike
HOG (only in (a))
0
0.2
0.4
0.6
0.8
1
(d) Car Fig. 6. PR-Curves of object detection results
(e) plot legend
1
Object Detection Combining Recognition and Segmentation
197
Fig. 7. Detection result on real images. The color indicates different segments. The last row contains cases of FPs for bikes and pedestrians.
198
L. Wang et al.
the inner edges dominate the SC histogram; so adding mask function makes only a little difference. Pedestrian Detection Compared with HOG. We also compare with HOG.using the implementation of the authors of [12] Figure 6-(a) shows that our method with FPP procedure are better than the results of HOG. Note that we only use a very limited number of training examples as shown in Table. 1 and we did not utilize any negative training examples.
6 Conclusion and Discussion In this paper, we developed an object detection method of combining top-down modelbased recognition with bottom-up image segmentation. Our method not only detects object positions but also gives the figure-ground segmentation mask. We designed an improved Shape Context feature for recognition and proposed a novel FPP procedure to verify hypotheses. This method can be generalized to many object classes. Results show that our detection algorithm can achieve both high recall and precision rates. However there are still some FPs hypotheses that cannot be pruned. They are typically very similar to objects, like a human-shape rock, or some tree trunks. More information like color or texture should be explored to prune out these FPs. Another failure case of SC detector is for very small scale object. These objects have very few edges points thus are not suitable for SC. Also our method does no work for severe occlusion where most local information is corrupted. Acknowledgment. This work is partially supported by National Science Foundation through grants NSF-IIS-04-47953(CAREER) and NSF-IIS-03-33036(IDLP). We thank Qihui Zhu and Jeffrey Byrne for polishing the paper.
References 1. Viola, P.A., Jones, M.J.: Rapid object detection using a boosted cascade of simple features. In: CVPR (2001) 2. Borenstein, E., Ullman, S.: Class-specific, top-down segmentation. In: Heyden, A., Sparr, G., Nielsen, M., Johansen, P. (eds.) ECCV 2002. LNCS, vol. 2351, Springer, Heidelberg (2002) 3. Levin, A., Weiss, Y.: Learning to combine bottom-up and top-down segmentation. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3951, Springer, Heidelberg (2006) 4. Leibe, B., Seemann, E., Schiele, B.: Pedestrian detection in crowded scenes. In: CVPR (2005) 5. Ferrari, V., Tuytelaars, T., Gool, L.J.V.: Object detection by contour segment networks. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3951, Springer, Heidelberg (2006) 6. Kokkinos, I., Maragos, P., Yuille, A.L.: Bottom-up & top-down object detection using primal sketch features and graphical models. In: CVPR (2006) 7. Zhao, L., Davis, L.S.: Closely coupled object detection and segmentation. In: ICCV (2005)
Object Detection Combining Recognition and Segmentation
199
8. Ren, X., Berg, A.C., Malik, J.: Recovering human body configurations using pairwise constraints between parts. In: Sebe, N., Lew, M.S., Huang, T.S. (eds.) Computer Vision in Human-Computer Interaction. LNCS, vol. 3766, Springer, Heidelberg (2005) 9. Mori, G., Ren, X., Efros, A.A., Malik, J.: Recovering human body configurations: Combining segmentation and recognition. In: CVPR (2004) 10. Srinivasan, P., Shi, J.: Bottom-up recognition and parsing of the human body. In: CVPR (2007) 11. Felzenszwalb, P.F., Huttenlocher, D.P.: Pictorial structures for object recognition. International Journal of Computer Vision 61(1) (2005) 12. Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: CVPR (2005) 13. Belongie, S., Malik, J., Puzicha, J.: Shape matching and object recognition using shape contexts. IEEE Trans. Pattern Anal. Mach. Intell. 24(4) (2002) 14. Mori, G., Belongie, S.J., Malik, J.: Efficient shape matching using shape contexts. IEEE Trans. Pattern Anal. Mach. Intell 27(11) (2005) 15. Thayananthan, A., Stenger, B., Torr, P.H.S., Cipolla, R.: Shape context and chamfer matching in cluttered scenes. In: CVPR (2003) 16. Rubner, Y., Tomasi, C., Guibas, L.J.: A metric for distributions with applications to image databases. In: ICCV (1998) 17. Ramanan, D.: Using segmentation to verify object hypotheses. In: CVPR (2007) 18. Shi, J., Malik, J.: Normalized cuts and image segmentation. In: CVPR (1997)
An Efficient Method for Text Detection in Video Based on Stroke Width Similarity Viet Cuong Dinh, Seong Soo Chun, Seungwook Cha, Hanjin Ryu, and Sanghoon Sull Department of Electronics and Computer Engineering, Korea University, 5-1 Anam-dong, Seongbuk-gu, Seoul, 136-701, Korea {cuongdv,sschun,swcha,hanjin,sull}@mpeg.korea.ac.kr
Abstract. Text appearing in video provides semantic knowledge and significant information for video indexing and retrieval system. This paper proposes an effective method for text detection in video based on the similarity in stroke width of text (which is defined as the distance between two edges of a stroke). From the observation that text regions can be characterized by a dominant fixed stroke width, edge detection with local adaptive thresholds is first devised to keep text- while reducing background-regions. Second, morphological dilation operator with adaptive structuring element size determined by stroke width value is exploited to roughly localize text regions. Finally, to reduce false alarm and refine text location, a new multi-frame refinement method is applied. Experimental results show that the proposed method is not only robust to different levels of background complexity, but also effective to different fonts (size, color) and languages of text.
1 Introduction The need for efficient content-based video indexing and retrieval has increased due to the rapid growth of video data available to consumers. For this purpose, text in video, especially the superimposed text, is the most frequently used since it provides highlevel semantic information about video content and it has distinctive visual characteristic. Therefore, the success in video text detection and recognition would have a great impact on multimedia applications such as image categorization [1], video summarization [2], and lecture video indexing [3]. Many efforts have been made for text detection in image and video. Regarding the way used to locate text regions, text detection methods can be classified into three approaches: connected component (CC)-based method [4, 5, 6], texture-based method [7, 8], and edge-based method [9, 10]. The CC-based method is based on the analysis of geometrical arrangement of edges or homogeneous color that belongs to characters. Alternatively, the texture-based method treats text region as a special type of texture and employs learning algorithms, e.g., neural network [8], support vector machine (SVM) [11], to extract text. In general, the texture-based method is more robust than the CC-based method in dealing with complex background. However, the main drawbacks of this method are its high complexity and inaccuracy localization. Y. Yagi et al. (Eds.): ACCV 2007, Part I, LNCS 4843, pp. 200–209, 2007. © Springer-Verlag Berlin Heidelberg 2007
An Efficient Method for Text Detection in Video Based on Stroke Width Similarity
201
Another popularly studied method is the edge-based method, which is based on the fact that text regions have abundant edges. This method is widely used due to its fast performance in detecting text and its ability to keep geometrical structure of text. The method in [9] detects edges in an image and then uses the fixed size horizontal, vertical morphological dilation operations to form text line candidate. Real text regions are identified by using the SVM. Two disadvantages of this method are its poor performance in case of complex background and the use of fixed size structuring element in dilation operations. To deal with the background complexity problem, edge detection-based method should be accompanied by a local threshold algorithm. In [10], the image is first divided into small windows. A window is considered to be complex if the “number of blank rows” is smaller than a certain specific value. Then, in the edge detection step, a higher threshold is assigned for these complex windows. However, the “number of blank rows” criterion appears sensitive to noise and not strong enough to handle different text sizes. Therefore, how to design an effective local threshold algorithm for detecting edge is still a challenging problem of text detection in video. The main problem of the above existing methods is that they are not robust to different text colors, sizes, and background complexity, since they simply use either general segmentation method or some prior knowledge. In this paper, we attempt to discover the intrinsic characteristic of text (namely the stroke width similarity) and then exploit it to build a robust method for text detection in video. From the knowledge of font system, it turns out that, if characters are in the same font type and size, their stroke widths are almost constant. In another view, a text region can be considered as a region with a dominant fixed stroke width value. Therefore, the similarity in stroke width can be efficiently used as a critical characteristic to describe the text region in video frame. The contributions of this paper can be summarized as follow: • Exploiting the similarity in stroke width characteristic of text to build an effective edge detection method with local adaptive threshold algorithm. • Implementing a stroke-based method to localize text regions in video. • Designing a multi-frame refinement method which can not only refine the text location but also enhance the quality of the detected text. The rest of this paper is organized as follows: Section 2 presents the proposed method for text detection in video. To demonstrate its effectiveness, experimental results are given in Section 3. In Section 4, the concluding remarks are drawn.
2 Proposed Method In the proposed method, text regions in video are detected through three processes. First, edge detection with local adaptive threshold algorithm is applied to reveal text edge pixels. Second, dilation morphological operator with adaptive structuring element size is exploited in the stroke-based localization process to roughly localize text regions. Finally, a multi-frame refinement process is applied to reduce false alarm, refine the location, and enhance the quality of each text region. Figure 1 shows the flow chart of the proposed system.
202
V.C. Dinh et al.
Video frames
Edge Detection with Local Adaptive Thresholds
Stroke-Based Text Localization
Multi-Frame Text Refinement
Detected Text Regions
Fig. 1. Flowchart of the proposed text detection method
2.1 Motivation From the knowledge of font system, it turns out that, if characters are in the same font type and font size, their stroke widths are almost constant. Therefore, in the proposed method, the stroke width similarity is used as a clue to characterize text regions in frame. Generally, the width of any stroke (of both text and non-text objects) can be calculated as distance (measured in pixel) in horizontal direction between its doubleedge pixels. Figure 2(a) shows an example of double-edge pixels (A and B). It can be seen from the figure that stroke widths from different characters are almost similar. Scan line A
B
(a)
(b)
Fig. 2. An example of text image. (a) Text image. (b) Edge values for the scan line in (a), wt is the stroke width value.
In general, the color of text often contrasts to its local background. Therefore, for any double-edge pixels of a stroke, this contrast makes an inversion in sign of the edge values, i.e. the gradient magnitude of edge pixels, in horizontal direction (dx) between two pixels on the left- and right-hand side of the stroke. Figure 2(b) shows the corresponding edge values in horizontal direction of a given horizontal scan line in Fig. 2(a); it is clear that the stroke can be modeled as double-edge pixels within a certain range, delimited by a positive and a negative peak nearby. By using the doubleedge pixel model to describe the stroke, we can take the advantages of: 1) Reducing the effect of noise; 2) Applicability even with low-quality edge image. 2.2 Edge Detection with Local Adaptive Threshold Algorithm First, the Canny edge detector with a low threshold is applied to video frame to keep all possible text edge pixels and each frame is divided into M × N blocks, typically 8 × 8 or 12 × 8. Second, by analyzing the similarity in stroke width corresponding to each block, blocks are classified into two types: simple blocks and complex blocks. Then, a suitable threshold algorithm for each block type is used to determine the proper threshold for each block. Finally, the final edge image is created by applying each block with the new proper threshold.
An Efficient Method for Text Detection in Video Based on Stroke Width Similarity
203
2.2.1 Block Classification For each block, we create a stroke width set which is the collection of all stroke width candidates contained in this block. Due to the similarity in stroke width of characters, the values in the stroke width set of the text region on the simple background are concentrated on some close values. Whereas stroke width candidates of the text region on the complex background or background region may also be created by other background objects. As a result, the element values in this set may spread over a wide range of values. Therefore, text regions on a simple background can be characterized by a smaller value of the standard deviation of stroke width than those on other regions. Based on this different characteristic, blocks in the frame are classified into two types: simple blocks and complex blocks. A block is classified as a simple one if the standard deviation of stroke width values is smaller than a given specific value. Otherwise, it will be classified as a complex one. For the simple block, the threshold of the edge detector should be relatively low to detect both low-contrast and high-contrast texts. On the contrary, the threshold for the complex block should be relatively high to eliminate background and highlight text. 2.2.2 Local Adaptive Threshold Algorithm In each block, the stroke width value corresponding to text objects often dominates in population of the stroke width set. Therefore, it can be estimated by calculating the stroke width with the maximum stroke width histogram value. Let wt denote the stroke width value of text, wt can be defined as:
wt = max H (l ) , l
(1)
where H(l) is the value on the block’s stroke width histogram with stroke width l. From the set of all double-edge pixels, we construct two rough sets: the text set St and the background set Sbg . The St represents the set of all pixels which are predicted as text edge pixels whereas the Sbg represents the set of all predicted background edge pixels. St and Sbg are constructed as follow:
St = {i, j | i, j ∈ E , w(i, j ) = wt } ,
(2)
Sbg = {i, j | i, j ∈ E , w(i, j ) ≠ wt } ,
(3)
where E is the edge map of the block and w(i, j) denotes the stroke width between the double-edge pixels i and j. Note that St and Sbg are only the rough sets of the text edge pixels and background edge pixels, since only edge pixels with gradient direction in horizontal are considered during the stroke width calculation process. Thresholds for the simple block and the complex block are determined as follow: • In the simple block case, the text lies on clear background. Therefore, the threshold is determined as the minimum edge value of all edge pixels belonging to St in order to keep text information and simplify the computational process. • In the complex block case, to determine the suitable threshold for the edge detector is much more difficult. Applying general thresholding methods does not often give
204
V.C. Dinh et al.
a good result since these methods are used for classifying general problems, not for such a specific problem as separating text from background. In this paper, by discovering the similarity in stroke width of text, we can roughly estimate the text set and background set as St and Sbg . Therefore, the problem of finding an appropriate threshold in this case can be converted into another but easier problem of finding appropriate threshold to correctly separate the two sets: St and Sbg . Image Block
Calculate stroke width wt Estimate the text set and background set as: St and Sbg Simple block case
Complex block case
Calculate edge value of St and Sbg Set the threshold as the smallest edge value of St
(a) Construct corresponding edge value histograms: ht(r) and hbg(r) Set the threshold as the edge value satisfying equation (4)
Edge detection with new threshold value
Edge Image Block
(b)
Fig. 3. Flowchart of the proposed local adaptive threshold algorithm
(c)
Fig. 4. Edge detection results. (a) Original Image. Edge detection using (b) constant threshold, (c) proposed local adaptive threshold algorithm.
Let r denote the edge value (gradient magnitude) of a pixel in a block, ht (r ) and hbg (r ) denote the histograms of the edge values corresponding to the text set St and background set Sbg , respectively. According to [12], if the form of the two distributions is known or assumed, it is possible to determine an optimal threshold (in term of minimum error) for segmenting the image into the two distinct sets. And the optimal threshold, denoted as T, can be revealed as the root of the equation: pt × ht (T ) = pbg × hbg (T ) ,
(4)
where pt and pbg ( pbg = 1 − pt ) are the probabilities of a pixel to be in St and Sbg sets, respectively. Consequently, the appropriate threshold for the complex block is determined as the value which satisfies or approximately satisfies equation (4). Figure 3 shows flowchart of the local adaptive threshold algorithm. Figure 4 shows the results of edge detection method on video frame in Fig. 4(a) by using only one constant threshold (Fig. 4(b)), in comparison with using the proposed local adaptive thresholds (Fig. 4(c)). The pictures show that the proposed method could eliminate more background pixels while still conserves text pixels.
An Efficient Method for Text Detection in Video Based on Stroke Width Similarity
205
2.3 Stroke-Based Text Localization
After edge detection process, dilation morphological operator is applied to the edge detected video frame for highlighting text regions. The size of the structuring element is adaptively determined by the stroke width value. When applying the dilation operator, one of the most important factors that need to be considered is the size of the structuring element. If this size is set too small, the text area cannot be filled wholly. As the result, this area can be regarded as non-text area. In contrast, if this size is set too large, text can be mixed with the surrounding background. This problem results in increasing the number of false alarms. Moreover, using only a fixed size of the structure element, as in Chen et al.’s [9] method, is not applicable for texts of different sizes. 2wt+1
1
1
….
1
1
Fig. 5. Structure element of the dilation operation (wt is the stroke width value)
In this paper, we determine the size of the structure element based on the stroke width value which is already revealed in the edge detection process. More specifically, for each block, we apply a dilation operator of the size: ( (2 × wt + 1) × 1 ) at which the stroke width is wt as shown in Fig. 5. This size is satisfactory to wholly fill the character as well as connect neighborhood characters together. Moreover, using block-based dilation with suitable structure element shape makes it applicable for text with different sizes, at different locations in video frame. Figure 6(a) shows the image using the proposed dilation operators.
(a)
(b)
(c)
Fig. 6. Text localization and refinement process (a) Dilated image (b) Text regions candidates (c) Text regions after being refined by multi-frame refinements
After dilation process, connected component analysis is performed to create text region candidates. Then, based on the characteristic of text, the following simple criteria for filtering out non-text regions are applied: 1) the height of the region is between 8 and 35; 2) the width of the region must be larger than the height; 3) the number of edge pixel must be two times larger than the width based on the observation that text
206
V.C. Dinh et al.
region should have abundant edge pixels. Figure 6(b) shows the text region candidates after applying these criteria. 2.4 Multi-frame Refinement
Multi-frame integration has been used for the purpose of text verification [13], or text enhancement [14]. However, temporal information for the purpose of text refinement in frame, which often plays an important role in increasing the accuracy of text segmentation and recognition steps afterward, has not been utilized so far. In this paper, we propose a multi-frame based method to refine the location of text by further eliminating background pixels in the rough text regions detected in the previous steps. Moreover, the quality of text is also improved by selecting the most suitable frame, i.e. the frame at which text is displayed clearest, in the frame sequence. By using our method, the enhanced text region doesn’t cause the blurring problem as in text enhancement of Li et al.’s method [14]. First, a multi-frame verification [13] is applied to reduce the number of false alarms. For each of m consecutive frames in a video sequence, a text region candidate is considered as a true text region only if existing at least n (n<m) similar text regions T0, T1, …, Tn-1 appearing in n different frames. Tk (k = 0, 1...n-1) is the region of the corresponding frame received after edge detection process. Let call T the stationary edge image of the corresponding text region candidate. The pixel value at location (x, y) of T is determined as follows: ⎧ if ⎪edge pixel , T ( x, y ) = ⎨ ⎪non edge pixel , ⎩
n-1
∑ I k ( x, y ) > θ
k=0
(5)
otherwise,
where θ is a specific threshold and Ik (x, y) is defined as:
⎧1, I k ( x, y ) = ⎨ ⎩0,
if Tk ( x, y ) is edge pixel otherwise.
(6)
Refer to (5), T(x, y) is an edge pixel if at the location (x, y), an edge pixel appears more than θ times, otherwise, T(x, y) is a non-edge pixel. In the proposed method, the θ is set equal to [n × 3 / 4] in order to reduce the effect of noise. Based on the stationary characteristic of text, almost all background pixels are removed in T. However, this integration process may also remove some text edge pixels. In order to recover the lost text edge pixels, a simple edge recovery process is performed. A pixel in T is marked as edge pixel if it’s two neighborhoods in the horizontal, vertical, or diagonal direction are edge pixels. After the recovery process, T can be seen as the edge image of the true text regions. Therefore, the precise text location of the corresponding text region can be obtained by calculating the bounding box of edge pixels contained in T. In order to enhance the quality of the text, we extract the most suitable frame in the frame sequence where text appears clearest. Based on the fact that a text region is clearest if the corresponding edge image contains almost text pixels, the most suitable frame is extracted if the edge image of its text region is the best matching with T. In
An Efficient Method for Text Detection in Video Based on Stroke Width Similarity
207
other words, we choose the frame whose edge image Tk (k = 0,.., n-1) is the most similar with T. The MSE (Mean Squared Error) measurement is used to measure the similarity between two regions. The effectiveness of using multi-frame refinement is manifested in Fig. 6(c). Comparing to Fig. 6(b), two false alarms are removed and all of true text regions have more precise bounding boxes.
3 Experimental Result Due to the lack of a standard database for the problem of text detection in video, in order to evaluate the effectiveness of the proposed method, we have collected a number of videos from various sources for a test database. Text appearance varies with different color, orientation, language, and character font size (from 8pt to 90pt). The video frame formats are 512×384 and 720×480 pixels. The test database can be divided into three main categories: news, sport, and drama. Table 1 shows the video length and the number of ground-truth text regions contained in each video category. Totally, there are 553 ground-truth text regions in the whole video test database. Table 1. Properties of video categories Drama Video length Text regions
15 minute 126
Sport 32 minute 202
News 38 minute 225
For quantitative evaluation, the detected text region is considered as the correct one if the intersection of the detected text region (DTR) and the ground-truth text region (GTR) covers more than 90% of this DTR and 90% of this GTR. The efficiency of our detection method is assessed in terms of three measurements (which are defined in [10]): Speed, Detection Rate, and Detection Accuracy. In order to assess the effectiveness of the proposed method, we compare the performance of the proposed method with that of the typical edge-based method proposed by Lyu et al. [10], and the method using 3 processes: edge detection with a constant threshold, text localization with fixed size dilation operations (similar to the algorithm in [9]), and multi-frame refinement. Let call it “constant threshold” method. Table 2 shows the number of correct and false DTRs for three video categories. It can be seen from the table that not only does the proposed method create the highest number of correct DTRs but it also produces the smallest number of false DTRs in every case. Our method is obviously stronger than the others even in the case of news category (the number of false DTRs is about only a half compared to other methods). It is more difficult to detect text in news video since the background is changing fast and texts have variable sizes with different contrast levels to the background. The proposed method could overcome these problems since it successfully exploits the self characteristic of text (the stroke similarity), which is invariant to the background complexity as well as different font sizes and colors of text. Table 3 gives a summary of the detection rate and the detection accuracy of the three methods tested on the whole video test database. The proposed method achieved
208
V.C. Dinh et al. Table 2. Number of correct and false DTRs
Correct DTRs False DTRs Correct DTRs False DTRs Correct DTRs False DTRs
Lyu et al. [10] Constant threshold Proposed Method
Drama
Sport
News
96 16 109 19 114 11
154 26 152 32 179 20
185 38 189 46 205 21
the highest accuracy with the detection rate of 90.1% and the detection accuracy of 90.5%. This encouraging result shows that our proposed method is an effective solution to the background complexity problem of text detection in video. It can be seen from the table that the proposed method is faster than Lyu et al.’s [10] method and a bit lower than using constant threshold method which is obviously clear since we need to scan the frame with different thresholds. Moreover, the processing time of 0.18s per frame meets the requirement for real-time applications. Figure 7 shows some more examples of the results we got. In these pictures, all the text strings are detected and their bounding boxes are relatively tight and accurate. Table 3. Text detection accuracy
Lyu et al. [10] Constant threshold Proposed Method
Correct DTRs 435 450 498
False DTRs 80 97 52
Detection Rate 78.7 % 81.4 % 90.1 %
Detection Accuracy 84.5 % 82.3 % 90.5 %
Fig. 7. Some pictures of detected text regions in frames
Speed (Sec/frame) 0.23s 0.16s 0.18s
An Efficient Method for Text Detection in Video Based on Stroke Width Similarity
209
4 Conclusion This paper presents a comprehensive method for text detection in video. Based on the similarity in stroke width of text, an effective edge detection method with local adaptive thresholds is applied to reduce the background complexity. The stroke width information is further utilized to determine the structure element size of the dilation operator in the text localization process. To reduce the false alarm as well as refine the text location, a new multi-frame refinement method is applied. Experimental results with a large set of videos demonstrate the efficiency of our method with the detection rate of 90.1% and detection accuracy of 90.5%. Based on these encouraging results, we plan to continue research on text tracking and recognition for a real time text-based video indexing and retrieval system.
References 1. Zhu, Q., Yeh, M.C., Cheng, K.T.: Multimodal fusion using learned text concepts for image categorization. In: Proc. of ACM Int’l. Conf. on Multimedia, pp. 211–220. ACM Press, New York (2006) 2. Lienhart, R.: Dynamic video summarization of home video. In: Proc. of SPIE, vol. 3972, pp. 378–389 (1999) 3. Fan, J., Luo, H., Elmagarmid, A.K.: Concept-oriented indexing of video databases: toward semantic sensitive retrieval and browsing. IEEE Trans. on Image Processing 13, 974–992 (2004) 4. Zhong, Y., Karu, K., Jain, A.K.: Locating text in complex color images. Pattern Recognition 28, 1523–1536 (1995) 5. Jain, A.K., Yu, B.: Automatic text location in images and video frames. In: Proc. of Int’l. Conf. on Pattern Recognition, vol. 2, pp. 1497–1499 (August 1998) 6. Ohya, J., Shio, A., Akamatsu, S.: Recognition characters in scene images. IEEE Trans. on Pattern Analysis and Machine Intelligence 16, 214–220 (1994) 7. Qiao, Y.L., Li, M., Lu, Z.M., Sun, S.H.: Gabor filter based text extraction from digital document images. In: Proc. of Int’l. Conf. on Intelligent Information Hiding and Multimedia Signal Processing, pp. 297–300 (December 2006) 8. Li, H., Doermann, D., Kia, O.: Automatic text detection and tracking in digital video. IEEE Trans. on Image Processing, 147–156 (2000) 9. Chen, D., Bourlard, H., Thiran, J.P.: Text identification in complex background using SVM. In: Proc. of Int’l. Conf. on Document Analysis and Recognition, vol. 2, pp. 621–626 (December 2001) 10. Lyu, M.R., Song, J., Cai, M.: A comprehensive method for multilingual video text detection, localization, and extraction. IEEE Trans. on Circuits Systems Video Technology, 243–255 (2005) 11. Jung, K.C., Han, J.H., Kim, K.I., Park, S.H.: Support vector machines for text location in news video images. In: Proc. of Int’l. Conf. on System Technology, pp. 176–189 (September 2000) 12. Gonzalez, R.-C., Woods, R.E.: Digital Image Processing, 2nd edn., pp. 602–608. PrenticeHall, Englewood Cliffs (2002) 13. Lienhart, R., Wernicke, A.: Localizing and segmenting text in images and videos. IEEE Trans. on Circuits Systems Video Technology, 256–268 (2002) 14. Li, H., Doermann, D.: Text enhancement in digital video using multiple frame integration. In: Proc. of ACM Int’l. Conf. on Multimedia, pp. 19–22. ACM Press, New York (1999)
Multiview Pedestrian Detection Based on Vector Boosting Cong Hou1, Haizhou Ai1, and Shihong Lao2 1
Computer Science and Technology Department, Tsinghua University, Beijing 100084, China Sensing and Control Technology Laboratory, Omron Corporation, Kyoto 619-0283, Japan
[email protected]
2
Abstract. In this paper, a multiview pedestrian detection method based on Vector Boosting algorithm is presented. The Extended Histograms of Oriented Gradients (EHOG) features are formed via dominant orientations in which gradient orientations are quantified into several angle scales that divide gradient orientation space into a number of dominant orientations. Blocks of combined rectangles with their dominant orientations constitute the feature pool. The Vector Boosting algorithm is used to learn a tree-structure detector for multiview pedestrian detection based on EHOG features. Further a detector pyramid framework over several pedestrian scales is proposed for better performance. Experimental results are reported to show its high performance. Keywords: Pedestrian detection, Vector Boosting, classification.
1 Introduction Pedestrian detection researches originated in the requirement of intelligent vehicle system such as driver assistance systems [1] and automated unmanned car systems [13], and become more popular in recent research activities including visual surveillance [2], human computer interaction, and video analysis and content extraction, of which the last two are in more general sense that involve full-body human detection and his movement analysis [14]. Pedestrian, by definition, means a person traveling on foot, that is, a walker. Pedestrian detection is to locate all pedestrian areas in an image, usually in the form of bounding rectangles. We all know as a special case in more general research domain “object detection or object category”, face, car and pedestrian are most researched targets. Nowadays, although face detection or at least frontal face detection is well accepted solved problem in academic society, car detection and pedestrian detection are not so well solved; they remain a big challenge to achieve a comparable performance to face detection in order to meet the requirement of practical applications in visual surveillance etc. In general, object detection or object category is still in its early research stage that is very far from real application. For previous works before 2005 see a survey [3] and an experimental study [4]. Recent works are mainly machine learning based approaches among which the edgelets method [5] and the HOG method [7] are most representative. The edgelets method [5] uses a new type of silhouette oriented feature called an edgelet that is a short segment of line or curve. Based on edgelets features, part (full-body, Y. Yagi et al. (Eds.): ACCV 2007, Part I, LNCS 4843, pp. 210–219, 2007. © Springer-Verlag Berlin Heidelberg 2007
Multiview Pedestrian Detection Based on Vector Boosting
211
head-shoulders, torso, and legs) detectors are learned by Real AdaBoost. Responses of part detectors are combined to form a joint likelihood model and the maximum a posteriori (MAP) method is used for post processing to deal with multiple, possibly inter-occluded humans detection problem. The HOG method [7] uses the histograms of oriented gradients features to characterize gradient orientation distribution in a rectangular block. A detection window is divided into several rectangular blocks and each block is divided into several small spatial regions called cells in which the HOGs are computed and combined to form the features for classification. A linear SVM is used for detector training. This method is improved in [9] for speed up by AdaBoost learning of a cascade detector, in which variable-size blocks are used to enrich the feature pool and each feature is a 36D vector (concatenated by 4 cells’ 9 orientation bins of histogram in a block) that is fed into a linear SVM to form a weak classifier for AdaBoost learning. The above works are for still images and there are also recent advance for video [6][8][12]. Pedestrian detection in video is quite different from that in still images although techniques developed in later case can be help to pedestrian detection in video, for example to initialize or trigger a tracking module. Anyway pedestrian detection in still images is more fundamental. In this paper, we will focus on multiview pedestrian detection (MVPD) in still images and present a method using extended HOG features and Vector Boosting originally developed for multiview face detection (MVFD) for MVPD. Although pedestrian detection seems similar to face detection, it is more difficult due to large variation caused by clothes in addition to other common factors like pose, illumination, etc. In last several years MVFD has achieved great success and found its ways in practical applications. Many MVFD methods have been developed including parallel cascades [15], pyramid [16], decision tree [17] and WFS tree [18], of which the WFS tree together with the Vector Boosting algorithm is proved to be one of the most efficient methods. In this paper, we develop a method to apply this technique to the MVPD problem. We quantify gradient orientations into three angle scales that divide gradient orientation space into totally 27 dominant orientations. The EHOG features of a block of rectangle or non-regular rectangle are used to represent statistical information of edges in that block. Therefore blocks with their dominant orientations constitute the feature pool. The Vector Boosting learning [18] is used to construct a tree-structure detector [18] for MVPD. Further a detector pyramid framework over several pedestrian scales is proposed for better performance. The main contributions are (1) Dominant orientation in combined rectangle block is introduced into HOG features to form a feature pool; (2) A high performance tree-structure detector is developed for MVPD based on Vector Boosting. The rest of the paper is organized as follows: in Section 2, an extension to the HOG feature is introduced. In Section 3, the tree-structure detector training is described. Experiments are reported in Section 4 and conclusions are given in Section 5.
2 Extended HOG Feature The HOG feature has been proved effective in pedestrian detection [7][9]. The feature makes statistics about magnitude of gradient in several orientations, which are called
212
C. Hou, H. Ai, and S. Lao
bins. In [7][9], the orientation over 0◦~180◦ is divided into 9 bins. The HOG is collected in local regions in a picture called cells. A block contains several cells whose HOGs are concatenated into a higher dimension vector. And then a SVM is used to construct a classifier for each corresponding block over training set. However, the SVM detector is computational intensive in detection. Therefore, boosted detector that has been proved successful in face detection can be a good choice. In [9] HOG features are fed into a linear SVM to form a weak classifier which results in a much faster detector. In this paper in order to achieve better performance in both detection rate and speed, we make an extension to the HOG feature which outputs a scale value. The feature can be directly used in boosting learning as a weak classifier, which avoids time consuming inner product of high dimension vectors as in SVM or LDA type of weak classifier. First, we calculate the HOG in a block itself without dividing it into smaller cells as in [7]. Therefore the block functions in fact as an ensemble cell:
Gb = ( gb (1), gb (2),..., g b (n))T where n is the dimension of the HOG (in [7][9], n = 9), and b is a block in an image. Then, we introduce the concept of dominant orientation D (for detail, see section 2.1) that is defined as a subset of the above basic level of bins, that is, D ⊆ {1, 2," , n} , and calculate the EHOG feature corresponding to D as:
Fb ( D) = ∑ gb (i ) / Z b i∈D
where Z b is the normalizing factor: n
Z b = ∑ g b (i ) i =1
With the help of the integral image of HOG [9],
gb (1), gb (2),..., gb (n) and Zb
can be calculated very fast. We will explain two important concepts in more detail: the dominant orientation D and the non-rectangle block b . 2.1 Dominant Orientation The dominant orientation is indeed a set of representative bins of the HOG. We have observed that in an area containing simple edges, most of gradients concentrate in a relatively small range of orientation. Therefore, we can use a small part of bins to represent these edges. In most situations, this treatment is acceptable as shown in Fig 1. In training, the dominant orientation is found by feature selection. In our implementation, we also divide the orientation over 0◦~180◦ into 9 bins as in [7], and the dominant orientation of each feature may contain 1, 2 or 3 neighboring bins as shown in Fig 2. Therefore, there are totally 27 different dominant orientations for each block.
Multiview Pedestrian Detection Based on Vector Boosting
213
Fig. 1. (a) A picture with a pedestrian. (b) The HOG calculated in the red rectangle in (a). The length of each line denotes the magnitude of gradient in each bin. It can be seen there are three main orientations (lines of these orientations are in bold). (c) We only pick these three bins out and use the normalized summation of their values as the output of the EHOG feature.
Fig. 2. Three levels of orientation partition between 0◦-180◦, and each partition has 9 different orientations (note that there are some overlaps between neighboring parts in (b) (c)). The dominant orientation in each level covers (a) 20◦ (1 bin), (b) 40◦ (2 bins), (c) 60◦ (3 bins).
2.2 Non-rectangle Blocks The HOG and EHOG feature are both calculated in a local region of an image which is called a block. In [7], the size of the block is fixed, and in [9] it is variable. We also use variable-size blocks, and make some extension that other than the rectangle blocks used in [7][9], we also adopt blocks with non-rectangle shapes like in Fig 3 (a) called combined blocks to enrich the feature pool in order to reflex geometry structure of feature representation. In addition, we add block pairs to capture symmetric feature of pedestrians (see Fig.7 for block pair examples). To avoid feature space explosion, we manage the feature space by way of selecting and expanding with a heuristic search strategy. The initial feature space contains only
Fig. 3. (a) Some blocks with irregular shapes. (b) Two types of expanding operators.
214
C. Hou, H. Ai, and S. Lao
rectangle blocks. After feature selection, we get a small set of best rectangle features as seeds for generating additional non-rectangle blocks. Two kinds of operation on these seeds to make shape changes are defined as illustrated in Fig 3 (b): sticking and pruning. To describe the operation, we differ the rectangle blocks into two types: positive one and negative one. To stick is to add a positive rectangle block beside the seed block, and to prune is to add a negative rectangle block in the seed block. After several such operations, a seed can be propagated into thousands of ones which constitute the new feature space for further training.
3 Multi-view Pedestrian Detection Although pedestrians of different poses are not so much discriminative as that in MVFD problem in which frontal, left-half-profile and left-full-profile, right-halfprofile and right-full-profile are common divided sub-views, we can still divide pedestrians into three relatively separated classes according to their views: frontal/rear, left-profile and right-profile views. We use Vector Boosting to learn a tree-structure detector for multiview pedestrian detection. 3.1 Vector Boosting The Vector Boosting algorithm was first proposed by Huang etc. in [19] to deal with the multi-view face detection. It deals with multi-class classification problems by means of the vectorization of hypothesis output space and the flexible loss function defined by intrinsic projection vectors, for detail see [19]. 3.2 Tree-Structure Detector The tree-structure detector is illustrated in Fig 4. Before the branching node, a series of nodes try to separate different views of positive samples and at the same time discard as many negative samples as possible. They functions as a cascade detector [11] in which each node performs a binary decision: positive or negative. The branching node outputs a 3D vector whose components determine which branch or branches the sample should be sent to. For example, the output (1, 1, 0) means the sample may be a left profile pedestrian or a frontal/rear one. After the branching node, again there comes a cascade detector for each branch.
Fig. 4. The tree-structure multi-view pedestrian detector. The gray node is a branching node which outputs a 3D binary decision.
Multiview Pedestrian Detection Based on Vector Boosting
215
3.3 Training Process There are three kinds of tree node to train: the nodes before the branching node, the branching node and the nodes after the branch node. Each node is a strong classifier learned by Vector Boosting which is denoted by F(x) . In our problem,
F (x) = ( Fl ( x), F f (x), Fr (x))T . The decision boundaries as stated in [19] will be as follows in our problem:
P (ω N | x) =
1 1 + exp(2 Fl ( x)) + exp(2 F f ( x)) + exp(2 Fr ( x))
P (ωL | x) = exp(2 Fl ( x)) P (ω N | x) P (ωF | x) = exp(2 F f ( x)) P (ω N | x) P (ωR | x) = exp(2 Fr ( x)) P (ω N | x) where P(ωN | x) , P(ωL | x), P(ωF | x) and P(ωR | x) are separately the posterior probability of negative samples and positive samples of three views. The first kind of node above only cares if the sample is positive or negative, so it only needs to calculate P(ωN | x) . In the training, we’ll find a threshold Pt (ωN )
Fig. 5. Distributions of 3 classes (negative samples, positive samples of left profile and frontal/rear views) in the output space of the first 9 nodes before the branching. It can be seen that after 6 nodes pedestrians of different views can be separated rather well.
216
C. Hou, H. Ai, and S. Lao
according to the detection rate and false alarm rate of the node. If P (ω N | x) > Pt (ω N ) , the sample is regarded as a negative one, else positive. The second kind of node, that is, the branching node, tries to separate positive samples with different views, so P(ωL | x), P(ωF | x) and P(ωR | x) are all needed. The output is a 3D vector, in which each dimension is a binary decision decided by a corresponding threshold. The node in branches deals with a two-class classification problem, so the normal Real AdaBoost [10] learning can be used. One question remained is how to determine when to branch. This is done by experiments in our practice. As shown in Fig 5, the first 9 nodes before branching node have their distributions of negative samples, positive samples with frontal/rear and left profile views. It can be seen that the pedestrians with different views have been well separated in the 9th node; therefore we choose this node as the branching node. 3.4 The Detector Pyramid Framework Generally speaking, the size of training samples has great impact on the performance of the learnt detector both in detection accuracy and in speed. In face detection research, a common used size is 24×24 pixel (19×19 and 20×20 are also used in earlier work) which has been demonstrated very effective. In pedestrian detection research, 15×20 [12], 24×58 [5], 64×128[7][9] have been used. Different from face detection research, there is no common accepted size widely used. In practice, we found that detectors trained by larger samples have better performance when detecting larger pedestrians possibly because larger samples offer more clear information for classification of more complex objects like pedestrians. So, we use samples of different scales (sizes) to build a detector pyramid. The small size detector in the pyramid deals with small pedestrians and the large size detector deals with large ones. The number of layers of the scale pyramid of the input image to be scanned accordingly decreases, which can speed up the detection compared with single scale detector case.
4 Experiments Our training set contains 9,006 positive samples for frontal/rear view, 2,817 positive samples for left/right profile view. Pedestrians in samples are upright standing or walking. Some samples are shown in Fig 6. The negative samples are sampled from more than 10,000 images without any human.
Fig. 6. Positive training samples: (a) frontal/real views; (b) left profile view; (c) right profile view
Multiview Pedestrian Detection Based on Vector Boosting
217
The detector pyramid has 3 layers whose sizes are 24×58, 36×87 and 48×116 pixel respectively. The number of features in each node of the three detectors decreases as the size of the detector increases. For example, the total numbers of features in the first 5 nodes of these three detectors are 75, 53 and 46 respectively. So the speed of the detector increases with its size grows. Because EHOG features with non-rectangle blocks are slower than those with rectangle blocks in computing, for efficiency the feature pool for the first several nodes only contains the rectangle ones that guarantees faster speed. Fig 7 shows the first three (pair) features selected in the 24×58 detector. It can be seen that the second feature captures the edge of shoulders and the third captures the edge of foot. The detection speed of our detector is about 1.2 FPS with a 320×240 pixel image on a 3.06GHz CPU.
Fig. 7. The first three features selected and their corresponding dominant orientations
We evaluate our detector on two testing sets: one is Wu et al [5]’s testing set which contains pedestrians with frontal/rear view and the other is the INRIA testing set [7]. Wu’s testing set contains 205 photos with 313 humans of frontal and rear view. Fig 8 (a) shows the ROC curves of our detector (including the detector pyramid and a
Fig. 8. (a) ROC curves of evaluation on Wu’s testing set [5]. (b) Miss-rate/FPPW curves on INRIA testing set [7].
218
C. Hou, H. Ai, and S. Lao
Fig. 9. Some detection results on Wu’s frontal/rear testing set [5]
Fig. 10. Some detection results on INRIA testing set [7]
24×58 detector) and Wu’s edgelet full-body detector and their combined detector. It can be seen that our detector pyramid is better than the full-body detector and the combined detector in accuracy, and is better than single detector too. Some detection results are shown in Fig 9 on Wu’s test set. The INRIA testing set contains 1805 64×128 images of humans with a wide range of variations in pose, appearance, clothing, illumination and background. Fig 8 (b)
miss-rate/FPPW
shows the comparative results in (False Positive Per Window) curves. We can see that our method is comparable with Zhu’s method when the FPPW is low. At 10-4 FPPW, the detection rate is 90%. Some detection results are shown in Fig 10.
5 Conclusion In this paper, a multiview pedestrian detection method based on Vector Boosting algorithm is presented. The HOG features are extended to form EHOG features via dominant orientations. Blocks of combined rectangles with their dominant orientations constitute the feature pool. The Vector Boosting algorithm is used to learn a tree-structure detector for multiview pedestrian detection based on EHOG features. Further a detector pyramid framework over several pedestrian scales is proposed for better performance. This results in a high performance MVPD system that can be very useful in many practical applications including visual surveillance. We are planning to extend this research to video for pedestrian tracking in future.
Acknowledgement This work is supported in part by National Science Foundation of China under grant No.60673107 and it is also supported by a grant from Omron Corporation.
Multiview Pedestrian Detection Based on Vector Boosting
219
References 1. Gavrila, D.M.: Sensor-based Pedestrian Protection. IEEE Intelligent Systems, 77–81 (2001) 2. Zhao, T.: Model-based Segmentation and Tracking of Multiple Humans in Complex Situations. In: CVPR 2003 (2003) 3. Ogale, N.A.: A survey of techniques for human detection from video, University of Maryland, Technical report (2005) 4. Munder, S., Gavrila, D.M.: An Experimental Study on Pedestrian Classification. TPAMI 28(11) (2006) 5. Wu, B., Nevatia, R.: Detection of Multiple, Partially Occluded Humans in a Single Image by Bayesian Combination of Edgelet Part Detectors. In: Sebe, N., Lew, M.S., Huang, T.S. (eds.) Computer Vision in Human-Computer Interaction. LNCS, vol. 3766, Springer, Heidelberg (2005) 6. Wu, B., Nevatia, R.: Tracking of Multiple, Partially Occluded Humans based on Static Body Part Detection. In: CVPR 2006 (2006) 7. Dalal, N., Triggs, B.: Histograms of Oriented Gradients for Human detection. In: CVPR 2005 (2005) 8. Dalal, N., Triggs, B., Schmid, C.: Human Detection Using Oriented Histograms of Flow and Appearance. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3951, Springer, Heidelberg (2006) 9. Zhu, Q., Avidan, S., et al.: Fast Human Detection Using a Cascade of Histograms of Oriented Gradients. In: CVPR 2006 (2006) 10. Schapire, R.E., Singer, Y.: Improved Boosting Algorithms Using Confidence-rated Predictions. Machine Learning 37, 297–336 (1999) 11. Viola, P., Jones, M.: Rapid Object Detection using a Boosted Cascade of Simple Features. In: CVPR 2001 (2001) 12. Viola, P., Jones, M., Snow, D.: Detecting Pedestrians Using Pattern of Motion and Appearance. In: ICCV 2003 (2003) 13. Zhao, L., Thorpe, C.E.: Stereo- and Neural Network-Based Pedestrian Detection. IEEE Trans. on Intelligent Transportation Systems 1(3) (2000) 14. Gavrila, D.M.: The Visual Analysis of Human Movement: A Survey. Computer Vision and Image Understanding 73(1), 82–98 (1999) 15. Wu, B., Ai, H., et al.: Fast rotation invariant multi-view face detection based on Real Adaboost. In: FG 2004 (2004) 16. Li, S.Z., Zhu, L., et al.: Statistical Learning of Multi-View Face Detection. In: Heyden, A., Sparr, G., Nielsen, M., Johansen, P. (eds.) ECCV 2002. LNCS, vol. 2352, Springer, Heidelberg (2002) 17. Jones, M., Viola, P.: Fast Multi-view Face Detection. MERL-TR2003-96 (July 2003) 18. Huang, C., Ai, H.Z., et al.: Vector Boosting for Rotation Invariant Multi-View Face Detection. In: Sebe, N., Lew, M.S., Huang, T.S. (eds.) Computer Vision in Human-Computer Interaction. LNCS, vol. 3766, Springer, Heidelberg (2005) 19. Huang, C., Ai, H.Z., et al.: High-Performance Rotation Invariant Multiview Face Detection. TPAMI 29(4), 671–686 (2007)
Pedestrian Detection Using Global-Local Motion Patterns Dhiraj Goel and Tsuhan Chen Department of Electrical and Computer Engineering Carnegie Mellon University, U.S.A.
[email protected],
[email protected]
Abstract. We propose a novel learning strategy called Global-Local Motion Pattern Classification (GLMPC) to localize pedestrian-like motion patterns in videos. Instead of modeling such patterns as a single class that alone can lead to high intra-class variability, three meaningful partitions are considered - left, right and frontal motion. An AdaBoost classifier based on the most discriminative eigenflow weak classifiers is learnt for each of these subsets separately. Furthermore, a linear threeclass SVM classifier is trained to estimate the global motion direction. To detect pedestrians in a given image sequence, the candidate optical flow sub-windows are tested by estimating the global motion direction followed by feeding to the matched AdaBoost classifier. The comparison with two baseline algorithms including the degenerate case of a single motion class shows an improvement of 37% in false positive rate.
1
Introduction
Pedestrian detection is a popular research problem in the field of computer vision. It finds its applications in surveillance, fast automatic video browsing for pedestrians, activity monitoring etc. The problem to localize pedestrians in image sequences, however, is extremely challenging owing to the variations in pose, articulation and clothing. The resulting high intra-class variability for the class of pedestrians is further exaggerated by the background clutter and the presence of pedestrian-like upright objects in the scene like trees and windows. Traditionally, appearance and shape cues have been the popular discernible features to detect pedestrians in a single image. Oren et al. [1] devised one of the first appearance based algorithms using wavelet response, while more recently, histogram of oriented gradients [2] have been used to learn a shapebased model to segment out humans. However, in an uncontrolled environment the appearance cues alone aren’t faithful enough for reliable detection. Recently, motion cues have been gaining a lot of interest for pedestrian detection. In general, pedestrians need to be detected in videos where high correlation between consecutive frames can be used to good effect. While human appearances can be deceptive in a single image, their motion patterns are significantly different from other kinds of motions like vehicles (Fig. 2). The articulation of the human body while in motion due to the movement of limbs and torso can Y. Yagi et al. (Eds.): ACCV 2007, Part I, LNCS 4843, pp. 220–229, 2007. c Springer-Verlag Berlin Heidelberg 2007
Pedestrian Detection Using Global-Local Motion Patterns
221
Fig. 1. Overview of the proposed system
provide useful cues to localize moving pedestrians, especially in a stationary cluttered background. To model such a phenomenon, spatio-temporal filters based on shifted frame difference were used by Viola et al. [3], thus, combining the advantages of both shape and motion cues. Fablet and Black [4] used dense optical flow to learn a generative human-motion model while a discriminative model based on Support Vector Machines was trained by Hedvig [5]. The common feature in all the above techniques is that they consider pedestrians as a single class. Though at one hand using human motion patterns circumvents many problems posed by appearance cues, considering all such patterns as a single class can still lead to a very challenging classification problem. In this paper, we present a novel learning strategy to partition the human motion patterns into natural subsets with lesser variability. The rest of the paper is organized as follows: Sect. 2 provides an overview of the proposed method, Sect. 3 introduces the learning strategy based on partitioning the human motion pattern space, Sect. 4 reports the comparison with two baseline algorithms and detection results, and Sect. 5 concludes with a discussion.
2
Overview
Figure 1 gives an overview of the proposed system to detect pedestrian-like motion patterns in the image sequences. Figure 2 illustrates some of the examples of such patterns. Due to high intra-class variability of the flow patterns generated by the pedestrians, modeling all such patterns using a single classifier is difficult. Hence, these are divided into meaningful subsets according to the global motion direction - left, right and frontal. As a result, the classification is divided into two stages. A linear three-class Support Vector Machines (SVM) classifier is trained to estimate the global motion direction. Next, a cascade of AdaBoost classifiers with the most discriminative eigenflow vectors is learnt for each of the global motion subsets. The motion patterns in the same partition share some similarity and hence, intra-class variability for each of these subsets is be less as compared to the whole set, rendering the classification less challenging.
222
D. Goel and T. Chen
(a)
(b)
Fig. 2. (a) Pedestrian sample images along with their horizontal optical flow for right, left and frontal motion subsets. (b) Sample labeled images from the non-pedestrian data and examples of non-pedestrian horizontal flow.
At the time of testing, the dense optical flow image is searched for pedestrianlike motion patterns using sub-windows of different sizes. For every candidate sub-window, first the global motion direction is estimated using the linear threeclass SVM classifier. Thereafter, it is tested against the matching AdaBooost classifier. 2.1
Computing Dense Optical Flow
Dense optical flow is used as a measure to estimate motion between consecutive frames. Though numerous methods exist in the literature to compute dense flow, 2-D Combined Local Global method [8] was chosen since it has been shown to provide very accurate flow. Furthermore, using bidirectional multi-grid strategy, it can work in real-time [9] at upto 40 fps for 200x200 pixels image. The final implementation used for pedestrian detection incorporates a slight modification in the weighting function of the regularization term as mentioned in [6]. 2.2
Training Data
The anatomy of the learning algorithm necessitates a pedestrian data set labeled according to the global motion. For this purpose, the CASIA Gait database [7] was chosen. A total of eight global motion directions were considered that were merged to give three dominant motions - left, right and frontal (Fig. 2(a)). The left and the right motion subset capture the lateral motion while the motion perpendicular to the camera plane is contained in the frontal motion subset. Dense optical flow was computed for the videos and the horizontal, u, and the vertical, v, flows for the labeled pedestrians were cropped. The collection of these flow patterns formed the training-test data for the classification. Specifically, the frontal motion subset had 2500 training data samples and 1000 test data samples. The other two motion subsets had 4800 training data samples and 2000 test data samples each. The cropped data samples were resized to 16x8 pixels, normalized to lie in the range [−1, 1] and concatenated to form a 256 dimension feature vector - [u1 , u2 , . . . , u128 , v1 , v2 , . . . , v128 ]. The non-pedestrian data was generated by hand-labeling sub-windows with non-zero flow in the videos containing moving vehicles. To automate the process,
Pedestrian Detection Using Global-Local Motion Patterns
223
an Adaboost classifier was trained for the set of all pedestrian and non-pedestrian data and was run on other videos to generate additional non-pedestrian flow patterns (from the false positives). The non-pedestrian data samples are resized and normalized in the same way as the pedestrian data. Approx. 120,000 such samples were generated, with some examples shown in Fig. 2(b).
3
Classification Strategy
This section describes the classification strategy to distinguish the motion patterns of pedestrians from other kinds of motions like that of vehicles etc. As illustrated in Fig. 1, it is divided into two stages - estimating the global motion direction (Section 3.1) followed by testing against the discriminative classifier (Section 3.2). Training procedure for the latter has been described in [6]. The final detection performance depends on the accuracy of both the stages and is greatly influenced by the taxonomy of the pedestrian motion patterns. A maximum of eight possible motion classes were considered as shown in the Fig. 2(a). Building a discriminative classifier for each of them results in a group of classifiers that are highly discriminative for the motion direction they are trained for. Thus, the accuracy in estimating the motion direction becomes crucial to the overall performance, i.e. the sub-window containing strictly left moving pedestrian should be fed to the classifier trained to detect strictly left moving pedestrians. However, it is very difficult to reliably estimate the motion direction in these eight subsets. Thus, the detection rate of the classifier as a whole degrades. The natural modification is to merge the different motion subsets such that the motion direction can be estimated faithfully but at the same time intraclass variability is kept low. Splitting the motion patterns into three subsets left, right and frontal - gave the best performance. 3.1
Estimating Global Motion
In order to decide which motion-specific discriminative classifier to use, it is important to first estimate the global motion. The mean motion direction for the pedestrian data was found to be unreliable in achieving such an objective. Hence, a linear three-class SVM classifier was trained. This classifier acts as more of a switch that assigns the queried data samples to their appropriate classifiers that have been specifically trained to handle those particular flow patterns. The labeled pedestrian data is used to train this switch classifier. The same number of training data samples, about 2000 each, was used for all the three classes to obviate bias towards any particular class. Further, each of the classes themselves contain the same proportion of different motions contained within them. For example, the left class contains the same number of samples for strict left motion, left front at 45o and left back at 45o . Figure 3 shows the class confusion matrix for the learned model. 348 support vectors were chosen by the model that is less than 6% of the number of training data samples, indicating a well generalized classifier.
Frontal
224
D. Goel and T. Chen Frontal
Right
Left
0.964
0.023
0.013
Left
Right
Frontal
0.022
0.978
0.00
0.019
0.00
0.981 Right
Fig. 3. Class confusion matrix for estimating the global motion direction using the three-class linear SVM classifier
Left
Fig. 4. Magnitude of the mean and the first two eigenflow vectors of the horizontal optical flow for the training pedestrian data
The trained switch classifier is used to allocate non-pedestrian data for each of the motion classes for training the discriminative motion-specific classifiers. Out of 120,000 data samples, about 75,000 got classified as belonging to the frontal motion, 25,000 were categorized as left motion class while the remaining 20,000 as having right motion. 3.2
Learning the Discriminative Classifiers
This section describes the learning procedure to train the discriminative motionspecific classifiers. In total, three separate classifiers are learnt, one for each global motion. The learning process is the same for all of them. Hence, for the sake of clarity, motion-specific term has been dropped in this section and whenever pedestrian and non-pedestrian data is mentioned, it refers to the data belonging to a particular global motion, unless stated otherwise. It is worth mentioning that the symmetrical properties of left and right classifiers can be exploited by training the classifier for one and using it’s mirror image (after changing the sign for horizontal motion) for the other. Weak Classifier. Principal Component Analysis was done separately on the pedestrian and non-pedestrian data to obtain the eigenvectors for the optical flow, known as eigenflow [10]. Figure 4 shows the magnitude of the mean and the first two u-flow eigenvectors for each of the three global motions. As is evident, the mean flows represent the global motion while the eigenflow vectors capture the poses and the articulation of the human body, especially the movement of the limbs. For the frontal motion, the mean is not that informative since it contains both front and backward moving pedestrians. Using all the eigenflow vectors, 256 for each of the pedestrian and nonpedestrian data, we have a total of 512 eigenflow vectors that act as a pool of features for AdaBoost. Taking the magnitude of correlation between the training
Pedestrian Detection Using Global-Local Motion Patterns
225
Table 1. Feature selection and training AdaBoost classifier – Given the training data (x1 , y1 ), (x2 , y2 ), . . . , (xn , yn ) where xi is the eigenflow and yi is 0 for non-pedestrian and 1 for pedestrian examples. 1 1 , 2m for yi = 0, 1 respectively, where l and m are – Initialize the weights w1,i = 2l the number of pedestrian and non-pedestrian examples. – for t = 1, . . . , T w 1. Normalize the weights wt,i ← n t,iwt,j j=1
2. Selectthe best weak classifier ht with respect to the weighted error: t = minj i wi |hj − yi | 3. Update the weights: wt+1,i = wt,i βt1−ei where ei = 0 if example xi is correctly classified by ht , ei = 1 otherwise, and t . βt = 1− t – The strong classifier is given by: T T 1 1, if t=1 αt ht (x) ≥ 2 t=1 αt (2) C(x) = 0, otherwise. where αt = log
1 βt
data x and an eigenflow vector zj and finding the optimum threshold θj that minimizes the overall classification error would yield a weak classifier hj . 1, if |xT zj | ≶ θj hj (x) = (1) 0, otherwise. Feature Selection and AdaBoost. The procedure to choose the most discriminative of the weak classifiers, as illustrated in Fig. 5(a) is motivated by the face detection algorithm proposed in [11]. Table 1 describes the complete algorithm. The final strong classifier is a weighted vote of the weak classifiers (Eq. (2)). Figures 5(b), (c) and (d) depict the horizontal component of the two eigenflow features selected by this algorithm for each of the global motion subset. The selection of the most discriminative vectors follows a similar trend in all the three cases. While the first one responds to motion near the boundary, the second one captures the motion within the window. It is also interesting to note the pattern at the bottom of the first eigenflow vectors - those belonging to the right and left subsets take into account the spread of the legs in the lateral motion while the one for the frontal motion restricts any such articulation. Individually, they may perform poorly but as a combination, they can perform much better. Table 2 juxtaposes the false positive rate (FPR) of the GLMPC classifier with two other classifiers for a fixed detection rate of 98%. The first one is the linear SVM classifier that is clearly outperformed in both speed and accuracy. 13,313 support vectors were chosen by the linear SVM that is more than 50% of the
226
D. Goel and T. Chen
Fig. 5. (a) Feature Selection using AdaBoost. (b), (c) and (d) Two u-eigenflow vectors selected by AdaBoost for the Right, Left and Frontal subsets respectively. Table 2. False positive rate for the different classifiers for the detection rate of 98% SVM LMPC GLMPC False Positives (%) 62.3 1.16 0.74
training data, an indication of a poorly generalized classifier. Besides, such a high number of support vectors would result in about 1.3 million dot products per frame, assuming 100 candidate sub-windows in a frame. On the other hand, classification using GLMPC requires only 348 dot products for the three-class SVM switch and 35 dot products for AdaBoost cascade (full cascade in the worst case). The other classifier considered for comparison is the degenerate case of the proposed algorithm, that we refer as Local Motion Pattern Classifier (LMPC) [6], when all the pedestrian data is considered as one single class. GLMPC provides a reduction of 37% in FPR that is further amplified by the fact that they may hundreds of candidate sub-windows in a frame. Cascade of AdaBoost Classifiers. In general, in any scene, flow patterns that share no resemblance with human motion should be discarded quickly, while those that share greater similarity require more complex analysis. A cascade of AdaBoost classifiers [11] can achieve this. The early stages in the cascade have a lesser number of weak classifiers and hence, aren’t too discriminative but are really fast at classification. The later stages consist of more complex classifiers with larger number of weak classifiers. To be labeled as a detection, a candidate data sample has to pass through all the stages. Hence, the classifier spends most of the time analyzing difficult motion patterns and rejects easy ones quickly. In our implementation, there are two stages in the cascade for each of the global motion classifiers. The same pedestrian data was used across all stages. For training the classifier, the ratio of pedestrian to non-pedestrian data (for both training and test data) was kept at one for the left and right motion subsets and 0.5 for the frontal motion. Non-pedestrian data for the next stage in the cascade is generated by collecting the false positives after running the existing classifier on different videos taken from both static and moving cameras. The
Pedestrian Detection Using Global-Local Motion Patterns
227
final frontal classifier has 5 weak learners in the first stage and 20 in the second. The corresponding numbers for the right and the left motion classifiers are 10 and 25, and 10 and 20 respectively.
4
Experiments
For detecting human motion patterns, the dense optical flow image is searched with sub-windows of different scales, seven in total. Every scale size also has an associated step size. Naturally, larger sub-windows have bigger steps size to prevent redundancy due to excessive overlap between neighboring sub-windows. Knowing a priori, the camera orientation can greatly reduce the search space since the pedestrians need to be looked for only on the ground plane. Exploiting such an information reduced the total number of scanned sub-windows in the image by almost half. Finally, only the candidate sub-windows that satisfy the minimum flow thresholds are resized and normalized, before feeding to the classifier. Again, these thresholds vary with the scale size as larger sub-windows search for near-by pedestrians that should appear to move faster due to parallax. Figure 6 depicts the detection results by linear SVM, LMPC and GLMPC classifier after the first stage in the cascade. The overlapping windows have not been merged to show the all the detected sub-windows. As is evident, the GLMPC is able to localize the pedestrians much better than any of the two methods and in addition, gives less false positives. The full cascade GLMPC classifier was tested for pedestrian patterns in different test videos and works at 2fps on a Core 2 Duo 2 GHz PC. Figure 7 shows some of the relevant results. The algorithm was tested with multiple moving pedestrians in the presence of other moving objects, mainly cars and is able to detect humans in different poses and moving at different pace (Fig. 7(a)). The occluding objects can lead to false rejections since the flow in the concerned sub-window doesn’t conform to the pedestrian motion. This is evident in the
(a) SVM
(b) LMPC
(c) GLMPC
Fig. 6. Comparison of the performance of GLMPC classifier with linear SVM and LMPC after Stage 1 in the cascade. Color coding - white if direction is not known, red for right moving pedestrians, yellow for left and black for frontal motion.
228
D. Goel and T. Chen
(a)
(b)
(c)
(d) Fig. 7. Final detection results without merging the overlapping detections
second image in Fig. 7(a). Stationary and far-off pedestrians that are moving very slowly can also be missed owing to their negligible optical flow. The system is also robust to illumination changes (Fig. 7 (b)) and can detect moving children (Fig. 7(c)) even though the training data was composed of only adult pedestrians. Moreover, notice the panning of the camera over time in the image sequence, illustrating the robustness of the system towards small camera motion. The videos captured from a slow moving car were also tested and the system still manages to detect pedestrians (Fig. 7 (d)).
5
Discussion
A novel learning strategy to detect moving pedestrians in videos using motion patterns was introduced in the paper. Instead of considering all human motion patterns as one class, they were split into three meaningful subsets dictated by the global motion direction. A cascade of AdaBoost classifiers with the most discriminative eigenflow vectors were learnt for each of these global motion
Pedestrian Detection Using Global-Local Motion Patterns
229
subsets. Further, a linear three-class SVM classifier was trained that acts as a switch to decide which Adaboost classifier to choose to determine if a pedestrian is contained in the candidate sub-window. It was shown that the proposed algorithm is far superior to the linear SVM and provides an improvement of 37% in FPR as compared to LMPC. Moreover, the proposed system has been shown to be robust to slow illumination changes, camera motion and can even detect children. Apart from conspicuous advantages of accuracy, GLMPC allows for extensibility to incorporate new pedestrian motion like jumping without retraining the whole classifier again. Only a couple of changes would be required. The first would be to retrain the motion switch multi-class SVM classifier to take into account the new motion type. The next would be to train a new AdaBoost classifier to discriminate between the jumping motion of the pedestrians and other kinds of motions. The already trained classifiers for left, right and frontal motion can be used in their original form. An important area of research for the future work would be to compute the ROC curve for the classifiers like GLMPC that don’t have a single global threshold. Work on similar lines has been done by Xiaoming et al. [10].
References 1. Oren, M., Papageorgiou, C., Sinha, P., Osuna, E., Poggio, T.: Pedestrian detection using wavelet templates. CVPR, 193–199 (1997) 2. Dalal, N., Triggs, B.: Histograms of Oriented Gradients for Human Detection. CVPR 1, 886–893 (2005) 3. Viola, P., Jones, M., Snow, D.: Detecting Pedestrians Using Patterns of Motion and Appearance. ICCV 2, 734–741 (2003) 4. Fablet, R., Black, M.J.: Automatic Detection and Tracking of Human Motion with a View-Based Representation. In: Heyden, A., Sparr, G., Nielsen, M., Johansen, P. (eds.) ECCV 2002. LNCS, vol. 2352, pp. 476–491. Springer, Heidelberg (2002) 5. Sidenbladh, H.: Detecting Human Motion with Support Vector Machines. ICPR 2, 188–191 (2004) 6. Goel, D., Chen, T.: Real-time Pedestrian Detection using Eigenflow. In: IEEE International Conference on Image Processing, IEEE Computer Society Press, Los Alamitos (2007) 7. http://www.cbsr.ia.ac.cn/Databases.htm 8. Bruhn, A., Weickert, J., Schn¨ orr, C.: Lucas/Kanade Meets Horn/Schunck: Combining Local and Global Optic Flow Methods. IJCV 61, 211–231 (2005) 9. Bruhn, A., Weickert, J., Kohlberger, T., Schn¨ orr, C.: A Multigrid Platform for Real-Time Motion Computation with Discontinuity-Preserving Variational Methods. IJCV 69, 257–277 (2006) 10. Liu, X., Chen, T., Kumar, B.V.: Face authentication for multiple subjects using eigenflow. Pattern Recognition 36, 313–328 (2003) 11. Viola, P., Jones, M.: Rapid Object Detection using a Boosted Cascade of Simple Features. CVPR (2001)
Qualitative and Quantitative Behaviour of Geometrical PDEs in Image Processing Arjan Kuijper Radon Institute for Computational and Applied Mathematics, Linz, Austria
Abstract. We analyse a series of approaches to evolve images. It is motivated by combining Gaussian blurring, the Mean Curvature Motion (used for denoising and edge-preserving), and maximal blurring (used for inpainting). We investigate the generalised method using the combination of second order derivatives in terms of gauge coordinates. For the qualitative behaviour, we derive a solution of the PDE series and mention its properties briefly. Relations with general diffusion equations are discussed. Quantitative results are obtained by a novel implementation whose stability and convergence is analysed. The practical results are visualised on a real-life image, showing the expected qualitative behaviour. When a constraint is added that penalises the distance of the results to the input image, one can vary the desired amount of blurring and denoising.
1
Introduction
Already in early years of image analysis the Gaussian filter played an important role. As a side effect of Koenderink’s observation that this filter relates to human observation due to the causality principle [1], it opened the way for application of diffusion processes in image analysis. This is due to the fact that the Gaussian filter is the Greens’ function of the heat equation, a linear partial differential equations (PDEs). Because of its linearity, details are blurred during evolution. Therefore, various non-linear PDEs were developed to analyse and process images. A desirable aspect in the evolution of images is independence of the Cartesian coordinate system by choosing one that relates directly to image properties. One can think of the famous Perona Malik equation [2] using edge-strength. Using such so-called gauge coordinates, Alvarez et al. derived the Mean Curvature Motion [3] by blurring only along edges. On the other hand, the opposite approach can be used in inpainting [4,5]: blurring perpendicular to edges. Perhaps surprisingly, when combining these to methods one obtains the heat equation (see section 2). In this paper we proposea series of PDEs obtained by a parameterised linear combination of these two approaches. By doing so, one is able to influence the
This work was supported by FFG, through the Research & Development Project ‘Analyse von Digitaler Bilder mit Methoden der Differenzialgleichungen’, and the WWTF ‘Five senses-Call 2006’ project ‘Mathematical Methods for Image Analysis and Processing in the Visual Arts’.
Y. Yagi et al. (Eds.): ACCV 2007, Part I, LNCS 4843, pp. 230–239, 2007. c Springer-Verlag Berlin Heidelberg 2007
Qualitative and Quantitative Behaviour of Geometrical PDEs
231
evolution of the methods discussed above by adjusting the parameters. This relates to in- or decreasing blurring that is locally either tangent or normal to isophotes. Although one cannot obtain a filter as the Greens’ function for the general case, solutions give insight into the qualitative behaviour of the PDE. This is done in section 2. Also relations with general diffusion processes are given. The PDEs need a stable numerical implementation, which is depending on the parameters. In section 3 a novel numeric scheme is given, including a stability analysis. This scheme allows larger time steps than conventional finite difference schemes, and remains stable at corner points, in contrast to standard finite difference schemes.
2
Geometric PDEs: Second Order Gauge Derivatives
An image can be thought of as a collection of curves with equal value, the isophotes. Most isophotes are non-self-intersecting. At extrema an isophote reduces to a point, at saddle points the isophote is self-intersecting. At the noncritical points Gauge coordinates (v, w) (or (T, N ), or (ξ, η), or . . .) can be chosen [6,7,8]. Gauge coordinates are locally set such, that the v direction is tangent to the isophote and the w directionpoints in the direction of the gradient vector. Consequently, Lv = 0 and Lw = L2x + L2y . Of interest are the following second order structures: Lvv =
L2x Lyy + L2y Lxx − 2Lx Ly Lxy L2x + L2y
(1)
Lww =
L2x Lxx + L2y Lyy + 2Lx Ly Lxy L2x + L2y
(2)
These gauge derivatives can be expressed as a product of gradients and the Hessian matrix H with second order derivatives: Lww L2w = ∇L · H · ∇T L ˜ −1 · ∇T L, Lvv L2w = ∇L · H
(3) (4)
˜ −1 = det H · H −1 . Note that with ∇L = (Lx , Ly ), H the Hessian matrix, and H the expressions are invariant with respect to the spatial coordinates. Combining the two different expressions for the second order derivatives in gauge coordinates, Eqs. (1)-(2), yield Lt = pLvv + qLww .
(5)
Several parameter settings have relations to PDEs and histogram operations [9]: – (p, q) = (1, 1): Gaussian scale space [1], repeated infinitesimal mean filtering, – (p, q) = (1, 0): Mean Curvature Motion [3,10], repeated infinitesimal median filtering, – (p, q) = (1, −2): infinitesimal mode filtering. – (p, q) = (0, 1): maximal blurring used for inpainting [4].
232
2.1
A. Kuijper
A General Solution
In this section we derive the general solution for Eq. (5). As is the case with gauge coordinates, it is assumed that the solution is independent of direction, 2 2 size, dimension, and orientation. Therefore the dimensionless variable ξ = x +y t is used. Second, an additional t-dependency is assumed. This is inspired by the observation that the solution for p = q = 1 (the Gaussian filter) contains the factor t−1 . So starting assumption is L(x, y; t) = tn f (ξ), and equation (5) becomes tn−1 (−nf (ξ) + (2(p + q) + ξ)f (ξ) + 4qξf (ξ)) = 0. (6) The solution of this ODE with respect to f (ξ) and ξ is given by p−q ξ ξ p+q p+q ξ − 4q 2q f (ξ) = e c1 U n + , , + c2 L− p+2nq+q 2q 2q 4q 4q 2q
(7)
Here U (a, b, z) is a confluent hypergeometric function and Lba (z) is the generalised Laguerre polynomial expression [11]. Taking r = p+q 2q , we find L(x, y; t) = e−
x2 +y2 4qt
2 x + y2 x2 + y 2 tn c1 U n + r, r, + c2 Lr−1 . (8) −n−r 4qt 4qt
The formula reduces dramatically for n = −r, since U (0, ., .) = L.0 (.) = 1. This gives the following positive solutions of Eq. (5): p+q
L(x, y; t) =
+y2 t− 2q − x24qt e 4πq
(9)
The simplified diffusion (p, q) = (b − 1, 1) Lt = Lww + (b − 1)Lvv
(10)
has solution
−x2 −y2 1 4t e . (11) 4πtb/2 Qualitatively these types of flows are just a rescaling of standard Gaussian blurring, albeit that linearity between subsequent images in a sequence with increasing scale t is lost. Only for b = 2 the filter is linear, resulting in the Gaussian filter. For b = 1 one obtains maximal blurring. Note that a solution can only be obtained when q = 0. This implies that the direction Lww (i.e. blurring) must be present in the flow. Solutions for pure Lvv flow - mean curvature motion - are given by L(x, y; t) = L x2 + y 2 + 2t , which is not dimensionless.
2.2
Nonlinear Diffusion Filtering
The general diffusion equation [12] reads Lt = ∇ · (D · ∇L).
(12)
Qualitative and Quantitative Behaviour of Geometrical PDEs
233
The diffusion tensor D is a positive definite symmetric matrix. With D = 1 (when D is considered a scalar - i.e. an isotropic flow), or better, D = In , we have Gaussian scale space. When D depends on the image, the flow is nonlinear, e.g. in the case of the Perona Malik equation [7,2] with D = k 2 /(k 2 + ∇L2 ). For D = Lp−2 we have the p-Laplacian [13,14]. To force the equality Eq. (5) = w Eq. (12), (13) ∇ · (D · ∇L) = pLvv + qLww , D must be a matrix that is dimensionless and that contains only first order derivatives. The most obvious choice for D is D = ∇L · ∇T L/L2w . This yields, perhaps surprisingly, the Gaussian scale space solution. This is, in fact, the only possibility as one can verify. 2.3
Constraints
An extra condition may occur in the presence of noise (assume zero mean, variance σ 2 ): 1 (L − L0 )2 d Ω = σ 2 (14) I= 2 Ω where L0 is the input image and L the denoised one. The solution of min E s.t. I is obtained by the Euler Lagrange equation δE + λδI = 0 with δI = L − L0 , 2 λ = <δE,δI> <δI,δI> , and < δI, δI > = 2σ (see Eq. (14)). The solution can be reached by an evolution determined by a steepest decent evolution Lt = −(δE + λδI) When we set λ = 0, an unconstrained blurring process is obtained. Alternatively, λ can be regarded as a penalty parameter that limits the L2 difference between the input and output images. A too small value will cause an evolution that forces the image to stay close to the input image.
3
Numerical Implementation
The PDE is implemented using Gaussian derivatives [6,15]. As a consequence, larger time steps can be taken. When the spatial derivatives are computed as a convolution ( ) of the original image L with derivatives of a Gaussian G, the −y following results hold: (G L)x = ( −x 2t G) L, (G L)y = ( 2t G) L, (G L)xx = xy y −2t ( x 4t−2t 2 G) L, (G L)xy = ( 4t2 )G L, and (G L)yy = ( 4t2 G) L. Consequently, 2
2
y 2 + x2 − 4t G) L 4t2 −1 =( G) L 2t y 2 + y 2 − 2t =( G) L 4t2
(16)
q(x2 + y 2 ) − 2t(p + q) G) L 4t2
(18)
(G L)xx + (G L)yy = ( (G L)vv (G L)ww
(15)
(17)
Then we have pLvv + qLww = (
234
A. Kuijper
and Eq. (5) is numerically computed by n Ln+1 j,k − Lj,k
Δt
=(
q(x2 + y 2 ) − 2t(p + q) G) Lnj,k 4t2
(19)
where Lnj,k = ξ n ei(jx+ky) is the Von Neumann solution. The double integral, the right hand side of Eq. (19) reads q (α − x)2 + (β − y)2 − 2t(p + q) n i(jα+kβ)− (α−x)2 +(β−y)2 4t ξ e dαdβ 16πt3 2 2 1 p + q 2t(j 2 + k 2 ) − 1 ξ n e−t(j +k )+i(xj+ky) , which and evaluates to − 2t equals 2 2 1 − (20) p + q 2t(j 2 + k 2 ) − 1 e−t(j +k ) · Lnj,k = Ψ · Lnj,k . 2t Consequently, after dividing by Lnj,k (= 0!), Eq. (19) reduces to ξ − 1 = Δt · Ψ
(21)
For stability we require ξ ≤ 1, so Δt · Ψ + 1 ≤ 1. The minimum for Ψ is −p+3q obtained by ∂j Ψ = 0, ∂k Ψ = 0, i.e. t = 2q(j 2 +q 2 ) , yielding the value Ψmin = −q p−3q 2q . t e
p−3q
− 2q For the maximum step size we find ξmax = 2t . q e Obviously, as the implementation is based on the solution of the heat equation, so the maximum step size is limited by the case p = q = 1, i.e. ξmax ≤ 2te. So for 3 the Lww flow (p = 0, q = 1) the step size 2te 2 would yield instabilities. Secondly, −t(j 2 +k2 )
for Lvv flow (p = 1, q = 0), Ψ reduces to − e 2t p . The minimum is obtained at (j, k) = (0, 0), which obviously makes no sense as the Von Neumann solution −t then simplifies to ξ n . We therefore can assume j 2 + k 2 ≥ 1. Then Ψmin = −1 2t e t and the maximum step size is min{4te , 2te}. 3.1
An Alternative Approach
∇L Niessen et al. [15, p196] used ∇L = (cos θ, sin θ) to derive a maximal time step of 2et for the Lvv flow. Here we follow their line of reasoning for the more general Eq. (5). Firstly, the derivatives become
Gvv = cos2 (θ) ∗ Gyy + sin2 (θ) ∗ Gxx − 2 sin(θ) cos(θ) ∗ Gxy
(22)
Gww = cos (θ) ∗ Gxx + sin (θ) ∗ Gyy + 2 sin(θ) cos(θ) ∗ Gxy .
(23)
2
2
Strictly, the Von Neumann stability analysis is only suitable for linear differential equations with constant coefficients. However, we can apply it to equations with variable coefficients by introducing new constant coefficients equal to the frozen values of the original ones at some specific point of interest and test the n . We then find: modified problem instead. Let θ denote θj,k y 2 − 2t x2 − 2t xy + sin2 (θ ) − 2 sin(θ ) cos(θ ) 2 2 2 4t 4t 4t 2 2 2 y − 2t 2 x − 2t xy = cos (θ ) + sin (θ ) + 2 sin(θ ) cos(θ ) 2 . 4t2 4t2 4t
Gvv = cos2 (θ ) Gww
(24) (25)
Qualitative and Quantitative Behaviour of Geometrical PDEs
235
Numerically, with Lnj,k as above, we derive (Lnj,k G)vv = (j sin(θ ) − k cos(θ )) e−(j 2
2
+k2 )t
2 −(j 2 +k2 )t
(Lnj,k G)ww = (j cos(θ ) + k sin(θ )) e
Lnj,k
(26)
Lnj,k .
(27)
Since p (j sin(θ ) − k cos(θ )) + q (j cos(θ ) + k sin(θ )) ≤ max(p, q)(j 2 + k 2 ) 2
2
(28)
we derive for the stability criterion ξ = 1 − Δt max(p, q)(j 2 + k 2 )e−(j
2
+k2 )t
(29)
2et where again the optimum is obtained for s = j 2 + k 2 , yielding Δt ≤ max(p,q) . n This derivation holds for all points Ljk and we find the same stability criterion for the Lww and Lvv flow and for Gaussian blurring.
4
Results
Figure 1 shows two standard shapes used to evaluate the given numerical recipes. Firstly, results for applying 10 time steps in a finite difference scheme is shown in Figure 2. Clearly artifacts can be seen at the corners, due to the directional preference of the first order derivatives. Clearly, the corner behaving “good” in Lvv behaves “bad” in Lww flow, vice versa. Secondly, the Gaussian derivatives implementation for the disk and square are shown in Figures 3 - 4. The scale is chosen as σ = .8, so Δt = 2et = 2e 21 σ 2 = 1.74. The predicted critical scale for the Lvv flow is 4tet = 1.64e.32 = 1.76. One clearly
Fig. 1. Disk and square with values √ 0,1, with uniform random noise on (0,1), and the results of a Gaussian filter at σ = 128, i.e. t=64
Fig. 2. Results of 10 time steps in a finite difference scheme for, from left to right, the Lvv flow for the noisy disk and square, and the Lww flow for these images
236
A. Kuijper timestep 1.82857
timestep 1.77778
timestep 1.72973
timestep 1.68421
timestep 1.64103
timestep 1.82857
timestep 1.77778
timestep 1.72973
timestep 1.68421
timestep 1.64103
timestep 1.82857
timestep 1.77778
timestep 1.72973
timestep 1.68421
timestep 1.64103
Fig. 3. Results of the noisy disk for Lvv flow (top row), Gaussian flow (middle row), and Lww flow (bottom row), for various time step ranges around the critical value Δt = 2et = 2e 12 σ 2 = 1.74 timestep 1.82857
timestep 1.77778
timestep 1.72973
timestep 1.68421
timestep 1.64103
timestep 1.82857
timestep 1.77778
timestep 1.72973
timestep 1.68421
timestep 1.64103
timestep 1.82857
timestep 1.77778
timestep 1.72973
timestep 1.68421
timestep 1.64103
Fig. 4. Results of the noisy square for Lvv flow (top row), Gaussian flow (middle row), and Lww flow (bottom row), for various time step ranges around the critical value Δt = 2et = 2e 12 σ 2 = 1.74
sees the change around the critical values. Since relatively much noise is added, the value is a bit lower that the predicted value. If a too large time step is taken, instability artifacts are visible: For Lvv flow the results become peaky, while the Lww flow shows ringing, and the Gaussian blurring is completely disastrous. Note that the rounding effect for the Lvv flow and the peaky results for the Lww flow are intrinsic to these flows.
Qualitative and Quantitative Behaviour of Geometrical PDEs
237
Fig. 5. Original image and a noisy one, σ = 20 p,q 0.5,1.
p,q 0.,1.
p,q 0.5,1.
p,q 1.,1.
p,q 0.5,0.5
p,q 0.,0.5
p,q 0.5,0.5
p,q 1.,0.5
p,q 0.5,0.
p,q 0.,0.
p,q 0.5,0.
p,q 1.,0.
p,q 0.5,0.5
p,q 0.,0.5
p,q 0.5,0.5
p,q 1.,0.5
Fig. 6. Geometrical evolution of Lt = pLvv + qLww for several values of p and q. The noise variance σ is set to 20. The result satisfies the noise constraint up to an error of 10−7 .
To see the effect of Lt = pLvv + qLww for several values of p and q, Figure 5 is used. The result of applying the Gaussian derivatives implementation in 50 time steps is shown in Figure 6 (with noise constraint) and Figure 7 (without one). As one can see in Figure 6, the choice of p and q enables one to steer between
238
A. Kuijper p,q 0.5,1.
p,q 0.,1.
p,q 0.5,1.
p,q 1.,1.
p,q 0.5,0.5
p,q 0.,0.5
p,q 0.5,0.5
p,q 1.,0.5
p,q 0.5,0.
p,q 0.,0.
p,q 0.5,0.
p,q 1.,0.
p,q 0.5,0.5
p,q 0.,0.5
p,q 0.5,0.5
p,q 1.,0.5
Fig. 7. Geometrical evolution of Lt = pLvv + qLww for several values of p and q. There is no constraint. For negative p there are spiky artifacts, for positive ones there is blurring. For negative q one sees the edges.
denoising regions and deblurring around edges (where the artifacts occurred). The evolution converges within 50 time steps, the error in the constraint is of order 10−7 . The unconstrained evolution shows the spiky artifacts for p ≤ 0, while q < 0 gives the edges. Note that for these values Ψ may become negative and local stability problems may occur. The diagonal gives Gaussian (de)blurring. Visually, q = 0 gives the best result, although here the number of time steps heavily influences the results.
5
Summary and Discussion
We presented a line of approaches to evolve images that unify existing methods in a general framework, by a weighted combination of second order derivatives in terms of gauge coordinates. The series incorporate the well-known Gaussian blurring, Mean Curvature Motion and Maximal Blurring. For the qualitative
Qualitative and Quantitative Behaviour of Geometrical PDEs
239
behaviour, a solution of the series was derived and its properties were briefly mentioned. Relations with general diffusion equations were given. Quantitative results were obtained by a novel implementation and its stability was analysed. The practical results are visualised on artificial images to study the method in detail, and on a real-life image showing the expected qualitative behaviour. The examples showed that positive values for p and q are indeed necessary to guarantee numerical stability (Fig. 7). Theoretically, this relates to the fact that q < 0 implies deblurring, notoriously ill-posed and unstable. However, when a reasonable constraint is added, this deblurring is possible (Fig. 6). Choosing optimal values of p and q depends on the underlying image and is beyond the scope of this paper.
References 1. Koenderink, J.J.: The structure of images. Biological Cybernetics 50, 363–370 (1984) 2. Perona, P., Malik, J.: Scale space and edge detection using anisotropic diffusion. IEEE Transactions on Pattern Analysis and Machine Intelligence 12, 629–639 (1990) 3. Alvarez, L., Lions, P., Morel, J.: Image selective smoothing and edge detection by nonlinear diffusion. SIAM Journal on Numerical Analysis 29, 845–866 (1992) 4. Caselles, V., Morel, J.M., Sbert, C.: An axiomatic approach to image interpolation. IEEE Transactions on Image Processing 7, 376–386 (1996) 5. Bertalmio, M., Vese, L., Sapiro, G., Osher, S.: Simultaneous structure and texture image inpainting. IEEE Transactions on Image Processing 12, 882–889 (2003) 6. Haar Romeny, B.M.t.: Front-end vision and multi-scale image analysis. Kluwer Academic Publishers, Dordrecht, The Netherlands (2003) 7. Aubert, G., Kornprobst, P.: Mathematical Problems in Image Processing: Partial Differential Equations and the Calculus of Variations, 2nd edn. Springer, Heidelberg (2006) 8. Kornprobst, P., Deriche, R., Aubert, G.: Image coupling, restoration and enhancement via PDE’s. In: Proc. Int. Conf. on Image Processing, vol. 4, pp. 458–461 (1997) 9. Griffin, L.: Mean, median and mode filtering of images. Proceedings of the Royal Society Series A 456, 2995–3004 (2000) 10. Yezzi, A.: Modified curvature motion for image smoothing and enhancement. IEEE Transactions on Image Processing 7, 345–352 (1998) 11. Abramowitz, M., Stegun, I.A.: Handbook of Mathematical Functions with Formulas, Graphs, and Mathematical Tables, 9th edn. Dover, New York (1972) 12. Weickert, J.A.: Anisotropic Diffusion in Image Processing. Teubner, Stuttgart (1998) 13. Aronsson, G.: On the partial differential equation u2x uxx + 2ux uy uxy + u2y uyy = 0. Arkiv f¨ ur Matematik 7, 395–425 (1968) 14. Kuijper, A.: p-laplacian driven image processing. In: ICIP 2007 (2007) 15. Niessen, W.J., ter Haar Romeny, B.M., Florack, L.M.J., Viergever, M.A.: A general framework for geometry-driven evolution equations. International Journal of Computer Vision 21, 187–205 (1997)
Automated Billboard Insertion in Video Hitesh Shah and Subhasis Chaudhuri Indian Institute of Technology Bombay, Powai, Mumbai 400 076, India
Abstract. The paper proposes an approach to superimpose virtual contents for advertising in an existing image sequence with no or minimal user interaction. Our approach automatically recognizes planar surfaces in the scene over which a billboard can be inserted for seamless display to the viewers. The planar surfaces are segmented in the image frame using a homography dependent scheme. In each of the segmented planar regions, a rectangle with the largest area is located to superimpose a billboard into the original image sequence. It can also provide a viewing index based on the occupancy of the virtual real estate for charging the advertiser.
1
Introduction
Recent developments in computer vision algorithms have paved the way for a novel set of applications in the field of augmented reality [1]. Among these, virtual advertising has gained considerable attention on account of its commercial implications. The objective of virtual advertising is to superimpose computer mediated advertising images or videos seamlessly into the original image sequence so as to give the appearance that the advertisement was part of the scene when the images were taken. It introduces possibilities to capitalize on the virtual space. Conventionally, augmentation of video or compositing has been done by skilled animators by painting 2D images onto each frame. This technique ensures that the final composite is visually credible, but is enormously expensive, and is also limited to relatively simple effects. Current state-of-art methods for introducing virtual advertising broadly fall into three categories. The first category consists of approaches which utilize pattern recognition techniques to track the regions over which the advertisement is to be placed. Patent [2] is an example of such an approach. It depends on human assistance to initially locate the region for placement of billboard which is tracked in subsequent frames using a Burt pyramid. The approaches in this category face problems when the region leaves the field of view and later reappears, requiring complete and accurate re-initialization. Medioni et al. [3] present an interesting approach which addresses this issue, but the approach is limited to substitution of billboards. The second category comprises of the methods which require access to the scene and/or to the equipment prior to filming like in [4,5,6,7]. In these approaches special markers are set up in the scene to identify the places for future billboard insertions. They may also require physical sensors on the camera to track the changes in the view (pan, Y. Yagi et al. (Eds.): ACCV 2007, Part I, LNCS 4843, pp. 240–250, 2007. c Springer-Verlag Berlin Heidelberg 2007
Automated Billboard Insertion in Video
241
tilt, zoom). However, this renders such approaches incapable of augmenting an existing video. The final category assumes the knowledge about the structure and geometry of the scene, e.g. the system proposed in patent [8] depends on a landmark model describing the set of natural landmarks in a given scene. Similarly, the techniques proposed in [9,10] assume the image sequence to be of a soccer and tennis game, respectively to make the best use of scene property. This makes such a solution very case specific. As opposed to the above methods, the proposed approach automatically locates a possible region for superimposing a billboard and does not require access or strong assumptions on the structure of the scene or equipment. Our approach exploits the inherent constraints introduced due to requirement of mapping of a planar surface (billboard) onto a planar surface (for e.g. wall) in this particular context. Further, we provide a viewing index for the price fixation for the advertisement.
2
Problem Formulation
An arbitrary sequence of n images I1 , ..., In of a scene and the billboard(s) to be placed are given. We need to automatically locate dominant planar regions in the scene and superimpose geometrically corrected billboards over each of the planar regions in each image frame (Figure 1). The scene is assumed to have textured and planar dominant regions which are not occluded in majority of the frames in the sequence. As indoor scene with walls, outdoor scenes of buildings or pre-existing physical billboards are the target for the approach, the requirement of dominant planar region is not at all restrictive.
(a)
(b)
(c)
Fig. 1. Illustration of virtual billboard insertion: (a) is a frame from input image sequences. The frame after it has been augmented by placing a virtual advertisement at a planar region in the scene are shown in (b) and (c).
3
Proposed Approach
Our approach consists of three stages: image analysis, reconstruction of a quadrilateral in 3D space and image synthesis. Image analysis stage is responsible for
242
H. Shah and S. Chaudhuri
finding and segmenting the planar surfaces in the scene. It consists of weak calibration, plane fitting, planar segmentation and then locating the largest rectangular area on each of the segmented regions. Back projecting each rectangle on the corresponding planar surface, a projective reconstruction of a quadrilateral in 3D space is obtained. The image synthesis stage maps texture on the quadrilateral with that of the required billboard and performs augmentation by projecting them on each of the given image frames. It also calculates the viewing index for each billboard inserted in the image sequence as a measure of price to be paid by the sponsor. 3.1
Weak Calibration
In weak calibration the structure and motion will be determined up to an arbitrary projective transformation. For this, the interest points in the image sequence, obtained by Harris corner detector are tracked over all the frames using normalized correlation based matching. The tracked interest points are then utilized to solve for the projection matrices P1 , ..., Pn corresponding to each image frame and for recovering the 3D positions of the interest points X1 , ..., Xm with projective ambiguity as explained in Beardsley et al. [11] or by Triggs [12]. In our approach the projection matrices and the recovered positions of the interest points are used to evaluate the homography between image frames. As co-planarity of points is preserved under any projective transformation, a projective reconstruction suffices; updating the reconstruction to affine or Euclidean is not needed to deal with the planar regions. 3.2
Plane Fitting
For recovering the planar surface in the scene, interest points X1 , ..., Xm are divided on the basis of the plane they support. Thus from a point cloud in 3D space, points which are coplanar are to be identified and grouped. Hough transform and RANdom SAmple Consensus (RANSAC) [13] are powerful tools to detect specified geometrical structures among a cluster of data points. However, any one of them when used individually has the following limitations. Accuracy of the parameters recovered using Hough transform is dependent on the bin size. To obtain higher accuracy the bin size has to be smaller implying a large number of bins and thus is computationally more expensive. RANSAC, on the other hand requires many iterations when the fraction of outliers is high and trying all possible combinations can be also computationally expensive. It is able to calculate the parameters for the plane with higher accuracy in reasonable time when it is to fit one instance of the model albeit with a few outliers in the data points, as in our case there might be multiple instances of the model, i.e. plane, in the data it performs poorly on its own. To overcome the above limitations, a Hough transform followed by RANSAC on the partitioned data is adopted for recognizing planes. In the first stage Hough transform with a coarse bin size is utilized to obtain the parameters of the planes. These parameters are then utilized to partition the input points
Automated Billboard Insertion in Video
243
X1 , ..., Xm into subsets of points belonging to individual planar regions. Each one of these subset of points support a plane whose parameters are calculated using the Hough transform. Note that there will be a number of points which cannot be fit to a planar surface and they should be discarded. Each subset of data forms the input to the RANSAC algorithm which then fits a plane to recover the accurate parameters for the plane. Such an approach can efficiently calculate the equations of planes fitting the data points. Thus at the end of plane fitting operation, equations of the dominant planes in the scene are obtained. In the following subsections we explain the details of Hough transform and RANSAC method used in this study. Data Partitioning. A plane P in XY Z space can be expressed with the following equation: ρ = xsinθcosφ + ysinθsinφ + zcosθ
(1)
where (ρ, θ, φ) helps define a vector from the origin to the nearest point on the plane. This vector is perpendicular to the plane. Thus under the Hough transform each plane in XY Z space is represented by a point in (ρ, θ, φ) parameter space. All the planes passing through a particular point B(xb , yb , zb ) in XY Z space can be expressed with the following equation from eq. (1) ρ = xb sinθcosφ + yb sinθsinφ + zb cosθ.
(2)
Accordingly all the planes that pass through the point B(xb , yb , zb ) can be expressed with a curved surface described by the eq. (2) in (ρ, θ, φ) space. A three dimensional histogram in (ρ, θ, φ) space is set up to find the planes to which a group of 3D data points belong. For each 3D data point B(xb , yb , zb ) ∈ Xi , all histogram bins that the curved surface passes through are incremented. To obtain the parameters of a particular plane a search for the local maxima in the (ρ, θ, φ) space is performed. The top k significant local maxima are obtained in the (ρ, θ, φ) space the input point cloud is divided into k + 1 subsets, each containing the points that satisfy the plane eq. (1) with a certain tolerance. The last subset contains points that do not fit into any of the above k planes. Accurate plane fitting is carried out on each set using RANSAC as explained in the next section. Plane Fitting Using RANSAC. The basic idea of RANSAC method is to compute the free parameters of the model from an adequate number of randomly selected samples. Then all samples vote whether they agree with the proposed hypothesis. This process is repeated until a sufficiently broad consensus is achieved. The major advantage of this approach is its ability to ignore outliers without explicit handling. We proceed as follows to detect a plane in the subsets of points obtained using the previous step: – Choose a candidate plane by randomly drawing three samples from the set. – The consensus on this candidate is measured.
244
H. Shah and S. Chaudhuri
– Iterate the above steps. – The candidate having the highest consensus is selected. At the end of above iterations, equations for k planes π1 , ..., πk , corresponding to each subset, are obtained. 3.3
Segmentation of Planar Regions
Having estimated the dominant planar structures in the scene, we now need to segment these regions irrespective of its texture. For a given plane πi , the image frame in which the minimum foreshortening of the plane occurs is selected as the reference image Iref . This ensures maximum visibility of the region on the plane in the image Iref . When any other image frame Iother from the sequence is mapped onto the reference image using a homography for the plane πi , the region of the image frame Iother on the plane πi is pixel aligned with the region of the image on the plane πi in Iref and the rest of it gets misaligned due to non belongingness to the selected plane πi . Figure 2(b) shows the resulting image obtained by applying homography (calculated for the plane coincident with the top of the box) to images in the sequences and then taking the average color at each pixel location over all back projected frames. The pixels in the region on the top of the box in the image frames projected using homography are aligned well with the region of the box top in Iref . Thus in the averaged image the top of the box appears sharp in contrast to the surrounding region. Hence to segment the region on the plane πi in image Iref , at each pixel location a set of color is obtained by mapping equally time spaced image frame (for e.g., every 10th frame) in the sequence using their respective homography for the plane πi . Homography calculation is explained in Appendix A. For each pixel in image Iref lying on the plane, variance of image texture for all re-projected points in the scene at this pixel will be very small as compared to pixels which are not on this planar region due to misalignment. Hence pixel wise variance over all re-projected image frames can be used as a measure to segment the regions in the image on the plane. For each pixel in the image Iref the variance of the above set is calculated and compared against a threshold to obtain a binary segmentation
(a)
(b)
(c)
(d)
Fig. 2. Illustration of homography based segmentation: (a) reference image to be segmented. (b) is obtained by projecting and averaging all the frames in the sequence on (a). Notice the extensive blurring of the region not coplanar to the top of the box. (c) Variance measured at each pixel (white represents larger variance). (d) Segmented planar region obtained after performing thresholding on the basis of variance.
Automated Billboard Insertion in Video
(a)
(b)
245
(c)
Fig. 3. (a), (b) and (c) are the augmented image frames where separate billboards have been placed on two dominant planes which were automatically identified by the approach
of the reference image for the particular planar region. Figure 2(c) represents the per pixel variance of the image frame in figure 2(a). It can be readily seen that the variance of pixels on the top of the box are less as compared to the surrounding region which appears in white region. Finally, figure 2(d) is the segmented planar region obtained by thresholding the variance image. There may be occasional holes in the segmented region as seen in figure 2(d). Such small regions are filled up using a morphological closing operation. Regions corresponding to each of the plane π1 , ..., πk are obtained similarly. 3.4
Billboard Placement and Augmentation
Having obtained the segmented regions corresponding to each of the dominant planes, the largest inscribed rectangular area within each of them is located using a dynamic programming Billboards are usually rectangular in shape and are horizontally (or vertically) oriented. Hence we try to fit the largest virtual real estate possible in the segmented region. In absence of any extrinsic camera calibration, it is assumed that the reference frame is vertically aligned. The end points of these rectangles are back projected using the projection matrix, as explained in Appendix B, of the corresponding reference image on the corresponding plane to obtain a quadrilateral in 3D space. Each quadrilateral represents a possible planar region for insertion of a billboard in the 3D projective space. The quadrilaterals can be texture mapped [14] with the required advertising material and then projected onto each of the image frames in the sequence using the respective projection matrices. To reduce aliasing artifact and to increase rendering speed mipmapping [15] is used for texture mapping. 3.5
Calculation of the Viewing Index
Total viewing index can also be calculated during augmentation for each billboard inserted in the video. The total viewing index for a billboard is directly proportional to the amount of time the billboard is on the screen and is equal to the sum of the viewing index calculated per frame. The per frame viewing index
246
H. Shah and S. Chaudhuri Viewing index per frame 0.4 Billboard on the front side of box Billboard on the top side of box 0.35
Viewing Index
0.3
0.25
0.2
(b)
0.15
0.1
0.05
0
0
100
200
300 400 Frame Number
500
600
700
(a)
(c)
Fig. 4. (a) Calculated viewing index for billboard on the top and front of the box for each frame. (b) & (c) are the frames with highest viewing indices (encircled in (a)) for billboard on the front and the top, respectively.
depends on the amount of area the billboard is occupying in the image frame as well as the part of the image frame where it appears, i.e. top, middle, bottom, corners as the location matters for advertising purposes. The total viewing index for a particular billboard reflects roughly the amount of impact the billboard has on the viewer. Thus, it can be utilized to develop a fair pricing policy for the sponsor of the advertisement billboard. To calculate viewing index per frame each pixel Pi,j in the image frame is assigned a weight 2
Weight(Pi,j ) =
− 12 ( (μx −i) 1 2 σx e 2πσx σy
+
(μy −j)2 2 σy
)
(3)
where μx = (height of frame)/2, μy = (width of frame)/2, σx = (height of frame)/6, σy = (width of frame)/6. This selection of parameter assigns higher weights to the pixel in the center of the frame and the weight slowly decreases as the pixels move away from the center. Also the selection of σx and σy ascertains that sum of weights of all the pixels in a frame is almost equal to unity. The viewing index for the billboard in a frame is then equal to the sum of the weights of each pixel over which it is projected. This ensures that the viewing index is directly proportional to the area of frame occupied by the billboard and also to the billboard’s position in the frame. Figure 4 shows computed the viewing indices for the billboard on the top and front sides of the box calculated in the above manner. Using the viewing index per frame a total viewing index for a particular billboard can be calculated by summing the viewing index in each frame. It can be observed that the billboard on the top of the box has higher total viewing index as compared to the one
Automated Billboard Insertion in Video
247
on the front side. Hence the sponsor of the billboard on the top can be charged relatively higher to account for the larger occupancy of prime virtual real estate.
4
Experimental Results
The proposed approach has been implemented in MATLAB and the image sequences used for tests have been captured using a hand held camera. Each of the image sequences had 300-700 frames. In our experimentation we used factorization method proposed in [12] for weak calibration. Figure 1 shows the qualitative result of our approach on two image sequences. In the first sequence one dominant plane was detected corresponding to the wall whereas in the second sequence two dominant planes, top and front of the box, were located. Figure 1 (b,c,e,f) are resulting frames with billboard added on one plane and figure 3 shows the altered image frames with separate billboard over the two dominant planes generated by the proposed approach. Videos captured using a mobile phone camera are difficult to augment using existing approaches due to inherent jitters, low frame rate and less spatial resolution. However, the proposed approach is able to insert billboards seamlessly into such videos also. Few results obtained by augmenting videos captured using a mobile phone camera are shown in figure 5.
(a)
(b)
(c)
(d)
(e)
(f)
Fig. 5. Augmented frames from three distinct videos captured using a mobile phone camera are shown in (a,b,c), (d,e,f) respectively. In each of the video one dominant planar region is identified and augmented with a billboard.
5
Conclusion
In this paper we have presented an automated approach for locating planar regions in a scene. The planar regions recovered are used to augment the image sequence with a billboard which may be a still image or a video. One of the
248
H. Shah and S. Chaudhuri
possibilities for the application of the approach is to use it in conjunction with the set top box. Before transmission the video is analyzed for the planar regions in the scene. Information about the identified planar regions is stored as meta data in each frame. At the receiving end before display the set top box can augment each of the image frames with billboards in real time. The billboards in this case may be adaptively selected by the set top box depending on the viewer habits learned by it or the video being shown, e.g. a health video may be augmented with a health equipment advertisement. While, evaluating the results of the current experimentation it is observed that the placement of the billboard is geometrically correct in each of the image frame. No significant drift or jitter has been observed. However, there may be photometric mismatches for the inserted billboard with its surrounding. We are currently looking into the photometric issues related to illumination, shadow and focus correction of the augmented billboard.
References 1. Azuma, R., Baillot, Y., Behringer, R., Feiner, S., Julier, S., MacIntyre, B.: Recent advances in augmented reality. IEEE Comput. Graph. Appl. 21(6), 34–47 (2001) 2. Rosser, R.J., Leach, M.: Television displays having selected inserted indicia. In: US Patent 5,264,933 (2001) 3. Medioni, G., Guy, G., Rom, H., Francois, A.: Real-time billboard substitution in a video stream. In: Proceedings of the 10th Tyrrhenian International Workshop on Digital Communications (1998) 4. Rosser, R., Tan, Y., Kennedy Jr., H., Jeffers, J., DiCicco, D., Gong, X.: Image insertion in video streams using a combination of physical sensors and pattern recognition. In: US Patent 6,100,925 (2000) 5. Wilf, I., Sharir, A., Tamir, M.: Method and apparatus for automatic electronic replacement of billboards in a video image. In: US Patent 6,208,386 (2001) 6. Gloudemans, J.R., Cavallaro, R.H., Honey, S.K., White, M.S.: Blending a graphic. In: US Patent 6,229,550 (2001) 7. Bruno, P., Medioni, G.G., Grimaud, J.J.: Midlink virtual insertion system. In: US Patent 6,525,780 (2003) 8. DiCicco, D.S., Fant, K.: System and method for inserting static and dynamic images into a live video broadcast. In: US Patent 5,892,554 (1999) 9. Xu, C., Wan, K., Bui, S.H., Tian, Q.: Implanting virtual advertisement into broadcast soccer video. In: Advances in Multimedia Information Processing - PCM, vol. 2, pp. 264–271 (2004) 10. Tien, S.C., Chia, T.L.: A fast method for virtual advertising based on geometric invariant-a tennis match case. In: Proc. of Conference on Computer Vision, Graphics, and Image Processing (2001) 11. Beardsley, P.A., Zisserman, A., Murray, D.W.: Sequential updating of projective and affine structure from motion. Int. J. Comput. Vision 23(3), 235–259 (1997) 12. Triggs, B.: Factorization methods for projective structure and motion. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 845–851. IEEE Computer Society Press, San Francisco, California, USA (1996) 13. Fischler, M.A., Bolles, R.C.: Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM 24(6), 381–395 (1981)
Automated Billboard Insertion in Video
249
14. Foley, J.D., van Dam, A., Feiner, S.K., Hughes, J.F.: Computer graphics: principles and practice. Addison-Wesley Longman Publishing Co. Inc., USA (1996) 15. Williams, L.: Pyramidal parametrics. In: SIGGRAPH 1983, pp. 1–11. ACM Press, New York (1983) 16. Hartley, R., Zisserman, A.: Multiple view geometry in computer vision. Cambridge University Press, New York, USA (2000)
A
Homography Calculation
Consider images Im and In with projection matrices given by Pm and Pn , respectively. Let A be any point on the plane π which projects to am and an on the images Im and In , respectively, thus πT A = 0 am = Pm A
(4) (5)
an = Pn A
(6)
where π is a 4x1 row vector and A is a 4x1 homogeneous representation of the point. Due to eq. (4), A lies in the null space ( NS ) of π T . A = NS(π T )C
(7)
where C are the coordinates of A with respect to the basis of the nullspace of A. From eq. (7) and eq. (6) C = [Pn NS(π T )]
−1
an .
(8)
Using eq. (5), eq. (7) and eq. (8) −1
am = Pm NS(π T )[Pn NS(π T )]
an
am = Hmn an where Hmn = Pm NS(π T )[Pn NS(π T )] ping an to am .
B
(9) (10)
−1
and is a 3x3 homography matrix map-
Back-Projection of a Point on a Plane
Let x be the homogenous representation of the point in the image, which is to back-projected on the plane. Let P be a 3x4 projection matrix of the image. It can be written as
250
H. Shah and S. Chaudhuri
P = [M |p4 ]
(11)
where M is a 3x3 matrix consisting of the first three columns and p4 is the last column of P . As per [16], the camera center C for the image is given by C = −M −1 p4
1
T
(12)
and if D be the point at intifinity in the direction of the ray from C passing through x. Then D = M −1 x
0
T
(13)
All the points on the line from C and D can be expressed parametrically by X(t) = C + tD
(14)
Let π = [ a b c d ]T be the equation of the plane on which the point x is to back-projected. Thus for a point, with homogenous representation Y , on the plane πT Y = 0 T
(15) T
It can also be written as π = [z d ] , where z = [a b c ] . The back projected point is on the line and the plane. Thus using eq. (14) and eq. (15) π T X(t) = π T C + tπ T D −1 −M −1 p4 M x 0 = [ z d ]T + t [ z d ]T 1 0 −1 −1 t zM x = zM p4 − d t=
zM −1 p4 − d zM −1 x
Thus the back-projection of a point x on a plane is given by zM −1 p4 − d M −1 x −M −1 p4 ∗ X = + . 1 0 zM −1 x
Improved Background Mixture Models for Video Surveillance Applications Chris Poppe, Ga¨etan Martens, Peter Lambert, and Rik Van de Walle Ghent University - IBBT Department of Electronics and Information Systems - Multimedia Lab Gaston Crommenlaan 8, B-9050 Ledeberg-Ghent, Belgium
Abstract. Background subtraction is a method commonly used to segment objects of interest in image sequences. By comparing new frames to a background model, regions of interest can be found. To cope with highly dynamic and complex environments, a mixture of several models has been proposed. This paper proposes an update of the popular Mixture of Gaussian Models technique. Experimental analysis shows a lack of this technique to cope with quick illumination changes. A different matching mechanism is proposed to improve the general robustness and a comparison with related work is given. Finally, experimental results are presented to show the gain of the updated technique, according to the standard scheme and the related techniques.
1
Introduction
The detection and segmentation of objects of interest in image sequences is the first processing step in many computer vision applications, such as visual surveillance, traffic monitoring, and semantic annotation. Since this is often the input for other modules in computer vision applications, it is desirable to achieve very high accuracy with the lowest possible false alarm rates. The detection of moving objects in dynamic scenes has been the subject of research for several years and different approaches exist [1]. One popular technique is background subtraction. During the surveillance of a scene, a background model is created and dynamically updated. Foreground objects are represented by the pixels that differ significantly from this background model. Many different models have been proposed for background subtraction, of which the Mixture of Gaussian Models (MGM) is one of the most popular [2]. However, there are a number of important problems when using background subtraction algorithms (quick illumination changes, initialization with moving objects, ghosts and shadows), as was reported in [3]. Sect. 2 elaborates on a number of techniques which improve the traditional MGM and try to deal with the above mentioned problems. This paper presents a new approach to deal with the problem of quick illumination changes (like clouds gradually changing the lighting conditions of the environment). We propose an updated matching mechanism for MGM. As such, Sect. 3 elaborates on the conventional mixture technique and its observed shortcomings. Subsequently, Sect. 4 shows the adjustments of the original scheme Y. Yagi et al. (Eds.): ACCV 2007, Part I, LNCS 4843, pp. 251–260, 2007. c Springer-Verlag Berlin Heidelberg 2007
252
C. Poppe et al.
and presents experimental results. Finally, some conclusions are formulated in Sect. 5.
2
Related Work
Toyama et al. discussed in detail several known problems when using background subtraction algorithms [4]. This list was adopted by Javed et al. and they selected a number of important problems which have not been addressed by most background subtraction algorithms [3]. Wu et al. gave a concise overview of background subtraction algorithms, of which they have chosen MGM to compare with their own technique [2]. They use a global gaussian mixture model, built upon a difference image between the current image and an estimated background. Although their system is better for localization and contour preserving, it is more sensitive to complex environmental movements (such as waving trees). Lee et al. improved MGM by introducing means to initialize the background models when moving objects are present in the environment [5]. They presented an online expectation maximization learning algorithm for training adaptive Gaussian mixtures. Their system allows to initialize the mixture models much faster then the original approach. Related to this topic, Zhang et al. presented an online background reconstruction method to cope with the initialization problem [6]. Additionally, they presented a change history map to control the foreground mergence time and make it independent of the learning rate. As such, they deal with the initialization problem and the problem of ghosts (background objects which are moved) but can not deal with quick illumination changes. In [3], Javed et al. presented a number of important problems when using background subtraction algorithms. Accordingly, they proposed a solution using pixel, region and frame level processing. They proposed a solution to the problem of the quick illumination changes, but their technique is based on a complex gradients-based algorithm. Conveniently, the paper does not provide any information about the additional processing times needed for this technique. Tian et al. [7] used a similar approach as the one used in [3]. They presented a texture similarity measure based on gradient vectors, obtained by the Sobel operator. A fixed window is used for the retrieval of the gradient vectors, which largely determines the performance (both in processing time and accuracy ) of their system. We present a conceptual simpler approach by extending the MGM algorithm. The results presented in Sect. 4.2 show that we obtain similar successes in coping with quick illumination changes. Numerous techniques have been proposed to deal with shadows. Since these are used to deal with the effects of the lighting of a scene, an evaluation is made of available shadow detection techniques to see how they can be used to manage the quick illumination changes. An interesting overview on the detection of moving shadows is given in [8]. Prati et al. divide the shadow detection techniques in four classes, of which the deterministic non-model based approach shows the
Improved Background Mixture Models for Video Surveillance Applications
253
best results for the entire evaluation set used in the overview. Since the two critical requirements of a video surveillance system are accuracy and speed, not every shadow removal technique is appropriate. MGM is an advanced object detection technique, however, the maintenance of several models for each pixel is computational expensive. Therefore, every additional processing task should be minimal. Furthermore, MGM was created to cope with highly dynamic environments, with the only assumption being the static camera. According to these constraints and following the results presented by Prati et al., we have chosen the technique described in [9] for a comparison with our system. Results hereof are presented in Sect. 4.2.
3
3.1
Background Subtraction Using Mixture of Gaussian Models The Mixture of Gaussian Models
MGM was first proposed by Stauffer and Grimson in [10]. It is a time-adaptive per pixel subtraction technique in which every pixel is represented by a vector, called Ip , consisting of three color component (red, green, and blue). For every pixel a mixture of Gaussian distributions, which are the actual models, is maintained and each of these models is assigned a weight. T −1 1 1 e− 2 (Ip −μp ) Σp (Ip −μp ) . G (Ip , μp , Σp ) = n (2π) |Σp |
(1)
(1) depicts the formula for a Gaussian distribution G. The parameters are μp and Σp , which are the mean and covariance matrix of the distribution respectively. For computational simplicity, the covariance matrix is assumed to be diagonal. For every new pixel a matching, an update, and a decision step are executed. The new pixel value is compared with the models of the mixture. A pixel is matched if its value occurs inside a confidence interval within 2.5 standard deviations from the mean of the model. In that case, the parameters of the corresponding distribution are updated according to (2), (3), and (4). μp,t = (1 − ρ) μp,t−1 + ρ (Ip,t ) .
(2)
Σp,t = (1 − ρ) Σp,t−1 + ρ (Ip,t − μp,t ) (Ip,t − μp,t )T .
(3)
ρ = αG (Ip,t , μp,t−1 , Σp,t−1 ) .
(4)
α is the learning rate, which is a global parameter, and introduces a trade-off between fast adaptation and detection of slow moving objects. Each model has a weight, w, which is updated for every new image according to (5). wt = (1 − α) wt−1 + αMt .
(5)
254
C. Poppe et al.
If the corresponding model introduced a match, M is 1 , otherwise it is 0. Formulas (2) to (5) represent the update step. Finally, in the decision step, the models are sorted according to their weights. MGM assumes that background pixels occur more frequently then actual foreground pixels. For that reason a threshold based on these weights is used to define which models of the mixture depict background or foreground. Indeed, if a pixel value occurs recurrently, the weight of the corresponding model increases and it is assumed to be background. If no match is found with the current pixel value, then the model with the lowest weight is discarded and replaced by a normal distribution with a small weight, a mean equal to the current pixel value, and a large covariance. In this case the pixel is assumed to be a foreground pixel. 3.2
Problem Description
Fig. 1 shows the results of applying MGM to the PetsD2Tec2 sequence (with a resolution of 384x288) provided by IBM Research at several time points [11]. Black pixels depict the background, white pixels are assumed to be foreground. The figure shows a fragment of the scene being subject of changing illumination circumstances causing a repetitive increase of certain pixel values in a relatively short time period (about 30s). The illumination change results in relatively small differences between the pixel values of consecutive frames. However, the consistent nature of these differences causes the new pixel values to eventually exceed the acceptance range of the mixture models. This is because the acceptance decision is based on the difference with the average of the model, regardless the difference with the previous pixel value. The learning rate of MGM is typically very small (α is usually less then 0.01), so gradual changes spread over long periods (e.g., day turning into night) can be learned into the models. However, the small learning rate makes the adaptation of the current background models not quick enough to encompass the short consistent gradual changes. Consequently, these changes, which are hard to distinguish by the human eye, result in numerous false detections. The falsely detected regions can range from small regions of misclassified pixels to regions encompassing almost half of the image.
Fig. 1. MGM output during quick illumination change
Improved Background Mixture Models for Video Surveillance Applications
4 4.1
255
Improved Background Mixture Models Advanced MGM
MGM uses only the current pixel value and the mixture model in the matching, update and decision steps. The pixel values of the previous image are not stored since they are only used to update the models. We propose to make the technique aware of the immediate past by storing the previous pixel value (prevI) and the previously matched model number (prevM odel). The matching step is then altered according to the following pseudocode: If (Model == prevModel) If (|I - prevI| < cgc * stdev) Match = true; Else checkMatch(Model,I); Else checkMatch(Model,I); update(Model,I); decide(Model); If ((Match == true) and (Decision == background)) { prevModel = Model; prevI = I; } In the matching step for each pixel, it is checked if the pixel value matches with one of the models in the mixture. For the model which matched the pixel values in the previous frame, the difference between the previous and current pixel value is taken. If this difference is small enough, a match is immediately effectuated. Otherwise the normal matching step is executed. If the matched model is considered to represent part of the background, then the model number and the current pixel value are stored, otherwise they remain unchanged. This way, passing foreground objects do not affect the recent history. If a new pixel value differs slightly from the previously matched one, but would fall out of the matching range of the model, a different outcome, compared with the original algorithm, will be obtained. Since the normal matching process is dependent on the specific model, more specifically on the standard deviation, it is better to enforce this for the threshold as well. Therefore, we have chosen for a per pixel threshold dependent on the standard deviation. We introduce a new parameter, cgc (from Consistent Gradual Change), to control the threshold. In Fig. 2 we have recorded the number of detection failures and false alarms for several values of cgc for the PetsD1TeC2 sequence (another sequence from the IBM benchmark which shows similar situations for the consistent gradual changes). A manual ground
256
C. Poppe et al. 300 2,6 290 2,4 280
2,2 2
Detection Failure
270
1,8 260
1,6
1,4
1,2 1
250
0,8
0,6
240 230 220 210 200 0
200
400
600
800
1000
1200
False Alarms
Fig. 2. ROC for different values of cgc
truth annotation has been used to calculate the false positives and negatives. The average values over the entire sequence were then plotted in the curve to find the optimal value for the parameter. A cgc of about 1.8 gives the best results. Lower values result in too much false alarms since many of the consistent gradual changes will not be dealt with then. If we increase the value of cgc, we see that the number of detection failures increases drastically; if the threshold is too high, too many foreground pixels will be mistaken for background pixels. Consequently, cgc = 1.8 is chosen and is further used in all experiments. 4.2
Experimental Results
We adopt the evaluation means of the related work [3,7] and compare the updated algorithm with the original scheme. The result of the proposed algorithm for the example frame of Fig. 1 is shown in Fig. 3. The left side shows the results of the original MGM, the right side shows the results of our system. The new matching process gives significantly less false positives, while it still detects foreground objects. In this case, no morphological post-processing has been applied, so further refinements can be done. Fig. 4 and 5 show a quantitative comparison of the regular MGM and the proposed scheme for the PetsD2TeC2 sequence (with a framerate of 30 frames per second). A manual ground truth annotation has been done for every 50th frame of the sequence. For each of these frames the ground truth annotation is matched with the output of the detection algorithms to find the amount of pixels which are erroneously considered to be foreground. As can be seen, a sudden increase occurs at the end of the video (which corresponds to a quick illumination change in the scene). We notice that
Improved Background Mixture Models for Video Surveillance Applications
257
Fig. 3. Results of MGM and the proposed scheme
14000
MGM Shadow_hsv Proposed
12000
False Positives
10000 8000 6000 4000 2000
95 0 11 00 12 50 14 00 15 50 17 00 18 50 20 00 21 50 23 00 24 50 26 00 27 50
80 0
65 0
50 0
35 0
50 20 0
0
Frame
Fig. 4. False Positives for MGM, a shadow removal technique and our proposed system
the proposed scheme succeeds to deal with the gradual lighting changes (frames 2100 to 2850) much better then the original scheme. The amount of false positives is largely reduced; in the best case we obtain a reduction of up to 95 % of the false positives compared with the normal scheme. The figure also shows that the updated technique obtains the same results as the original technique in scenes without gradual changes (frames 0 to 2050). Fig. 5 shows the false negatives recorded during the sequence. Our updated algorithm gives only a slight increase in the number of false negatives. In Sect. 2 we elaborated on alternate techniques which also give a solution for the quick illumination change problem. These methods are based on complex region level processing whereas our technique is solely pixel-based. Javed et al. presented their results on their website.1 Fig. 6 shows a scene in which a sudden 1
http://www.cs.ucf.edu/∼vision/projects/Knight/background.html
258
C. Poppe et al. 800
MGM Shadow_hsv Proposed
700
False Negatives
600 500 400 300 200 100
95 0 11 00 12 50 14 00 15 50 17 00 18 50 20 00 21 50 23 00 24 50 26 00 27 50
80 0
65 0
50 0
35 0
50 20 0
0
Frame
Fig. 5. False Negatives for MGM, a shadow removal technique and our proposed system
Fig. 6. From left to right: captured image, MGM output, result of [3], result of proposed scheme
illumination change occurs. The second image is the output from MGM and the third is the result of the system of Javed et al. The fourth image is the output of our proposed system. As can be seen, our conceptually simpler approach achieves similar results in coping with the illumination changes. As discussed in Sect. 2, some shadow techniques might provide a solution for the problem of quick illumination changes. We have evaluated the technique described in [9]. This technique uses the HSV color space since this corresponds closely to the human perception of color. Since the hue of a pixel does not change significantly when a shadow is cast and the saturation is lowered in shadowed points, the HSV color space indeed looks interesting for shadow detection. Consequently, the decision process is based on the following equation: IpV ≤ β ∧ IpS − BgpS ≤ τs ∧ IpH − BgpH ≤ τh . (6) Sp = α ≤ BgpV In (6), Bgp are the pixel values for the current background model. If Sp = true the pixel is assumed to be covered by a shadow. α should be adjusted according
Improved Background Mixture Models for Video Surveillance Applications
259
to the strength of the light source causing the shadows, β is needed to cope with certain aspects of noise, τs and τh are thresholds which respectively decide how large the difference in saturation and hue can be. This technique is therefore vastly dependent on the actual environment, but works well for shadow detection if the individual parameters can be fine-tuned according to the scene. In highly dynamic scenes as discussed in this paper, this approach would not be optimal. The illumination changes, in our situation, can cause shadows, but will mostly result in the opposite effect; pixel values get lighter color values. Therefore, we use the adjusted formula (7) for the detection of the lighting change. IpV (7) Sp = 1/β ≤ V ≤ 1/α ∧ IpS − μSp ≥ −τs ∧ IpH − μH p ≤ τh . μp Fig. 4 and 5 also show the false positives and false negatives of the adjusted shadow removal technique (Shadow hsv), respectively. We see that the shadow detection results in less false positives then the original scheme, but it cannot manage the entire change. Moreover, there is a strong increase of the false negatives.
5
Conclusion
This paper presents an updated scheme for object detection using a Mixture of Gaussian Models. The original scheme has been discussed in more detail and the incapability of dealing with quick illumination changes has been detected. An update of the matching mechanism has been presented. Furthermore, a comparison has been made with existing relevant object detection techniques which are able to deal with the problem. Experimental results show that our algorithm has significant improvements compared with the standard scheme, while only introducing minor additional processing.
Acknowledgments The research activities that have been described in this paper were funded by Ghent University, the Interdisciplinary Institute for Broadband Technology (IBBT), the Institute for the Promotion of Innovation by Science and Technology in Flanders (IWT), the Fund for Scientific Research-Flanders(FWO-Flanders), the Belgian Federal Science Policy Office (BFSPO), and the European Union.
References 1. Dick, A., Brooks, M.J.: Issues in automated visual surveillance. In: Proceedings of International Conference on Digital Image Computing: Techniques and Applications, pp. 195–204 (2003)
260
C. Poppe et al.
2. Wu, J., Trivedi, M.: Performance Characterization for Gaussian Mixture Model Based Motion Detection Algorithms. In: Proceedings of the IEEE International Conference on Image Processing, pp. 97–100. IEEE Computer Society Press, Los Alamitos (2005) 3. Javed, O., Shafique, K., Shah, M.: A Hierarchical Approach to Robust Background Subtraction using Color and Gradient Information. In: Proceedings of the Workshop on Motion and Video Computing, pp. 22–27 (2002) 4. Toyama, K., Krumm, J., Brumitt, B., Meyers, B.: Wallflower: Principles and Practice of Background Maintenance. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 255–261. IEEE Computer Society Press, Los Alamitos (1999) 5. Lee, D.: Online Adaptive Gaussian Mixture Learning for Video Applications. Statistical Methods in Video Processing. LNCS, pp. 105–116 (2004) 6. Zhang, Y., Liang, Z., Hou, Z., Wang, H., Tan, M.: An Adaptive Mixture Gaussian Background Model with Online Background Reconstruction and Adjustable Foreground Mergence Time for Motion Segmentation. In: Proceedings of the IEEE International Conference on Industrial Technology, pp. 23–27. IEEE Computer Society Press, Los Alamitos (2005) 7. Tian, Y., Lu, M., Hampapur, A.: Robust and Efficient Foreground Analysis for Real-time Video Surveillance. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 1182–1187. IEEE Computer Society Press, Los Alamitos (2005) 8. Prati, A., Mikic, I., Trivedi, M.M., Cucchiara, R.: Detecting Moving Shadows: Algorithms and Evaluation. IEEE Transactions on Pattern Analysis and Machine Intelligence 25, 918–923 (2003) 9. Cucchiara, R., Grana, C., Neri, G., Piccardi, M., Prati, A.: The Sakbot System for Moving Object Detection and Tracking. Video-Based Surveillance Systems Computer Vision and Distributed Processing, pp. 145–157 (2001) 10. Stauffer, C., Grimson, W.E.L.: Learning Patterns of Activity Using Real-Time Tracking. IEEE Transactions on Pattern Analysis and Machine Intelligence 22, 747–757 (2000) 11. Brown, L.M., Senior, A.W., Tian, Y., Connell, J., Hampapur, A., Shu, C., Merkl, H., Lu, M.: Performance Evaluation of Surveillance Systems Under Varying Conditions. In: Proceedings of IEEE International Workshop on Performance Evaluation of Tracking and Surveillance (2005) http://www.research.ibm.com/peoplevision/performanceevaluation.html
High Dynamic Range Scene Realization Using Two Complementary Images Ming-Chian Sung, Te-Hsun Wang, and Jenn-Jier James Lien Robotics Laboratory, Dept. of Computer Science and Information Engineering National Cheng Kung University, No. 1, Ta-Hsueh Road, Tainan, Taiwan {qwer,jjlien}@csie.ncku.edu.tw http://robotics.csie.ncku.edu.tw
Abstract. Many existing tone reproduction schemes are based on the use of a single high dynamic range (HDR) image and are therefore unable to accurately recover the local details and colors of the scene due to the limited information available. Accordingly, the current study develops a novel tone reproduction system which utilizes two images with different exposures to capture both the local details and color information of the low- and high-luminance regions of a scene. By computing the local region of each pixel, whose radius is determined via an iterative morphological erosion process, the proposed system implements a pixel-wise local tone mapping module which compresses the luminance range and enhances the local contrast in the low-exposure image. And a local color mapping module is applied to capture the precise color information from the high-exposure image. Subsequently, a fusion process is then performed to fuse the local tone mapping and color mapping results to generate highly realistic reproductions of HDR scenes. Keywords: High dynamic range, local tone mapping, local color mapping.
1 Introduction In general, a tone reproduction problem occurs when the dynamic range of a scene exceeds that of the recording or display device. This problem is typically resolved by applying some form of tone mapping technique, in which the high dynamic range (HDR) luminance of the real world is mapped to the low dynamic range (LDR) luminance of the display device. Various uniform (or global) tone mapping methods have been proposed [19], [21]. However, while these systems are reasonably successful in resolving the tone reproduction problem and avoid visual artifacts such as halos, the resulting images tend to lose the local details of the scene. By contrast, non-uniform (or local) tone mapping methods such as those presented in [1], [3], [4], [5], [7], [16] and [18] not only provide a good tone reproduction performance, but also preserve the finer details of the original scene. Such approaches typically mimic the human visual system by computing the local adaptation luminance in the scene. When computing the local adaptation luminance, the size of the local region is a crucial consideration and is generally estimated using some form of local contrast measure. Center-surround Y. Yagi et al. (Eds.): ACCV 2007, Part I, LNCS 4843, pp. 261–270, 2007. © Springer-Verlag Berlin Heidelberg 2007
262
M.-C. Sung, T.-H. Wang, and J.-J.J. Lien
functions such as the difference of Gaussian blurring images in [2] and [16] provides one means of estimating the size of this region. However, the local region size determined using this method is generally too local (or small) to reveal sufficient details. By contrast, piece-wise methods tend to improperly emphasize the local details. Furthermore, if the dynamic range of the real-world scene is very large, some of the image regions will be over-exposed, while some will be under-exposed, and hence the details in these regions will be lost. When processing such scenes using only a single image, the use of luminance compression techniques to recover the scene details achieves only limited success due to the lack of scene information available. Modern digital cameras invariably feature a built-in exposure bracketing functionality which allows a series of images to be obtained at different exposure levels via a single depression of the shutter release mechanism. This functionality is exploited to generate two simultaneous images of a high contrast scene with different levels of exposure such that the color and details of the scene can be accurately reproduced. Let the term IL refer to the low-exposure image, in which the brighter region is well-exposed, but the darker region is under-exposed. The brighter region contains good detail definition and has an abundance of color information. However, in the dark region of the image, the scene details are hidden and the true color of the scene cannot be recovered. Furthermore, let IH denote the high-exposure image, in which the darker region is well-exposed, but the brighter region is over-exposed. In this case, the darker region retains good detail definition and accurately reproduces the true color, while the brighter region is saturated such that the scene details cannot be perceived and the true color is not apparent. Although IL and IH have different exposure levels and may be not perfectly overlapped geometrically due to taken by unstable hand-held camera, coherence nevertheless exists in their color statistics and spatial constraints because they are taken simultaneously of the same scene [10]. The basic principle of the tone reproduction system developed in this study is to exploit these coherences in order to derive an accurate reproduction of the scene. In many image processing applications, the performance can be enhanced by using multiple input images to increase the amount of available information. Typical applications which adopt this approach include noise removal using flash and no-flash pairs [6], [12]; motion deblurring using normal and low-exposure pairs [10]; color transfer [15], [17]; and gray scale image colorization [11]. Goshtasby proposed an excellent method for realizing HDR reduction via the maximization of image information by using many images with different exposures [8]. However, the proposed method required the contents of the two images to be perfectly overlapped. Thus, the use of a fixed tripod was required, with the result that the practicality of the method was rather limited. In an attempt to resolve the problems highlighted above, the current study develops a novel tone reproduction system in which two input images acquired under different exposure conditions are input to a local adaptation mechanism which takes into account both the color statistics and the spatial constraint between the two images in order to reproduce the details and color of the scene. The proposed system performs a local tone mapping and local color mapping process to refine the advantage of IL and IH, respectively. Subsequently, a fusion procession is applied to make a compromise between the local optimum and the global optimum.
High Dynamic Range Scene Realization Using Two Complementary Images
(a)
(b)
(c)
(d)
(e)
(f)
263
Fig. 1. (a) Low-exposure image IL; (b) high-exposure image IH; (c) segmentation of IL into four regions based upon entropy; (d) illustration of morphological erosion operation following three iterations; (e) iteration map of IL; and (f) example of pixels and their corresponding local regions as determined by their iteration map values
2 Iteration Map Creation and Local Region Determination In order to perform the local mechanism, this study commences by finding the local region of each pixel. A histogram-based segmentation process is applied to group the luminance values of the various pixels in the image into a small number of discrete intervals. The radius of the local region of each pixel is then determined by iteration map which is derived from a morphological erosion operation. 2.1 Histogram-Based Region Segmentation Using Entropy Theorem The entropy theorem provides a suitable means of establishing an optimal threshold value T when separating the darker and brighter regions of an image [13]. According to this theorem, by maximizing the entropy of an image by maximizing the entropy of both the darker and the brighter regions of the image, an optimal threshold value can be obtained via the following formulation: t
255
T = arg max ( − ∑ p i log p i − ∑ p i log p i ) , t
i=0
d
d
b
b
(1)
i=t
where t is the candidate threshold, Pid and Pib are the probability of the darker pixels with luminance value i and brighter pixels with luminance i, respectively. Adopting a dichotomy approach, the segmentation procedure is repeated three times, yielding three separate threshold values, i.e. Llow, Lmiddle and Lhigh , which collectively segment the histogram into four subintervals, namely [Lmin, Llow], [Llow, Lmiddle], [Lmiddle, Lhigh] and [Lhigh, Lmax], respectively.
264
M.-C. Sung, T.-H. Wang, and J.-J.J. Lien
2.2 Iteration Map Creation Using Morphological Erosion Operation Having segmented the image, the proposed tone reproduction system then determines the circular local region Rx,y of each pixel (x, y). The radius of this region is found by performing an iterative morphological erosion operation in each luminance region, and creating an iteration map to record the iteration number at which each pixel is eroded. Clearly, for pixels located closer to the region boundary, the corresponding iteration value is lower, while for those pixels closer to the region center, the iteration value is higher. Hence, by inspection of the values indicated on the iteration map, it is possible not only to determine the radius of the circular local region of each pixel, but also to modulate the tone mapping function as described later.
3 Tone Reproduction System Since the light range in the brighter regions of an image is greater than that in the darker regions, the details in the over-exposed regions in IH are usually lost. Hence, in the current approach, more detailed IL is executed using the color and tone information associated with IH. 3.1 Luminance: Pixel-Wise Local Tone Mapping The proposed tone reconstruction method commences by applying a non-uniform luminance scaling process to IL to generate an initial middle-gray tone image. Due to the under-exposed darker region and well-exposed brighter region of IL, it is necessary to apply a greater scaling effect to the darker region to brighten the concealed details and a reduced scaling to the well-exposed brighter region, i.e. ⎛1 ⎞ Lk = exp⎜⎜ ∑ log(δ + Lk ( x, y)) ⎟⎟ Lk ∈ {LL , LH } ⎝ N x, y ⎠
⎛ L L( x, y ) = ⎜⎜ 2 H ⎝ LL
⎞⎛ LL ( x, y ) ⎞ ⎟ LL ( x, y ) , ⎟⎟⎜1 − 2 ⎜ Lwhite ⎟⎠ ⎠⎝
(1)
(2)
LL and LH are the log-average luminance (referred to as the key values in [9], [19] and [20]) of IL and IH , respectively, and are used to objectively measure whether the scene is low-gray, middle-gray or high gray tone. Furthermore, LL(x, y) is the luminance value of pixel (x, y) in IL, and is normalized within the interval [0, 1]. Finally, Lwhite is the maximum luminance value in IL. By applying Eqs. (1) and (2), the luminance of LL can be scaled to an overall luminance L. To mimic the human visual system which attains visual perception of a scene by locally adapting luminance differences, the system proposed in this study performs a local tone mapping process which commences by computing the local adaptation luminance. Since the radius of the circular local region Rx,y has already been determin ed for each pixel (x, y), the value of the local adaptation luminance can be obtained
High Dynamic Range Scene Realization Using Two Complementary Images
(a)
(b)
265
(c)
Fig. 2. (a) Local adaptation luminance result. Note result is normalized into interval [0, 255] for display purposes; (b) detailed term H; and (c) luminance compression term V’.
simply by convoluting the luminance values in the local region using a weighted mask, i.e. V ( x, y ) =
⎞ ⎛ ⎟ 1 ⎜ L ( x, y)G x , y (i, j ) K x, y (i, j ) ⎟ , ⎜ Z x, y ⎜ (i, j∈R ) ⎟ x, y ⎠ ⎝
∑
(4)
The significance of each neighborhood pixel (i, j) when performing this convolution, is evaluated using Gx,y and Kx,y , which are Gaussian weights corresponding to the spatial distance between pixels (x, y) and (i, j) and the difference in luminance of the two pixels, respectively. And Zx,y in Eq. (4) is a normalization term. A method known as local tone mapping was proposed by Reinhard et al. [16] for addressing the tone reproduction problem. This simple non-uniform mapping technique compresses the luminance range of the scene such that all of the luminance values fall within the interval [0,1]. The system presented in the current study goes a step further in modulating the local contrast and luminance compression by extracting the detailed term (denoted as H) and the local adaptation luminance compression term (denoted as V’) as Fig. 2 (a~c) and then modulating them in accordance with [3]: ρ
Ld =
ρ
L ⎛ L⎞ ⎛ V ⎞ = HxV ′ = ⎜ ⎟ x⎜ ⎟ 1+V ⎝V ⎠ ⎝1+ V ⎠
γ≦1
γ
(5)
ρ
. The value of controls the degree of sharpness of the where 0< <2 and 0< reproduced image, with a larger value generating a sharper result. In the current study, the value of is varied in direct proportion to the iteration value of each pixel specified in the iteration map to ensure a smooth boundary while simultaneously revealing most of the image details near the region center. Meanwhile, the value of determines the degree of luminance compression. As its value is reduced, the luminance of the darker regions in the image is compressed into a larger display is inversely proportional to the range of the interval [Lmin, range. As a result, Lmiddle]. Finally, in order to obtain the final local tone mapping result It, the value of each RGB channel is scaled according to the change ratio of the luminance which is obtained by dividing the output luminance Ld by input scaled luminance L, i.e.
ρ
γ
γ
It = I L x
Ld L
(6)
266
M.-C. Sung, T.-H. Wang, and J.-J.J. Lien
3.2 Color: Pixel-Wise Local Color Mapping Tone mapping method mentioned in last section modulates only the luminance value, and hence the original imprecise RGB color information of IL is still reserved. If a low-exposure image is modulated such that the color and detailed information of the brighter regions can be captured, the darker regions of the image will inevitably be under-exposed. Even if the scene is characterized by an extremely broad light range, the darker regions will become so dark that the true color information has not been recorded by camera and thus only an unsatisfied result of local tone mapping can be obtained. Moreover, the darker regions generally contain considerable noise and thus obtaining a crisp result is difficult. To resolve these problems, the current approach modulates IL using the color information relating to IH, and pixel-wise local color mapping module is applied to acquire the ground-truth color. Reinhard et al. [15] proposed the following simple but highly effective method, implemented in the Aαβ color space, for transferring the color statistics from IH to IL [14]: I L ′ ( x, y) = g (I L ( x, y )) = μ H +
σH ( I L ( x, y) − μ L ), σL
(7)
where IL(x, y) is the color value of pixel (x, y) in each Aαβ channel in the lowexposure image, L and L are the mean and standard deviation of the color value in IL, respectively, and H and H are the mean and standard deviation of the color value in IH., respectively. In the current system, this color transfer process is performed on each Aαβ channel and the results are then converted back to the RGB color space to obtain the preliminary result IL’. However, this global color transfer approach fails to obtain the precise local color when the source or target image contains many different color regions because it cannot distinguish which particular region each color statistic derives from, and thus mixes them all up, yielding a uniform transfer [17]. Hence, the current system applies a further pixel-wise local color mapping process in RGB channel to find the true local color result of pixel (x, y) from pixel (i*, j*) in IH in the local color mapping region Sx,y whose radius is inversely proportional to the value of pixel (x, y) in the iteration map because the color near the region center is more
μ
μ
σ
σ
Weight
LL
Fig. 3. Local color mapping region Sx,y is inversely proportional to the value of pixel (x, y) in the iteration map
Fig. 4. Finding fusion weight αx,y using double sigmoid function
High Dynamic Range Scene Realization Using Two Complementary Images
267
reliable, so we can search its precise color in IH within a smaller local color mapping region Sx,y as shown in Fig. 3.. This suggests following equation: ⎛ ( I ' ( x, y ) − I H (i, j )) 2 ⎞⎟ (i* , j* ) = arg max ⎜ exp(− L ) , ⎜ ⎟ 2σ 2 (i , j )∈S x , y ⎝ ⎠
(8)
σ
is specified as half the value of pixel (x, y) in the iteration map. where the value of When (i*, j*) in the Eq. (8) is obtained, the color value of IH (i*, j*) is used in place of IL’(x, y) to construct the local mapping result Ic. This local color mapping method resolves the image shift problem and gives the ground-truth color appearance of IH. Input Image Pairs
Output Image
IH
IL IL
Pixel-Wise Local Tone Mapping
Iteration map Creation and Local Region Determination Region Segmentation By Entropy
Id
IH
LL Lψ L
Iteration Map
Result Fusion
Local Adaptation Luminance V
Local Tone Mapping Result Local Adaptation Luminance V
IL Segmentation Result
Morphological Erosion Operation
Local Tone Mapping Result It
It
Pixel-Wise Local Color Mapping
IL
IH
Global Color ’ Transfer Result IL Local Color Mapping Result
Iteration Map
Ic IH & IL
’
Iteration Map
Local Color Mapping Result Ic
Fig. 5. Flowchart of the proposed system
268
M.-C. Sung, T.-H. Wang, and J.-J.J. Lien
I d = α x , y I t + (1 − α x , y ) I c
α x, y
β ⎧ ⎪ ⎛ ⎞ ⎪1 + exp⎜ − 2 LL ( x, y ) − Lmiddle ⎟ ⎜ | Lmiddle − Llow | ⎟⎠ ⎪⎪ ⎝ =⎨ β ⎪ ⎛ ⎪ L x, y ) − Lmiddle ⎞⎟ ( L ⎪1 + exp⎜⎜ − 2 | Lmiddle − Lhigh | ⎟⎠ ⎝ ⎩⎪
(9)
if LL ( x, y ) < Lmiddle
(10)
otherwise.
α
The function used to generate the weight x,y is the double sigmoid function as shown in Fig. 4. This function provides a virtually linear fusion over the interval [Llow, Lhigh] and a non-linear fusion over the intervals [Lmin, Llow] and [Lhigh, Lmax], respectively. In the darker regions, Ic is assigned a higher weight to obtain the true color appearance and to reduce the noise in It. Meanwhile, in the brighter regions, It is assigned a greater weight to compensate the color and details in the saturated region of Ic. The parameter in Eq. (10) is inversely proportional to the value of pixel (x, y) in the iteration map serves to further control the color appearance of the fusion result. The flowchart of proposed method is shown in Fig. 5.
β
4 Experimental Results and Conclusion The current tone reproduction experiments were performed on a PC equipped with an Intel Pentium 4 (3.2GHz) processor. The execution time of the proposed method depends on the number and size of segmentation regions. In general, the results showed that a 640 x 480 image could be processed within 10 seconds on average. When implementing the local tone reproduction methods presented in the literature, the size of the local region is a crucial factor. Figure 6 compares the local regions estimated by the schemes presented in [16] and [3], respectively, with that estimated by the current method. It can be seen that the local region size derived using the method proposed by Reinhard et al. [16] may be too small to reveal sufficient local details. Conversely, the region size estimated using the region-wise method presented in [3] is not always adaptive for each pixel in that region and may therefore result in an unnatural emphasis. However, the morphological erosion method proposed in the current study enables the derivation of a more adaptive local region size.
(a)
(b)
(c)
Fig. 6. Example of pixel and corresponding local region measured using: (a) local region method proposed by Reinhard et al. [16]; (b) region-wise method [3]; and (c) current morphological erosion operation
High Dynamic Range Scene Realization Using Two Complementary Images
(a.1)
(b.1)
(c.1)
(a.2)
(b.2)
(c.2)
(a.3)
(b.3)
(c.3)
(a.4)
(b.4)
(c.4)
269
Fig. 7. Typical image pairs and corresponding tone reproduction results: (a.1~c.1) input image pairs IL and IH; (a.2~c.2) tone mapping results It; (a.3~c.3) local color mapping results Ic; and (a.4~c.4) fusion results Id
Besides the ability of the proposed method to successfully process the HDR images as shown in Fig. 7(a.1~a.4), our method is also useful to deal with the LDR images and non-overlapped image pairs as shown in Fig. 7(b.1~b.4). The local color mapping process effectively overcomes such shift position problem between image pairs while simultaneously obtaining the true color from IH. In Fig. 7(c.1~c.4), it is seen that the proposed method effectively smoothes the noise within the darker regions. In conclusion, tone reproduction methods are essential techniques in realizing HDR scenes in LDR display devices. Many previous tone reproduction techniques fail to accurately recover the color and details of HDR scenes since they use only a single image and therefore have only limited information at their disposal. However, in the tone reproduction technique presented in this paper, two images are acquired simultaneously at different exposures and are supplied to an automatic local adaptation mechanism which takes account of both the color statistics and the spatial constraint between the images in order to maintain the color and detailed information of the original scene. The experimental results confirm that the proposed system
270
M.-C. Sung, T.-H. Wang, and J.-J.J. Lien
provides promising results full of rich detailed and color information content and is capable of generating highly realistic reproductions of HDR scenes.
References 1. Ashikhmin, M.: A Tone Mapping Algorithm for High Contrast Images. 13th Eurographics, 145–156 (2002) 2. Blommaert, F.J.J., Martens, J.-B.: An Object-Oriented Model for Brightness perception. Spatial Vision 5(1), 15–41 (1990) 3. Chen, H.T., Liu, T.L., Fuh, C.S.: Tone Reproduction: A Perspective from luminanceDriven Perceptual Grouping. In: IJCV, vol. 65, pp. 73–96 (2005) 4. DiCarlo, J.M., Wandell, B.A.: Rendering High Dynamic Range Images. SPIE: Image Sensors 3965, 392–401 (2000) 5. Durand, F., Dorsey, J.: Fast Bilateral Filtering for the Display of High-Dynamic-Range Images. ACM SIGGRAPH, 257–266 (2002) 6. Eisemann, E., Durand, F.: Flash photography enhancement via intrinsic relighting. ACM Trans. Graph. 23(3), 673–678 (2004) 7. Fattal, R., Lischinski, D., Werman, M.: Gradient Domain High Dynamic Range Compress. ACM SIGGRAPH, 249–256 (2002) 8. Goshtasby, A.: A High dynamic range reduction via maximization of image information, http://www.cs.wright.edu/~agoshtas/hdr.html 9. Holm, J.: Photographic Tone and Colour Reproduction Goals. CIE Expert Symposium ’96 on Colour Standard for Image Technology, 51–56 (1996) 10. Sun, J., Tang, J.: Bayesian Correction of Image Intensity with Spatial Consideration. In: Pajdla, T., Matas, J(G.) (eds.) ECCV 2004. LNCS, vol. 3024, pp. 342–354. Springer, Heidelberg (2004) 11. Levin, A., Lischinski, D., Weiss, Y.: Colorization using optimization. ACM Trans Graph. 23(3), 689–694 (2004) 12. Petschnigg, G., Szeliski, R., Agrawala, M., Choen, M., Hoppe, H., Toyama, K.: Digitial photography with flash and no-flash image pairs. ACM Trans. Graph. 23(3), 664–672 (2004) 13. Pardo, A., Sapiro, G.: Visualization of High Dynamic Range Images. IEEE Trans. on Image Proc. 12(6), 639–647 (2003) 14. Ruderman, D.L., Chonin, T.W., Chiao, C.C: Statistic of cone responses to natural images: implications for visual coding. J. Optical Soc. Of America 15(8), 2036–2045 (1998) 15. Reinhard, E., Ashikhmin, M., Gooch, B., Shirley, P.: Color Transfer between Images. IEEE CG&A 21, 34–41 (2001) 16. Reinhard, E., Stark, M., Shirley, P., Ferwerda, J.: Photographic Tone Reproduction for Digital Images. ACM SIGGRAPH, 267–276 (2002) 17. Tai, Y.W., Jia, J., Tang, C.K.: Local Color Transfer via Probabilistic Segmentation by Expectation-Maximization. In: CVPR, vol. 1, pp. 747–754 (2005) 18. Tumblin, J., Turk, G.: LCIS: A Boundary Hierarchy for Detail-Preserving Contrast Reduction. ACM SIGGRAPH, 83–90 (1899) 19. Tumblin, J., Rushmeier, H.: Tone Reproduction for Computer Generated Images. IEEE CG&A 13(6), 42–48 (1993) 20. Ward, G.: A Contrast-Based Scale factor for Luminance Display. In: Heckbert, P. (ed.) Graphics Gems IV, pp. 415–421. Academic Press, Boston (1994) 21. Ward, G., Rushmeuer, H.E., Piatko, C.D.: A visibility matching tone reproduction operator for high dynamic range scenes. IEEE Trans. Visualization and Computer Graphics 3(4), 291–306 (1997)
Automated Removal of Partial Occlusion Blur Scott McCloskey, Michael Langer, and Kaleem Siddiqi Centre for Intelligent Machines, McGill University {scott,langer,siddiqi}@cim.mcgill.ca
Abstract. This paper presents a novel, automated method to remove partial occlusion from a single image. In particular, we are concerned with occlusions resulting from objects that fall on or near the lens during exposure. For each such foreground object, we segment the completely occluded region using a geometric flow. We then look outward from the region of complete occlusion at the segmentation boundary to estimate the width of the partially occluded region. Once the area of complete occlusion and width of the partially occluded region are known, the contribution of the foreground object can be removed. We present experimental results which demonstrate the ability of this method to remove partial occlusion with minimal user interaction. The result is an image with improved visibility in partially occluded regions, which may convey important information or simply improve the image’s aesthetics.
1
Introduction
Partial occlusions arise in natural images when an occluding object falls nearer to the lens than the plane of focus. The occluding object will be blurred in proportion to its distance from the plane of focus, and contributes to the exposure of pixels that also record background objects. This sort of situation can arise, for example, when taking a photo through a small opening such as a cracked door, fence, or keyhole. If the opening is smaller than the lens aperture, some part of the door/fence will fall within the field of view, partially occluding the background. This may also arise when a nearby object (such as the photographer’s finger, or a camera strap) accidentally falls within the lens’ field of view. Whatever its cause, the width of the partially-occluded region depends on the scene geometry and the camera settings. Primarily, the width increases with increasing aperture size (decreasing f -number), making partial occlusion a greater concern in low lighting situations that necessitate a larger aperture. Fig. 1 (left) shows an image with partial occlusion, which has three distinct regions: complete occlusion (outside the red contour), partial occlusion (between the green and red contours), and no occlusion (inside the green contour). As is the case in this example, the completely occluded region often has little highfrequency structure because of the severe blurring of objects far from the focal plane. In addition, the region of complete occlusion can be severely underexposed when the camera’s settings are chosen to properly expose the background. In [7], it was shown that it is possible to remove the partial occlusion when the location and width of the partially occluded region are found by a user. Because Y. Yagi et al. (Eds.): ACCV 2007, Part I, LNCS 4843, pp. 271–281, 2007. c Springer-Verlag Berlin Heidelberg 2007
272
S. McCloskey, M. Langer, and K. Siddiqi
Fig. 1. (Left) Example image taken through a keyhole. Of the pixels that see through the opening, more then 98% are partially occluded. (Right) The output of our method, with improved visibility in the partially-occluded region.
of the low contrast and arbitrary shape of the boundary between regions of complete and partial occlusion, this task can be challenging, time consuming, and prone to user error. In the current paper we present an automated solution to this vision task for severely blurred occluding objects and in doing so significantly extend the applicability of the method in [7]. Given the input image of Fig. 1 (left), the algorithm presented in this paper produces the image shown in Fig. 1 (right). The user must only click on a point within each completely-occluded region in the image, from which we find the boundary of the region of complete occlusion. Next, we find the width of the partially occluded band based on a model of image formation under partial occlusion. We then process the image to remove the partial occlusion, producing an output with improved visibility in that region. Each of these steps is detailed in Sec. 4.
2
Previous Work
The most comparable work to date was presented by Favaro and Soatto [3], who describe an algorithm which reconstructs the geometry and radiance of a scene, including partially-occluded regions. While this restores the background, it requires several registered input images taken at different focal positions. Tamaki and Suzuki [12] presented a method for the detection of completely occluded regions in a single image. Unlike our method, they assume that the occluding region has high contrast with the background, and that there is no adjacent region of partial occlusion. A more distantly related technique is presented by Levoy et al. in [5], where synthetic aperture imaging is used to see around occluding objects. Though this
Automated Removal of Partial Occlusion Blur
273
ability is one of the key features of their system, no effort is made to identify or remove partial occlusion in the images. Partial occlusion also occurs in matte recovery which, while designed to extract the foreground object from a natural scene, can also recover the background in areas of partial occlusion. Unlike our method, matte recovery methods require either additional views of the same scene [8,9] or substantial user intervention [1,11]. In the latter category, users must supply a trimap, a segmentation of the image into regions that are either definitely foreground, definitely background, or unknown/mixed. Our method is related to matte recovery, and can be viewed as a way of automatically generating, from a single image, a trimap for images with partial occlusion due to blurred foreground objects.
3
Background and Notation
In [7], it was shown that the well-known matting equation, Rinput (x, y) = α(x, y)Rf + (1 − α(x, y))Rb (x, y),
(1)
describes how the lens aperture combines the radiance Rb of the background object with the radiance Rf of the foreground object. The blending parameter α describes the proportion in which the two quantities combine and Rinput is the observed radiance. Notionally, the quantity α is the fraction of the pixel’s viewing frustum that is subtended by the foreground object. Since that object is far from the plane of focus, the frustum is a cone and α is the fraction of that cone subtended by the occluding object. In order to remove the contribution of the occluding object, the values of α and Rf must be found at each pixel. Given the location of the boundary between regions of complete and partial occlusion, the distance d between each pixel and the nearest point on the boundary can be found. From d and the width w of the partially occluded region, the value of α is well-approximated [7] by √ 2d 1 l 1 − l2 + arcsin(l) , where l = min 2, α= − − 1. (2) 2 π w This can be done if the user supplies both w and the boundary between regions complete and partial occlusion, as in [7]. Unfortunately, this task is time consuming, difficult, and prone to user error. In this paper, we present an automated solution to this vision problem, from which we compute the values α and Rf .
4
Method
To state the vision problem more clearly, we refer to the example1 in Fig. 2. In this example, the partial occlusion is due to the handle of a fork directly in front 1
The authors of [7] have made this image available to the public at http://www.cim. mcgill.ca/∼scott/research.html
274
S. McCloskey, M. Langer, and K. Siddiqi
Fig. 2. To remove partial occlusion from a foreground object, the vision problem is to determine the boundary of the completely occluded region (green curve) and the width of the partially-occluded region (the length of the red arrow)
of the lens. In order to remove the contribution of the occluding object, we must automatically find the region of complete occlusion (outlined in green) and the width of the partially occluded band (the length of the red arrow). In order to find the region of complete occlusion within the image, we assume that the foreground image appears as a region of nearly constant intensity. Note that this does not require that the object itself have constant radiance. Because the object is far from the plane of focus, high-frequency radiance variations will be lost due to blurring in the image. Moreover, when objects are placed against the lens they are often severely under-lit, as they fall in the shadow of the camera or photographer. As such, many occluding objects with texture may appear to have constant intensity in the image. A brief overview of the method is as follows. Given the location of a point p that is completely occluded (provided by the user), we use a geometric flow (Sec. 4.2) to produce a segmentation with a smooth contour such as the one outlined in Fig. 2, along which we find normals facing outward from the region of complete occlusion. The image is then re-sampled (Sec. 4.3) to uniformly-spaced points on these normals, reducing an arbitrarily-shaped occluding contour to a linear contour. Low variation rows in the resulting image are averaged to produce a profile from which the blur width is estimated (Sec. 4.4). Once the blur width is estimated, the method of [7] is used to remove the partial occlusion (Sec. 4.5). 4.1
Preprocessing
Two pre-processing steps are applied before attempting segmentation: 1. Because our model of image formation assumes that the camera’s response is linear, we use the method of [2] to undo the effects of its tone mapping function, transforming camera image Iinput to a radiance image Rinput . 2. Before beginning the segmentation, we force the completely occluded region to be the darkest part of the image by subtracting Rp , the radiance of the user-selected point, and taking the absolute value. This gives a new image R = Rinput − Rp ,
(3)
Automated Removal of Partial Occlusion Blur
275
which is nearly zero at points in the region of complete occlusion, and higher elsewhere. As a result of this step, points on the boundary between the regions of partial and complete occlusion will have gradients ∇R that point out of the region of complete occlusion. This property will be used to find the boundary between the regions of complete and partial occlusion. 4.2
Foreground Segmentation
While the region of complete occlusion is assumed to have nearly constant intensity, segmenting this region is nontrivial due to the extremely low contrast at the boundary between complete and partial occlusion. In order to produce good segmentations in spite of this difficulty, we use two cues. The first cue is that pixels on the boundary of the region of complete occlusion have gradients of R that point into the region of partial occlusion. This is assured by the pre-processing of Eq. 3, which causes the foreground object to have the lowest radiance in R. The second cue is that points outside of the completely occluded region will generally have intensities that differ from the foreground intensity. To exploit these two cues, we employ the flux maximizing geometric flow of [13], which evolves a 2D curve to increase the outward flux of a static vector field through its boundary. Our cues are embodied in the vector field ∇R − → , where φ = (1 + R)−2 . V =φ |∇R|
(4)
∇R The vector field |∇R| embodies the first cue, representing the direction of the gradient, which is expected to align with the desired boundary as well as be orthogonal to it2 . The scalar field φ, which embodies the second cue, is near 1 in the completely-occluded region and smaller elsewhere. As noted in [6], an exponential form for φ can be used to produce a monotonically-decreasing function of R, giving similar results. The curve evolution equation works out to be ∇R →− − → → − Ct = div( V )N = ∇φ, (5) + φκR N , |∇R|
where κR is the Euclidean mean curvature of the iso-intensity level set of the image. The flow cannot leak outside the completely occluded region since by construction both φ and ∇φ are nearly zero there. This curve evolution, which starts from a small circular region containing the user-selected point, may produce a boundary that is not smooth in the presence of noise. In order to obtain a smooth curve, from which outward normals can be robustly estimated, we apply a few iterations of the euclidean curve-shortening flow [4]. While it is possible to include a curvature term in the flux-maximizing flow to evolve a smooth contour, we separate the terms into different flows which are computed in sequence. Both flows are implemented using level set methods [10]; details are given in the Appendix. 2
It is important to normalize the gradient of the image so that its magnitude does not dominate the measure outside of the occluded region.
276
S. McCloskey, M. Langer, and K. Siddiqi
Fig. 3. (Left) Original image with segmentation boundary (green) and outward-facing normals (blue) along which the image will be re-sampled. (Right) The re-sampled image (scaled), which is used to estimate the blur width.
Once the curve-shortening flow has terminated, we can recover the radiance Rf of the foreground (occluding) object by simply taking the mean radiance value within the segmented (completely occluded) region. Note that we use this instead of Rp , the radiance of the user-selected point, as there may be some low-frequency intensity variation within the region of complete occlusion. 4.3
Boundary Rectification and Profile Generation
One of the difficulties in measuring the blur width is that the boundary of the completely occluded region can have an arbitrary shape. In order to handle this, we re-sample the image R along outward-facing normals to the segmentation boundary, reducing the shape of the occluding contour to a line along the left edge of a re-sampled image Rl . The number of rows in Rl is determined by the number of points on the segmentation boundary, and pixels in the same row of Rl come from points on the same outward-facing normal. Pixels in the same column come from points the same distance from the segmentation boundary on different normals and thus, recalling Eq. 2, have the same α value. The number of columns in the image depends on the distance from the segmentation boundary to the edge of the input image. We choose this quantity to be the largest value such that 80% of the normals remain within the image frame and do not re-enter the completely-occluded region (this exact quantity is arbitrary and the method is not sensitive to variations in this choice). Fig. 3 shows outward-facing surface normals from the contour in Fig. 2, along with the re-sampled image. The task of measuring the width of the partially occluded region is also complicated by the generality of the background intensity. In the worst case, it is impossible (for human observers or our algorithm) to estimate the width if the background has an intensity gradient in the opposite direction of the intensity gradient due to partial occlusion. The measurement is straightforward if the background object had constant intensity, though this assumption is too strong. Given that the blurred region is a horizontal feature in the re-sampled image, we average rows of Rl in order to smooth out high-frequency background
Automated Removal of Partial Occlusion Blur
277
2 0.03
1.8
1.6
0.025
1.4 0.02
Fitting Error
P(x)
1.2
1
0.8
0.015
0.01
0.6
0.4 0.005
0.2
0
0
20
40
60
80
100
120
140
160
0 20
40
60
80
x
100 w′
120
140
160
Fig. 4. [Left] Profile generated from the re-sampled image in Fig. 3 (black curve). Model 50 141 with relatively high error (red curve). Model profile Pm with minimum profile Pm error (green curve). [Right] Fitting error as a function of w .
texture. While we do not assume a uniform background, we have found it useful to eliminate rows with relatively more high-frequency structure before averaging. In particular, for each row of Rl we compute the sum of its absolute horizontal derivatives Rl (x + 1, y) − Rl (x − 1, y). (6) x
Rows with an activity measure in the top 70% are discarded, and the remaining rows are averaged to generate the one dimensional blur profile P . 4.4
Blur Width Estimation
Given a 1D blur profile P , like the one shown in Fig. 4 (black curve), we must estimate the width w of the partially occluded region. We do this by first expressing P in terms of α. Recalling Eq. 3 and the fact that Rf ≈ Rp , we rearrange Eq. 1 to get (7) Rl (x, y) = (1 − α(x, y))Rbl (x, y) − Rf , where Rbl is the radiance of the background object defined on the same lattice as the re-sampled image. The profile P (x) is the average of many radiances from pixels with the same α value, so P (x) = (1 − α(x))Rbl (x) − Rf ,
(8)
where Rbl (x) is the average radiance of background points a fixed distance from the segmentation boundary (which fall in a column of the re-sampled image). As we have removed rows with significant high-frequency structure and averaged the rows of the re-sampled image, we assume that the values Rbl (x) are relatively constant over the partially-occluded band, and thus P (x) = (1 − α(x))Rbl − Rf .
(9)
Based on this, the blur width w is taken to be the value that minimizes the average fitting error between the measured profile P and model profiles. The
278
S. McCloskey, M. Langer, and K. Siddiqi
w model profile Pm for a given width w is constructed by first generating a linear ramp l and then transforming these values into α values by Eq. 2. An example is shown in Fig. 4, where the green curve shows the model profile for which the error is minimized with respect to the measured profile (black curve), and the red curve shows another model profile which has higher error. A plot of the error as a function of w is shown in figure 4. We see that it has a well-defined global minimum, which is at w = 141 pixels.
4.5
Blur Removal
Once the segmentation boundary and the width w of the partially-occluded region have been determined, the value of α can be found using Eq. 2. In order to compute α at each pixel, we must find its distance to the nearest point on the segmentation boundary. We employ the fast marching method of [10]. Recall that the radiance Rf of the foreground object was found previously, so we can recover the radiance of the background at pixel (x, y) according to Rb (x, y) =
Rinput (x, y) − α(x, y)Rf . 1 − α(x, y)
(10)
Finally, the processed image Rb is tone-mapped to produce the output image. This tone-mapping is simply the inverse of what was done in section 4.1.
5
Experimental Results
Fig. 5 shows the processed result from the example image in Fig. 2. The userselected point was near the center of the completely occluded region though, in our experience, the segmentation is insensitive to the location of the initial point. We also show enlargements of a region near the occluding contour to illustrate the details that become clearer after processing. Near the contour, as α → 1, noise becomes an issue. This is because we are amplifying a small part of the input signal, namely the part that was contributed by the background.
Fig. 5. (Left) Result for the image shown in Fig. 2. (Center) Enlargement of processed result. (Right) Enlargement of the corresponding region in the input image.
Automated Removal of Partial Occlusion Blur
279
Fig. 6. Example scene through a small opening. (Left) Input wide-aperture image. (Middle) Output wide-aperture image. (Right) Reference small-aperture image. Notice that more of the background is visible in our processed wide-aperture image.
Fig. 6 shows an additional input and output image pair, along with a reference image taken through a smaller aperture. The photos were taken through a slightly opened door. It is important to note that processing the wide aperture photo reveals background detail in parts of the scene where a small aperture is completely occluded. Namely, all pixels where α > .5 are occluded in a pinhole aperture image, though many of them can be recovered by processing a wide aperture image. In this scene, there are two disjoint regions of complete occlusion, each of which has an adjacent region of partial occlusion. This was handled by having the user select two starting points from which the segmentation flow was initialized, though the method could also have been applied separately to each occluded region. The method described in this paper can also be extended to video processing. In the event that the location of the camera and the occluding object are fixed relative to one another, we need only perform the segmentation and blur estimation on a single frame of the video. The recovered value of α at each pixel (the matte) can be used to process each frame of the video separately. A movie, keyholevideo.mpg, is included in the supplemental material with this submission, and shows the raw and processed frames side-by-side (as in Fig. 1).
6
Conclusion
The examples in the previous section demonstrate how our method automatically measures the blur parameters and removes partial occlusion due to nearby objects. Fig. 6 shows that pictures taken through small openings (such as a fence,
280
S. McCloskey, M. Langer, and K. Siddiqi
keyhole, or slightly opened door) can be processed to improve visibility. In this and the case of the text image shown in Fig. 5, this method reveals important image information that was previously difficult to see. The automated nature of this method makes the recovery of partially-occluded scene content accessible to the average computer user. Users need only specify a single point in each completely occluded region, and the execution time of 10-20 seconds is likely acceptable. Given such a tool, users could significantly improve the quality of images with partial occlusions. In order to automate the recovery of the necessary parameters, we have assumed that the combination of blurring and under-exposure produces a foreground region with nearly constant intensity. Methods that allow us to relax this assumption are the focus of ongoing future work, and must address significant additional complexity in each of the segmentation, blur width estimation, and blur removal steps.
References 1. Chuang, Y., Curless, B., Salesin, D.H., Szeliski, R.: A Bayesian Approach to Digital Matting. In: CVPR 2001, pp. 264–271 (2001) 2. Debevec, P., Malik, J.: Recovering High Dynamic Range Radiance Maps from Photographs. In: SIGGRAPH 1997, pp. 369–378 (1997) 3. Favaro, P., Soatto, S.: Seeing Beyond Occlusions (and other marvels of a finite lens aperture). In: CVPR 2003, pp. 579–586 (2003) 4. Grayson, M.: The Heat Equation Shrinks Embedded Plane Curves to Round Points. Journal of Differential Geometry 26, 285–314 (1987) 5. Levoy, M., Chen, B., Vaish, V., Horowitz, M., McDowall, I., Bolas, M.: Synthetic Aperture Confocal Imaging. In: SIGGRAPH 2004, pp. 825–834 (2004) 6. Perona, P., Malik, J.: Scale-Space and Edge Detection using Anisotropic Diffusion. IEEE Trans. on Patt. Anal. and Mach. Intell. 12(7), 629–639 (1990) 7. McCloskey, S., Langer, M., Siddiqi, K.: Seeing Around Occluding Objects. In: Proc. of the Int. Conf. on Patt. Recog. vol. 1, pp. 963–966 (2006) 8. McGuire, M., Matusik, W., Pfister, H., Hughes, J.F., Durand, F.: Defocus Video Matting. ACM Trans. Graph. 24(3) (2005) 9. Reinhard, E., Khan, E.A.: Depth-of-field-based Alpha-matte Extraction. In: Proc. 2nd Symp. on Applied Perception in Graphics and Visualization 2005, pp. 95–102 (2005) 10. Sethian, J.A.: Level Set Methods and Fast Marching Methods. Cambridge University Press, Cambridge (1999) 11. Sun, J., Jia, J., Tang, C., Shum, H.: Poisson Matting. ACM Trans. Graph. 23(3) (2004) 12. Tamaki, T., Suzuki, H.: String-like Occluding Region Extraction for Background Restoration. In: Proc. of the Int. Conf. on Patt. Recog. vol. 3, pp. 615–618 (2006) 13. Vasilevskiy, A., Siddiqi, K.: Flux Maximizing Geometric Flows. IEEE Trans. on Patt. Anal. and Mach. Intell. 24(12), 1565–1578 (2002)
Appendix: Implementation Details For the experiments shown here, we down-sample the original 6MP images to 334 by 502 pixels for segmentation and blur width estimation. Blur removal is
Automated Removal of Partial Occlusion Blur
281
performed on the original 6MP images. Based on this, blur estimation and image processing takes approximately 10 seconds (on a 3 GHz Pentium IV) to produce the output in Fig. 5. Other images take more or less time, depending on the size of the completely-occluded region. Readers should note that some of the code used in this implementation was written in Matlab, implying that the execution time could be further reduced in future versions. As outlined in section 4.2, we initially use a flux-maximizing flow to perform the segmentation, followed by a euclidean curve-shortening flow to produce a smooth contour. For the flux-maximizing flow, we evolve the level function with speed Δt = 0.1. This parameter was chosen to ensure stability for our 6MP images; in general it depends on image size. The evolution’s running time depends on the size of the foreground region. The curve evolution is terminated if it fails to increase the segmented area by 0.01% over 10 iterations. As the flux-maximizing flow uses an image-based speed term, we use a narrow-band implementation [10] with a bandwidth of 10 pixels.
High Capacity Watermarking in Nonedge Texture Under Statistical Distortion Constraint Fan Zhang, Wenyu Liu, and Chunxiao Liu Huazhong University of Science and Technology, Wuhan, 430074, P.R. China {zhangfan,liuwy}@hust.edu.cn
[email protected]
Abstract. High-capacity image watermarking scheme aims at maximize bit rate of hiding information, neither eliciting perceptible image distortion nor facilitating special watermark attack. Texture, in preattentive vision, delivers itself by concise high-order statistics, and holds high capacity for watermark. However, traditional distortion constraint, e.g. just-noticeable-distortion (JND), cannot evaluate texture distortion in visual perception and thus imposes too strict constraint. Inspired by recent work of image representation [9], which suggests texture extraction and mix probability principal component analysis for learning texture feature, we propose a distortion measure in the subspace spanned by texture principal components, and an adaptive distortion constraint depending on image local roughness. The proposed spread-spectrum watermarking scheme generates watermarked images with larger SNR than JND-based schemes at the same distortion level allowed, and its watermark has a power spectrum approximately directly proportional to the host image’s and thereby more robust against Wiener filtering attack.
1 Introduction Image watermarking is applied to copyright protection and covert communication. Efficiency of watermarking scheme can be measured by hiding capacity. Generally, a watermarking scheme achieves a high hiding capacity by qualifying the tradeoff between the achievable information-hiding rates and the allowed distortion constraint for the information hider and the attacker [1]. Specially, for additional watermarking scheme, e.g. spread-spectrum watermarking scheme [2], the embedding algorithm is designed to add maximal intensity watermark into the host image satisfying the distortion constraint for information hider. In this way, hiding capacity is restricted by distortion level allowed. Information-theoretical analysis [1] has proved the augment. A variety of distortion constraints have been proposed to incorporate certain psychovisual properties of the human visual system (HVS). The most popular method maybe is Just-noticeable-distortion (JND), which is firstly applied to image compression [3]. JND provides each signal a threshold level of error visibility, below which reconstructed signals are rendered without noticeable distortion. JND profile of a still image is a function of local signal properties, including background intensity, activity of luminance changes and dominant spatial frequency. For more accurate JND estimation, edge region and nonedge region should be distinguished [4]. Edge is Y. Yagi et al. (Eds.): ACCV 2007, Part I, LNCS 4843, pp. 282–291, 2007. © Springer-Verlag Berlin Heidelberg 2007
High Capacity Watermarking in Nonedge Texture Under Statistical Distortion Constraint
283
directly related to image content that demarcates object boundaries, surface crease, and reflectance change, and thus distortion around edge is easier to be noticed. Nonedge texture, therefore, can hide more error than smooth or edge areas. Psychovisual studies by Julesz [5] suggest that image textures with the same second-order statistics are perceived as identical by the human visual system. Among the work of texture synthesis, some synthesis-by-analysis methodologies [6~8] verify that texture can be reconstructed by matching features’ statistics, where the features are generally texture’s projection to a set of direction determined by a bank of suitable filters. Meanwhile, other resampling approaches [11,12] implies local neighborhoods can be alternated without inciting distortion. These works suggest that distortion constraint should tolerance local diversity but guard the global difference of statistics. This paper holds a hypothesis that local texture can be warped more intensely across the direction where the own texture has larger variance. The augment may be explained as (1) noise (watermark) having the same statistics with the texture will cause less perceptible distortion than white noise, and (2) when warped across the direction respect to large variance, the texture patch has more probability to resemble another patch somewhere, hence the warp is equivalent to alternation and may be allowable. Inspired by HIMPA (Hybrid ICA-Mixture of PPCA Algorithm) [9], which use an independent component analysis (ICA) model for edge representation followed by a mixture of probabilistic principal components analyzers (MPPCA) for textural surface representation, this paper proposes (1) a nonedge texture extraction method by a three-factor image model, and (2) a texture distortion measure based on its principal components and an adaptive distortion constraint depending on image local roughness, so that we can embed spread-spectrum watermark with maximal intensity into the nonedge regions of the host image to achieve a high capacity watermarking scheme. The watermarking scheme is introduced in section 2. Selected experimental results are presented in section 3. The paper closes with concluding remarks in section 4.
2 Proposed Scheme Our scheme is shown in Figure 1. Without loss of generality, we embed one bit of message m whose value is either –1 or +1 into image blocks vector x. Corresponding to secret key k, p is a pseudorandom noise sequence with zero mean, and its values are equal to +1 or –1. Modulated by m, the sequence p is weighted added to or subtracted from x to form the watermarked image block vector y as
y = x + mdiag( p)γ .
(1)
where diag(A) is a diagonal matrix having vector A as its diagonal. The pixelwise weighted coefficient γ represent the intensity of watermark, and it is maximized under the adaptive distortion constraint as formula (2), which will be analyzed in section 2.1. d ( y, x ) < Dw .
(2)
The watermarked image is possibly corrupted by an attacker’s noise. We only consider the additive noise, as formula (3)
284
F. Zhang, W. Liu, and C. Liu
z = y + n.
(3)
The receiver knows the secret key k and can recover the p, and then the detection is performed. Firstly, the (normalized) correlation is calculated as
r =< z,p >= m
∑ γ + < x ,p > + < n ,p > .
(4)
where
denotes correlation, i.e., inner product of sequence A and sequence B. The two latter terms in (4) can be neglected due to independency between x and p and independency between n and p, then watermark is estimated as the sign of r mˆ = sign (r ).
key adaptive distortion constraint
(5)
PRN
PRN
γ
p
p m y
x
Nonedge texture extraction
key
z
correlation detector
mˆ
n channel & attack noise Fig. 1. The proposed watermarking scheme
2.1 Nonedge Texture Extraction
For extraction of nonedge texture, we propose a three-factor image model. Three factors are assumed to contribute to the appearance of nature image: (a) objects’ intrinsic luminance and shading effects, (b) dominant edges, and (c) nonedge texture. Different from HIMPA model [9], our model isolates low frequency band of image, xL, as the factor (a), and uses HIMPA model to analyze the high frequency band xH, so as to discriminate factor (b) and factor (c). Factor (b) is edge region of xH while factor (c) is the surface region. The reason for isolating factor (a) is that clustering of MPPCA will not be affected by local average luminance, and thereby will depend more on local contrast. According to HIMPA [9], we use independent component analysis (ICA) to extract nonedge texture from the high frequency band of image. An image patch xH is represented as a linear superposition of columns of the mixing matrix A and a residue noise which can be neglected. x H = As.
(6)
The columns of A are learned from nature images by making the components of the vector s statistically independent. In the learning algorithm, A is constrained to be an orthogonal transformation. So the vector s is calculated by
High Capacity Watermarking in Nonedge Texture Under Statistical Distortion Constraint
285
s = AT x H .
(7)
The elements of s with large magnitudes signify the presence of structure in an image block, corresponding to edges, oriented curves, etc. The elements of s with smaller magnitudes represent the summation of a number of weak edges to yield nonedge texture. For each image block, we divide A into two groups, AE and AT, keeping all the absolute values of s corresponding to columns of AE are larger than AT. The image block is also decomposed into two subcomponents. xH = xE + xT .
(8)
x E = AE ( AE T AE ) −1 AET x H .
(9)
where
The ratio between the column amounts of AT and A is determined by ICA threshold. In this paper, ICA threshold is 0.375 and A has 160 columns, therefore AE has 100 columns, while AT has 60 columns. Due to ratio control by the ICA threshold, nonedge texture is the residual after removing the relatively dominant edge instead of the absolutely sharp edge in HIMPA [9], therefore, much intenser nonedge texture will be extracted from the rough region than the smooth region. Figure 2 illustrates the process of nonedge texture extraction. 2.2 Statistical Distortion Measure
The basic idea of the proposed measurement is to valuate distortion in the subspace which is spanned by the texture’s principal components. PPCA is a suitable stochastic model for a homogenous nonedge region as suggested by HIMPA [9]. MPPCA is able to model more complex nonedge regions in a real image scene due to utilization of clustering. We describe the salient feature of MPPCA model and the detail is contained in [9]. The nonedge texture can be assumed to sample from a limited number of clusters. Each cluster is assumed homogenous, and efficient basis can be constructed, where a texture block from cluster k is generated using the generative model
∈
x k = W k s k + μ k + ε k , k = 1,..., K .
(10)
where xk Ra is a column vector elongated from the host image block and has a dimension of a, sk Rq is the lower dimensional source manifold assumed to be Gaussian distributed with zero mean and identity covariance. Note that, in this section, image block always refers to the nonedge texture block. The dimension of sk is q and a > q. Wk is a mixing matrix of a by q. μk is the cluster observation mean, εk is Gaussian white isotropic noise with zero mean, i.e. εk ~ N (0, σ2I). Hence, Ra conforms to a distribution N (μk, WkWkT+σ2I). Wk, Rq, μk, and εk are hidden variables that need to be estimated from the data of observed texture. μ and columns of W in one cluster are visualized as 8×8 block at the right bottom of Figure 2.
∈
286
F. Zhang, W. Liu, and C. Liu
xT
MPPCA analyzing
ICA xH
coding
averaging x
host image
xE
filtering classified 8×8 blocks xL μk
Wk
Fig. 2. Three-factor model for nonedge extraction and features learned by MPPCA
We define the distortion between image block vector x and y as their quadratic,
d ( y, x ) = E xy2 = ( x − y )T C −1 ( x − y )
.
(11)
where C is covariance of the maximum-likelihood cluster which x belongs to, and
C = σ 2 I + WW T
(12)
Formula (11) can be transformed into a standard quadratic form according to [10]
d ( y, x ) = v T D −1v
(13)
where v is the projection of x – y, v = UT(x – y). If sample covariance matrix learned in MPPCA has an eigenvalues matrix arranged in descending order denoted by diag(λ1, λ2, …, λq, …, λa), U is the corresponding eigenvectors matrix, D–1 is a diagonal matrix, noted as diag(1/λ12, 1/λ22, …, 1/λq2, 1/σ2,…, 1/σ2), and σ is obtained by averaging the last a – q eigenvalues. Similar with principal component analyzer (PCA), the transform UT projects the observation into a set of orthogonal directions, across which the observations have a large variation. In fact, formula (13) is equal to mean square error metric, when D is a unit matrix. So the measurement is a general case of mean square error metric. It is noteworthy that formula (13) looks like Karhunen Loeve Transform (KLT) norm, but substantial difference is that formula (13) is a measurement in an orthogonal space estimated by the whole observations’ covariance, instead of a single observation’s covariance. Therefore, formula (13) is a statistical measurement. 2.3 Adaptive Distortion Constraint
We define adaptive distortion constraint by setting Dw as Dw = α ( x T − μ) T C −1 ( x T − μ).
(14)
High Capacity Watermarking in Nonedge Texture Under Statistical Distortion Constraint
287
where α is a positive coefficient scaling the upper limit of distortion, μ is mean of the maximum-likelihood cluster which xT belongs. Because xT more tends to μ in the smooth region than in the rough region, Dw for smooth region will be generally smaller than for rough region. Therefore Dw is loosened for rough region which is often full of intenser texture. According to (4), optimization of watermark hiding capacity is to maximize sum of γ under quadratic inequality constraint from (2) and (13). By Lagrange multiplier, we obtain the solution,
γ~ =
Dw diag ( p)Cp . pT Cp
(15)
We also consider luminance masking effect in JND theory, and the luminance masking TL conforms to definition in [4]. So the final solution γ for each pixel is
γ = γ~ + TL .
(16)
3 Experimental Results ICA mixing matrix is estimated beforehand using samples from a training set of 13 natural images downloaded with the FastICA package [17]. Note that ICA mixing matrix is independent with host images and need not be held by watermark detector. At the watermark encoder, after 8×8 averaging filtering and extracting by ICA coding, nonedge texture is obtained and then, partitioned into 8×8 blocks and vectorized into 64×1 vectors. Parameters of MPPCA model has eight clusters with 4 principal components in each cluster, where the program code was developed based on the NETLAB Matlab package [13]. Then, adaptive distortion constraint Dw and the maximal intensity γ for each pixel is calculated, considering luminance masking of JND. Finally, spread-spectrum watermark is embedded. At the watermark decoder, only correlation detection is enough to extract the watermark message without any image analysis or attack estimation. So the proposed scheme is a blind watermarking scheme without need of any side information. 3.1 Distortion We compare our scheme with the spread-spectrum watermarking schemes based on spatial JND model [4] and wavelet JND model [14]. We measure the watermark intensity by signal-to-noise-ratio (SNR) of watermarked image related to the host image. SNR is also used to measure indirectly the hiding capacity for spread-spectrum watermark because intenser watermark scheme supports higher bit ratio of watermark to host image under the same detecting error rate at the watermark decoder. Our experiment makes the watermarked images of three schemes have the same SNR (Baboon 20.1dB, Bridge 21.7dB, and Lena 27.3dB) by adjusting their parameters, so that experimenters can make subject distortion evaluation and further compare the distortion levels allowed under different constraints. The watermarked image patches are shown in Figure 3.
288
F. Zhang, W. Liu, and C. Liu
Baboon
Bridge
Lena (a)
(b)
(c)
(d)
Fig. 3. Watermarked patches. (a) host image. (b) our scheme, (c) spatial JND, (d) wavelet JND.
(a)
(b)
(c)
Fig. 4. Watermark of our scheme. The top row contains host images, and the bottom contains watermarks, where dark pixels denote the negative signals and the light denote the positive.
In the experiment, two JND-based schemes expose more distortion than our scheme when embedding saturated watermark. Spatial JND scheme (Figure 3c) reveals noticeable noise either across sharp edges or at smooth surfaces. Wavelet JND
High Capacity Watermarking in Nonedge Texture Under Statistical Distortion Constraint
289
scheme (Figure 3d) fails to keep image quality around fine edges, which correspond to high frequency wavelet bands. Moreover, wavelet JND scheme generates glaring dark or light speckles in texture region, where, in fact, inverse wavelet transform of the coefficients watermarked exhaustedly is overbound. Our scheme has followed advantages. Firstly, it does not incite watermark intensity at sharp edges due to the three-factor image coding before watermarking. Secondly, it prefers to embed watermark into rough region rather than smooth region due to the adaptive distortion constraint Dw. Lastly, it activates the intensity of watermark conforming to the principal components of texture and inhibit the unconformable watermark. The characters are clearly exhibited in the watermark signal as shown in Figure 4. Since our scheme exploits the texture region, its advantages are not obvious for images with sparse texture, e.g. image Lena. 3.2 Robustness Against Estimation-Based Attack Sophisticate attackers can make estimation-based attack if they can obtain some prior knowledge of host image or watermark’s statistics [16]. Image denoising provides a natural way to develop estimation-based attacks, where watermark is treated as a noise. Given the power spectrum of host image and watermark, one of the most malevolent attacks is the denoising method by adaptive Wiener filter. So the most robust watermark should have a power spectrum directly proportional to the power spectrum of the host image [16]. We compare the power spectrum of watermark by three schemes in Figure 5.
(a)
(b)
(c)
(d)
Fig. 5. Power spectrum of image Baboon by Fourier analysis. (a) host image. (b) ~ (d) watermark signals, (b) our scheme, (c) spatial JND scheme, (d) wavelet JND scheme. Signal is normalized to 0 mean and unit variance before Fourier analysis. The luminance denotes logarithmic amplitude of Fourier components. Zero-frequency component is shifted to the center.
It is clear that spatial JND scheme generates a nearly white watermark, and wavelet JND scheme embed watermark focusing on the middle frequency components. Among the three schemes, watermark power spectrum of our scheme mostly resembles the power spectrum of the host image, and it is nearly directly proportional to the power spectrum of the host image. So our scheme gives a suboptimal watermarking solution against the estimation-based attack of Wiener filter.
4 Conclusion The proposed watermarking scheme can hide more information than traditional schemes because it loosens distortion constraint for nonedge texture, and it is also
290
F. Zhang, W. Liu, and C. Liu
robust against Wiener filtering attack due to the watermark power spectrum resembling the host image. Meanwhile, the watermark detection can be blind and fast. High-capacity watermark should qualify the tradeoff between the bit ratio of watermark to host image and the error rate of detector. Although this paper guides a one-bit watermarking scheme with maximal watermark intensities, it still provides a potential design for high-capacity watermarking scheme, since multiple-bit watermark is to choose a suitable number of image blocks for embedding each one bit of watermark. Our schemes only consider the attack of additional noise. Some well known strategies against geometric attack and some smart detection algorithms may be merged into the proposed scheme so as to resist more sophisticate attacks. Acknowledgement. This work is supported by NSFC (NO: 60572063), Doctoral Fund of Ministry of Education of China (NO: 20040487009), and the Cultivation Fund of the Key Scientific and Technical Innovation Project Ministry of Education of China (NO: 705038). The author would like to thank Nikhil Balakrishnan and Yizhi Wang for discussions.
，
References 1. Mounlin, P., O’Sullivan, J.A.: Information-theoretic analysis of information hiding. IEEE Trans. Information Theory 49, 563–593 (2003) 2. Cox, I.J., Kilian, J., Leighton, F.T., Shamoon, T.: Secure spread spectrum watermarking for multimedia. IEEE Trans. Image Proc. 6, 1673–1687 (1997) 3. Jayant, N.: Signal compression: Technology targets and research directions. IEEE J. Select. Areas Commun. 10, 314–323 (1992) 4. Yang, X.K., Li, W.S., Lu, Z.K., et al.: Motion-compensated residue preprocessing in video coding based on just-noticeable-distortion profile. IEEE Trans. Circuits & System Video Tech. 15, 742–752 (2005) 5. Julesz, B.: Visual pattern discrimination. IRE Trans. Inf. Theory 8, 84–92 (1962) 6. Heeger, D., Bergen, J.: Pyramid-based texture analysis/synthesis. In: Proc. ACM SIGGRAPH, pp. 229–238 (1995) 7. Zhu, S.C., Wu, Y.N., Mumford, D.B.: Filter, Random fields, and Maximum Entropy (FRAME) –Towards a Unified Theory for Texture Modeling. Int’l Journal of Computer Vision 27, 107–126 (1998) 8. Portilla, J., Simoncelli, E.P.: A parametric texture model based on joint statistics of complex wavelet coefficients. Int’l Journal of Computer Vision 40, 49–71 (2000) 9. Balakrishnan, N., Hariharakrishnan, K., Schonfeld, D.: A new image representation algorithm inspired by image submodality models, redundancy reduction, and learning in biological vision. IEEE Trans. Pattern Analysis & Machine Intelligence 27, 1367–1378 (2005) 10. Tipping, M., Bishop, C.: Mixtures of Probabilistic Principal Component Analyzers. Neural Computation 11(2), 443–482 (1999) 11. Lefebvre, S., Hoppe, H.: Parallel controllable texture synthesis. In: ACM SIGGRAPH, pp. 777–786 (2005) 12. De Bonet, J.S.: Multiresolution sampling procedure for analysis and synthesis of texture images. In: ACM SIGGRAPH, pp. 361–368 (1997)
High Capacity Watermarking in Nonedge Texture Under Statistical Distortion Constraint
291
13. Nabney, I., Bishop, C.: Netlab Neural Network Software (July 2003), http://www.ncrg. aston.ac.uk/netlab 14. Huang, X.L., Zhang, B.: Perceptual watermarking using a wavelet visible difference predictor. IEEE ICASSP 2, 817–820 (2005) 15. Liu, W.Y., Zhang, F., Liu, C.X.: Spread-Spectrum Watermark by Synthesizing Texture. In: Pacific-Rim Conf on Multimedia (to be published, 2007) 16. Voloshynovskiy, S., Pereira, S., Pun, T., et al.: Attacks on Digital Watermarks: Classification, Estimation-Based Attacks, and Benchmarks. IEEE Communications Magazine 39, 118– 126 (2001) 17. Hyrarinen, A.: Fast ICA Matlab package (April 2003), http://www.cis.hut.fi/projects/ica /fastlab
Attention Monitoring for Music Contents Based on Analysis of Signal-Behavior Structures Masatoshi Ohara1,2, Akira Utsumi1 , Hirotake Yamazoe1 , Shinji Abe1 , and Noriaki Katayama2 ATR Intelligent Robotics and Communication Laboratories 2-2-2 Hikaridai, Seikacho, Sorakugun, Kyoto 619-0288, Japan 2 Osaka Prefectural College of Technology 26-12 Saiwaicho Neyagawashi, Osaka 572-8572, Japan
1
Abstract. In this paper, we propose a method to estimate user attention to displayed content signals with temporal analysis of their exhibited behavior. Detecting user attention and controlling contents are key issues in our “networked interaction therapy system” that effectively attracts the attention of memory-impaired people. In our proposed method, user behavior, including body motions (beat actions), is detected with auditory/vision-based methods. This design is based on our observations of the behavior of memory-impaired people under video watching conditions. User attention to the displayed content is then estimated based on body motions synchronized to auditorial signals. Estimated attention levels can be used for content control to attract deeper attention of viewers to the display system. Experimental results suggest that the proposed method effectively extracts user attention to musical signals.
1
Introduction
Human behavior is considered to have a close relation with mental and/or physical states, intention, and individual interests. For instance, when a person is shopping, human eye movement may provide significant information regarding interest in specific products in the shop. Since the issue of human state/intention estimation based on behavior can be related with a wide range of application domains, estimation mechanisms have been widely investigated in the field of computer vision and pattern recognition [1,2]. However, generally speaking, achieving reliable behavior-based state/intention estimation is hard without any domain specific knowledge because there are a wide variety of human behaviors and environment/context dependencies. On the other hand, estimating human attention and interest to a specific stimuli is relatively easy when the timing, position, and content of the stimuli are known. In this paper, we propose a system to estimate viewer’s interest in a displayed content based on viewer behavior. We focus on a method to estimate viewer attention levels to music contents by observing the synchronous behavior of users to the music signals. In a human-computer interaction task, for instance, attracting and retaining the motivation of users becomes significant for extracting positive reactions. To Y. Yagi et al. (Eds.): ACCV 2007, Part I, LNCS 4843, pp. 292–302, 2007. c Springer-Verlag Berlin Heidelberg 2007
Attention Monitoring for Music Contents
293
achieve this, the system has to estimate individual concentration levels and control the style and amount of displayed information. Since reactions vary from user to user, such control must occur dynamically. The same situation prevails in our “networked interaction therapy system,” which effectively attracts the attention of memory-impaired people and lightens the burden of helpers or family members [3]. “Networked interaction therapy” requires that the system provide remote communication with helpers and family members as well as video contents and other services. To attract the attention of users for long periods of time, the system has to detect user behaviors and control the order and timing of provided services based on estimates of their concentration levels. Before the above research, we presented several video contents to memoryimpaired people and analyzed their behaviors when they watched video and music contents. We observed the following positive reactions to video and musical contents: head nodding, laughing, singing, and clapping. Based on the results, we implemented a content switching mechanism based on user head movements. Then we estimated user concentration levels to the displayed video contents due to head directions and controlled the switching of contents to retain more user attentions to the video. However, since human behavior can change quickly based on content categories, such a tendency is remarkable for musical contents that do not require user attention to the video display. Therefore, in this paper, we examine the possibility of estimating viewer concentration levels to presented musical contents using user “beat” actions. “Beat” is a basic property by which humans recognize music, and there are close relations between music’s “beat” and human behavior. For example, Shiratori et al. analyzed the relation between the “beat” structure in music and human dance behavior. Since synchronized motion to music is a commonly observed human behavior when listening music, it should function as a useful observable feature to estimate user attention to and interest in music. As we mention below, synchronized motion to music contents is behavior that is not always observed for all users. However, we still consider that this useful cue complements other cues such as head movements in concentration level estimation. In the next section, we briefly summarize our “networked interaction therapy” project. Section 3 describes our observations of the TV watching behavior of dementia patients. Section 4 explains the framework of our attention estimation, and Section 5 shows the experimental results. In Section 6, we conclude this paper.
2
Information Therapy Project
Memory is frequently impaired in people with such acquired brain damage problems as encephalitis, head trauma, subarachnoid haemorrhage, dementia, and cerebral vascular injury. Such people have difficulty leading normal lives due to memory impairment or higher brain dysfunction; consequently, constant care and attention impose a heavy burden on their families. Networked Interaction Therapy, the name of our method that relieves the stress suffered by memory-impaired
294
M. Ohara et al.
people and their family members, provides easy access to the services of networked support groups [3]. The main goals of Networked Interaction Therapy include supporting the daily activities of memory-impaired people and reducing the burden on their families. Its primary function is to automatically detect the intentions of a memoryimpaired person. This therapy provides the necessary information to guide the individual to comfortable situations on behalf of family members before beginning to experience behavioral problems such as wandering at night, incontinence, temper tantrums, and so on. These anxieties are often caused by a lack of information. A system based on Networked Interaction Therapy needs to detect situations where communication can help the individual overcome the difficulties encountered in daily activities. In this paper, we describe a method to estimate user attention/interest levels to presentation contents. For instance, estimation results can be used for effectively switching contents to hold user interest/attention. In the next section, we give some typical behavior of memory-impaired people that was observed through experiments using video contents display.
3
Video Watching Behavior of Dementia People
Before system implementation, we examined the behavior of actual dementia patients watching TV through observation experiments [4]. Here we briefly summarize the results. In the experiments, we prepared both interesting/uninteresting video programs for subjects based on their preexamined preferences and personalized reminiscence video produced by photographs selected from their photo albums [5]and sequentially displayed such video contents to them. We performed experiments with eight subjects including three subjects (A, B, C) living at home with their families and five in nursing facilities (D-H). Table 1 shows brief profiles of the subjects. Though observed behavior depends on individuals and their symptom levels, behavior where subjects look away from the TV due to “uninterested” contents is very common. We observed significant differences of watching time for particular subjects. Based on this observation, we consider face orientation a crucial behavior replated for attention/interest to video contents. The following are additional typical behavior related to level of attention/ interest. Example of uttterances: – presenting positive or negative utterances to displayed video contents. – explaining the contents of the reminiscence video to family members. – singing songs from the video. From subject utterances, both positive and negative words are observed. Thus, it is difficult to estimate subject interest to the displayed contents only from the existence of utterances.
Attention Monitoring for Music Contents
295
Table 1. Brief subject profiles Subjects A B C D E F G H
Age 62 81 69 83 90 89 89 92
Case History Preference cerebral contusion Japanese chess, music Alzheimer’s disease train travel, music cerebral infarction baseball games, music cerebral dementia children’s songs senile dementia N/A Alzheimer’s disease movies Alzheimer’s disease music senile dementia N/A
Example of hand motions: – beating time with hands to music contents. – pointing at TV while viewing reminiscence videos and news programs. The “beating time with hands to music” action is considered a typical positive reaction to the displayed contents. However subject A, for instance, presented the “beat” action without gazing at the TV and left the room just after making this reaction. In addition, in one case, a “pointing” action was observed with a negative utterance to the contents. So hand motions can be either “positive” or “negative” reactions. Therefore, estimating user concentration levels is difficult from only the existence of hand motions. The entire summary of observations is as follows. – Effective content switching attracts user attention to the displayed contents longer and can be realized by measuring user attention levels with face directions. – Although user utterances contain information regarding user interest in the displayed contents, estimating the user interest level is hard from only the presence of utterances. – Although frequency depends on individuals, such reactions as hand beckoning, clapping, and singing were observed while subjects watched the video contents. The above results suggest that interest in user contents can be estimated from the direction of user faces, the content of utterances, and hand motions. We previously developed a content switching system based on user face orientations and confirmed its effectiveness through experiments. However, in some cases, estimating user interest levels is difficult from face orientations, such as music contents that can be enjoyed without being constantly watched. Therefore, the following sections examine a method to estimate user interest levels from another cue, i.e., synchronous body motions to the displayed music signals.
296
4
M. Ohara et al.
Attention Monitoring Using Image and Audio Analysis
From the results of the previous section, we found that laughing, singing, and clapping can be observed as “positive” reactions. Therefore, we developed a system to estimate user attention to the displayed music contents based on synchronization between user behavior and displayed music contents. Here, the “tempo” of user behavior is extracted with image and auditorial analysis. Assuming a known temporal structure for music signals like MIDI signals, we can directly compare user behavior with such structures. However, since temporal structures are generally unknown for normal TV/radio programs, CDs, DVDs, etc., we first extract the “beat” structures from music signals through frequency analysis and then compare them with viewer body motions. 4.1
System Configurations
Figure 1 shows a diagram of the proposed system. The music signal output from a speaker system is simultaneously taken to the system with an A/D converter, where it is then processed with FFT to extract the frequency elements and sent to the beat extraction process. During beat extraction, a voting process determines the dominant (fundamental) tempo value of the input music signal. User cyclic behavior is detected as follows. User motions are observed through both image and audio analysis. Image analysis detects user movements as numbers of fluctuated pixels in inter-frame subtractions, and user “beat” actions can be extracted as cyclic changes. Audio analysis extracts sound related with user motions (e.g., clapping) from input audio signals with a subtraction process of original music signals through an LMS algorithm. Then the “beat” of the user motion is determined from the extracted sound signals. Finally, we compare the tempos of user behavior and music signals and estimate the attention (concentration) level of users to the music contents by the degree of synchronicity. Sections below explain the details of the above processes.
♪ Music
Motion
Music Data
FFT
Beat Extraction
FFT
Beat Extraction
Rhythm Estimation
Speaker
Microphone
Clapping Noise Extraction
Frame Subtraction
Beat Extraction
Camera
Fig. 1. System diagram
Rhythm Estimation
Attention Monitoring for Music Contents
297
Frequency 9 6 0-250Hz
3 250-500Hz
0
500Hz-1KHz
1-2KHz
25
40
2-4KHz 0
100
200
300
400
500
[frame]
Fig. 2. Input music signal and results of frequency analysis
4.2
60
30
35
20 15 PHASE 80 100 10 5 120 BPM 140 160 0
Fig. 3. Voting results for tempo estimation (Music No. 3)
Beat Detection
First, we describe how to detect the beat information in a music signal. The proposal method employs a frequency-based beat extraction algorithm [6,7]. Here, the input signal is converted into frequency domain with FFT analysis every 1/32 sec. It separates the input signal into the following five frequency bands: 0-250 Hz, 250-500 Hz, 500-1 kHz, 1-2 kHz, and 2-4 kHz. We detect the envelope of each frequency band and extract the “beat” as common rising points in multiple frequency bands. Figure 2 shows the example results for a pop song. 4.3
Tempo Estimation
In this section, we explain our tempo estimation method. Current implementation assumes that the music tempo stays in a range of 60 to 145 bpm (the range most pop music falls), and the tempo and the phase of the “beat” are determined using a voting algorithm. Figure 3 shows the result for a country song of 81 bpm. 4.4
Detection of Body Behavior by Image
For body motion detection, a silhouette-based method [8,9], a model-based method [10], etc. have previously been proposed. However, since it is difficult to specifically target a body part for body motion extraction in advance and only the relatively lower frequency one-dimensional signal must be extracted in “beat” extraction, we employed simple motion analysis based on inter-frame extraction. Here, the magnitude of motion is measured as the number of moving pixels Nt . “Beat action” in which the user takes hand and feet action can be detected as the times when number Nt of the pixels becomes minimum: dNt−1 t 1 ( dt < 0, dN dt 0 and Nt 0) beat = (1) 0 (otherwise).
298
M. Ohara et al.
(1) Original Music Signal 200 100 0 0
100
200
300
400
500
[frame]
-100 -200
(2) Sound data captured with microphone 200 100 0 -100 -200 0
100
200
300
400
500
[frame]
(3) Extracted user origin sound 200 100 0 -100 -200 60000 65000 70000 75000 80000 85000 90000 95000 100000 0
100
200
300
400
500
frame
[frame]
Fig. 4. Examples of “beat” action extractions (top: hand, middle: foot, bottom: head)
Fig. 5. Separation of user origin sound through LMS algorithm
Figure 4 shows an example of an extracted beat action. Here, the person was expressing the beat by using hand (left), foot (middle), and nodding (right). 4.5
Extraction of User Origin Sound
Audio signal observed contains the displayed music signal output from the speaker in addition to the sound related with user actions such as clapping. Here, we separate these signals by an LMS algorithm [11] that is commonly used for echo canceling in TV conferencing systems. Generally, there are multiple reflection paths for sound output from a speaker before it reaches a microphone. The LMS algorithm models this echo path as an linear system and determines its impulse response with a steepest descent method as follows: hN (k + 1) = hN (k) + αe(k)x(k − N ).
(2)
Here, x(k) is observation at time k. hN (k) is a N -th coefficient at time k. e(k) is estimation error for an observation value at time k. α is a step gain. We estimate the music signal level in an input audio signal through the estimated model and extract sound related to user motions by a subtraction process. Figure 5 shows the results of the extraction process. Figure 5 (top) shows the original music signal. Figure 5 (middle) shows the observed signal with a microphone. Figure 5 (bottom) shows the subtraction results based on an LMS algorithm. As can be seen, the sound signal related with user motion is properly extracted by the above process. We estimate the “tempo” of user motions with a similar process in Section 4.2.
Attention Monitoring for Music Contents
4.6
299
Synchronous Judgment
Finally, we judge the synchronicity between the estimated tempos of the music signal and user “beats.” First, we estimate the “tempo” of user motions by applying the voting algorithm described in Section 4.3 to the “beat” information (image and sound). Then, we compare the estimation results with music “tempo,” and if the difference is less than 10 bpm, we determine that both signals are synchronized. As mentioned in the next section, we estimate the song to which a user is listening as one that has a tempo difference with user motions less than 10 bpm and the difference is minimum in the multiple candidates (if any). ⎧ |bmusic,i − b| < threshold and ⎨ i |bmusic,i − b| < |bmusic,j − b|(j = i) result = (3) ⎩ −1 otherwise. In addition, there is a case where a user’s “beat” action happened once every two beats of the music signal. Therefore, we may have to consider two signals synchronized if the tempo of one signal is an integral multiple of another.
5
Experiment Results
First, the example of extracting user “beat” motion was synchronized to displayed music signals. Figure 6 shows the voting results of tempo extraction for clapping while listening to the music of Figure 3 (Figure 6 left) and random clapping (Figure 6 right). As can be seen, the “beat” action case has a clear peak signal, although the non-“beat” action case does not. This suggests that the presence of synchronized behavior can be distinguished from others. Figure 7 shows the difference of the number of detected synchronous “beats” in “beat” and non-“beat” actions for a longer sequence. Since non-“beat” action does not have a cyclic property and is not synchronized to the music signal, “beat” and non-“beat” motions can be distinguished with the proposed method. Next, we performed the following experiments to confirm the possibility of behavior-based estimation of user attention to music signals. We prepared five Frequency
Frequency 9
9
6
6
3
3 0
0 30
40
60
80 100 10 5 120 140 BPM 160 0
Beat action
35
25 20 15 PHASE
40
60
80 100 10 5 120 140 BPM 160 0
non-beat action
Fig. 6. Results of beat and non-beat actions (body motion)
35 30 25 20 15 PHASE
300
M. Ohara et al.
Number of Sync. Beats Per Minute
75 70 65 60 55 50 45 40 35 30
Beat Action
Random
Fig. 7. Discrimination of beat/non-beat actions Table 2. Genre and extracted tempo for sample music Music No. Genre Tempo extraction results [bpm] 1 country 113 fusion 74 2 country 81 3 Japanese pop 130 4 popular 113 5
music signals. Table 2 shows their genres and estimated tempos that are distributed from 74 to 130 bpm. Music 1 and 5 have similar tempos. We employed five subjects (healthy adults) who listened and clapped to the displayed music. The tempos of their body motions were estimated by the proposed method. Table 3 shows the tempo of subject 1’s motion estimated with both image and sound cues. We obtained similar estimation results for both cues. Table 4 shows the estimation results of all subjects (sound cues). Here, denotes cases where the system correctly estimated the music from user behavior. × denotes cases where estimation failed, and denotes others. In the following, we discuss the “” cases. Music 1 has two instruments related to “beat.” The extracted tempo changes along with the instrument’s part to which the user clapped. As in the above case, the extracted tempo may change depending on the part (instrument) to which the user is attracted. The result for music 2 resembles that of subject 3. Therefore, we may have to consider two signals synchronized if the tempo of one signal is a integral multiple of another. On the other hand, since music 1 and 5 have similar tempos, deciding what song the user is listening to is difficult. However, since the main purpose of the proposed method is to estimate attention levels to currently displayed music contents, synchronization with other music does not matter. Consequently, we accurately estimated synchronous properties between user “beat” actions and displayed music in more than 90% of all cases. This suggests the effectiveness of the proposed method for estimating user attention levels to displayed music contents.
Attention Monitoring for Music Contents
301
Table 3. Tempo extraction results (subject 1, image, and sound cues) Music No. 1 2 3 4 5 tempo extracted with image cues 61 77 88 130 130 tempo extracted with sound cues 61 77 81 130 113 Table 4. Tempo extraction for all subjects (sound cues) Music No. Subject 1 Subject 2 Subject 3 Subject 4 Subject 5
1 61 67 60 61 61
2 77 74 145 77 74
3 81 130 × 81 81 81
4 130 64 × 135 135 135
5 113 61 61 113 113
Furthermore, since two false cases belong to the same subject, perhaps the temporal accuracy of this subject’s clapping was less than others. We must investigate this point in the future.
6
Summary
In this paper, we examined a method to estimate user interest in music contents from the synchronicity of the presented music and human behavior. The results are presented while switching two or more video contents for memory-impaired people and observing user attention behavior. User reactions differ based on presented contents: clapping, speaking, and sideways turning of the face. We examined an effective content switch technique for reflecting user intention. Future work includes improvements of the estimation accuracy of user interests by integrating other user behaviors with beat actions and developing a contents control system to switch displayed contents based on user attention levels to entertain users longer. This research was supported in part by the National Institute of Information and Communications Technology.
References 1. Boiman, O., Irani, M.: Detecting irregularities in images and in video. In: Proceedings of International Conference on Computer Vision, pp. 462–469 (2005) 2. Osawa, T., Wu, X., Wakabayashi, K., Yasuno, T.: Human tracking by particle filtering using full 3d model of both target and environment. In: Porceedings of International Conference on Pattern Recognition, pp. 25–28 (2006) 3. Kuwahara, N., Kuwabara, K., Utsumi, A., Yasuda, K., Tetsutani, N.: Networked interaction therapy: Relieving stress in memory-impaired people and their family members. In: Proc. of IEEE Engineering in Medicine and Biology Society, IEEE Computer Society Press, Los Alamitos (2004)
302
M. Ohara et al.
4. Utsumi, A., Kawato, S., Abe, S.: Attention monitoring based on temporal signalbehavior structures. In: Sebe, N., Lew, M.S., Huang, T.S. (eds.) Computer Vision in Human-Computer Interaction. LNCS, vol. 3766, pp. 100–109. Springer, Heidelberg (2005) 5. Kuwahara, N., Kuwabara, K., Tetsutani, N., Yasuda, K.: Reminiscence video helping at-home caregivers of people with dementia. In: HOIT 2005, pp. 145–154 (2005) 6. Scheirer, E.D.: Tempo and beat analysis of acoustic musical signals. Journal of Accoustical Society of America 103(1), 588–601 (1998) 7. Goto, M.: An audio-based real-time beat tracking system for music with or without drum-sounds. Journal of New Music Research 30(2), 159–171 (2001) 8. Kanade, T., Rander, P., Narayanan, P.J.: Virtualized reality: Constructing virtual worlds from real scenes. IEEE MultiMedia 4(1), 34–47 (1997) 9. Waggg, D.K., Nixon, M.S.: On automated model-based extraction and analysis of gait. In: Proc. of the 6th IEEE International Conference on Automatic Face and Gesture Recognition, pp. 11–16. IEEE Computer Society Press, Los Alamitos (2004) 10. Lim, J., Kriegman, D.: Tracking humans using prior and learned representations of shape and appearance. In: Proc. of the 6th IEEE International Conference on Automatic Face and Gesture Recognition, pp. 869–874. IEEE Computer Society Press, Los Alamitos (2004) 11. Widrow, B., Hoff, M.E.: Adaptive switching circuit, 96–104 (1960)
View Planning for Cityscape Archiving and Visualization Jiang Yu Zheng and Xiaolong Wang Department of Computer Science Indiana University Purdue University Indianapolis (IUPUI), USA
Abstract. This work explores full registration of scenes in a large area purely based images for city indexing and visualization. Ground-based images including route panoramas, scene tunnels, panoramic views, and spherical views are acquired in the area and are associated with geospatial information. In this paper, we plan distributed locations and paths in the urban area based on the visibility, image properties, image coverage, and scene importance for image acquisition. The criterion is to use a small number of images to cover as large scenes as possible. LIDAR data are used in this view evaluation and real data are acquired accordingly. The extended images realize a compact and complete visual data archiving, which will enhance the perception of spatial relations of scenes. Keywords: Multimedia, panoramas, visibility, urban space, LIDAR data, model based vision, heritage.
1 Introduction This work explores a framework to represent spaces with images. With densely taken images in a space, one can establish a complete database for scene access. This image registration task is required at excavation sites, museums, markets, real estate property, heritage districts, and large urban areas. Moreover, a full archive of scenes will allow navigation, location finding, guidance, crisis preparing, etc. Employing 3D models, either manually constructed or automatically extracted from LIDAR (Light Detection and Ranging) data, have been a long-standing trend in showcasing an urban area [6]. The process is usually laborious particularly in the model construction and texture mapping. To obtain higher resolutions and more proper viewing angles than overhead images (satellite or aerial) and map, we emphasize ground-based-views in this work. Although video clips can capture walkthrough views continuously [12], they have not been extended to large areas because of the vast data size. Mosaicing translating images is also difficult because of the disparity inconsistencies from the depth variations [8]. A multi-perspective image has been composed only for scenes close to a planar surface [1]. On the other hand, the slit scanning approaches including push-bloom [14] and X-slit [9][16] have been implemented to create route panorama in order to avoid inter-frame matching, depth estimation, morphing and interpolation. Y. Yagi et al. (Eds.): ACCV 2007, Part I, LNCS 4843, pp. 303–313, 2007. © Springer-Verlag Berlin Heidelberg 2007
304
J.Y. Zheng and X. Wang
Our work in this paper aims at a ground-based image database representing largescale spaces. The goals to achieve are: (1) Less images for rich spatial information, and (2) Pervasive view archive of an environment. Taking images at every location in a large area can be very inefficient due to data redundancy, and sometimes impossible. We plan viewpoints to cover as large spaces as possible to achieve data reduction. The view significance is evaluated according to the scene visibility and distribution. The sparse viewpoints and paths are then planned for acquiring cityscapes using route panoramas [14], scene tunnels [15][20], panoramic view, spherical views [4][5][11] and traditional digital images. As these images are compact in data size, as well as complete and continuous in the scene coverage, we can enlarge the area to model significantly. In the following sections, we propose a view significance measure using LIDAR data in Section 2, for various types of spaces. Section 3 describes view selection approaches based on the viewpoint evaluations, followed with a real process of view acquisition to form an image based virtual environment.
2 Ground-Based Views and View Significance 2.1 Views Categorization According to Virtual Actions In archiving a space at a city level, the image storage and transmission demand a scene selection scheme to cover more scenes with less data. Taking images at every viewpoint in a space is not necessary. Extended from normal digital images, we use local panoramas, route panoramas, and object panoramas to cover sites, paths and architectures, respectively. In a real environment, a visitor takes different actions in spaces such as looking around, walking, and examining things from various directions. It is preferable that our images are categorized according to these actions. In addition, a map view (or satellite image) can provide global locations. Figure 1 shows a diagram to register an area with these images. All these views can be classified more formally as follows: A local panorama V(0) is taken from a single viewpoint outward with an FOV possibly up to 360 degree in orientations. Examples include wide/fish-eye images, panoramic/spherical images, which employ perspective, central, cylindrical, and spherical projections respectively. A route panorama V(1) is a long image along a straight or mildly curved path for representing positions. Examples include slit-scanning route panoramas and scene tunnels, which employ parallel-perspective and parallel-central projections. Multiperspective projection (mosaicing multiple images on a path) can also be used if scenes are close to an equal-depth [1]. An object panorama V(2) is a set of inward images taken around an object or building (also called object movie and similar to aspect views). They are useful in showing all building facades or full appearances of an object. Now, it is necessary to investigate where these views should be located first in a large urban area if we cannot take images everywhere, i.e., how to cover large areas pervasively with a limited number of ground-based images. We use Airborne LIDAR data as an elevation map for view prediction. LIDAR is an optical remote sensing
View Planning for Cityscape Archiving and Visualization
305
(a)
(b)
(c) Fig. 1. A visualization model of urban area. Real scenes are projected towards a grid of paths and positions. (a) Image framework containing V(0), V(1), and V(2). (b) An object panorama V(2), (c) A section of route panorama V(1).
technology that measures properties of scattered light to find range and intensity of a remote target. The resolution of airborne LIDAR data with height and texture measured by laser scanning can be as fine as half meter per dot. 2.2 Viewpoint Evaluation Based on Visibility To take images in a large space, we have to plan the camera positions. We propose a view significance estimation based on how much of 3D surfaces an image can cover. This view significance can be measured in an urban area using LIDAR data or be examined on typical structure layouts. Our main interest is on walkthrough views taken roughly at eye-level; compared to an overhead image, ground-based-images capture more vertical surfaces and details. Intuitively, a view with large portions as horizon is not as visually significant as a view full of objects in conveying spatial information. Similarly, a view with a large sight from an overlook is more significant than a view from a narrow valley of buildings in telling global locations. Figure 2 illustrates an idea to compute the view significance from a view shed that is a view-covered region on 3D surfaces. Let P(X,Y,Z) denote a position in the space and a ray from P is defined by n(φ,ϕ), in orientation φ∈[0, 2π] and azimuth angle ϕ∈[-π/2, π/2]. If the ray hits a surface point at distance D(ϕ, φ), a sign function λ(φ, ϕ) takes value 1 and otherwise 0. The visible
306
J.Y. Zheng and X. Wang
surfaces from P thus form a view shed. We define view significance σ(P) to be the area of the view shed (∑λ), which is calculated by σ (P) =
D (φ , ϕ )
∫∫ w λ (φ , ϕ ) D (φ , ϕ ) + D
φ ,ϕ
d φd ϕ
(1)
0
where w is a weight of importance assigned on each surface, and D0 is a large constant (e.g., 100m). The denominator of Eq. 1 counts for the image quality degradation on distant scenes due to atmospheric haze. It avoids a close-to-infinity scene to be integrated largely into σ(P). The weight w, assigned building-wise in a space, takes a uniform value in this paper to show the influence from the geometric layout only, unless some important façades or landmarks need emphasis particularly.
View shed View volume P
Fig. 2. Computing view significances from the view shed at a viewpoint for V(0)
We calculate a continuous distribution of σ(P) using an elevation map H(X,Z). For an urban area, LIDAR data are reduced in resolution to a hole-less map first. At each small grid region (e.g., 1∼5m2) in the LIDAR data, non-zero elevation points are median-filtered to yield an integer value in discrete metrics H(X,Z) (Fig. 4a). Second, all reachable points P(X,Y,Z) at eye-level are marked (if Y>0). Third, we compute lines of sight in all discrete orientations originating from every viewpoint P(X,Y,Z). Each line is stretched out until it hits an obstacle, where the front tip of the line, Pl(Xl,Yl,Zl), satisfies Yl
Fig. 3. Typical spatial structures and distributions of the view significances with spherical views. A crossing, square, square with roads, and two connected yards are enclosed with walls.
View Planning for Cityscape Archiving and Visualization
307
Figure 4 shows a city area with buildings of different heights. Three spherical views in Fig. 4a are evaluated in depth (Fig. 4b). The σ(P) of entire reachable space are calculated with D0=100m in Fig. 4c, d. According to σ(P) evaluated for spherical views [11], the positions marked in red and yellow at a round square and an open crossing in Fig. 4a are more significant than the third position marked in green in a parking lot. This can be confirmed in Fig. 4b where a large sky appears in Fig. 4(b.3) and the high buildings at distance degrade in the significance estimation. For cylindrical panoramas [4] (limited FOV at ϕ∈[-45°,45°]), the distribution of σ(P) is shown Fig. 4d. The significance value is lower at positions close to high rises than at positions farther away now because the close positions miss high surfaces on buildings.
(a)
(b.1)
(b.2)
(c)
(b.3)
(d)
Fig. 4. View significance evaluations at all positions in an area. (a) LIDAR elevation map with the intensity representing height. (b.1)(b.2)(b.3) 360°×180° spherical depths at three positions marked in (a). FOVs of cylindrical projection are in dashed frames. (c) Reachable positions and their view significances in gray levels. (d) View significance evaluated with cylindrical panoramas.
308
J.Y. Zheng and X. Wang
(a)
(b)
(c) Fig. 5. Depth map from LIDAR data and corresponding views. (a) The mechanism to scan a scene tunnel. (b) Depths along a camera path calculated from LIDAR data (intensity inversely proportional to depth). Vertical axis indicates azimuth angle of rays and the horizontal axis indicates the route distance. (c) Blocks of scene tunnel taken along the real street.
2.3 View Significance Evaluation for Paths and Buildings The view significance can also be defined for a route panorama and object panorama similarly. Figure 2b depicts the geometry in getting a scene tunnel when a camera facing sideways on a vehicle moves along a street [15]. Denote a ray by n(ϕ) in the Plane of Scanning (PoS) at position l on the path (Fig. 5a). A surface point at distance D(l,ϕ) is projected to the scene tunnel. The view significance is measured at the position is defined as σ ( P(l )) = ∫ wλ (l , ϕ ) ϕ
D (l , ϕ ) dϕ D(l , ϕ ) + D0
(2)
where ϕ∈[-π/2, π/2] is the azimuth angle of the ray. The significance of an entire street is σ (L) =
∫ σ ( P ( l )) dl L
L
(3)
where L is the street length. Figure 5 shows the depth map D(l,ϕ) visible from a street. It is predicted using LIDAR data and the half-side scene tunnel along the real street is displayed for comparison as well. The σ(L) is high in this section. On the contrary, we can predict that very open streets such as highways at suburb may be insignificant for their monotonic scenes and large sky area. An architecture can displayed with its object panoramas V(2), i.e., side views adjacent to each other. Here, the object panorama does not strictly fall into the definition of aspect views that are divided according to object surfaces. In evaluating significant viewpoints for buildings or landmarks, visibility is more meaningful than the shooting distance because distance can be compensated by using different lenses. We define view significance σ(P,B) at a surrounding point P for watching B by
View Planning for Cityscape Archiving and Visualization σ (P, B) =
∫∫ w λ
φ ,ϕ
2
(φ , ϕ )
D (φ , ϕ ) d φd ϕ D (φ , ϕ ) + D 0
309
(4)
where λ2(φ,ϕ) is 1 if a ray reaches building B from P and 0 otherwise. As an example, we use a convex block, a building complex, and scattered towers in Fig. 6a to calculate their view significance distributions in Fig. 6b,c. The block in Fig. 6a can be captured with a normal lens (Fig 6b.1), while the scattered architecture group is suitable to be captured in a wide image from a point at the central area (Fig. 6b.3). In general, the view significance is continuous in the reachable area of H(X,Z) and decreases as the viewing distance gets farther away. The significance value also tends to be higher in an orientation from which multiple facades become visible, and drops to zero when the target is occluded completely. Moreover, the smaller the vertical field of view of a camera, the farther the significant viewpoints tend to be, as we compare individual pairs in Fig. 6b and 6c. In real situation, many buildings are aligned in street blocks, which is simpler to be captured in V(1) images.
(a)
(b.1)
(c.1)
(b.2)
(c.2)
(b.3)
(c.3)
Fig. 6. View significances of buildings displayed in levels. (a) Elevation map of three groups of architecture. (b.1)(b.2)(b.3) distributions of view significances in (a) when a cylindrical panorama (vertical FOV is 90°) is used. (c.1)(c.2)(c.3) as references, the view significances are counted with spherical views for each building group again.
3 View Selections and Pervasive Image Acquisition 3.1 Street View Acquisition from a Moving Vehicle We start with the scanning of route panoramas V(1) because imaging from a moving vehicle is the most efficient way to obtain cityscapes. Route panoramas and scene tunnels are typically planned along streets with rich scenes (σ(L) is high). The route panoramas are scanned continuously with a pixel line in the video frame. This slit scanning method is much more efficient in capturing translating scenes with depth changes than most mosaicing methods that need to merge discrete images at consecutive positions, because it avoids image correspondence, depth estimation, and image integration.
310
J.Y. Zheng and X. Wang
According to the simulated depth map along a street, we select a lens up to 180 degree FOV for a street. The lens scope may exclude tops of some high rises if other parts of the street are low architectures in average. The route panoramas also have some deformations. Because of the employed parallel-perspective projection in composing a long image, the aspect ratios of objects at different depths from the path are not constant in the route panorama. The aspect ratio can be adjusted well at one depth but is hard to satisfy other depths. At an adjusted depth, scenes are exposed to the route panorama sharply, while beyond the depth the stationary blur [10] appears along horizontal direction in the route panorama. Therefore, we select a sampling rate and vehicle speed after the camera lens is determined from the height coverage of side scenes. If we denote the curvature of the path by κ, the depth of scene by z, the vehicle speed by v, the angle of the plane of sight with respect to the motion vector by α, and the focal length by f, the blurring rate from the original image (It/Ix) can be calculated as ⎛1 Il κ ⎞ = fv⎜⎜ − 2 ⎟⎟ Ix z sin α⎠ ⎝
(5)
where It and Ix are contrast in the route panorama and video image respectively. The detailed proof is in [21]. Setting the ratio to be 1, we can inversely obtain v from known κ, α, f and average z. This will reduce the distortion of aspect ratio as well as the stationary blur on major scenes. On the other hand, we can also keep the vehicle velocity as constant as possible and normalize the length of route panorama/scene tunnel according to GPS output or satellite images. 3.2 Placing Local Panoramas in Urban Area After route panoramas V(1) cover major streets, we place a number of local panoramas V(0) in the urban area. There are various strategies of viewpoint selection. For representing larger 3D surfaces, we plan V(0) at peaks of σ(P) distribution. A viewpoint close to a selected peak may share a large portion of scenes with the peak. Therefore, we avoid local maxima of σ(P) in the same hill with a selected peak. As an automatic procedure, we decrease a threshold level gradually over the σ(P) distribution in Fig. 4d, in order to locate peaks and island regions, E(level)={Pe |σ(Pe)≥level}. Assuming two viewpoints should not be closer than a predefined distance r, we select an emerging local maximum of σ(P) as a viewpoint if it is not closer to any island region than r. We locate those peaks in σ(P) for panoramic view in order according to the following algorithm. Set E to be an empty set For level decreasing from max(σ) to min(σ) For every point P∈H(X,Z) satisfying σ(P)≥level and P∉E If P is a peak and is father away from E than r then select P as a viewpoint; // Mark P as a point in E Add point P to E;
This algorithm selects peaks of σ(P) on major hills and ignores local maxima on the examined hills of the view significance distribution. Figure 7 gives an output of the algorithm for placing local panoramas. After selecting viewpoints, images are
View Planning for Cityscape Archiving and Visualization
311
Fig. 7. Planned positions in white spots are plotted for local panoramas in the elevation map of the area
taken with digital cameras [17] and spherical views are taken with a fish-eye lens if necessary. In real cases, the camera positions have to be deviated from the planned ones due to busy traffics. Also, V(0) images here are more suitable to represent spatial layouts rather than symbolic scenes with cultural and sightseeing values. Additional V(2) images can always be taken to emphasize meaningful scenes. 3.3 Locating Images Around Objects Now, we locate multiple discrete images around buildings of interest to generate an image group namely object panorama (in QTVR). After the calculation of σ(P,B) for a building or building block, picking viewpoints manually at positions with high σ(P,B) values is feasible. Alternatively, we can also calculate viewpoints automatically. First, the center of heights, P0(X0, Z0), of a building is obtained from H(X,Z) as (X 0 , Z 0 ) = ∑ ( X , Z ) H ( X , Z ) P∈ B ( 2 )
∑ H (X ,Z )
P∈ B ( 2 )
(6)
In each orientation φ around P0, we find a distance d(φ) where the viewpoint has the highest significance, i.e., d (φ ) = arg max σ (P , d
B)
(7)
which results in a closed curve surrounding B. We then select several orientations φ1, φ2, φ3, ... on the curve which local maxima of σ(P,B) to imaging B inward. The images can be dense to share partial views (or facades) so that the orientations are more conceivable through images. Various lenses can be used for these images as long as scenes are located in the frame properly. Beside above three types of images, discrete images can always be taken at spots of interest or partial scenes on the buildings to highlight details. We did experiments at an urban area of 1.6×1 km2, which includes 50 panoramic images, 34 route panoramas of both sides and 44 buildings. The image has been normalized to a fixed height for display and the lengths are proportional to their real lengths [22].
312
J.Y. Zheng and X. Wang
4 Conclusion This work creates a framework to pervasively record scenes in a large-scale area for image archive and visualization. The compact nature of the employed images has significantly enlarged the scale of cityscape modeling. We proposed an evaluation of viewpoints, namely view significance, according to the visibility and effective image coverage of cityscapes. Based on the significance distribution estimated from LIDAR data, we precisely located different types of images so that they can include as many scenes as possible but avoid data redundancy. The visual data can be displayed in a web and loaded further to portable navigation devices for on-site urban area guidance.
References [1] Agarwala, A., et al.: Photographing long scenes with multi-viewpoint panoramas. ACM Transactions on Graphics 25(3), 853–861 (2006) [2] Aliaga, D.G., Carlbom, I.: Plenoptic Stitching: A scalable method for reconstructing 3D interactive walkthroughs. In: SIGGRAPH 2001, pp. 443–450 (2001) [3] Aliaga, D.G., Funkhouser, T., Yanovsky, D., Carlbom, I.: Sea of Images. In: IEEE Conf. Visualization, pp. 331–338 (2002) [4] Chen, S.E., Williams, L., Quicktime, V.R.: An image-based approach to virtual environment navigation. In: SIGGRAPH 1995, pp. 29–38 (1995) [5] Coorg, S., Master, N., Teller, S.: Acquisition of a large pose-mosaic dataset. In: IEEE CVPR 1998, pp. 23–25 (1998) [6] Frueh, C., Zakhor, A.: Constructing 3D city models by merging ground-based and airborne views. In: IEEE CVPR 2003, pp. 562–569 (2003) [7] McMillan, L., Bishop, G.: Plenoptic modeling: an image based rendering system. In: ACM SIGGRAPH 1995 (1995) [8] Peleg, S., Rousso, B., Rav-Acha, A., Zomet, A.: Mosaicing on adaptive manifolds. IEEE Trans. PAMI 22(10), 1144–1154 (2000) [9] Roman, A., Garg, G., Levoy, M.: Interactive design of multi-perspective images for visualizing urban landscapes. In: IEEE Conf. Visualization 2004, pp. 537–544 (2004) [10] Shi, M., Zheng, J.Y.: A slit scanning depth of route panorama from stationary blur. In: IEEE CVPR (2005) [11] Szeliski, R., Shum, H.-Y.: Creating full view panoramic image mosaics and texturemapped models. In: ACM SIGGRAPH 1997, pp. 251–258 (1997) [12] Uyttendaele, M., et al.: Image-based interactive exploration of real-world environments. IEEE Computer Graphics and Application 24(3) (2004) [13] Zhao, H., Shibasaki, R.: A Vehicle-borne urban 3D acquisition system using single-row laser range scanners. IEEE Trans. on SMC, B 33(4), 658–666 (2003) [14] Zheng, J.Y.: Digital Route Panorama. IEEE Multimedia 10(3), 57–68 (2003) [15] Zheng, J.Y., Zhou, Y., Mili, P.: Scanning Scene Tunnel for city traversing. IEEE Trans. Visualization and Computer Graphics 12(2), 155–167 (2006) [16] Zomet, A., Feldman, D., Peleg, S., Weinshall, D.: Mosaicing New Views: The crossedslits projection. IEEE Trans. on PAMI, 741–754 (2003) [17] Li, S.: Full-View Spherical Image Camera. ICPR (4), 386–390 (2006) [18] Li, S., Nakano, M., Chiba, N.: Acquisition of spherical image by fish-eye conversion lens. IEEE Virtual Reality 235–236 (2004)
View Planning for Cityscape Archiving and Visualization
313
[19] Zheng, J.Y., Tsuji, S.: Panoramic Representation for route recognition by a mobile robot. IJCV 9(1), 55–76 (1992) [20] Zheng, J.Y., Li, S.: Employing a fish-eye camera in scanning scene tunnel. In: 7th ACCV, vol. 1, pp. 509–518 (2006) [21] Zheng, J.Y., Shi, M.: Depth from stationary blur with adaptive filtering. In: 8 th ACCV (2007) [22] http://www.cs.iupui.edu/ jzheng/ACCV07
Synthesis of Exaggerative Caricature with Inter and Intra Correlations Chien-Chung Tseng and Jenn-Jier James Lien Robotics Laboratory, Dept. of Computer Science and Information Engineering National Cheng Kung University, No. 1, Ta-Hsueh Road, Tainan, Taiwan {ed,jjlien}@csie.ncku.edu.tw http://robotics.csie.ncku.edu.tw
Abstract. We developed a novel system consisting of two modules, statisticsbased synthesis and non-photorealistic rendering (NPR), to synthesize caricatures of exaggerated facial features and other particular characteristics, such as beards or nevus. The statistics-based synthesis module can exaggerate shapes and positions of facial features based on non-linear exaggerative rates determined automatically. Instead of comparing only the inter relationship between features of different subjects at the existing methods, our synthesis module applies both inter and intra (i.e. comparisons between facial features of the same subject) relationships to make the synthesized exaggerative shape more contrastive. Subsequently, the NPR module generates a line-drawing sketch of original face, and then the sketch is warped to an exaggerative style with synthesized shape points. The experimental results demonstrate that this system can automatically, and effectively, exaggerate facial features, thereby generating corresponding facial caricatures. Keywords: Exaggerative rate, Exaggerative caricature synthesis, Eigenspace, Non-photorealistic rendering (NPR).
1 Introduction Caricatures often appear in newspapers, comic, and even at popular tourist spots where some artists draw for sightseers. Generally, caricature is a kind of exaggerative representation. It exhibits the funny and extraordinary characteristics of a person, and it also can be a human-like agent because it contains lots of recognizable facial features. The basic definition of facial caricature is that it is the exaggeration for all the facial features which are found by comparing the impression features of the subject with the average face. The first caricature generation system is made by Brennan [4], who used the interactive algorithm to create the sketch with exaggeration. Following works base on two approaches: with or without training process. The works with training process usually build some standards which are used to compare with the testing data to find the difference between their features. Koshimize et al [11] proposed a template-based approach to create caricature with exaggerative rates. However, the method is linebased drawing; the result will be unrecognizable when the exaggerative rates are too Y. Yagi et al. (Eds.): ACCV 2007, Part I, LNCS 4843, pp. 314–323, 2007. © Springer-Verlag Berlin Heidelberg 2007
Synthesis of Exaggerative Caricature with Inter and Intra Correlations
315
large. The works [12], [13], [17] developed example-based approaches by using partial least squares or neural networks to learn the drawing style of an artist, but they can not exhibit some particular characteristics like beards, nevus and etc. without training them in advance. Chiang et al [7] analyzed facial features and warped the color caricature created by artist to the exaggerative style with analyzed result. However, the representation of result is limited by the prototype drawn by an artist. Xu [18] exaggerated both the shapes and the positions of facial features by eigenspace, but the parameters used to exaggerate facial features are determined empirically. Except the approach with training process, some works for caricature creation has no training process. Gooch et al [9] proposed an approach to create an exaggerative black-and-white drawing by using Blommaert and Martens’ model and manual exaggeration [3], and Akleman [1] created a caricature by a forward triangle morphing. Those methods involve not only exaggerated facial features but also other particular characteristics. However, they also need a lot of manual works. Besides, there are other works [5], [6], [10], [19] creating good-quality cartoon face without exaggeration, and our work can be views as the extension of their results. In order to solve existing problems discussed above, we developed a novel system based on the advantages of processes with training and without training by implementing statistics-based synthesis (with training process) and non-photorealistic rendering (without training process). The exaggerative facial feature caricatures generated by our method contain exaggerative facial features and other particular characteristics which are always eliminated in order to simplify the training process. In the statistics-based synthesis module, we find the major principle components of features by using the principal component analysis (PCA) process, and expand them to emphasize personal specialties. In addition, the exaggeration is based on two relationships proposed by Redman [16]: one is the relationship between different subjects’ features, and we define this relationship as inter process; the other is the relationship between facial features of the same subject ignored by previous works, and we define it as intra process.
2 Statistics-Based Synthesis Module: Training Process In the training process of the statistics-based synthesis module, we compiled 69 male and 55 female images from the AR face database [14], 90 images are for training and 34 images are for testing. All the facial images are taken at the same distance from camera, and each training facial image is normalized as a canonical image (512×512 pixels) by affine transformation including only translation and rotation factors by two inner corner points of the eyes. The reason of eliminating scaling factor is that all the position of eyes of training images will be the same if applying scaling factor, thus we can’t exaggerate the position of eyes by comparing the difference from the position of average eyes. For each canonical image, the feature shape points of eyebrows (20 points), eyes (16 points), nose (12 points), mouth (20 points), and facial contour (19 points) are selected manually and denoted by Fi where i = 1 to 7. Also, we choose 7 control points corresponding to left eyebrow, right eyebrow, left eye, right eye, nose, mouth,
316
C.-C. Tseng and J.-J.J. Lien
and face contour respectively and denoted by P where P = (p1x, p1y, p2x … p7x, p7y). Each control point represents the position of each feature shape. Furthermore, all the feature shape and position vectors are separated into x and y coordinates denoted by Fix, Fiy, Px, Py respectively to avoid mutual effects of x and y coordinates of featureshapes or positions. Based on all feature shape vectors, Fix and Fiy, we can calculate their corresponding average vectors, MFix and MFiy, and by applying the PCA, the corresponding eigenspaces, UFix and UFiy, can be obtained. Subsequently, the average position point of each feature is calculated in stead of generating eigenspaces, because there is only one position point for one facial feature. The information is not enough to build position eigenspaces. The framework of training process is shown in Fig. 1.
Geometric normalization
I
S
Feature shape F1
F3
F2
F at x and y coordinates. F1x Eigenspaces at x and y UF 1x coordinates. I: Canonical image.
FF1y1y F
1y
UF1y
F4
F5
…
F1y
…
UF7x
F7x
F6
F7
F1y
Feature position
F7y
P at x and y Px coordinates Px
PPxy
UF7y
Average P x, P y
MPy
MPx
S: Feature shape points.
Fig. 1. The training framework of statistics-based synthesis module
3 Statistics-Based Synthesis Module: Testing Process To start with the testing process of statistics-based synthesis module, as shown in Fig. 2, we extract the facial feature points by applying an active appearance model (AAM) [8]. If the extracting result is not good, we label the feature points manually. Then, as the training process, we can have TFi where i = 1 to 7, TP = (tp1x, tp1y, tp2x … tp7x, tp7y), TFix, TFiy, TPx, TPy. Subsequently, the eigenspaces and average position points which have been generated at training process are used to perform statistics-based synthesis module. The statistics-based synthesis module is divided into two parts: one is the exaggeration of feature shape points on x and y coordinates; and the other one is the exaggeration of feature position points on x and y coordinates. Both two parts further apply inter and intra processes. By inter and intra process we can automatically determine which features should be exaggerated. The exaggeration of feature position points is performed after the exaggeration of feature shape points. And to simplify the explanation, we take feature shape exaggeration on x coordinate as examples to explain inter and intra processes for facial feature shape. As for the feature position exaggeration, fuller discussion will be presented in Section 3.4.
Synthesis of Exaggerative Caricature with Inter and Intra Correlations
317
Statistics-based synthesis module Fig. 4
FFiyiy TFix TFi
TP
T
Iaf TFi’
Ini
TPx
TS
AAM
Irf
FFiyiy TFiy
Irp
TPy
Iap
TS’ TP’
Non-photorealistic rendering module Line-drawing sketch generation
Metamorphosis ’
T
Caricature
Ini: Exaggerative rates initialization. Irf: Inter exaggeration for feature shape. Iaf: Intra exaggeration for feature shape. Irp: Inter exaggeration for feature position. Iap: Intra exaggeration for feature position. Fig. 2. The framework of exaggerative caricature creation
3.1 The Initialization of Exaggerative Rates In Fig. 3, there are two ellipses, a small one and a big one. Having the same horizontal variation, the small ellipse is more distinctive than the big ellipse, and so does the variation on displacement. This example indicates that the degree of shape or position exaggeration is negatively relative to the length or width of feature itself [15]. Therefore, we develop following equation to model such condition:
Dix = exp(1 −
length(TFix )
∑i=1length(TFix ) 7
)
(1)
where Dix represents the ratio which is negatively and exponentially relative to the horizontal or vertical length of feature; and length(TFix) is the horizontal length of feature where length(TFix)=max(TFix) min(TFix). The reasons we use exponential function are that the exponential function is always positive (never violates the truth of feature), and the rate of growth is proportional to its value, namely the larger variation the feature has, the higher degree of exaggeration it should have. Basically, the exaggeration of feature is in fact the extension of the difference from corresponding average feature. In order to implement the exaggeration process, the “exaggerative rate” is defined to be the scalar. There are two kinds of exaggerative rates: the shape exaggerative rates and the position exaggerative rates. The initial
－
shape exaggerative rate of x coordinate of i-th feature is
ef ixk =1 = 1 + c1 ⋅ Dix , and it
satisfies the rule of exaggeration, c1 is the constant to controlling the degree of
318
C.-C. Tseng and J.-J.J. Lien
(a)
(b)
Fig. 3. (a) Ellipses having the same horizontal variances. (b) Ellipses having the same horizontal displacement.
exaggeration and c1=0.2 in this study. The initialization of position exaggerative rates will be discussed later. 3.2 Exaggerative Feature Shape Creation: Inter Exaggeration After initializing the shape exaggerative rates, the unbiased shape vector of i-th facial feature is calculated by subtracting corresponding average shape MFix, and then projected to the corresponding eigenspace UFix to get the weights of eigenvectors. Then we expand the weights by multiplying shape exaggerative rates. Finally, the exaggerative feature shape is obtained by the reconstruction with the expended weights:
Eixk = ∑ j =1 ( ρ ⋅ ef ixk ⋅ w jx ⋅ u jx ) + MFix
(2)
w jx = (TFix − MFix )T ⋅ u jx
(3)
n
T
Eixk
is the result of inter exaggeration at the k-th iteration. ρ is the proportion of each eigenvector, which means the higher proportion of eigenvector, the bigger shape exaggerative rate the eigenvector has. wjx is the projection weight of the j-th eigenvector, and ujx is the j-th eigenvector of the eigenspace. The result of inter process is shown in Fig. 5.(d). After comparing with the average face, we further apply the intra exaggeration process which considers the relationship with other features of the same subject to increase the contrast between all the facial features. where
3.3 Exaggerative Feature Shape Creation: Intra Exaggeration As we have said above, an artist considers the difference not only from the average face but also other neighbor features of the same subject. In other words, the major facial features of the subject are expected to be extracted, enhanced and then exaggerated in order to emphasize personal style of this face. As for other non-major facial features, the degree of exaggeration will decrease in order to enhance the contrast between the major features and the non-major features. For example, if the subject’s mouth is bigger than average mouth, but his/her nose is much bigger and
Synthesis of Exaggerative Caricature with Inter and Intra Correlations
319
rounder. Then the artist will draw the mouth smaller, but still bigger than average mouth, to emphasize the variation of nose. In order to increase the contrast between features, all the variances of exaggerative features should be calculated by the following equation: 1 ∑ (Eix − TFix ) 2 . n vix = length(TFix )
(4)
Here, vix represents the variance of the x coordinate of the i-th feature normalized by the feature length. n is the number of points of each feature. After calculating all the variances of features, these variances are sorted to determine which shape exaggerative rate of feature should be increased and which one should be decreased. Then, we update the shape exaggerative rates by the equations based on the sorted variances. Indeed, the updated process should satisfy the condition mentioned in Section 3.1, and the power of effect will decrease based on a Gaussian weighting function.
∑ (r ⋅ d ( x, y)) ⋅ D − A⋅ ∑ d ( x, y ) 7
ef
( k +1) ix
= ef
k ix
j =1
j
7
ix
(5)
j =1
⎧ 1 if v jx > vix ⎪ r j = ⎨ 0 if v jx = vix ⎪− 1 if v < v jx ix ⎩
(6)
⎡ tpix − tp jx 2 tpiy − tp jy 2 ⎤ ) +( ) ⎥) d ( x, y ) = exp(− ⎢( σx σy ⎣⎢ ⎦⎥
(7)
where A is a constant controlling the power of effect and A = 0.2 in this study. rj is a switch which increases the shape exaggerative rate of feature with bigger variance and decreases the shape exaggerative rate of feature with smaller variance. tpix and tpiy are the i-th elements of vector TPx and TPy used to calculate the distance between facial features in the 2-D Gaussian weighting function d(x,y). After the intra exaggeration process, we use the new shape exaggerative rates and original feature points TF consists of all facial feature points to perform inter exaggeration process again in order to increase the contrast of these features and find major features as shown in Fig. 4. Thus, we can determine all of the shape exaggerative rates automatically instead of setting them one by one. As long as any exaggerative rate equals to zero, the iteration process terminates to prevent the feature from exaggerating in the contrary direction. For example, if we draw a nose small but in fact it is bigger than general. Obviously, the feature is exaggerated in the contrary direction. Additionally, we need to set the maximum iteration time to avoid none of the shape exaggerative rates being zero. When satisfying the termination condition
320
C.-C. Tseng and J.-J.J. Lien
after n-1 time iteration, original feature shape TF and last shape exaggerative rates ef n are used in inter process to generate the x coordinate of exaggerative shape. As for the exaggeration of feature shapes on y coordinate, inter and intra processes mentioned above is applied in the same way. The result of iteration is shown in Fig. 5.(e). ef 1 TF
ef 2 TF
ef n-1 TF
ef n TF
Inter process
Inter process
Inter process
Inter process
E1 Intra process
E2 Intra process
…
E
n-1
Intra process
Exaggerative shape
ef k: shape exaggerative rate at k-th iteration Ek: the intermediate at k-th iteration Fig. 4. The diagram of entire iteration process for inter and intra shape exaggerations
3.4 Exaggerative Feature Position Creation Having finished the exaggeration of feature shape, the positions of features are further exaggerated. For the position vectors TPx and TPy, the difference from average feature position is calculated by subtracting MPx, MPy, and then expended by the equations as follow with the position exaggerative rate which also contains two parts: inter position exaggerative rate epinter and intra position exaggerative rate epintra:
⎧tp 'ix = mpix + epix (tpix − mpix ).Dix ⎨ ⎩tp 'iy = mpiy + epiy (tpiy − mpiy ).Diy
(8)
⎧⎪ep ix = ep ixint er + ep ixint ra ⎨ int er int ra ⎪⎩ep iy = ep iy + ep iy
(9)
⎧ int ra length(TF7' x ) − length( MF7 x ) ⎪epix = length( MF7 x ) ⎪ . ' ⎨ ( ) − length( MF7 y ) length TF 7 y int ra ⎪epiy = ⎪⎩ length( MF7 y )
(10)
Here, inter position exaggerative rate is given by the initial process where epixint er = 1 + c2 ⋅ Dix , c2 is a constant to control the exaggerative degree and c2 = 0.3 in this study. After comparing the difference from average feature positions, artists always adjust the results of comparisons based on the width or the length of face. If the face is wider or longer than average face, the distance between the two features will be increased horizontally or vertically, and vise versa. Thus, the intra exaggerative rate only depends on the width and length of face. Indeed, Dix, Diy are added to ensure that the displacement of each position of feature is also satisfies the
Synthesis of Exaggerative Caricature with Inter and Intra Correlations
321
condition discussed in Section 3.1. After the feature position exaggeration, we translate all the exaggerative feature shape points to the specific location based on exaggerative positions, and the result TS’ is shown in Fig. 5.(f).
4 Non-photorealistic Rendering Module Since we have exaggerated the shapes and positions of facial features, we will apply non-photorealistic rendering processes to generate a caricature with exaggerative style. First, the facial contour is emphasized by modifying a method proposed by Gooch [9]. Instead of generating a binary image, the values of pixels of result are replaced with gray values of original image to make the result more lifelike. Then after creating a line-drawing sketch T’ without exaggeration, as shown in Fig. 5.(g), the line-drawing sketch is warped to an exaggerative style by using the image metamorphosis [2] with exaggerative feature shape. Finally, a caricature with exaggerative feature positions and shapes and other particular characteristics, as shown in Fig. 5.(h), are generated.
(b)
(c) (f)
(d)
(e)
(g)
(h)
Fig. 5. (a) Original input image. (b) Original feature shape. (c) Average feature shape. (d) The result after only inter exaggeration process for shape. (e) The result after inter and intra exaggeration processes only for shape. (f) The result after inter and intra exaggeration processes for both shape and position. (g) The line-drawing sketch of original face. (h) The exaggerative caricature of original face keeping the information of nevus.
5 Experimental Results For the testing image of frontal face, as shown in Fig. 5.(a), the diagrams of shape exaggerative rates are shown in Fig. 6. These two diagrams indicate that the contrast between shape exaggerative rates increases by iteration, and major features can be extracted with high exaggerative rates. In other words, our system can automatically set the exaggerative rates and extract major facial features. In addition, by comparing with Xu’s method [18], shown in Fig. 7, our result can keep the contour of original feature after the increase of exaggerative rates, but the result of Xu’s method occurs distortion which decreases the likeness of features. Therefore, our result is subtler than Xu’s. The experimental results demonstrate that our system can automatically and effectively exaggerate facial features, and generate corresponding facial caricatures of other particular characteristics. More results with exaggerative sizes and positions of the facial features are shown in Fig 8.
322
C.-C. Tseng and J.-J.J. Lien
Left Eyebrow Left Eye Nose 3 Face
Right Eyebrow Right Eye Mouth
Left Eyebrow Left Eye Nose 3.5 Face
ER: Exaggerative rate
2.5
ER
ER
2 1.5 1 0.5 0 1
3
5
7
9 11 13 15 17 19 Iteration
Right Eyebrow Right Eye Mouth ER: Exaggerative rate
3 2.5 2 1.5 1 0.5 0 1
(a)
3
5
7
9 11 13 15 17 19 Iteration
(b)
Fig. 6. The variation of shape exaggerative rates on (a) x coordinate (b) y coordinate
Fig. 7. (a) The results of our approach (b) The results of Xu’s approach [18]
(a)
(b)
(c)
(a)
(b)
(c)
(a)
(b)
(c)
(a)
(b)
(c)
Fig. 8. More results of (a) original face (b) line-drawing sketch (c) exaggerative caricature
6 Conclusion We have developed a novel system serving to generate an exaggerative caricature which involves not only exaggerative facial features but also other particular characteristics, like beards or nevus. Besides, we model the drawing skills of artists to
Synthesis of Exaggerative Caricature with Inter and Intra Correlations
323
simplify the determination of exaggerative rates. However, our system can only handle the image data taken from fixed distance and under fixed lighting. The variation of lightness and distance will affect our result. In the future work, we want to solve the factors of environment by simulating the image with visual distance and appropriate lighting to make our system more robust.
References 1. Akleman, E.: Making Caricature with Morphing. In: Proc. of ACM SIGGRPH, p. 145 (1997) 2. Beier, T., Neely, S.: Feature-based Image Metamorphosis. In: Proceedings of ACM SIGGRAPH, pp. 35–42 (July 1992) 3. Blommaert, F.J.J., Martens, J.-B.: An Object-Oriented Model for Brightness Perception. Spatial Vision 5(1), 15–41 (1990) 4. Brennan, S.: Caricature Generator. Master’s thesis, Cambridge, MIT (1982) 5. Chen, H., Xu, Y.Q., Shum, H.Y., Zhu, S.C., Zheng, N.N.: Example-based Facial Sketch Generation with Non-parametric Sampling. In: Proceedings of International Conference on Computer Vision, Vancouver, Canada (July 2001) 6. Chen, H., Zheng, N.N., Liang, L., Li, Y., Xu, Y.Q., Shum, H.Y.: PicToon: A Personalized Image-based Cartoon System. In: Proc. of ACM Int’l Conf. on Multimedia (2002) 7. Chiang, P.Y., Liao, W.H., Li, T.Y.: Automatic Caricature Generation by Analyzing Facial Features. In: Proceedings of Asia Conf. on Computer Vision (2004) 8. Cootes, T.F., Edwards, G.J., Taylor, C.J.: Active Appearance Models. IEEE Trans. on Pattern Analysis and Machine Intelligence 23(6) (2001) 9. Gooch, B., Reinhard, E., Gooch, A.: Human Facial Illustrations: Creation and Psychophysical Evaluation. ACM Trans. on Graphics 23(1), 27–44 (2004) 10. Hsu, R.L., Jain, A.K.: Generating Discriminating Cartoon Faces Using Interacting Snakes. IEEE Trans. on Pattern Analysis and Machine Intelligence 25, 1388–1398 (2003) 11. Koshimize, H., Tominaga, M., Fujiwara, T., Murakami, K.: On KANSEI Facial Processing for Computerized Facial Caricaturing System Picasso. In: Proceedings of IEEE International Conference on Systems, Man and Cybernetics, pp. 294–299 (1999) 12. Lai, K.H., Edirisinghe, E.A., Chung, P.W.H.: A Facial Component based Hybrid Approach to Caricature Generation using Neural Networks. In: Proceedings of ACTA on Computational Intelligence (2006) 13. Liang, L., Chen, H., Xu, Y.Q., Shum, H.Y.: Example-based Caricature Generation with Exaggeration. In: Proc. 10th Pacific Conf. on Computer Graphics and Applications (2002) 14. Martinez, A.M., Benavente, R.: The AR Face Database. CVC Technical Report #24 (June 1998) 15. Mo, Z., Lewis, J.P., Neumann, U.: Improved Automatic Caricature by Feature Normalization and Exaggeration. In: Proceedings of ACM SIGGRAPH Conf. on Abstracts and Applications, New York (2004) 16. Redman, L.: How to Draw Caricatures. Contemporary Books (1984) 17. Shet, R.N., Lai, K.H., Edirisinghe, E.A., Chung, P.W.H.: Use of Neural Networks in Automatic Caricature Generation: An Approach Based on Drawing Style Capture. In: IEE International Conference on VIE, UK, pp. 23–29 (2005) 18. Xu, G.Z., Kaneko, M., Kurematsu, A.: Synthesis of Facial Caricature Using Eigenspaces. In: Proceedings of Electronics and Communications in Japan. Part 3, vol. 87(8) (2004) 19. Xu, Z., Chen, H., Zhu, S.C.: A High Resolution Grammatical Model for Face Representation and Sketching. In: Proceedings of Computer Vision and Pattern Recognition, pp. 470–477 (2005)
Pose-Invariant Facial Expression Recognition Using Variable-Intensity Templates Shiro Kumano1 , Kazuhiro Otsuka2 , Junji Yamato2 , Eisaku Maeda2 , and Yoichi Sato1 Institute of Industrial Science, The University of Tokyo, 4–6–1 Komaba, Meguro-ku, Tokyo, 153–8505 Japan {kumano,ysato}@iis.u-tokyo.ac.jp 2 NTT Communication Science Laboratories, NTT 3–1 Morinosato-Wakamiya, Atsugi-shi, Kanagawa, 243–0198 Japan {otsuka,yamato}@eye.brl.ntt.co.jp, [email protected] 1
Abstract. In this paper, we propose a method for pose-invariant facial expression recognition from monocular video sequences. The advantage of our method is that, unlike existing methods, our method uses a very simple model, called the variable-intensity template, for describing different facial expressions, making it possible to prepare a model for each person with very little time and effort. Variable-intensity templates describe how the intensity of multiple points defined in the vicinity of facial parts varies for different facial expressions. By using this model in the framework of a particle filter, our method is capable of estimating facial poses and expressions simultaneously. Experiments demonstrate the effectiveness of our method. A recognition rate of over 90% was achieved for horizontal facial orientations on a range of ±40 degrees from the frontal view.
1
Introduction
Facial expression recognition is attracting a great deal of attention because of its usefulness in many applications such as human-computer interaction and the analysis of conversation structure [1]. Most existing methods for facial expression recognition assume that the person in the video sequence does not make any large movements and that the image shows a nearly frontal view of the face [2][3][4][5]. However, in situations such as multi-party conversations (e.g. meetings), people will often turn their faces to look at other participants. Hence, we must simultaneously handle the variations in facial pose as well as facial the expression changes. Facial expression recognition methods handling facial pose variations require a facial shape model of the user’s neutral expression and a model of facial expression to treat the variations of facial pose and expression separately. The shape model and the facial expression model are together referred to as the face model in this paper. The face model expresses facial pose variations by globally translating and rotating the shape model in three-dimensional space, and facial Y. Yagi et al. (Eds.): ACCV 2007, Part I, LNCS 4843, pp. 324–334, 2007. c Springer-Verlag Berlin Heidelberg 2007
Pose-Invariant Facial Expression Recognition
325
expression changes by locally deforming the shape model according to the facial expression model. Existing methods require an accurate face model, because image variations caused by facial expression change are often smaller than those caused by facial pose change. Accordingly, the use of inaccurate shape models degrades the accuracy of the facial pose and expression estimates because those two components cannot be separated reliably. Face models are divided broadly into two groups: person-dependent models and person-independent models. Previous methods generate a person-dependent face model for each user by using stereo cameras [6][7]. Accordingly, this approach cannot be applied to monocular video sequences. On the other hand, personindependent models can be applied to arbitrary users [8][9][10]. However, it has been reported that person-independent models cannot cover large interpersonal variations of face shape and facial expression with sufficient accuracy [11]. Motivated by these problems, we propose a novel method for facial expression recognition; based on variable-intensity templates, it offers the following advantage and goals: 1. It supports monocular video capture systems. 2. It can easily generate a face model for each person. 3. Facial expressions can be estimated even with a large change in facial pose. The variable-intensity template consists of three components: (1) a simple shape model, (2) a set of interest points, (3) an intensity distribution model. We use a cylinder as the shape model because it can be easily generated. However, to use such a simple shape model, we must handle the following problem. The interest points are defined on a face image in the frontal view (white points in the right figure of Fig.3). Their image positions for arbitrary facial poses are then calculated by projecting them onto the shape model, translating and rotating the shape model according to the pose, and projecting the resulting three-dimensional positions onto the image plane. Hence, if there is an error in the shape model, the calculated image position of an interest point shifts from its actual position as the facial pose angle increases. This problem can be effectively avoided by employing pairs of points that straddle the edge of different facial parts (the left part of Fig.1) [1][12]. Even if the interest points are shifted due to the error in the model, the change in the intensity of the points is small because the points are defined away from the edges where the intensity changes significantly (the center part of Fig.1). The intensity distribution model describes how interest point intensity varies for different expressions of a face. Our method prepares it to recognize facial expressions and to improve the robustness of facial pose estimataion against changes in facial expressions which cause large changes in intensity (the right part of Fig.1). Our contribution is that we propose a facial expression recognition method for varying facial poses based on the key idea that facial expressions can be correctly recognized by knowing how interest point intensity varies for different facial expressions. The main advantage of our method is that a face model for each person is created simply by capturing frontal face images of the person in
S. Kumano et al.
Enlarged
326
The shift of interest points due to error in shape model Intensity is not largely changed
Interest points
The shift of facial parts due to change in facial expression Intensity is largely changed
Face image Eyebrow Eye Expression
Neutral
Neutral
Angry
Fig. 1. Our method absorbs errors in shape models and recognizes facial expressions by treating the changes in intensity of multiple points defined around facial parts. Interest points are composed of pairs of points that straddle the edges of facial parts.
the target facial expressions, unlike existing methods. Furthermore, we implement a particle filter utilizing the variable-intensity template as a face model to simultaneously estimate facial poses and expressions. The remainder of this paper is organized as follows. First, our proposed method is described in Section 2. Then, in Section 3, experimental results are given. Finally, a summary and future works are given in Section 4.
2
Proposed Method
Our method consists of two stages: a calibration stage and a test stage (see Fig.2). The calibration stage generates a variable-intensity template. In the test stage, our method estimates the facial pose and expression simultaneously within the framework of a particle filter. 2.1
Variable-Intensity Template
The variable-intensity template M consists of the following three components: M = {S, P, L} Calibration Stage
Variable-Intensity Template
Face detection
Shape Model
Interest Points Extraction
Interest Points
Calibration
Intensity Distribution Model
Fig. 2. System flow chart
(1) Test Stage Input Video Sequences Estimation (Particle Filter ) Estimated States
Pose-Invariant Facial Expression Recognition
327
Edge detection
Extraction of pairs of points
Eyebrow Input image
Edge image
Extraction results
Interest points
Enlarged
Face image in neutral expression
Fig. 3. Left: The method to extract interest points P. Right: The extraction result. Small white rectangles represent interest points; point pairs are indicated by the lines. The set of large rectangles represents the regions holding the interest points.
where S, P, and L denote a rigid facial shape model, a set of interest points, and an intensity distribution model, respectively. The intensity distribution model describes the intensity distribution of each interest point for different facial expressions. Shape Model S. A cylinder is used as the facial shape model because of its geometric simplicity. The radius of the cylinder is calculated as the width of the face region detected in a neutral expression image by the method of [13] multiplied by a fixed constant. Set of Interest Points P. The set of interest points P is described as follows: P = {p1 , · · · , pN }
(2)
where pi denotes the image coordinates of point i and N denotes the number of interest points in the same image used to generate shape model S. An interest point constitutes a pair of points. The pairs that satisfy the following conditions are extracted, from the region including four kinds of facial parts (eyebrows, eyes, nose, and mouth) in ascending order of the interpair difference in intensity until the number of pairs in each facial part reaches the limit [12]: (1) Each pair of interest points straddles and is centered on an edge. (2) Pairs are separated by at least a predefined distance (see Fig.3). The edges are obtained as zero-cross boundaries of the Laplacian-Gaussian filtered image. Intensity Distribution Model L. The intensities of interest points are assumed to follow independent normal distributions, because each point is apart from the other point and the normal distribution can adequately express the effect of the position shifts due to shape model error and imaging noise. The intensity distribution model L describes how the means and standard deviations of the distributions vary for discrete facial expressions. That is, the intensity distribution model of each interest point is the mixture distribution that consists of distributions for each facial expression.
328
S. Kumano et al. Interest point
(2) Eyebrow lowerer
Probability
Eyebrow
Face Expression Neutral
Angry
(3) Change in intensity of interest point
Intensity of interest point
Intensity distribution model (1) Change in facial expression
Fig. 4. Intensity distribution model L: The intensity distributions of interest points, described as normal distributions, change in facial expressions. The colors in the right part correspond to the interest points in the left part.
The intensity distribution model L is described as follows: L = {N1 , · · · , NN } , Ni = N μi (e), σi2 (e) , σi (e) = kμi (e)
(3) (4)
where N (μ, σ 2 ) denotes a normal distribution with mean μ and standard deviation σ, e ∈ {0, · · · , Ne − 1} denotes facial expression, Ne denotes the number of target expressions, e = 0 expresses neutral expression, μi (e) and σi (e) denote the mean and standard deviation of the intensity of point i for expression e, respectively. Standard deviation σi is assumed to be proportional to mean μi with a constant of proportionality k. The changes in facial expressions cause large changes in intensity around facial parts (see Fig.4). The intensity distribution model is used to these changes for different facial expressions. Our method generates intensity distribution models for each person by using one frontal face image for each facial expression without any head movement during the capture process. By using this neutral expression image to also generate shape model S and extract the set of interest points P, the intensity means of the interest points in each expression can be set to be the intensities on the pixels where these points were defined. For the standard deviation, our method employs a large k, to reduce the effect of calibration errors and changes in intensity of the face caused by changes in facial poses which alter the illumination direction of the face. 2.2
Simultaneous Estimation of Facial Pose and Expression by Using a Particle Filter
Our method simultaneously estimates the facial pose and expression by calculating the likelihood of the intensity of interest points for the intensity distribution model. The joint distribution of facial pose and expression at time t given all face images up to that time (z 1:t ) is recursively represented as follows:
Pose-Invariant Facial Expression Recognition
P (ht , et |z 1:t ) = αP (z t |ht , et )
P (ht |ht−1 )
329
P (et |et−1 )
et−1
P (ht−1 , et−1 |z 1:t−1 )dht−1
(5)
where the facial pose state ht and expression state et follow first order Markov processes; ht and et are assumed to be conditionally independent given image z t ; Bayes’ rule and conditional independence are used along with marginalization [14], and α = 1/P (z t ) is a normalization constant. The facial pose state ht consists of the following six continuous variables: the coordinate of the center of the template on the image plane, three-dimensional rotation angles (roll, pitch, and yaw), and scale. We adopt a random walk model for each parameter of facial pose yielding state transition model P (ht |ht−1 ), and set P (et |et − 1) to be equal for all facial expression combinations. Equation (5), unfortunately, cannot be calculated exactly, because parameters of facial pose ht are continuous, and their distributions are complex due to occlusion, etc. Hence, we use a particle filter, which calculates Equation (5) by approximating the posterior density as a set of weighted samples called particles. Each particle expresses a state and its weight. In our method, the state and (l) (l) (l) (l) weight of the lth particle are expressed as [ht , et ] and ωt , where ωt is propor (l) (l) (l) tional to P (z t |ht , et ) calculated using Equation (6) and satisfies l ωt = 1. Likelihood of Observation. The likelihood of face image z t for facial pose ht and expression et is expressed as P (z t |ht , et ). Since the intensity of each interest point zi,t is assumed to be independent as described in Section 2.1, likelihood P (z t |ht , et ) is transformed as follows: P (z t |ht , et ) = P (zi,t |ht , et ) (6) i∈P
where P denotes the set of non-occluded interest points. Here, we consider that the interest point is not occluded if the surface normal of its corresponding point on the facial shape model is pointing toward the camera. We define the likelihood of point i in facial pose ht and expression et , P (zi,t |ht , et ), by adopting a robust estimation as follows:
zi,t − μi (et ) 1 1 exp − ρ P (zi,t |ht , et ) = √ , (7) 2 σi (et ) 2πσi (et ) 2 if x2 < x , (8) ρ(x) = , otherwise where μi (et ) and σi (et ) are the mean and standard deviation of the intensity of interest point i in facial expression et , respectively, and ρ(·) denotes a robust function. Intensity zi,t is the intensity of image z t at the image coordinate of point i at time t, q i,t . The image coordinate q i,t is calculated as q i,t = f (pi , S, ht ), where function f returns q i,t by projecting the image coordinate pi onto the shape model S, with translation and rotation of S according to pose ht , and performing weak perspective projection of the resulting threedimensional position onto the image plane.
330
S. Kumano et al.
Estimator of Facial Pose and Expression. Estimators of facial pose and
t and e t ) are calculated as follows: expression at time t (h (l) (l)
t = h ωt ht , (9) l
e t = arg max et
(l)
ωt
(10)
(l) l∈et =et
where estimated facial expression e t is defined as the maximum probability in facial expression P (et |z1:t ).
3
Experimental Results
To evaluate the usefulness of our method, we performed two types of tests on video sequences wherein subjects exhibited multiple facial expressions with the head fixed (Test 1) and freely changed (Test 2). The target facial expressions were neutral, angry, sad, surprise and happy. Grayscale video sequences with a size of 512 × 384 pixel were captured by an IEEE1394 camera at 15 fps for the subjects. The number of particles was set to 1,500, and the processing time was about 80 ms/frame on a Pentium D processor at 3.73GHz with 3.0GB RAM. 3.1
Details of the Experiment
Five male subjects participated in Test 1 once, and one male subject participated in Test 2 once. In Test 1, the subject showed five facial expressions one by one with the head fixed to an horizontal direction for a duration of 60 frames followed by a 60 frame interval, according to instructions displayed on a monitor. We targeted the following five yaw angles of the face relative to the camera direction: −40, −20, 0, 20, and 40 (degrees). In Test 2, the subject freely showed five facial expressions one by one while shaking the head left and right. 3.2
Facial Expression Recognition Results
The recognition rates of facial expression were calculated for Test 1. Ground truth of facial expression at every frame was defined to be the expression indicated by the instruction. In consideration of the time lag between the instruction and the exhibition of the facial expression, we excluded the first 20 frames of each expression just after the instruction was displayed. Figure 5 shows some estimation results of facial poses and expressions in Test1, where facial poses and expressions were correctly estimated in the images of all subjects. Recognition rate was calculated as the ratio between the number of frames wherein the estimated expression equaled the ground truth and the total number of target frames. Table 1 shows the average facial expression recognition rates of the five tests for each target yaw angle. The average rate in the range of ±40 degree yaw angles exceeded 90(%), although the recognition rate decreased as the yaw
Pose-Invariant Facial Expression Recognition Neutral Angry Sad Surprise Happy
331
Estimated probabilities of facial expressions
Grids of shape model (White lines)
Interest points (Small points)
Ground truth: Neutral
Ground truth: Angry
Sad
Ground truth: Surprise
Happy
Fig. 5. Some estimation results of facial poses and expressions in Test 1: White grids and small points on each face denote the shape model and interest points in the estimated facial pose. The width of each bar in the upper right part of each image denotes the estimated probability of each facial expression, P (et |z1:t ).
Table 1. Average recognition rates of facial expressions for each yaw angle: Test 1 Yaw angle (degree) total -40 Recognition rate (%) 90.7 91.4
-20 100.0
0 99.9
20 84.3
40 78.1
332
S. Kumano et al.
Table 2. Confusion matrix of average recognition rate of facial expressions in Test 1: The expressions in the rows are the ground truths and the expressions in the columns describe the recognition results Expression Neutral Angry Sad Surprise Happy Neutral 97.0 0.0 2.8 0.1 0.1 1.9 80.1 12.3 1.0 4.7 Angry 9.2 0.6 77.7 12.2 0.3 Sad 0.3 0.1 0.3 99.0 0.3 Surprise 0.1 0.0 0.0 0.0 99.9 Happy
(a) Input video sequence (from left to right, frame number 100, 290, 400, 560, 660). GT 1 NEUTRAL
Probability
0 1 ANGRY 0 1 SAD 0 1 SURPRISE 0 1 HAPPY 0 0
100
200
300
400
500
600
Frame Number
Angle (degrees)
(b) Ground truth (top) and recognition results (others) of facial expression: The probability of correct expression is remarkably higher than that of other expressions.
80 60 40 20 0 −20 −40 −60 −80 0
PITCH
100
200
300
YAW
ROLL
400
500
600
Frame Number
(c) Estimation results of facial pose (horizontal axis equals to that of (b)): Facial poses are estimated with enough accuracy to detect three cycles of head shake movement (red solid line). Fig. 6. Images and estimation results in Test 2
angle increased. It seems that the recognition rates decrease more with positive yaw angles than with negative yaw angles because of lighting asymmetry. That is, if the lighting condition is horizontally symmetric against the face, it
Pose-Invariant Facial Expression Recognition
333
is expected that the recognition rates with positive yaw angles will increase to a similar extent as negative yaw angles. Table 2 shows the confusion matrix for the recognition rates. It seems that angry and sad expressions were most similar, because they were sometimes confused each other. Five frames of the video sequence and the estimated results of facial expression and pose in each frame in Test 2 are shown in (a), (b) and (c) of Fig.6. Moreover, ground truth of the facial expression at every frame was determined manually. Figure 6(b) shows that facial expressions were recognized correctly in almost all frames. In addition, the correct expressions were assigned remarkably higher probabilities than other expressions. Fig.6(c) also shows that facial poses were estimated with enough accuracy to detect three cycles of head shake movement. Video sequences for the result in Test 1 and 2 are available from [15].
4
Summary and Future Works
In this paper, we presented a particle filter-based method for estimating facial pose and expression simultaneously by using a novel face model called the variable intensity template. Our method has the distinct advantage that a face model for each person can be prepared very easily with a simple calibration step. With our method, five facial expressions were recognized with 90.7% of accuracy for horizontal facial orientations over the range of ±40 degrees from the frontal view. In the future, we would like to conduct more experiments with additional subjects to complete the statistical evaluation. In the current framework, our method cannot correctly estimate facial poses and expressions under the large variation in intensities caused by large head movements, especially vertical direction, or lighting variations. Hence, we are planning to handle such intensity variations by updating the intensity of each interest point (e.g. [16]). Another goal is to achieve a fully automatic system by applying an online clustering technique for extracting target facial expressions, instead of calibrating the intensities for facial expressions in advance. In addition, We would also like to improve the way interest points are defined by using the component corresponding to the point of a projection matrix that maximizes the ratio of between and within class scatter of the facial expression images (i.e. Fisherfaces [17]) for optimizing point selection.
References 1. Otsuka, K., Yamato, J., Takemae, Y., Murase, H.: Conversation scene analysis with Dynamic Bayesian Network based on visual head tracking. In: Proc. of the IEEE International Conference on Multimedia and Expo, pp. 949–952. IEEE Computer Society Press, Los Alamitos (2006) 2. Cohen, I., Sebe, N., Chen, L., Garg, A., Huang, T.: Facial expression recognition from video sequences: Temporal and static modeling. Computer Vision and Image Understanding 91, 160–187 (2003)
334
S. Kumano et al.
3. Kaliouby, R., Robinson, P.: Generalization of a vision-based computational model of mind-reading. In: Proc. of the First International Conference on Affective Computing and Intelligent Interatction, pp. 582–589 (2005) 4. Chang, Y., Hu, C., Feris, R., Turk, M.: Manifold based analysis of facial expression. Image and Vision Computing 24, 605–614 (2006) 5. Bartlett, M., Littlewort, G., Frank, M., Lainscsek, C., Fasel, I., Movellan, J.: Automatic recognition of facial actions in spontaneous expressions. Journal of Multimedia 1, 22–35 (2006) 6. Gokturk, S.B., Tomasi, C., Girod, B., Bouguet, J.Y.: Model-based face tracking for view-independent facial expression recognition. In: Proc. of the Fifth IEEE International Conference on Automatic Face and Gesture Recognition, pp. 287–293. IEEE Computer Society Press, Los Alamitos (2002) 7. Oka, K., Sato, Y.: Real-time modeling of face deformation for 3D head pose estimation. In: Proc. of the IEEE International Workshop on Analysis and Modeling of Faces and Gestures, pp. 308–320. IEEE Computer Society Press, Los Alamitos (2005) 8. Dornaika, F., Davoine, F.: Simultaneous facial action tracking and expression recognition using a particle filter. In: Proc. of the Tenth IEEE International Conference on Computer Vision, vol. 2, pp. 1733–1738 (2005) 9. Zhu, Z., Ji, Q.: Robust real-time face pose and facial expression recovery. In: Proc. of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 681–688. IEEE Computer Society Press, Los Alamitos (2006) 10. Lucey, S., Matthews, I., Hu, C., Ambadar, Z., Torre, F., Cohn, J.: AAM derived face representations for robust facial action recognition. In: Proc. of the 7th International Conference on Automatic Face and Gesture Recognition, pp. 155–160 (2006) 11. Gross, R., Matthews, I., Baker, S.: Generic vs. person specific Active Appearance Models. Image and Vision Computing 23, 1080–1093 (2005) 12. Matsubara, Y., Shakunaga, T.: Sparse template matching and its application to real-time object tracking. IPSJ Transactions on Computer Vision and Image Media 46(9), 17–40 (2005) 13. Viola, P.A., Jones, M.J.: Rapid object detection using a boosted cascade of simple features. In: Proc. of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 511–518. IEEE Computer Society Press, Los Alamitos (2001) 14. Rich, E., Knight, K.: Artificial intelligence, pp. 537–583. McGraw-Hill Book Company, New York (1991) 15. http://www.hci.iis.utokyo.ac.jp∼ kumano/papers/accv2007.html 16. Jepson, A.D., Fleet, D.J., El-Maraghi, T.F.: Robust online appearance models for visual tracking. IEEE Transactions on Pattern Analysis and Machine Intelligence 25, 1296–1311 (2003) 17. Belhumeur, P.N., Hespanha, J., Kriegman, D.J.: Eigenfaces vs. Fisherfaces: Recognition using class specific linear projection. IEEE Transactions on Pattern Analysis and Machine Intelligence 19, 711–720 (1997)
Gesture Recognition Under Small Sample Size Tae-Kyun Kim1 and Roberto Cipolla2 2
1 Sidney Sussex College, University of Cambridge, Cambridge, CB2 3HU, UK Department of Engineering, University of Cambridge, Cambridge, CB2 1PZ, UK
Abstract. This paper addresses gesture recognition under small sample size, where direct use of traditional classifiers is difficult due to high dimensionality of input space. We propose a pairwise feature extraction method of video volumes for classification. The method of Canonical Correlation Analysis is combined with the discriminant functions and Scale-InvariantFeature-Transform (SIFT) for the discriminative spatiotemporal features for robust gesture recognition. The proposed method is practically favorable as it works well with a small amount of training samples, involves few parameters, and is computationally efficient. In the experiments using 900 videos of 9 hand gesture classes, the proposed method notably outperformed the classifiers such as Support Vector Machine/Relevance Vector Machine, achieving 85% accuracy.
1
Introduction
Gesture Recognition Review Gesture recognition is an important topic in computer vision because of its wide ranges of applications such as human-computer interfaces, sign language interpretation and visual surveillance. Not only spatial variation but also temporal variation among gesture samples make this recognition problem difficult. For instance, different subjects have different hand appearance and may sign gesture in different pace. Recent work in this area tends to handle the above variations separately and therefore leads to two smaller areas, namely posture recognition (static) and hand motion or action recognition (dynamic). In posture recognition, the pose or the configuration of hands is recognised using silhouette [5] and texture [6]. By contrast, hand motion or action recognition interprets the meaning of the movement using full trajectory [9], optical flow [4] and motion gradient [11]. Compared with hand motion recognition, posture recognition is easier in the sense that state-of-the-art classifiers, e.g. Support Vector Machine, Relevance Vector Machine [11] or Adaboot [6] can be directly applied to it. Gesture recognition, on the other hand, has adopted rather different approaches, e.g. Hidden Markov Model [9]) or Dynamic Time Warping [3]), to discriminate dynamic/or temporal information which is typically highly non-linear in a data space. These methods, especially the Hidden Markov Models, have many parameters to set, a large amount of training examples, and difficulty for extension to large vocabulary [2]. Besides, these traditional methods have not integrated the posture and Y. Yagi et al. (Eds.): ACCV 2007, Part I, LNCS 4843, pp. 335–344, 2007. c Springer-Verlag Berlin Heidelberg 2007
336
T.-K. Kim and R. Cipolla
temporal information and thus are difficult to differentiate gestures of similar movements signed by different hand shapes. Some recent works [8] directly operate with full spatiotemporal volume considering both posture and temporal information of gestures to a certain degree, but are still unreliable in cases of motion discontinuities and motion aliasing. Also, the method [8] requires the manual setting of the important parameters such as positions and scales of local space-time patches. Another important line of methods exploits visual code words (for representation) with either a Support Vector Machine (SVM) or a probabilistic generative model [12,13]. Again, for their good performance, it is critical to properly set the parameters associated with the representation, for e.g. space-time interest points and code book size. Motivation and Summary of This Study To avoid empirical setting of the parameters in the existing methods, it seems obvious to seek a more generic and simpler learnable approach for gesture recognition. Note that many of critical parameters in the previous methods are incurred in the step of representing gesture videos prior to using classifiers. In that case, it could be better to apply learnable classifiers directly to the videos which can be simply converted into column vectors. Unfortunately, this is not a good way either. Vectorization of a video by concatenating all pixels in the three-dimensional video volume causes a high dimension of N 3 , which is much larger than N 2 of an image. Also, it may be more difficult to collect sufficient number of video samples for classifiers than images (see that a single video consists of multiple images). So called small sample size problem is more serious in learning classifiers with videos than images. Getting back to the representation issue, this work focuses on how to learn useful features from videos for classification, discussing its benefits over direct using classifiers. With the given discriminative features, even a simple Nearest Neighbor classifier (NN) achieved a very good accuracy. An extension of Canonical Correlation Analysis (CCA) [1,15]-a standard tool of inspecting linear relationships of two sets of vectors- is proposed to yield robust pairwise features of any two gesture videos. The proposed method is closely related to our previous framework of Tensor Canonical Correlation Analysis [14], which extends the classical CCA into multidimensional data arrays by sharing either a single axis or two axes. The method of sharing two axes, i.e. planes between two video data, is updated and combined with the discriminative functions and the Scale-Invariant-Feature-Transform for further improvements. The proposed method does not require any significant meta-parameters to be adjusted and can learn both posture and temporal information for gesture classification. The rest of the paper is organized as follows: Next section explains the proposed method with the discriminant functions, discussing the benefit of the method over traditional classifiers. The SIFT representation for video data is combined to the method for improvements in Section 3. Section 4 shows the experimental results and Section 5 draws conclusion.
Gesture Recognition Under Small Sample Size
2
337
Discriminative Spatiotemporal Canonical Correlations
Canonical Correlation Analysis (CCA) has been a standard tool of inspecting linear relationships of two random variables, or two sets of vectors. This was recently extended to two multidimensional data arrays in [14]. The method of spatiotemporal canonical correlations (which is related to the previous work in exploiting planes rather than scan vectors of two videos) is explained as follows: A gesture video is represented by firstly decomposing an input video clip (i.e. a spatiotemporal volume) into three sets of orthogonal planes, namely XY-, YT- and XT-planes as shown in Figure 1. This allows posture information in XY-planes and joint posture/dynamic information in YT and XT-planes. Three kinds of subspaces are learnt from the three sets of planes (which are converted into vectors by raster-scanning). Then, gesture recognition is done by comparing these subspaces with the corresponding subspaces from the models by classical canonical correlation analysis, which measures principal angles between subspaces1 . By comparing subspaces of an input and a model, robust gesture recognition can be achieved up to pattern variations on the subspaces. The similarity of any model Dm and query spatiotemporal data Dq is defined as the weighted sum of the normalized canonical correlations of the three subspaces by 3 wk N k (Pkm , Pkq ) F (Dm , Dq ) = Σk=1
(2)
N k (Pkm , Pkq ) = (G(Pkm , Pkq ) − mk )/σ k ,
(3)
where, P1 , P2 , P3 denotes a matrix containing the first few eigenvectors in its columns of XY-planes, XT-planes, YT-planes respectively and G(Pm , Pq ) sum of the canonical correlations computed from Pm , Pq . The normalization parameters with index k are mean and standard deviation of matching scores, i.e. G of all pairwise videos in a validation set for the corresponding planes. The discriminative spatiotemporal canonical correlation is defined by applying the discriminative transformation [10] learnt from each of the three data domains as 3 wk N k (h(QkT Pkm ), h(QkT Pkq )), (4) H(Dm , Dq ) = Σk=1 1
Canonical correlations between two d-dimensional linear subspaces L1 and L2 are uniquely defined as the maximal correlations between any two vectors of the subspaces [1]: (1) ρi = cos θi = max max uTi vi ui ∈L1 vi ∈L2
= = 1, = viT vj = 0, j = 1, ..., i − 1. We will refer to subject to: ui and vi as the i-th pair of canonical vectors. Multiple canonical correlations are defined by having next pairs of canonical vectors orthogonal to previous ones. The solution is given by SVD of PT1 P2 as uTi ui
viT vi
uTi uj
PT1 P2 = LΛRT
where
Λ = diag{ρ1 , ..., ρd }.,
where P1 , P2 are the eigen-basis matrix, L, Λ, R are the outputs of SVD.
338
T.-K. Kim and R. Cipolla
where h is a vector orthonormalization function and Qk are the discriminative transformation matrix learnt over the corresponding sets of planes. The discriminative matrix is found to maximize the canonical correlations of within-class sets and minimizes the canonical correlations of between-class sets by analogy to the optimization concept of Linear Discriminant Analysis (LDA) (See [10] for details). On the transformed space, gesture video classes are more discriminative in terms of canonical correlations. In this paper, this concept has been validated not only for the spatial domain (XY-subspaces) but also for the spatiotemporal domains (XT-, YT-subspaces). Discussions The proposed method is a namely divide-and-conquer approach by partitioning original input space into the three different data domains, learning the canonical correlations on each domain, and then aggregating them with proper weights. By this way, the original data dimension N 3 , where N is the size of each axis, is reduced into 3 × N 2 so that the data is conveniently modelled. As shown in Figure 2a-c, each data domain is well-characterized by the corresponding lowdimensional subspace (e.g. hand shapes in XY-planes, joint spatial and temporal information in YT-, and XT- planes).
Y X T
Spatiotemporal Volume
XY-planes
XT-planes
YT-planes
Fig. 1. Spatiotemporal Data Representation
(a)
(b)
(c)
(d)
(e)
(f)
Fig. 2. Principal Components and Canonical Vectors: The first few principal components of the (a) XY (b) XT (c) YT subspaces of the two different illumination sequences of the same gesture class (See Figure 5) are shown at the top and bottom row respectively. The corresponding pairwise canonical vectors are visualized in (d) (f). Despite the different lighting conditions of the two input sequences, the canonical vectors in the pair (top and bottom) are very much alike, capturing common modes.
Gesture Recognition Under Small Sample Size
339
Moreover, the method is robust using mutual (or canonically correlated) components of the pairwise subspaces. By finding the mutual components of maximum correlations, which are canonical correlations, some undesirable information for classification can be filtered out. See Figure 2 for the principal components and canonical vectors for the given two sequences of the same gesture class which were captured under the different lighting conditions. Whereas the first few principal components mainly corresponded to the different lighting conditions (in Figure 2a-c), the canonical vectors (in Figure 2d-f) well captured the common modes of the two sequences, being visually same in each pair. In other words, the lighting variations across the two sets were removed in the process of CCA, as it is invariant to any variations on the subspaces. Many previous studies have told that lighting variations are often confined to a low-dimensional subspace. In summary, the proposed method has a benefit over direct learning classifiers under small sample size as drawn in Figure 3. High dimensional input space and a small training set often cause overfitting of classifiers to the training data and poor generalization to new test data. Distribution of the test samples taken under different conditions can be largely deviated from that of the training set, resulting in the majority of the test samples of class 1 misclassified in Figure 3. Nevertheless, the two Fig. 3. Canonical Correlaintersection sets of the train and test sets are still tion Based Classification placed in the correct decision regions learnt over the training sets. As discussed above, canonical correlation analysis can be conceptually seen as a process to find mutual information (or an intersection set) of any two sets.
3
SIFT Descriptor for Spatiotemporal Volume Data
Edge-based description of each plane of videos can help the method achieve more robust gesture recognition. In this section we propose a simple and effective SIFT (Scale-Invariant Feature Transform) [7] representation for a spatiotemporal data by a fixed grid. As explained, the spatiotemporal volume is broken down into three sets of orthogonal planes (XY-, YT- and XT-planes) in the method. Along each data domain, there is a finite number of planes which can be regarded as images. Each of these images is further partitioned into M × N patches in a predefined fixed grid and the SIFT descriptor is obtained from each patch (see Figure 4a). For each image, the feature descriptor is obtained by concatenating the SIFT descriptors of several patches in a predefined order. The SIFT representation of the three sets of planes is directly integrated into the proposed method in Section 2 by replacing the sets of image vectors with the sets of the SIFT descriptors prior to canonical correlation analysis. The experimental results tell that the edge-based representation generally improves the intensity-based
340
T.-K. Kim and R. Cipolla
representation in both of the joint space-time domain (YT-, XT-planes) and the spatial domain (XY-planes).
Region of Interest (ROI)
Histogram "Mask" on XY-plane
(a)
SIFT on XY-plane Region of Interest (ROI)
Histogram "Masks" on XY- and YT-plane
Composite descriptor of size 256
Analysis on dx, dy and dt under ROI
SIFT on YT-plane
dx dy Analysis on dx and dy under the ROI
SIFT Descriptor of Size 128
SIFT obtained from 3D blocks. This section presents a general 3D extension of SIFT features. Traditional classifiers such as Support Vector Machine (SVM)/ Relevance Vector Machine (RVM) are applied to the video data represented by the 3D SIFT so that they can be compared with the proposed method (with SIFT) in the same data domain. Given a spatiotemporal volume representing a gesture sequence, the volume is firstly partitioned into M × N × T tiny blocks. Within each tiny block, further analysis is done along XY-planes and YT-planes (see Figure 4b). For analysis on a certain plane, say XY-planes, derivatives along X- and Y- dimensions are obtained and accumulated to form several regional orientation histograms (under a 3D Gaussian weighting scheme). For each tiny block, the resultant orientation histograms of both planes are then concatenated to form the final SIFT descriptor of dimension 256. The descriptor for the whole spatiotemporal volume can then be formed by concatenating the SIFT descriptors of all tiny blocks in a predefined order. The spatiotemporal volume is eventually represented as a single long concatenated vector.
(b)
Fig. 4. SIFT Representation: (a) SIFT used in [7]. (b) SIFT from 3D blocks (refer to text).
4
Empirical Evaluation
4.1
Cambridge Hand Gesture Data Set and Experimental Protocol
We have acquired the hand-gesture data base 2 consisting of 900 image sequences of 9 gesture classes. Each class has 100 image sequences (5 different illuminations × 10 arbitrary motions of 2 subjects). Each sequence was recorded in a frame rate of 30fps and a resolution of 320×240. The 9 classes are defined by the 3 primitive hand shapes and 3 primitive motions (See Figure 5). See Figure 5c for the example images captured under the 5 different illumination settings. The 2
The database is available upon request. Contact e-mail: [email protected]
Gesture Recognition Under Small Sample Size
Flat/Leftward
class1
Flat/Rightward
class2
Flat/Contract
class3
Spread/Leftward
class4
Spread/Rightward
class5
Spread/Contract
class6
V-shape/Leftward
class7
V-shape/Rightward
class8
V-shape/Contract
class9
341
(a) 9 gesture classes formed by 3 shapes and 3 motions.
(b) 5 illumination settings. Fig. 5. Hand-Gesture Database
data set has temporally isolated gesture sequences which exhibit variations in initial positions, postures of hands and speed of movements in different sequences. All training was performed on the data acquired in a single illumination setting while testing was done on the data acquired in the remaining settings. The 20 sequences in the training set were randomly partitioned into the 10 sequences for training and the other 10 for the validation. 4.2
Results and Discussions
We compared the accuracy of 9 different methods: – Applying Support Vector Machine (SVM) or Relevance Vector Machine (RVM) on Motion Gradient Orientation Images [11] (MGO SVM or MGO RVM), – Applying RVM on the 3D SIFT vectors described in Section 3 (3DSIFT RVM), – Using the canonical correlations (CC) (i.e. the method using G(P1m , P1q ) in (2), spatiotemporal canonical correlations (ST-CC), discriminative ST-CC (ST-DCC), – Using the canonical correlations of the SIFT descriptors (SIFT CC), spatiotemporal canonical correlations of the SIFT vectors (SIFT ST-CC), and SIFT ST-CC with the discriminative transformations (SIFT ST-DCC).
342
T.-K. Kim and R. Cipolla
Fig. 6. Recognition Accuracy: The identification rates (in percent) of all comparative methods are shown for the plain lighting set used for training and all the others for testing
In the proposed method, the weights wk were set up proportionally to the accuracy of the three subspaces for the validation set and Nearest Neighbor classification (NN) was done with the defined similarity functions. Figure 6 shows the recognition rates of the 9 methods, when the plain lighting set (the leftmost in Figure 5c) was exploited for training and all the others for testing. The approaches of using SVM/RVM on the motion gradient orientation images are the worst. As observed in [11], using RVM improved the accuracy of SVM by about 10% for MGO images. However, we got much poorer accuracy than those in the previous study [11] mainly due to the following reasons: The gesture classes in this study were defined by hand shapes as well as motions. Both methods often failed to discriminate the gestures which exhibit the same motion of the different shapes, as the methods are mainly based on motion information of gestures. A much smaller number of sequences (of a single lighting condition) used in training is another reason to get the performance degradation. The accuracy of the RVM on the 3D-SIFT vectors was also poor. The high dimension of the 3D-SIFT vectors and small sample size might prevent the classifier from learning properly, as discussed. We measured the accuracy of the RVM classifier for the different numbers of the blocks in the 3D-SIFT representations (2-2-1,3-3-1,4-4-1,4-4-2 for X-Y-T) and obtained the best accuracy for the 2-2-1 case, which yields the lowest dimension of the 3D-SIFT vectors. Canonical correlation-based methods significantly outperformed the previous approaches. The proposed spatiotemporal canonical correlation method (ST-CC) improved the simple canonical correlation method by about 15%. The proposed discriminative method (ST-DCC) unexpectedly decreased the accuracy of STCC, possibly due to overfitting of discriminative methods. The train set did not reflect the lighting conditions in the test set. However, note that the discriminative method improved the accuracy when it was applied to the SIFT representations rather than using intensity images (See SIFT ST-CC and SIFT
Gesture Recognition Under Small Sample Size
343
Table 1. Evaluation of the individual subspace
(%) mean std
XY 64.5 1.3
CC XT 40.2 5.9
YT 56.2 5.3
ST 78.9 2.4
XY 70.3 2.1
SIFT CC XT 61.8 3.3
YT 58.3 4.0
ST 80.4 3.2
Table 2. Evaluation for different numbers of blocks in the SIFT representation: E.g. 2-2-1 indicates the SIFT representation where X,Y,and T axes are divided into 2,2,1 segments respectively 2-2-1 3-3-1 4-4-1 4-4-2 (%) ST-CC ST-DCC ST-CC ST-DCC ST-CC ST-DCC ST-CC ST-DCC mean 80.3 80.0 78.9 83.8 80.4 85.1 75.9 83.4 1.9 2.5 3.6 2.7 3.2 2.8 2.4 0.7 std
ST-DCC in Figure 6). The proposed three methods using the SIFT representations are better than the respective three methods of the intensity images. The best accuracy was achieved by the SIFT ST-DCC at 85%. Table 1 and Table 2 show more results about the proposed method, where all 5 experimental results (corresponding to each illumination set used for training) are averaged. As shown in Table 1 canonical correlations of the XY subspace obtained better accuracy with smaller standard deviations than the other two subspaces, but all three are relatively good compared with the traditional methods, MGO SVM/RVM and 3DSIFT RVM. Using the SIFT representation considerably improved the accuracy of the intensity images for each subspace, whereas the improvement for the joint representation was relatively small. Table 2 shows the accuracy of ST-CC and ST-DCC for the different numbers of the blocks of the SIFT representation. The best accuracy was obtained for the case of 4-4-1 for XYT (each number indicates the number of divisions along one axis). Generally, using the discriminative transformation improved the accuracy of ST-CC for the SIFT representation. Note that accuracy of the method is not sensitive about settings in number of the blocks, which is practically important. Also, the proposed approach based on canonical correlations is computationally cheap taking computations O(3 × d3 ), where d is the dimension of each subspace (which was 10), and thus facilitates efficient gesture recognition in a large data set.
5
Conclusion
A new method based on subspace has been proposed for gesture recognition under small sample size. Unlike typical classification approaches directly operating with input space, the proposed method reduces input dimension using the three sets of orthogonal planes. The method provides robust spatiotemporal volume
344
T.-K. Kim and R. Cipolla
matching by analyzing mutual information (or canonical correlations) between any two gesture sequences. Experiments for the 900 gesture sequences showed that the proposed method significantly outperformed the traditional classifiers and yielded the best classification result using the discriminative transformations and SIFT descriptors jointly. The method is also practically attractive as it does not involve significant tuning parameters and is computationally efficient.
References 1. Bj¨ orck, ˚ A., Golub, G.H.: Numerical methods for computing angles between linear subspaces. Mathematics of Computation, 27(123) 579–594, 1973. 2. Bowden, R., Windridge, D., Kadir, T., Zisserman, A., Brady, M.: A linguistic feature vector for the visual interpretation of sign language. In: ECCV, pp. 390– 401 (2004) 3. Darrell, T., Pentland, A.: Space-time gestures. In: Proc. of CVPR, pp. 335–340 (1993) 4. Efros, A.A., Berg, A.C., Mori, G., Malik, J.: Recognizing action at a distance. In: Proc. of ICCV, pp. 726–733 (2003) 5. Freeman, W., Roth, M.: Orientation histogram for hand gesture recognition. In: Int’l Conf. on Automatic Face and Gesture Recognition (1995) 6. Just, A., Rodriguez, Y., Marcel, S.: Hand posture classification and recognition using the modified census transform. In: Int’l Conf. on Automatic Face and Gesture Recognition (2006) 7. Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vision 60(2), 91–110 (2004) 8. Shechtman, E., Irani, M.: Space-time behavior based correlation. In: Proc. of CVPR 2005, pp. 405–412 (2005) 9. Starner, T., Pentland, A., Weaver, J.: Real-time american sign language recognition using desk and wearable computer based video. IEEE Trans. Pattern Anal. Mach. Intell. 20(12), 1371–1375 (1998) 10. Kim, T., Kittler, J., Cipolla, R.: Discriminative learning and recognition of image set classes using canonical correlations. IEEE Trans. on PAMI 29(6), 1005–1018 (2007) 11. Wong, S., Cipolla, R.: Real-time interpretation of hand motions using a sparse bayesian classifier on motion gradient orientation images. In: Proc. of BMVC 2005, pp. 379–388 (2005) 12. Niebles, J.C., Wang, H., Fei-Fei, L.: Unsupervised learning of human action categories using spatial-temporal words. In: BMVC (2006) 13. Schuldt, C., Laptev, I., Caputo, B.: Recognizing human actions: A local SVM approach. In: ICPR 2004, pp. 32–36 (2004) 14. Kim, T., Wong, S., Cipolla, R.: Tensor Canonical Correlation Analysis for Action Classification. In: CVPR (2007) 15. Hardoon, D., Szedmak, S., Taylor, J.S.: Canonical correlation analysis; An overview with application to learning methods. Neural Computation 16(12), 639–2664 (2004)
Motion Observability Analysis of the Simplified Color Correlogram for Visual Tracking Qi Zhao and Hai Tao Department of Computer Engineering University of California at Santa Cruz, Santa Cruz, CA 95064 {zhaoqi,tao}@soe.ucsc.edu
Abstract. Compared with the color histogram, where the position information of each pixel is ignored, a simplified color correlogram (SCC) representation encodes the spatial information explicitly and enables an estimation algorithm to recover the object orientation. This paper analyzes the capability of the SCC (in a kernel based framework) in detecting and estimating object motion and presents a principled way to obtain motion observable SCCs as object representations to achieve more reliable tracking. Extensive experimental results demonstrate the reliability of the tracking procedure using the proposed algorithm.
1
Introduction
The computer vision community has witnessed the development of several excellent tracking algorithms using color statistics, e.g., color histogram, as representations. There statistics features can be convolved with an isotropic kernel to allow gradient estimation of the representation [1]. One inherent limitation of such kernel based methods is the singularity problem, where the representation is blind to certain motion. Most existing kernel based tracking algorithms are concerned only with the tracking of object locations [1], or object locations and scales [2]. Since isotropic kernels are often used [1,3,4], rotational motion can not be estimated using these methods. Recently, Zhao and Tao [5] proposed the simplified color correlogram (SCC) representation to efficiently track location and orientation simultaneously. Although the kernel is also rotationally-symmetric, the underlying SCC representation is sensitive to orientation changes. This property makes the representation capable of tracking rotational motion as well as translational motion. As in most kernel based algorithms, the assumption is that the statistics of the SCC feature be sufficient to determine the motion of the object [4]. However, this assumption needs to be validated. This paper shows that under certain degenerated cases, ill-conditioned cases may occur, i.e., translational and/or rotational motion may not cause changes to the SCC, therefore the motion is not observable. In this study, we derive a criterion to evaluate the numerical stability of the tracking solution, according to which, schemes for the SCC selection are designed. The paper is organized as follows. Section 2 reviews the simplified color correlogram (SCC). Section 3 analyzes the properties of the SCC in a kernel based Y. Yagi et al. (Eds.): ACCV 2007, Part I, LNCS 4843, pp. 345–354, 2007. c Springer-Verlag Berlin Heidelberg 2007
346
Q. Zhao and H. Tao
framework, and proposes the solution to obtain the motion observable SCCs. Section 4 discusses implementation details. Section 5 shows experimental results and demonstrates the improvement over the standard mean shift algorithm, and section 6 concludes the paper.
2
Introduction to Simplified Color Correlogram (SCC)
Color correlogram expresses the correlation between color pairs in an image, which has been commonly used in the literature of image retrieval/indexing [6]. Zhao and Tao [5] has recently proposed a simplified version of color correlogram (SCC) for tracking purpose. Instead of including pixel pairs along all directions and with a set of distances, the SCC only counts pairs along one or several axes, i.e., pre-selected directions, with predefined distances. Formally, the SCC with L axes are defined as Su,v = P r(I(p1 ) = u ∧ I(p2 ) = v | f (p1 − p2 ) = (θ, d)).
(1)
Here, f is a function to obtain the direction and the distance of a pixel pair, representing the spatial relationship of two pixels p1 and p2 . θ ∈ {θl , l = 1, ..., L} and d ∈ {dl , l = 1, ..., L}, where L is the number of axes, θl is the direction of axis l and dl is the pair distance along axis l. We use the L2 norm to measure the distance between pixels. Though the conclusions about the singularity problem in the kernel based representations made in this study is not restricted to the SCC representation, we focus our discussions on the SCC due to the following reasons [5]: 1. The SCC achieves a natural integration of color and spatial information. 2. Since certain directions are emphasized, the SCC is effective in manifesting rotational variations. 3. The SCC is computationally inexpensive. 4. Being a middle ground of the template based methods and the pure statistics based representations, the SCC can have desired properties from both sides. Similar to the conventional color histogram, the SCC is also integrated into a kernel framework to allow efficient localization, thereby the singularity problem exists. This paper presents two methods to approach this issue in the SCC context, where both translation and rotation are concerned. One is to select more than one axis to form a multi-axis SCC, therefore motion ignored by pairs along one axis can be recovered along other axes, as will be justified later. This strategy suffices in most cases. However, efficiency consideration suggests an alternative of obtaining one optimal axis and its corresponding pair distance, so that the resulting SCC is the most sensitive to all different motion. Details of the two approaches are provided in the next section.
3
Motion Observability Analysis
The SCC based tracking method enables the detection of both translational and rotational motion, therefore reliable tracking in this context requires that both types of motion be distinctly observed and reliably recovered.
Motion Observability Analysis of the SCC for Visual Tracking
3.1
347
Representing Objects Using SCC
In this subsection, we first address necessary notations for the later analysis. Specifically, each pair of pixels counted in the SCC representation can be parameterized using a 3-dimensional vector Φ = [cx, cy, θ]T , where (cx, cy) are the image coordinates of the midpoint of each pair and θ is the angle between the axis of the object and the object coordinate system. Consider for a moment a target model/candidate for the SCC with one axis l and the index l is omitted to keep the notation simple. Target Model. Similar to [4], we define the matrix form of the target model as T M = αUM K(0). (2) In Eqn.2, UM = [u11 , u12 · · · u1m , u21 · · · u2m · · · um1 , um2 · · · umm ], where urs = [δ(I(Φ111 ) − Irs ), δ(I(Φ112 ) − Irs ) · · · δ(I(ΦW HO ) − Irs )]T , r, s = 1 · · · m. I(Φijk ) represents the colors of the pixel pair Φijk in the image I. W , H and O are the numbers of cxs, cys and θs considered in the SCC. If the colors of pixels in the pair Φijk are r and s, then the corresponding element in the vector is assigned 1, otherwise it is 0. The subscript M represents “model” and α normalizes the Φ112 ΦW HO T )] , where K is a kernel representation. K(0) = [K( Φ111 h ), K( h ) · · · K( h function that assigns a smaller weight to the locations and orientations that are farther from the center of the object, h is the kernel radius, and the kernel is centered at 0. By definition, L different axes yield L different target models M 1 , ..., M L . Target Candidate. Similarly, the target candidate is defined as C(Φ0 ) = βUCT K(Φ0 ),
(3)
where Φ0 is the initialized location and orientation in the current frame. β and UC are defined the same way as α and UM for the target model, and the subscript −Φ0 T C denotes “candidate”. K(Φ0 ) = [K( Φ111h−Φ0 ), K( Φ112h−Φ0 ) · · · K( ΦW HO )] . h 1 L L target candidates C , ..., C can be defined for L axes. 3.2
The Objective Function and Solution for Reliable Tracking
We focus on the single-axis case in this subsection and the multi-axis cases would be analyzed in the next subsection. For the mean shift based tracking algorithms [1,5], the objective is to seek the maximum of the Bhattacharyya coefficient [7]. Its well known connection with the Matusita metric opens the possibility that we analyze the Matusita metric other than the Bhattacharyya coefficient to better illustrate the inherent problem of kernel based tracking [4]. Using the notations given in section 3.1, the objective of tracking using the Matusita metric is defined as √ minΦ (DM (Φ)) = minΦ || M − C(Φ)||2 = minΦ ( ( Mu,v − Cu,v (Φ))2 ), u,v
(4) where Φ is the object location and orientation in the current frame.
348
Q. Zhao and H. Tao
A Newton-style iterative procedure is applied to convert this optimization problem to a more explicit form (Derivations provided in the Appendix) √ 1 d(C(Φ0 ))− 2 UCT JK (Φ0 )ΔΦ = 2( M − C(Φ0 )), (5) where Φ0 is the initialized object location and orientation for the current frame and d(C(Φ0 )) denotes the matrix with C(Φ0 ) on its diagonal. In Eqn.5, JK (Φ0 ) = Φ −Φ −Φ0 T [∇Φ K( Φ111h−Φ0 ), ∇Φ K( Φ112h−Φ0 ) · · · ∇Φ K( ΦW HO )] , where ∇Φ K( ijkh 0 ) = h Φijk −Φ0 2 2 || ), and g(x) = −k (x), k(||x||2 ) = K(x). h2 (Φijk − Φ0 )g(|| h 1 Denoting A = d(C(Φ0 ))− 2 (U )TC JK (Φ0 ) and converting the matrix before ΔΦ to a square matrix for further analysis, we obtain √ (6) AT AΔΦ = 2AT ( M − C(Φ0 )). Therefore the solution to the optimization problem in this 3-dimensional case is unique if and only if the 3 × 3 matrix AT A is of full rank. Additionally, the stability of the solution depends on the magnitude of its condition number. In the single-axis cases, the SCC with the parameters (θ, d) corresponding to the smallest condition number of AT A is the optimal SCC. 3.3
SCC with Multiple Axes
If multi-axis correlograms are used, denote ⎡ ⎤ 1 1 (Φ0 ) d(C 1 (Φ0 ))− 2 (UC1 )T JK ⎢ ⎥ .. AL = ⎣ ⎦, . L − 12 L T L d(C (Φ0 )) (UC ) JK (Φ0 ) and
l Al = d(C l (Φ0 ))− 2 (UCl )T JK (Φ0 ), l = 1, .., L, 1
then we have ATL AL =
L l=1
(Al )T Al .
In this paper, we explore further into this multi-axis problem considering the simple yet effective two-axis cases. A useful property of the semi-positive definite matrices (A1 )T A1 and (A2 )T A2 states as min(cond((A1 )T A1 ), cond((A2 )T A2 )) ≤ cond(AT2 A2 ) ≤ max(cond((A1 )T A1 ), cond((A2 )T A2 )),
(7)
where AT2 A2 = (A1 )T A1 + (A2 )T A2 . These inequalities indicate that the condition number of a two-axis SCC is between the two condition numbers of the corresponding single-axis SCCs. A consequence is that for a SCC defined with two axes, unfavorable condition numbers are less possible to be generated, since it requires both corresponding single-axis SCCs to have sufficiently large condition numbers. To make full use of this point, the two axes should be as independent as possible and in our work, two orthogonal axes are used.
Motion Observability Analysis of the SCC for Visual Tracking
3.4
349
Visual Interpretations and Verifications on Example Patterns
Using the SCC as an object representation, an image patch can have different SCCs, with some being more favorable than others in terms of motion observability. To provide a visual interpretation on the SCC representation, we make further analysis into the matrix A, which is ⎡ ⎤ − 12 Φ −Φ 2 (Φijk − Φ0 )g(|| ijkh 0 ||2 ) h2 C1,1 ⎢ ⎥ I(Φijk )=I11 ⎢ ⎥ ⎢ ⎥ .. ⎢ ⎥ . ⎢ ⎥ − 12 ⎢ Φijk −Φ0 2 ⎥ 1 2 −2 T C (Φ − Φ )g(|| || ) ⎢ ⎥. 2 ijk 0 d(C(Φ0 )) UC JK (Φ0 ) = ⎢ h 1,m h ⎥ I(Φijk )=I1m ⎢ ⎥ ⎢ ⎥ . .. ⎢ ⎥ ⎢ ⎥ ⎣ 2 − 12 Φijk −Φ0 2 ⎦ C (Φ − Φ )g(|| || ) m,m 2 ijk 0 h h I(Φijk )=Imm
(8) To obtain a unique solution for ΔΦ in Eqn.5, at least three of the row vectors of the above matrix need to be linearly independent. In this paper, we analyze two typical image patterns that are inherently unobservable to certain motion.
Fig. 1. Illustration for visual interpretation of SCC choice
Concentric Circles (Fig.1(a)): In color histogram based kernel methods, concentric circles are regarded as a degenerated case [4], where translation can not be detected. In this SCC based kernel methods, due to the spatial information encoded in the pixel pairs, translation along the SCC axis can now be observed. Without loss of generality, we set the SCC axis to be along the y direction, as shown in Fig.1(a), and validate the translation observability through Eqn.8 by Φ −Φ examining the weighted distance vectors (Φijk −Φ0 )g(|| ijkh 0 ||2 ) for pixel pairs of two certain distinct colors. The cx component (recall that Φ = [cx, cy, θ]T ) of this term is cancelled out by every two corresponding pixel pairs, i.e., two symmetric pairs w.r.t. the SCC axis, like pairs a and b in Fig.1(a). However, the cy component of the term can not be always cancelled out by corresponding pairs, i.e., symmetric ones w.r.t. the x direction. For example, pair c in Fig.1(a) is of color (i, j), while its corresponding pair (pair d ) is of color (j, i), therefore they
350
Q. Zhao and H. Tao
can neither cancel out each other for Cij , nor for Cji . Since the representation is rotationally-symmetric, the θ components are all cancelled out. As a result, the row vectors in Eqn.8 is κ[0, 1, 0]T . The intuition behind this is that among the three degrees of motion, only the translation along the SCC axis can cause sufficient changes to the SCC. Although one may use two axes to detect translation in both dimensions, the blindness to rotation is the inherent limitation of Concentric Circles. Parallel Stripes (Fig.1(b)): Independent of the axis choice in the SCC, the parallel stripes pattern is sensitive to motion along the x direction, while blind to motion along the y direction. However, its observability to rotation depends on the direction of the SCC axis. If the axis is defined to the be along the x direction, Φ −Φ then the elements for the orientation dimension in (Φijk − Φ0 )g(|| ijkh 0 ||2 ) cancel out. Intuitively, this means that slight rotation does not cause enough change to the SCC. On the other hand, if the axis is some degrees away from the x direction, then rotation makes a difference in the SCC by causing some pixels in the boundary of two stripes to be the other color.
Fig. 2. Quantitative relationship between condition numbers and parameters (θ, d): (a) Image patch (96 × 96), (b)-(d) Condition numbers w.r.t. axis directions with different pair distances: (b) d = 5 (Symbols at the top line represent overflow values), (c) d = 10, (d) d = 40
We evaluated the quantitative relationship between condition numbers and parameters (θ, d) for the image patch of the Parallel Stripes pattern, shown in Fig.2(a). The axis direction θ is defined in the object coordinate system, i.e., with along the x direction being 0 degree, and increases counterclockwise. We observe the condition numbers of the single-axis SCCs with different axis directions and those of two-orthogonal-axis SCCs, where the directions of the axes are θ and θ + 90. Fig.2(b-d) show the relationship of condition numbers w.r.t.
Motion Observability Analysis of the SCC for Visual Tracking
351
different axis directions with pair distances of 5, 10, and 40, respectively. From the illustrated outputs, following conclusions can be made: 1. The pair distance can not be too small. This is due to the discrete nature of the image and the fact that the colors for the SCC are assigned in a nearestneighbor manner. For short pixel pairs, small image rotation may not cause any changes to the SCC. As shown in Fig.2, the results with pair distance of 5 pixels (Fig.2(b)) are much poorer than those with longer distances(Fig.2(c-d)). 2. Some axis directions are more favorable than others in terms of stability. For the patch shown in Fig.2(a), the most favorable axis direction for single-axis SCCs is along the y direction (θ = 90 in Fig.2(c-d)). 3. When two axes are used, due to the inequalities given in Eqn.7, no matter what the directions of the axes are, the condition number is between the two condition numbers generated independently by the two single-axis SCCs. Fig.2(c-d) show that the average condition numbers provided by the two-orthogonal-axis SCCs are significantly smaller than those generated by the single-axis ones.
4
Implementation Details
For most real objects, textures are irregular enough to avoid the extreme cases, as shown in section 3.4, thereby a two-orthogonal-axis SCC suffices in most cases. However, in applications where speed is an important factor, single-axis SCC is greatly favored for efficiency considerations. In this case, we search for the optimal axis direction in the orientation space to obtain the SCC with the smallest condition number. The pair distance is an important parameter in that it influences not only the SCC’s sensitivity to rotation, but also the stability of the solution. On one hand, the larger the pair distance, the more observable the orientation changes, as discuessed in section 3.4; on the other hand, a SCC with a large pair distance tends to have too few pairs counted (both pixels should be in the tracking window), which decreases the stability of the tracking. By trial and error, we set the default distance to be max(1/8(l + w), 10), where l and w are the length and width of the kernel size. The lower bound of 10 pixels ensures the stability of the tracker when the object is small. In this paper, similar to [5], mean shift algorithm is extended to a translationrotation joint domain to locate the object position and orientation simultaneously, in a gradient descent manner. However, the proposed idea can be easily incorporated into any tracking framework other than the mean shift one.
5
Experiments
The usefulness of the proposed schemes to ensure reliable tracking have been demonstrated on vehicle and pedestrian sequences under various environmental conditions. In the following, the first two real-time tracking tasks compute and use the optimal single-axis SCCs as representations, and the two-orthogonal-axis SCCs are used for the other sequences.
352
Q. Zhao and H. Tao
a-1 a-2 b-1 b-2 c-1 c-2
Fig. 3. Car-Chasing Sequence
Car-Chasing: The Car-Chasing is a live video sequence of 2250 frames. It has been tested using the standard mean shift (MS) based tracking algorithm, the single-axis SCC based tracking algorithms with and without the optimal SCC selection. The possible problems in vehicle tracking using the MS tracker are revealed in Fig.3: (1) Loss of tracking tends to occur when the car makes turns (Fig.3(a-1)); (2)Fixed orientation of the tracking window causes scale adaptation difficult to realize and the mismatch of the window to the object makes the tracker sensitive to background clutter (Fig.3(b-1)). In Fig.3(c-1), although the window direction adapts to the object, using a non-optimal single-axis SCC, the tracker is less stable. In Fig.3(a-2),(b-2),(c-2), we show results of the optimal single-axis SCC based tracker, which tracks the car throughout the entire sequence. Results demonstrate that challenging issues like object rotations, heavy occlusions, background clutters, scale changes and motion blur are handled elegantly. PETS 2001 Data: The proposed schemes are further evaluated on the PETS 2001 Dataset [8]. Compared with the color histogram based methods, the SCC based tracker successfully removes the restrictions brought up by certain object shapes and/or camera viewpoints. The proposed optimal single-axis SCC selection scheme further ensures both reliability and efficiency of the tracker. Sample frames of the experimental outputs are shown in Fig.4. Multiple Human Parts: Other experiments are shown in Fig.5. The twoorthogonal-axis SCCs are used to track multiple human parts. Although only
Motion Observability Analysis of the SCC for Visual Tracking
353
Fig. 4. PETS 2001 Sequences
Fig. 5. Stretching and Walking Sequences
color information is extracted, the reliable tracking results indicate the algorithm’s potential in being a useful module in any human tracking or behavior analysis tasks.
6
Conclusion
This paper analyzes the capability of the SCC (in a kernel based framework) in recovering both translation and rotation. A criterion to evaluate the SCC in terms of motion estimation is provided to guide the SCC selection. Twoorthogonal-axis SCCs are proved to be practical sufficient, while in tasks where speed requirements are high, optimal single-axis SCCs are desired. The discussion is focused on, but not limited to, the SCC representation. The SCC in an extended mean shift tracking framework is not computationally expensive. The tracker runs comfortably at 30 fps on a PIV 3.20GHz PC.
354
Q. Zhao and H. Tao
References 1. Comaniciu, D., Ramesh, V., Meer, P.: Kernel-based object tracking. PAMI 25(5), 564–577 (2003) 2. Collins, R.: Mean-shift blob tracking through scale space. In: CVPR, pp. 234–240 (2003) 3. Fan, Z., Wu, Y.: Multiple collaborative kernel tracking. In: CVPR II, pp. 502–509 (2005) 4. Hager, G., Dewan, M., Stewart, C.: Multiple kernel tracking with SSD. In: CVPR I, pp. 790–797 (2004) 5. Zhao, Q., Tao, H.: Object tracking using color correlogram. In: VS-PETS, pp. 263– 270 (2005) 6. Huang, J.: Color-spatial image indexing and applications. PhD thesis, Cornell University (1998) 7. Kailath, T.: The divergence and Bhattacharyya distance measures in signal selection. IEEE Trans. on Comm. Tech. 15(1), 52–60 (1967) 8. http://visualsurveillance.org/PETS2001/
Appendix: Derivation for Eqn.5
√ Optimization on the objective function of min|| M − C(Φ)||2 results the fol√ 1 lowing equation: d(C(Φ0 ))− 2 UCT JK (Φ0 )ΔΦ = 2( M − C(Φ0 )), where ΔΦ is the motion vector in terms of both translation and rotation, and Φ0 is the initialized object center in the current frame. Applying the Taylor Expansion on C(Φ) and dropping higher order terms yields d C(Φ) C(Φ) = C(Φ0 ) + (9) |Φ=Φ0 ΔΦ. dΦ T Since C(Φ0 ) = UCT K(Φ0 ), therefore dC(Φ) dΦ |Φ=Φ0 = UC ∇K(Φ0 ). Introducing this into Eqn.9, we have 1 1 C(Φ) = C(Φ0 ) + d(C(Φ0 ))− 2 UCT ∇K(Φ0 )ΔΦ, (10) 2 where d(C(Φ0 )) is the matrix with C(Φ0 ) on its diagonal. Rewrite the objective function in terms of the motion vector ΔΦ, we obtain √ (11) argminΔΦ || M − C(Φ0 + ΔΦ)|| Substituting Eqn.10 into Eqn.11, the resulting objective function is √ 1 1 (12) argminΔΦ || M − C(Φ0 ) − d(C(Φ0 ))− 2 UCT ∇K(Φ0 )ΔΦ||, 2 the solution of which equates to the solution of the linear system √ 1 1 d(C(Φ0 ))− 2 UCT ∇K(Φ0 )ΔΦ = M − C(Φ0 ). (13) 2 Denoting ∇K(Φ0 ) as JK (Φ0 ) and scaling both sides up by a factor of 2 results √ 1 (14) d(C(Φ0 ))− 2 UCT JK (Φ0 )ΔΦ = 2( M − C(Φ0 )).
On-Line Ensemble SVM for Robust Object Tracking Min Tian1, Weiwei Zhang2, and Fuqiang Liu1 1
Broadband Wireless Communication and Multimedia Laboratory, Tongji University, Shanghai, China 2 Microsoft Research Asia, Beijing, China [email protected], [email protected], [email protected]
Abstract. In this paper, we present a novel visual object tracking algorithm based on ensemble of linear SVM classifiers. There are two main contributions in this paper. First of all, we propose a simple yet effective way for on-line updating linear SVM classifier, where useful “Key Frames” of target are automatically selected as support vectors. Secondly, we propose an on-line ensemble SVM tracker, which can effectively handle target appearance variation. The proposed algorithm makes better usage of history information, which leads to better discrimination of target and the surrounding background. The proposed algorithm is tested on many video clips including some public available ones. Experimental results show the robustness of our proposed algorithm, especially under large appearance change during tracking.
1 Introduction Visual tracking is an important subject in computer vision with a variety of applications. One of the main challenges that limit the performance of the tracker is appearance change caused by the variances of pose, illumination or view-point. In order to develop a robust tracker, lots of former work has been done to address those problems, however, robust object tracking still remains a big challenge. Object tracking can be considered as an optimization problem. Tracking algorithm is used to find a region with the local maximum similarity score. In [1], the similarity is defined as the SSD between the observation and a fixed template. In [6], Mean-shift is proposed as a nonparametric density gradient estimator to search the most similar region by computing the similarity between color histogram of the target and the search window. Object tracking can also be considered as a state estimation problem. In early works, Kalman filter or its variants are frequently used. However, Kalman can’t solve the non-Gaussian and non-linear cases well. In order to solve the nonGaussian and non-linear cases well, sequential Monte Carlo methods are applied for tracking, among which Particle Filter (PF) [3,8,12,4] is the most popular one. Object tracking can also be regarded as a template updating problem. The classical subspace tracking method is proposed by Black et al. [2]. Ross and Lim [13] extended Eigen-tracking by on-line incremental subspace updating. Along the other direction, in [9] Jepson models the target as a mixture of stable component, outliers, and two frame transient component. And an on-line EM algorithm is use for the parameters of each Y. Yagi et al. (Eds.): ACCV 2007, Part I, LNCS 4843, pp. 355–364, 2007. © Springer-Verlag Berlin Heidelberg 2007
356
M. Tian, W. Zhang, and F. Liu
component. Considering the tracker as a binary classify problem is very popular recently. [18] proposed a transductive learning method for tracking, in which D-EM algorithm is used for transducing color classifiers and selecting a good color space to determine each pixel whether belongs to the foreground or background. The constraints of these color trackers are clear because they can only work on color image sequences. In [10] an off-line SVM is used for distinguish the target vehicle from the background. Similar to [10], in [12] an Adaboost classifier is trained off-line to detect the hockey players for a proposal distribution to improve the robustness of the tracker. Since they need large mount training data, it’s not easy to extend those approaches to general objects tracking. In order to get a general tracker, [16] proposed an online learning of ensemble pixel based weak classifier, and tracking is done by classifying pixels into foreground and background. However, in his method the features are limited to an 11D low dimension vector including pixel colors and local orientation histogram. Helmun Grabner proposed an on-line version of the Adaboost algorithm in [17], which can on-line select features to get a strong classifier. Their work is roughly similar to the idea of [16], however it can on-line choose the features with the most discriminative ability. Among all of the topics that have been discussed in existing classifier based trackers, we found that the history information has not been paid much attention, which is very important for tracking. In order to better use those important information, we proposed an ensemble SVM classifier based tracking algorithm. In our algorithm, the linear SVM can automatically select the “Key Frames” of the target as support vectors. Moreover, through ensemble combining several linear SVM classifiers, history information can be used more reasonable, and the risk of drift can be decreased more effectively. Finally, because the ensemble method can automatically adjust each SVM classifiers weight on-line, so some off-line classifiers can also be trained and add into the framework to form a more robust tracker. The paper is organized as following, section 2 give out the proposed on-line updating of SVM based tracker. Section 3 give out the ensemble of SVM based tracker. Experiment and conclusion are given in section 4 and section 5 respectively.
2 On-Line SVM Tracker In this paper, we take the tracking as a binary classification problem, where we choose SVM as our basic classifier. In the following sections, we will show that SVM can automatically select “Key frame” of the target from the historic frames, and that is one of the most important factors for the proposed algorithm in this paper. First of all, let’s introduce our on-line SVM classifier based tracker. 2.1 SVM Classifier Based Tracking Within the context of object tracking, we define the target object region and its surroundings as positive data source and negative data source respectively, as shown in fig. 1(b). Our target is to learn a SVM classifier which can classify the positive data and negative data in the new frame. Starting from first frame, the positive and the negative samples are used to training the SVM classier. Then the search region (fig. 1(b)) can be estimated in the next frame. Finally, the target region in next frame is located with local maximum score within the search region.
On-Line Ensemble SVM for Robust Object Tracking
(a)
(b)
357
(c)
Fig. 1. (a) The confidence map of the search region. (b) The object region, search region and the context region. (c) Demonstration of a linear SVM. The filled circles and rectangles are the “support vectors”.
2.2 On-Line Updating Linear SVM One of the most difficult tasks for a tracker is how to on-line update the tracker to make it adapt to the appearance change of the target. Some former methods use a rapid update model, saying that xt = xˆ t −1 , but it’s dangerous and may cause drift problem [11]. Some off-line tracker, such as [7], takes benefit of the off-line selected “Key Frames”, which will let the tracker be more robust to against drift problem. Our idea is inspired by the former ones, and the difference is that our tracker can not only record the “Key Frames” of the target as the history information, but can also update on-line to decrease the risk of drift. We consider the updating of the classifier as an on-line learning process. Here we propose a simple yet effective way for on-line updating the linear SVM classifier. The details are described as Algorithm 1. Algorithm 1. On-line Linear SVM Tracking & Updating Input: In Video frames for processing ( n=1,…, L) R Rectangle region of the target region Output: Rectangles of target object’s region Rn ( n=1,…, L) Initialization for the first frame In ( n=1): z Extract positive and negative samples S1 = {xi , yi }iN=1 , where yi ∈ {−1, +1} , corresponding to the target region R. z Train a linear SVM to get f1 (x) = w1x + b1 and its support vectors V1 = {xi , yi }iM=1 For each new frame In ( n>1): z Find region Rn with the local maximum score given by f n−1 (x) . Here x denotes the search window’s feature vector. x Rn = arg max f n−1 (x) = arg max(w n−1x + bn−1 ) x
x
z If f n−1 (x R ) > 0 go to the next step to get a new SVM. n
Else stop updating and guess Rn is the target region, and go to the next frame. z Refresh positive samples Pn = Vn+−1 ∪ Sn+ and negative samples N n = Vn−−1 ∪ S n− . Here, Vn+−1 and Vn−−1 are the positive and negative support vectors of f n−1 (x) , Sn+ and Sn− are the
positive and negative samples of current frame. z Retrain the SVM using new samples for updating to get f n (x) = w n x + bn
358
M. Tian, W. Zhang, and F. Liu
By on-line updating, the SVM tracker can adjust its hyper-plane for the maximum margin between the new positive and negative samples. The support vectors transferred frame by frame contain important “Key Frames” of the target object in the previous tracking process (see fig.2 (b)). Figure 2 show the tracking result and part of selected “Key frames” by SVM in the final frame, the video is provided by Jepson [9]. The proposed tracker can adapt to the face with variation in appearance and distinguish it from the cluttered background (fig. 2(a)).
Fig. 2. (a) Tracking results of frame 1, 206, 366, 588, 709, 761, 973 and 1131. (b) In the end of the tracking task, 182 positive support vectors contain enough history information. Images displayed on the bottom are some of these support vectors.
3 Ensemble SVM Classifier Based Tracking Although the proposed SVM based tracker is powerful for tracking by on-line updating, we found there still existing several issues need to be addressed in the real world video clips. (a) The variance of the target is very large. (b) The tracker is disturbed by scale variation, partial occlusion or movement blur on a certain frame. Those two issues may lead the tracking algorithm drifting and finally fail. In order to further address those two issues, we proposed an ensemble SVM algorithm in this section. 3.1 On-Line Building the Ensemble of SVMs Our algorithm starts with a SVM trained with labeled data in the first frame. After then, in each frame new SVM may be added, the current tracking result with previous SVM classifiers. The match ratio rm is defined as equation (1), where U ( x) is a step function that equals to 1 when x is above zero, and otherwise it equals to 0. +
Here {xk }kN=1 are the positive samples, and
N+
is their number. The larger rm is, the
On-Line Ensemble SVM for Robust Object Tracking
359
better current component matches the positive samples. If rm below a ratio threshold, a new SVM should be added. N+
rm =
∑ U ( f m (x k ) − 1)
(1)
k =0
N+
N + − ∑ U ( f m (x k ) − 1) k =0
So after several frames, many SVM classifiers are generated and updated during different periods, which is shown as fig. 3.
Fig. 3. This flowchart is the demonstration of our framework of on-line ensemble SVM Tracker
After the number of SVM classifier is larger than one, we combine the linear SVM classifiers in the pool to get better classify result. Each SVM classifier is assigned with a coefficient α m , which is defined as following: N
1 α m = log 2
∑ U (Pm (xi , yi ) ⋅ ωi )
i =1 N
(2)
∑ U (− Pm (xi , yi ) ⋅ ωi )
i =1
Here ωi is the samples’ weight. And Pm (xi , yi ) is the output of each SVM classifier to evaluate its discriminative ability for every sample. Here we define Pm (xi , yi ) as following: 1 ⎧ ⎪ ⎪ ( fm (x) − T ) ⋅ fm (x) − T Pm ( x i , y i ) = ⎨ T2 ⎪ ⎪ −1 ⎪⎩
if y i ⋅ f m ( x ) ≥ 1 if 1 > y i ⋅ f m ( x ) > 0
(3)
if y i ⋅ f m ( x ) ≤ 0
When Pm (xi , yi ) is positive, it means the right classifying probability. Meanwhile, when it is negative, it means the wrong classifying probability. Here, T ∈ (0,1) is a threshold in the determining the rule, and we set is as 0.5 in our method. The details of on-line ensemble SVM tracker are described as Algorithm 2.
360
M. Tian, W. Zhang, and F. Liu
Algorithm 2. On-line Ensemble SVM Classifiers for Tracking Input: Output:
In Video frames for processing ( n=1,…, L) R Rectangle region of the target region Rectangles of target object’s region Rn ( n=1,…, L)
Initialization for the first frame In ( n=1): z Use the target and other random chosen regions without overlap with the target in the first frame to form a ground truth classifier f 0 (x) = w 0 x + b0 , which will not be updated during tracking. This classifier can also be other off-line trained classifier.
z Extract the positive and the negative samples as Algorithm 1. z Train a SVM classifier f1 (x) = w1x + b1 by using the extracted samples. z Initialize the samples’ weights ωi = 1/ N , i = 1, 2,..., N . z For m = 0 to 1 a)
Make {ωi }iN=1 a distribution
b)
Chose the most strong SVM with the largest α m by using equation (2)
c)
If α m <0, α m =0 and break
d) e)
Remove the chosen SVM Update samples’ weight ωi = ωi exp[−α m ⋅ yi Pm (xi , yi )] , i = 1, 2,..., N
z Normalize α i to make ∑ αi = 1 . The output of the ensemble one is F(x) = α 0 f0 (x) + α1 f1 (x) i
For each new frame In (n>1): z Use F(x) to search the target region and extract samples S = S + ∪ S − .
z z z z
Push some of S + into the positive sample queue by random sampling. Check whether a new SVM should be built by
rm in (1).
Choose the last K SVMs to update as Algorithm 1. Here we set K=5. Radom chose M samples S′ from the sample history queue, M equals to the number of S + . New group of samples are determined as: S ′′ = S ∪ S ′
z Initialize the samples’ weights ωi = 1/ N , i = 1, 2,..., N . z For m = 0 to K max ( K max = 10 in ours) a)
Make {ωi }iN=1 a distribution
b)
Chose the most strong SVM with the largest α m by using equation (2)
c)
If α m <0, α m =0 and break
d) e)
Remove the chosen SVM from the SVM queue Update samples’ weights ωi = ωi exp[−α m ⋅ yi Pm (xi , yi )] , i = 1, 2,..., N
z Normalize α i to make ∑ αi = 1 . The output of the ensemble one is: F(x) = i
K
∑ α m f m (x)
m =0
Compared with a single on-line SVM, the ensemble tracker can get a more reliable result, especially when the appearance of the target changes frequently (as fig. 4,5). From fig. 4 and fig.5, we can clearly find that a single on-line SVM is useful. However, it record all the history information as its support vectors to achieve the global optimization, which may cause it can difficultly handle large appearance variation in a short period. This phenomenon is also appeared in the incremental
On-Line Ensemble SVM for Robust Object Tracking
361
-
(a)
(b)
(c)
(d) Fig. 4. (a) Tracking results of frame 100,152,171,183 and 366 by using single on-line linear SVM tracker. (b) Tracking results of on-line ensemble SVMs tracker. The confidence map of the search region is on the right-bottom of each frame. (c) The ensemble weight of each SVM in the mixture model. (d) The updating period of each SVM (from being generated to stop updating). Mind that No.1 SVM is the ground truth SVM, and it will not be updated during tracking. frame 1
frame 121
frame 207
frame 282
frame 345
frame 384
frame 409
frame 429
frame 440
frame 481
(a)
(b)
(c)
Fig. 5. Sequences provided by Lim and Ross (a) Tracking results of incremental subspace learning tracker. The tracker failed after frame 345. (b) Tracking results of our single on-line linear SVM tracker. The tracker almost failed on frame 345, and then it drifts away from the target. (c) Tracking results of on-line ensemble SVMs tracker. The tracker finished the whole video with accurate results.
362
M. Tian, W. Zhang, and F. Liu
subspace learning tracker shown as fig.5 (a). The ensemble SVM tracker, which we proposed here, can choose the SVM classifiers with the best discriminative ability to the chosen samples, and on-line combines them together by adjusting their weight. Using this method, the ensemble classifier can use the history information more reasonable, at the same time the final tracker has an especially strong distinguish ability, which makes the tracking result more reliable.
4 Experiments In this section, several experimental results are carried out by our algorithm. Region patterns we used here are some common features: histograms of oriented gradients (HOG) [14] and local binary patterns (LBP) [5]. Integral histograms [15] are built for extracting region feature efficiently. Similar to the method used in [14], we construct a 9 bins HOG histogram for each cell, each block contains four cells with a 36-D HOG feature vector that is normalized to an L2 unit and a 59-D feature vector for LBP. Different form [14], the pixels including in a cell is not a constant, because the object region is scalable. The positive samples are captured by scaling the target region from 0.8 to 1.5 and rotating it form -8 degree to +8 degree. The negative samples are captured within the context region, and the negatives can have some overlaps with the positive one (below 1/3 area of positive sample). The sample rate between each negative region is set as 5 pixels per step in our method. In our method, the scale problem is solved by a naive way. The suitable scale is got by searching different scales around the centre of local maximum region. First of all, we captured some videos by ourselves to demonstrate the robustness of our framework (fig. 6(a)).Then we carried out our method on some frequently used public available sequences (fig. 6(b,c)). Compared with some other popular methods,
(a)
(b)
(c) Fig. 6. (a) A glass bottle with illumination and appearance change while moving in cluttered background. Beside the target, there is another bottle, which looks like the target bottle. However, the tracker can discriminate them very well without confusion. (b) A moving doll with large pose and illumination change, frames 1, 454, 728, 1162 and 1343. (c) A moving vehicle with disturbance of the light around, frames 1, 129, 245, 295 and 391.
On-Line Ensemble SVM for Robust Object Tracking
363
(a)
(b)
(c)
Fig. 7. Results of a boy with large head pose change. (a) Results of incremental subspace learning tracker. The tracker failed after frame 96 (b) Results of ensemble tracker, it runs well for 203 frames, but failed later, which may cause by size problem. (c) Results of our method.
the incremental subspace learning tracker [13] of Ross and Lim and the ensemble tracker [16] of Shai Avidan, we get the results in fig. 7. The incremental learning tracker is based on updating the sample mean and the eigenbasis over time. However, when the variation is very large, the updating can’t adapt the change quickly enough and an imprecise position may be got. So after several frames’ updating, the target may drift very quickly because of error accumulation. The ensemble is powerful, however it is a pixel based tracker (as [18]), the information for a pixel is little and the feature vector may have a large variation when the target is colorful, and the tracker may get confused when the color of the target and the background is similar. Our method, as mentioned before, is based on the region patterns which is more stable while tracking. Meanwhile, it contains and chooses the most useful “key frames” of the target by ensemble of SVMs, which have the most discriminative ability. Because of that, the performance of the tracker is especially good on some challenging videos with large appearance variation.
5 Conclusion In this paper, we build a novel framework to track general object. The ensemble SVM tracker proposed here is made up of several SVM classifiers, which are proved especially strong in selecting and recording the “Key Frames” of the object. These classifiers are generated and updated during different periods with different historical
364
M. Tian, W. Zhang, and F. Liu
information. By on-line adjusting each SVM’s weight, the ensemble classifier can distinguish the target and the background better than any single component. With the selected useful historical information and the strong discriminative ability, the tracker performs especially well on some difficult videos with large appearance variation.
Acknowledgments This work was done when the author visited visual computing group in Microsoft Research Asia. The author would like to appreciate all the researchers in that group for their supports. Thanks for Lim and Ross’s image sequences and matlab code of incremental subspace learning tracker provided in their website. Furthermore, thanks to Shai Avidan’s help and the results of ensemble tracker provided by him.
References 1. Hager, G.D., Belhumeur, P.N.: Efficient region tracking with parametric models of geometry and illumination. PAMI 20(10), 1025–1039 (1998) 2. Black, M.J., Jepson, A.: EigenTracking: Robust matching and tracking of articulated objects using a view-based representation. International Journal of Computer Vision 26(1), 63–84 (1998) 3. Isard, M., Blake, A.: Condensation-Conditional Density Propagation for Visual Tracking. International Journal of Computer Vision 29(1), 5–28 (1998) 4. Perez, P., et al.: Color-Based Probabilistic Tracking. In: ECCV, pp. 661–675 (2002) 5. Ojala, T., Pietikainen, M., Maenpaa, T.: Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. PAMI 24, 971–987 (2002) 6. Comaniciu, D., Ramesh, V., Meer, P.: Kernel-based object tracking. PAMI 25(5), 564–577 (2003) 7. Vacchetti, L., Lepetit, V., Fua, P.: Fusing online and offline information for stable 3D tracking in real-time. In: CVPR 2003, vol. 2, pp. 241–248 (2003) 8. Nummiaroa, K., Koller-Meierb, E., Gool, L.V.: An Adaptive Color-Based Particle Filter. Image and Vision Computing 99–110 (2003) 9. Jepson, A.D., Fleet, D.J., El-Maraghi, T.F.: Robust online appearance models for visual tracking. PAMI 25(10), 1296–1311 (2003) 10. Avidan, S.: Support Vector Tracking. PAMI 26(8), 1064–1072 (2004) 11. Matthews, I., Ishikawa, T., Baker, S.: The Template Update Problem. PAMI 26, 810–815 (2004) 12. Okuma, K., Taleghani, A.: A Boosted Particle Filter: Multitarget Detection and Tracking. In: Pajdla, T., Matas, J(G.) (eds.) ECCV 2004. LNCS, vol. 3024, pp. 28–39. Springer, Heidelberg (2004) 13. Ross, D., Lim, J., Yang, M.H.: Probabilistic visual tracking with incremental subspace update. In: Pajdla, T., Matas, J(G.) (eds.) ECCV 2004. LNCS, vol. 3022, pp. 470–482. Springer, Heidelberg (2004) 14. Dalal, N., Triggs, B.: Histograms of Oriented Gradients for Human Detection. In: CVPR 2005, vol. 1, pp. 886–893 (2005) 15. Porikli, F.: Integral histogram: a fast way to extract histograms in Cartesian spaces. In: CVPR 2005, vol. 1, pp. 829–836 (2005) 16. Avidan, S.: Ensemble tracking. In: Proceedings of CVPR 2005. vol.2, pp. 494–501 (2005) 17. Grabner, H., Bischof, H.: On-line Boosting and Vision. In: CVPR 2006, vol. 1, pp. 260– 267 (2006) 18. Wu, Y., Huang, T.S.: Color Tracking by Transductive Learning. In: Proceedings of CVPR 2000, vol. 1, pp. 133–138 (2000)
Multi-camera People Tracking by Collaborative Particle Filters and Principal Axis-Based Integration Wei Du and Justus Piater University of Li`ege, Department of Electrical Engineering and Computer Science Montefiore Institute, B28, Sart Tilman Campus, B-4000 Li`ege, Belgium weidu.montefiore.ulg.ac.be, [email protected]
Abstract. This paper presents a novel approach to tracking people in multiple cameras. A target is tracked not only in each camera but also in the ground plane by individual particle filters. These particle filters collaborate in two different ways. First, the particle filters in each camera pass messages to those in the ground plane where the multi-camera information is integrated by intersecting the targets’ principal axes. This largely relaxes the dependence on precise foot positions when mapping targets from images to the ground plane using homographies. Secondly, the fusion results in the ground plane are then incorporated by each camera as boosted proposal functions. A mixture proposal function is composed for each tracker in a camera by combining an independent transition kernel and the boosted proposal function. Experiments show that our approach achieves more reliable results using less computational resources than conventional methods.
1
Introduction
Tracking people in multiple cameras is a basic task in many applications such as video surveillance and sports analysis. A commonly-used fusion strategy is to detect people in each camera with bottom-up approaches such as background subtraction and color segmentation, and then to calculate the correspondences between cameras using the camera calibrations, or more often, the ground homographies. In order to reason about occlusions between targets, this fusion strategy usually requires all targets to be correctly detected and tracked [8,4,2,6,5]. However, sometimes we may be interested in the trajectories of only a few key targets, for instance, the star players in a soccer game or a few suspects in a surveillance scenario. Top-down approaches are preferable in such situations. In this paper, we present a novel top-down approach to people tracking by multiple cameras. The approach is based on collaborative particle filters, i.e., we track a target not only in each camera but also in the ground plane by individual particle filters. These particle filters collaborate in two different ways. First, the particle filters in each camera pass messages to those in the ground plane where the multi-camera information is integrated using the homographies Y. Yagi et al. (Eds.): ACCV 2007, Part I, LNCS 4843, pp. 365–374, 2007. c Springer-Verlag Berlin Heidelberg 2007
366
W. Du and J. Piater
of each camera. Such a fusion framework usually relies on precise foot positions of the targets, which are often not provided by the particle filters in the cameras. To overcome the imprecise foot positions as well as the uncertainties of the camera calibrations, we exploit the principal axes of the targets during integration, which greatly improves the precision of the fusion results. These fusion results are then incorporated by the trackers in each camera as boosted proposal functions. A mixture proposal function is composed for each tracker in a camera by combining an independent transition kernel and the boosted proposal function, from which new particles are generated for the next time instant. Our approach has several distinct features. First, it doesn’t require all targets to be tracked simultaneously. Instead of having different target trackers interact, we compute the consensus between cameras by having different camera trackers communicate. Second, it has a fully distributed architecture. All the computations are performed locally and only the filter estimates are exchanged between the cameras and the fusion module. Third, the fusion of the multi-camera information is done by intersecting the targets’ principal axes. Experiments on both surveillance and soccer scenarios show that our approach achieves more reliable results using less computational resources than conventional methods. Particle filters are conventional in multi-camera tracking. Most previous work performed particle filtering in 3D so that precise camera calibration is required to project particles into the image plane of each camera [9,7]. The multi-camera information is often integrated by either the product of the likelihoods in all cameras [7] or a selection of the best cameras that contain the most distinctive information [9]. In our previous work, we proposed a different approach to fusion that combined particle filters and belief propagation, where particle filters collaborated with each other via a message passing procedure [1]. To match ground-plane target positions using homographies, the foot positions of the tracked people have to be detected. This, however, is a difficult and errorprone task if done separately for each camera. In this paper, we address the precision and computational issues. We relax the dependence on precise foot positions by exploiting the principal axes of the targets, the intersections of which give better ground positions. At the same time, we improve the speed over our previous system [1] by incorporating the fusion results from the ground plane as proposal functions into each camera. The rest of the paper is organized as follows. Section 2 formulates the multicamera tracking problem. Section 3 introduces the collaborative particle filters, including the principal axis-based integration and the boosted proposal functions. Experiments on sequences of video surveillance and soccer games are shown in Section 4.
2
Problem Formulation
Suppose L cameras are used and each camera collects one observation for each target at each time instant. Denote the target state on the ground plane by xt,0 and its states in different cameras by xt,j , j = 1, . . . , L. Let zt,j denote
Multi-camera People Tracking by Collaborative Particle Filters
(a) tree-structured graphical model
367
(b) dynamic Markov model
Fig. 1. Graphical models for modeling the dependencies at time t and for modeling the evolution of the system in time
the observation in camera j at time t, Zt = {zt,1 , . . . , zt,L } the multi-camera observation at time t, and Z t = {Z1 , . . . , Zt } the multi-camera observations up to time t. Fig. 1(a) shows the graphical model that models the dependencies between target states in the ground plane and at different cameras at time t. We assume that the xt,j , j = 1, . . . , L, are independent given xt,0 so that a tree-structured model is formed. Note that xt,0 is associated with no observation. Connecting the graphical models at different times results in a dynamic Markov model, shown in Fig. 1(b), that describes the evolution of the system over time. As all the xt,j depend on xt,0 , we add temporal links from xt−1,0 to xt,j . The addition of these temporal links is beneficial to the design of the proposal functions, shown in the next section. In both models in Fig. 1, each directed link from xt,0 to xt,j , j = 1, . . . , L, represents a message passing process and is associated with a potential function t (xt,0 , xt,j ). The directed link from xt,j to zt,j , j = 1, . . . , L, represents the ψ0,j observation process and is associated with a likelihood function pj (zt,j |xt,j ). In Fig. 1(b), the directed links from xt−1,i to xt,i , i = 0, . . . , L, and from xt−1,0 to xt,j , j = 1, . . . , L represent the state transition processes and are associated with motion models p(xt,i |xt−1,i ) and p(xt,j |xt−1,0 ) respectively. Thus, we infer each xt,i , i = 0, . . . , L, based on all Z t . A message passing scheme, the same as is used in belief propagation, is adopted to pass messages from each camera to the ground plane. The message from camera j is defined as m0j (xt,0 ) ←
t pj (zt,j |xt,j )ψ0,j (xt,0 , xt,j )
p(xt,j |xt−1,j )p(xt−1,j |Z t−1 )dxt−1,j dxt,j . (1)
The belief p(xt,0 |Z t ) is computed recursively by the message product and the propagation of the previous posterior, t p(xt,0 |Z ) ∝ m0j (xt,0 ) × p(xt,0 |xt−1,0 )p(xt−1,0 |Z t−1 )dxt−1,0 . (2) j=1,..., L
Note that the same message and belief update equations are used in our previous work [1].
368
W. Du and J. Piater
The inference of xt,j , j = 1, . . . , L, is done by nearly standard particle filters, except that the fusion results at t − 1 are taken into consideration. The belief p(xt,j |Z t ) is computed as (3) p(xt,j |Z t ) ∝ p(xj |zj ) × p(xt,j |xt−1,j )p(xt−1,j |Z t−1 )p(xt,j |xt−1,0 )p(xt−1,0 |Z t−1 )dxt−1,0 dxt−1,j . The underlined terms incorporate the fusion results as a boosted proposal function. In other words, the fusion module is used by each camera as a coupled process.
3
Collaborative Particle Filters
All the inference processes formulated above, in the ground plane and for each camera, are performed by individual but collaborative particle filters. Details are given below. 3.1
Principal Axis-Based Integration
The ground-plane particle filter integrates the multi-camera information according to Eqs. 1 and 2. For tracking ground targets, homographies are often used to map the foot positions from each camera to the ground plane. However, a large number of particles are required to estimate precise foot positions, which significantly slows down the tracking system. With a small number of particles, usually the sizes of the targets cannot be estimated precisely. We overcome this problem by exploiting the principal axes of the targets. The principal axis of a target is defined as the vertical line from the head of the target to the feet. It has been shown that the principal axes of a target in different cameras intersect in the ground plane, and computing the intersection point yields very robust fusion results [4,6], illustrated in Fig. 3. We exploit this effect in our multi-camera integration. The idea is to sample particles in the ground plane by importance sampling, and to evaluate these particles by passing messages from each camera. Here, p(xt,0 |xt−1,0 ) is used as the proposal function from which new particles for xt,0 are sampled. Each of these ground-plane particles receives messages from each camera, and a message weight is computed using Eq. 1. The principal axes are t incorporated in the potential function ψ0,j (xt,0 , xt,j ) in Eq. 1. In general, the principal axes of the particles in a camera are projected to the ground plane using the homographies. The potential function measures the distances of the ground particles to these projected principal axes and converts them to probability densities, given by 2 n t m (xnt,0 , xm ψ0,j t,j ) ∝ exp(−dist (xt,0 , project(Hj , xt,j ))),
(4)
where xnt,0 and xm t,j are the nth ground-plane particle and mth particle in camera j, Hj is the homography from camera j to the ground plane, dist() computes
Multi-camera People Tracking by Collaborative Particle Filters
369
Fig. 2. The particle distributions in four cameras at a time instant. It can be seen that the foot positions are not precise although all the particles are placed at the right location.
(a) Mapping particles to the ground.
(b) Mapping principal axes to the ground.
Fig. 3. Comparison between homography-based integration and principal axis-based integration. In (a), the projections of the particles (the red stars) from the images in Fig 2 to the ground have a large variance, making the integration imprecise. In contrast, in (b), the intersection of the principal axes (the red lines) of four selected particles yields a more precise foot position (the white square).
the distance between a point and a line, and project() maps the principal axis to the ground. The message and belief weights are then computed by j,n ∝ wt,0
N
m t n πt,j ψ0,j (xnt,0 , xm t,j ), πt,0 ∝
m=0
L
j,n wt,0 ,
(5)
j=1
j,n n m is the message weight of xnt,0 from camera j, and πt,0 and πt,j are where wt,0 the belief weights of xnt,0 and xm . Intuitively, the closer a ground-plane particle t,j is to all the principal axes, the larger its weight is, as illustrated in Fig. 4.
3.2
Boosted Proposal Functions
A target is tracked in each camera by a particle filter. Due to the occlusions or other image noise, feedback from the fusion module is expected to improve
370
W. Du and J. Piater
(a) Camera 1 passes messages
(b) Camera 2 passes messages
Fig. 4. An illustration of evaluating ground-plane particles using two cameras. The ground-plane particles are evaluated according to the distances to the projected principal axes. (a) After the first camera passes messages to the ground plane, all the particles along the principal axes (red dots) have larger weights than those further away (blue dots). The weights of the camera particles are shown at one end of the corresponding principal axes. (b) After the second camera passes messages, only those ground-plane particles that are close to the intersections have large weights.
the tracking performance in a camera. A similar message passing procedure was adopted in our previous work to pass messages from the ground plane to each camera, which proved computationally expensive. We propose here a different method to incorporate this feedback. Note that in the dynamic Markov model in Fig. 1(b), for each xt,j , j = 1, . . . , L, there is an extra temporal link from xt−1,0 besides that from xt−1,j . This enables us to design a mixture proposal function for importance sampling, p(xt,j |xt−1,j , xt−1,0 ) ∝ αp(xt,j |xt−1,j ) + (1 − α)p(xt,j |xt−1,0 ).
(6)
Thus, we sample particles from both p(xt,j |xt−1,j ) and p(xt,j |xt−1,0 ), i.e., αN particles are sampled from p(xt,j |xt−1,j ) and the other (1 − α)N from p(xt,j |xt−1,0 ). Parameter α specifies a trade-off between two proposal functions and is set to 0.5 in our experiments. To sample from p(xt,j |xt−1,0 ), we fit a Gaussian distribution to xt−1,0 and propagate it to each camera using the homographies. In a sense, the fusion results at t− 1 are used as boosted proposal functions by each camera. This is beneficial not only in maintaining consistency between the particle filters at different nodes but also in speeding up the tracking algorithm. The sampled particles are evaluated using the image likelihood as is done in standard particle filters.
4
Results
We tested our method on both video surveillance and soccer game sequences. We manually initialized the targets of interest in the first frames of the sequences and sampled 100 particles for each filter. Figure 5 shows the results of tracking a pedestrian in PETS sequences and a comparison with a reference method [7], which tracks a target in 3D by a particle
Multi-camera People Tracking by Collaborative Particle Filters
371
Fig. 5. The results of tracking a pedestrian in PETS sequences with our approach (top rows) and with a reference method [7] (bottom rows). In the latter method [7], we initialize a tracker in one camera and project the particles to another camera using the homographies. Here, due to imprecise foot positions, the estimates are projected to wrong positions.
filter and evaluates the particles by the product of the likelihoods in all cameras. In this experiment, we adopted a classic color observation model and evaluate each particle by matching the color histogram to a reference model [11]. The figure shows that particle filters do not estimate precise foot positions; thus, mapping the particles between cameras or between cameras and the ground plane using homographies is imprecise. As a result, using this method [7], most particles in one camera are projected to wrong positions in another camera so that only one camera contributes to the tracking. On the other hand, due to the use of the principal axes, our method integrates information from both cameras and achieves more reliable results. Figure 6 shows the results of tracking two selected people in an indoor environment with four cameras. In this experiment, we adopted a hierarchical multi-cue observation model and evaluated each particle first by a color likelihood function and then by a background-subtraction likelihood function [10]. We also assumed that the sizes of the people were fixed and could be inferred from their ground positions [2]. Thus, the only parameters of interest were the positions in the images and in the ground plane. A comparison with our previous work [1] shows that the new approach achieved similar results but was approximately twice faster.
372
W. Du and J. Piater
Fig. 6. The results of tracking two people in an indoor environment with four cameras. Each row shows four simultaneous views. In this experiment, both the head and the ground homographies of each camera are available. The fixed-size assumption significantly improved the robustness of the algorithm.
Figure 7 shows the results of tracking several soccer players in three cameras. Due to the interactions between the players, the feedback from the fusion module to each camera becomes critical, without which the trackers in different cameras fail one by one. In this experiment, the same observation model as in the PETS experiment was used and the homographies of each camera were obtained on-line by using a field model and by accumulating motion estimates between consecutive frames [3]. Note in the figure that the estimated foot positions do not coincide with the bottom of the bounding boxes, but are more precise than these thanks to the multiple-camera fusion using principal axes. At one point, due to a heavy occlusion that occurs in all cameras, a tracker jumps from one
Multi-camera People Tracking by Collaborative Particle Filters
373
Fig. 7. The results of tracking several soccer players in the last frames of the three sequences. The ellipses under the rectangles are the fusion results in the ground plane.
Fig. 8. The particle distributions at the time when the tracker is about to jump to a different player, which happens here because the players involved are very close both in space and in appearance in all three views. The green rectangles are the sampled particles, the blue are the estimates, and the red are the predictions of the fusion results at the previous time.
target to another. In such situations, multi-camera systems without feedback between cameras are susceptible to mismatched targets. In our system, thanks to the feedback from the ground-plane tracker, the trackers at each camera remain consistent, even if they collectively follow the wrong target. Figure 8 shows the particle distributions at the time instant when the jump begins. This problem can be partially solved by tracking multiple targets simultaneously.
5
Conclusion and Future Work
This paper presents a novel approach to ground-plane tracking of targets in multiple cameras. Different from previous work, our approach is not based on bottom-up detection or segmentation methods. Instead, we infer target states in each camera and in the ground plane by collaborative particle filters. Message passing and boosted proposal functions are incorporated in the collaboration between the trackers in each camera and the fusion module. Principal axes are exploited in the multi-camera integration, which enables us to handle the imprecise foot positions and some calibration uncertainties. In doing so, we achieve robust results using relatively little computational resources. We are currently adapting this approach to multi-target, multi-camera tracking, which involves the modeling of the target interactions and data association across cameras.
374
W. Du and J. Piater
Acknowledgement The authors wish to thank J. Berclaz and F. Fleuret for sharing their data.
References 1. Du, W., Piater, J.: Multi-view object tracking using sequential belief propagation. In: Asian Conference on Computer Vision, Hyderabad, India (2006) 2. Fleuret, F., Berclaz, J., Lengagne, R., Fua, P.: Multi-camera people tracking with a probabilistic occupancy map. IEEE Transactions on Pattern Analysis and Machine Intelligence (2007) 3. Hayet, J.-B., Piater, J., Verly, J.: Robust incremental rectification of sports video sequences. In: British Machine Vision Conference, Kingston, UK, pp. 687–696 (2004) 4. Hu, W.-M., Hu, M., Zhou, X., Tan, T.-N., Lou, J., Maybank, S.J.: Principal axisbased correspondence between multiple cameras for people tracking. IEEE Transactions on Pattern Analysis and Machine Intelligence 27(4), 663–671 (2006) 5. Khan, S.M., Shah, M.: A multiview approach to tracking people in crowded scenes using a planar homography constraint. In: ECCV, pp. 98–109 (2006) 6. Kim, K., Davis, L.S.: Multi-camera tracking and segmentation of occluded people on ground plane using search-guided particle filtering. In: ECCV, pp. 98–109 (2006) 7. Kobayashi, Y., Sugimura, D., Sato, Y.: 3d head tracking using the particle filter with cascaded classifiers. In: BMVC (2006) 8. Mittal, A., Davis, L.S.: M2tracker: A multi-view approach to segmenting and tracking people in a cluttered scene. Internatial Journal of Computer Vision 51(3), 189– 203 (2003) 9. Nummiaro, K., Koller-Meier, E., Svoboda, T., Roth, D., van Gool, L.: Color-based object tracking in multi-camera environment. In: Michaelis, B., Krell, G. (eds.) Pattern Recognition. LNCS, vol. 2781, Springer, Heidelberg (2003) 10. P´erez, P., Vermaak, J., Blake, A.: Data fusion for visual tracking with particles. Proceeding of the IEEE 92(3), 495–513 (2004) 11. P´erez, P., Hue, C., Vermaak, J., Gangnet, M.: Color-based probabilistic tracking. In: European Conference on Computer Vision, Copenhagen, Denmark, vol. 1, pp. 661–675 (2002)
Finding Camera Overlap in Large Surveillance Networks Anton van den Hengel, Anthony Dick, Henry Detmold, Alex Cichowski, and Rhys Hill School of Computer Science University of Adelaide Adelaide, 5005, Australia {anton,ard,henry,alex,rhys}@cs.adelaide.edu.au
Abstract. Recent research on video surveillance across multiple cameras has typically focused on camera networks of the order of 10 cameras. In this paper we argue that existing systems do not scale to a network of hundreds, or thousands, of cameras. We describe the design and deployment of an algorithm called exclusion that is specifically aimed at finding correspondence between regions in cameras for large camera networks. The information recovered by exclusion can be used as the basis for other surveillance tasks such as tracking people through the network, or as an aid to human inspection. We have run this algorithm on a campus network of over 100 cameras, and report on its performance and accuracy over this network.
1
Introduction
Manual inspection is an inefficient and unreliable way to monitor large surveillance networks (see Figure 1 for example), particularly when coordination across observations from multiple cameras is required. In response to this, several systems have been developed to automate inspection tasks that span multiple cameras, such as following a moving target, or grouping together related cameras. A key part of any multi-camera surveillance system is to understand the spatial relationships between cameras in the network. In early surveillance systems, this information was manually specified or derived from camera calibration, but recent systems at least partly automate the process by analysing video from the cameras. These systems are demonstrated on networks containing of the order of 10 cameras, but have requirements that mean they do not scale well to networks an order of magnitude larger. For example: [1] requires manually marked correspondences between images; [2] requires a training stage where only one object is observed; and [3,4,5] require many correct detections of objects as they appear and disappear from cameras over a long period of time. An important step towards recovering spatial camera layout is to determine where cameras overlap. The approach taken in [6] is to estimate motion trajectories for people walking on a plane, and then match trajectories between cameras. However, this assumes planar motion, and accurate tracking over long periods Y. Yagi et al. (Eds.): ACCV 2007, Part I, LNCS 4843, pp. 375–384, 2007. c Springer-Verlag Berlin Heidelberg 2007
376
A. van den Hengel et al.
Fig. 1. Snapshot of video feeds from the network. Some cameras are offline.
of time. It also does not scale well, since track matching complexity increases as O(n2 ) with the number of cameras n. In [7] evidence for overlap is accumulated by estimating the boundary of each camera’s field of view in all other cameras. Again, this does not scale well to large numbers of cameras, and assumes that all cameras overlap. Because they start with an assumption of non-connectedness, and gradually accumulate evidence for connections, most methods for determining spatial layout rely on accurately detecting and/or tracking objects over a long time period. They also require comparisons to be made between every pair of cameras in a network. The number of pairs of cameras grows with the square of the number of cameras in the network, rendering exhaustive comparisons infeasible. This paper describes the implementation of a method called exclusion for determining camera overlap that is designed to quickly home in on cameras that may overlap. The method is computationally fast, and does not rely on accurate tracking of objects within each camera view. In contrast to most existing methods, it does not attempt to build up evidence for camera overlap over time. Instead, it starts by assuming all cameras are connected and uses observed activity to rule out connections over time. This is an easier decision to make, especially when a limited amount of data is available. It is also based on the observation that it is impossible to prove a positive connection between cameras—any correlation of events could be coincidence—whereas it is possible to prove a negative connection by observing an object in one camera while not observing it at all in another.
Finding Camera Overlap in Large Surveillance Networks
2
377
The Exclusion Algorithm
Consider a set of c cameras that generates c images at time t. By applying foreground detection [8] to all images we obtain a set of foreground blobs, each of which can be summarised by an image position and camera index. Each image is partitioned into a grid of windows, and each window can be labelled “occupied” or “unoccupied” depending on whether it contains a foreground object. Exclusion is based on the observation that a window which is occupied at time t cannot be an image of the same area as any other window that is simultaneously unoccupied. Given that windows tend to be unoccupied more often than they are occupied, this observation can be used to eliminate a large number of window pairs as potentially viewing the same area. The process of elimination can be repeated for each frame of video to rapidly reduce the number of pairs of image windows that could possibly be connected. This is the opposite of most previous approaches: rather than accumulate positive information over time about links between windows, we seek negative information allowing the instant elimination of impossible connections. Such connections are referred to as having been excluded [9]. 2.1
Exclusion over Multiple Timesteps
Rather than calculate exclusion separately at each timestep, it is more efficent to gather occupancy information over multiple frames and then calculate exclusion over all of them at once. Let the set of windows over all cameras be W = {w1 . . . wn }. Corresponding to each window wi is an occupancy vector oi = (oi1 , . . . , oiT ) with oit set to 1 if window wi is occupied at time t, and 0 if not. If two windows are images of exactly the same region in the world, we would expect their corresponding occupancy vectors to match exactly. This can be tested by applying the exclusiveor operator ⊕ to elements of the occupancy vectors: K
a ⊕ b = max ak ⊕ bk k=1
It can be inferred that two windows wi and wj do not overlap if oi ⊕ oj = 1. This comparison is very fast to compute, even for long vectors. 2.2
Exclusion with Tolerance
Exclusion as described so far assumes that: 1. corresponding windows in overlapping cameras cover exactly the same visible area in the scene, 2. all cameras are synchronised, so they capture frames at exactly the same time, and 3. the foreground detection module never produces false positives or false negatives.
378
A. van den Hengel et al.
In reality none of these assumptions is likely to hold completely. It is thus possible that two overlapping windows might simultaneously register as occupied and vacant and therefore that the exclusive-or of the corresponding occupancy vectors might incorrectly indicate that they do not overlap. Assumptions 1 and 2 can be relaxed by including the neighbours of a particular window when registering its occupancy. We use a padded occupancy vector pi which has element pit set to 1 when window wi or any of its neighbours is occupied at time t. A more robust mechanism for determining whether two windows wi and wj overlap is thus to calculate oi pj on the basis of the occupancy vector oi and the padded occupancy vector pj . The operator is a uni-directional version of the exclusive-or defined such that K
a b = max ak bk . k=1
(1)
where ak bk is 1 if and only if ak is 1 and bk is 0. Note that this means exclusion calculation is no longer symmetric. To account for detection errors (assumption 3), we calculate exclusion based on accumulated results over multiple tests, rather than relying on a single contradictory observation. Assuming that the detector has a constant failure rate, the evidence for exclusion is directly related to the number of contradictory observations in a fixed time period t = 1...T [9], which we call the exclusion count: Eij =
T
oit pjt .
(2)
t=1
2.3
Normalised Exclusion
The exclusion count has two main shortcomings as a measure for deciding window overlap/non-overlap: – As the operator ab will only return true when a is true, the exclusion count Eij between windows wi and wj is bounded by the number of detections in wi , and is likely to be higher for windows wi that register more detections. – In a large network, it will frequently occur that data sent from a camera will be lost, or not arrive in time to be included in the exclusion calculation, or that a camera will go offline. Thus the maximum value of Eij also depends on how often data from wj is available. To address these problems we define a padded availability vector v for each window that is set to 1 when occupancy data for the window and its neighbours is available, and 0 otherwise. We can then define an exclusion opportunity count between each pair of windows: Oij =
T t=1
oit vjt
(3)
Finding Camera Overlap in Large Surveillance Networks
379
Based on this we define an overlap certainty measure from each window with opportunity count at least 1 to every other window: Cij =
Oij − Eij Oij
(4)
which measures the number of times that an exclusion was not found between wi and wj as a proportion of the number of times an exclusion could possibly have been found given the available data. In general, exclusion estimates for windows that are only occupied a small number of times are dominated by noise such as erroneous detection. We therefore include a penalty term for such windows: log(Oij ) = Cij × min 1, (5) Cij log(Oref ) where Oref is a number of detections empirically determined to result in reliable exclusion calculation. We set this to 20 in our experiments.
3
Implementing Exclusion
In this section we describe how the exclusion algorithm is implemented in order to find overlap in a large network of cameras. This is done in two steps: – Object detection (Section 3.1): after each frame is captured, it is processed to detect objects within it. These detections are then converted to occupancy data for each window and sent to a central server. The main challenge for large camera networks is to detect objects quickly and reliably. – Exclusion calculation/update (Section 3.2, 3.3): at regular intervals of the order of several seconds, the stored occupancy data is used to calculate exclusion between each window pair. This exclusion result is then merged with exclusion results from earlier time periods, resulting in an updated estimate of camera overlap. The main challenge here is to synchronise data from different cameras, and to mitigate the memory requirements of exclusion data. 3.1
Distributed Foreground Detection
We detect foreground objects within each camera image using the Stauffer and Grimson background subtraction method [8]. To derive a single position from a foreground blob, we use connected components and take the midpoint of the low edge of the bounding box of each blob. This corresponds approximately to the lowest visible extent of the object in the image, assuming that the camera is approximately upright. Foreground detection is the most computationally intensive part of exclusion, but is also the stage that is easiest to parallelise. Presently, cameras are assigned to one of several processors that perform background subtraction on each image they capture. Eventually, though, we aim to implement detection on the cameras themselves.
380
A. van den Hengel et al.
3.2
Calculating Exclusion
Each occupancy result is tagged with the timestamp of the frame of video on which it is based and sent to a central server. After a fixed time interval, typically several seconds, these results are assembled to form an occupancy vector oi for each window wi . Each element of oi is indexed by a time offset t within the time interval, and can be one of three values: – oit = 2 if no occupancy data is available for wi within the time interval [t − tˆ, t + tˆ) – oit = 1 if wi is occupied within the time interval [t − tˆ, t + tˆ) – oit = 0 if wi is not occupied within the time interval [t − tˆ, t + tˆ) where tˆ is a tolerance to account for inaccuracies in camera synchronisation. These occupancy vectors are then used to calculate exclusion and opportunity counts as described in Sections 2.2 and 2.3 for each window pair within the time interval. These counts are then added to counts from previous time intervals, giving an updated estimate of exclusion confidence for each window pair. 3.3
Exclusion Data Compression
The central server stores both an exclusion count Eij and an exclusion opportunity count Oij for each pair of windows. Both counts are stored as a byte. This means that for a network of 100 cameras, each containing a 10 × 10 window grid, the counts require approximately 2 × 108 bytes of storage. Initially, Eij = 0 for all i and j. Consider how the exclusion counts are affected when a single person is observed in one window wD , and no other person is detected across the network. This will result in the exclusion count EDj being incremented for all windows j = D in the network. If the exclusion counts are stored in a matrix whose ij th element is the exclusion count between wi and wj , this results in an entire row of the matrix being incremented. Situations similar to this are quite common and suggest that a run length encoding scheme could effectively compress the matrix. Similarly, exclusion opportunity counts Oij are initially 0 for all i and j. Like exclusion counts, neighbouring opportunity counts are likely to be incremented at identical times, since an increment to Oij requires that wi is occupied and all data in the neighbourhood of wj is available. Again, this suggests the use of a run length encoding scheme to store exclusion opportunity data.
4
Testing Exclusion
We tested our exclusion implementation on network containing 100 Axis IP cameras, distributed across a university campus. Frames are captured from each camera as JPEG compressed 320×240 images using the Axis API. Each frame is divided into a 9×12 grid of windows, for a total of 10800 windows. As previously mentioned, the computational cost of foreground detection over a large number of
Finding Camera Overlap in Large Surveillance Networks
381
cameras far outweighs that of exclusion. This coarse level of foreground detection is well suited to implementation on board a camera, but for the purposes of testing a cluster of 16 dual core Opteron PCs has been used to process the footage from the 100 cameras in real time. By contrast, the central server, where occupancy results are assembled and exclusion is calculated, is a single desktop PC (Dell Dimension 4700, 3.2GHz Pentium 4, 1GB memory). 4.1
Performance Testing
We first test how the performance of exclusion scales, both over long time periods and large numbers of cameras. It was found that due to the optimisations described previously, the performance of exclusion does not depend strongly on the number of cameras on the network. Rather, it depends on the amount of activity in the network. Thus we observed the performance of exclusion during high and low activity periods, over a period of one hour. The memory required by exclusion increases over time, as shown in Figure 2. This is largely due to the decreased effectiveness of RLE compression of the exclusion counts (EC) as more activity is observed. The opportunity counts (EOC) are still well compressed by RLE after one hour, as camera availability changes rarely during this time. However, notice that the increase in EC elements, and corresponding increase in memory usage, is less than linear. Even after an hour of observation, only 29.56MB of memory is being used, compared to over 200MB that would be required to store the uncompressed data. Figure 3 shows the time taken to calculate exclusion at intervals over the one hour period. Notice that the time to compute exclusion remains fairly constant over the time period, and is consistently faster than real time, even using a standard desktop PC. In fact the exclusion is calculated for the hour’s footage 30000000
40 35
Memory Use (MB)
20000000
25
15000000
20 Data Size (MB)
15
Allocated Size (MB)
10000000
Compressed EC Elements
10
Compressed EOC Elements
Number of elements
25000000
30
5000000
5 0
0 0
15
30
45
60
Frame time (minutes)
Fig. 2. Memory usage over one hour of processing. The exclusion element count (EC) shows how the RLE compression becomes less effective over time.
382
A. van den Hengel et al. 16
14 Avg Occupancy Count / Camera
14
12
Speed Relative To Real Time Time Taken (min)
12
8 8 6 6
Time taken (min)
10
10
4
4
2
2
0
0 0
15
30
45
60
Frame time (minutes)
Fig. 3. Timing information for one hour of processing. The time required to process each frame remains approximately constant over time, although it increases slightly during periods of higher activity. Exclusion for 100 cameras is consistently calculated at over 4 times real time on a desktop PC, and an hour’s video takes under 13 minutes to process.
using less than 13 minutes of processor time. It is also evident that the time taken to calculate each exclusion count does depend on the amount of activity, measured by the number of occupancies detected per available camera. This can be seen by the slight increase in “Avg Occupancy Count per Camera” between about 30 and 50 minutes, and the corresponding decrease in “Speed relative to real time”. 4.2
Ground Truth Verification
It is difficult to verify that exclusion captures all overlap in a large camera network, and excludes all non-overlap. For example, Figure 1 shows a set of images captured from across the network at one moment. After some exclusion processing, the grid is rearranged to group together related cameras as shown in Figure 4. Connections are drawn between a window pair when the overlap certainty measure (Equation 5) exceeds a threshold C ∗ . The link must pass the threshold in both directions for the connection to be established, i.e. a link > C ∗ and Cji > C ∗ . In our is drawn between wi and wj if and only if Cij ∗ experiments we set C = 0.8. To verify the exclusion results we manually inspected the groups that were found. Close up views of some groups can be seen in Figure 5. It can be seen that overlap has been detected correctly in a variety of cases despite widely differing viewpoints and lighting conditions. These are correspondences that would be
Finding Camera Overlap in Large Surveillance Networks
383
Fig. 4. Video feeds from Figure 1 after running exclusion on one hour of footage. The cameras are arranged on screen so that related cameras are near each other, to aid human inspection.
Fig. 5. Overlapping groups detected by exclusion
very difficult to detect by tracking people, and attempting to build up correlations between tracks. The lighting conditions are often very poor, and the size of people in each camera varies greatly. Figure 4 also includes four camera groups that have been erroneously linked. Each of these groups has only one or two links between windows in each image, and view low traffic areas. These errors would thus disappear as more traffic is viewed. To correct these groups until enough traffic has been seen, a filter can be implemented that only links cameras when more than window in each camera is linked. Alternatively, a human operator can sever the links manually.
384
A. van den Hengel et al.
Some camera overlap was not detected because of low traffic during the hour that footage was captured. However, all overlap between cameras monitoring areas with enough detections (relative to Oref ) to calculate exclusion was correctly determined. This leads us to believe that remaining overlap can be detected when the system is run over a longer time period.
5
Conclusion
This paper describes a method for automatically determining camera overlap in large surveillance networks. The method is based on the process of eliminating impossible connections rather than the slower process of building up positive evidence of activity. We describe our implementation of the method, and show that it runs faster than real time on an hour of footage from a 100 camera network, using a single desktop PC. Future work includes testing the system over a period of several days, adding more cameras to the network, and implementing a more efficient foreground detector.
References 1. Javed, O., Rasheed, Z., Shafique, K., Shah, M.: Tracking across multiple cameras with disjoint views. In: ICCV 2003, pp. 952–957 (2003) 2. Dick, A.R., Brooks, M.J.: A stochastic approach to tracking objects across multiple cameras. In: Webb, G.I., Yu, X. (eds.) AI 2004. LNCS (LNAI), vol. 3339, pp. 160– 170. Springer, Heidelberg (2004) 3. Ellis, T.J., Makris, D., Black, J.K.: Learning a multi-camera topology. In: Joint IEEE Workshop on Visual Surveillance and Performance Evaluation of Tracking and Surveillance (VS-PETS), pp. 165–171. IEEE Computer Society Press, Los Alamitos (2003) 4. Stauffer, C.: Learning to track objects through unobserved regions. In: IEEE Computer Society Workshop on Motion and Video Computing, pp. 96–102. IEEE Computer Society Press, Los Alamitos (2005) 5. Tieu, K., Dalley, G., Grimson, W.E.L.: Inference of non-overlapping camera network topology by measuring statistical dependence. In: ICCV 2005, pp. 1842–1849 (2005) 6. Lee, L., Romano, R., Stein, G.: Monitoring activities from multiple video streams: Establishing a common coordinate frame. IEEE Trans. Pattern Analysis and Machine Intelligence 22(8), 758–767 (2000) 7. Khan, S., Javed, O., Rasheed, Z., Shah, M.: Human tracking in multiple cameras. In: IEEE International Conference on Computer Vision, pp. 331–336 (2001) 8. Stauffer, C., Grimson, W.E.L.: Learning patterns of activity using real-time tracking. IEEE Trans. Pattern Analysis and Machine Intelligence 22(8), 747–757 (2000) 9. van den Hengel, A., Dick, A., Hill, R.: Activity topology estimation for large networks of cameras. In: AVSS 2006. Proc. IEEE International Conference on Video and Signal Based Surveillance, pp. 44–49. IEEE Computer Society Press, Los Alamitos (2006)
Information Fusion for Multi-camera and Multi-body Structure and Motion Alexander Andreopoulos and John K. Tsotsos York University, Dept. of Computer Science & Engineering, Toronto, Ontario, M3J 1P3, Canada {alekos,tsotsos}@cse.yorku.ca Abstract. Information fusion algorithms have been successful in many vision tasks such as stereo, motion estimation, registration and robot localization. Stereo and motion image analysis are intimately connected and can provide complementary information to obtain robust estimates of scene structure and motion. We present an information fusion based approach for multi-camera and multi-body structure and motion that combines bottom-up and top-down knowledge on scene structure and motion. The only assumption we make is that all scene motion consists of rigid motion. We present experimental results on synthetic and nonsynthetic data sets, demonstrating excellent performance compared to binocular based state-of-the-art approaches for structure and motion.
1
Introduction
Multi-body and multi-camera structure and motion establishes the structure and motion of a scene that consists of multiple moving rigid objects that are observed from multiple views [1], [2]. Stereo vision analysis and image motion analysis provide information with complementary uncertainties which can depend on the motion of the camera platform, the scene structure and the spatiotemporal baselines. There are four fundamental problems with the extractable information from motion data or from stereo data [3]: (i) Image motion and disparity, with an unknown camera translation, allow us to infer object range only up to a scale ambiguity, since image motion and disparity depend on the ratio of camera translation to object range. (ii) Image motion and disparity tend towards zero near the focus of expansion (FOE). Since object range is inversely proportional to image motion and disparity, scene structure estimation is ill-conditioned near the FOE. (iii) The more closely aligned the local image structure is with the epipolar directions – i.e., directions pointing towards the FOE – the more ill-conditioned scene structure estimation becomes in those regions. (iv) Whereas large spatio-temporal baselines give better depth estimates for distant objects, the greater disparity and occlusion makes such cameras unsuitable for nearby objects. The severity of these problems is reversed when dealing with small baselines. The spatio-temporal baselines might be defined with respect to a monocular camera in motion –structure from motion–, a static stereo camera, or some other combination of static and non-static cameras. Y. Yagi et al. (Eds.): ACCV 2007, Part I, LNCS 4843, pp. 385–396, 2007. c Springer-Verlag Berlin Heidelberg 2007
386
A. Andreopoulos and J.K. Tsotsos
A method for fusing the structure and motion estimates of different cameras by preserving the accurate estimates and diminishing the effect of inaccurate estimates is highly desirable. For example, whereas in one camera the optical flow near the FOE might be poorly estimated, from another camera’s viewpoint the optical flow of the same scene region might not be as ill-conditioned, since the FOE will likely have changed. We present an information fusion based approach for dealing with all these problems in a unified framework. We model the above mentioned errors as originating from ambiguities in the estimation of stereo image correspondences and in the optical flow across all cameras. The only assumption we make as to the scene motion is that we are dealing with rigidly moving objects. The rest of the paper is organized as follows. Section 2 presents some related work. Section 3 introduces an approach for representing the motion and stereo data from a network of cameras. Section 4 describes how to combine this data in a single reference frame. Section 5 outlines a simple extension of the approach to camera rigs with arbitrary intrinsic and extrinsic parameters. Section 6 presents experimental results demonstrating the robustness of the approach. Section 7 concludes the paper.
2
Related Work
Richards [4] shows how the integration of changing disparity and object velocity can solve many of the ambiguitites inherent in stereopsis and motion under orthographic projection. Waxman [5] demonstrates the importance of the ratio of the rate of change of disparity over disparity, by using this quantity to unify stereo and motion analysis. As it is elaborated in [6], the importance of this ratio has been demonstrated numerous other times. Hanna and Okamoto [3] demonstrate how motion and stereo could be combined in a multi-camera system for egomotion and scene structure estimation. Their work is further expanded upon by Mandelbaum et al. [7]. Zhang and Kambhamettu [8] present a system which integrates 3D scene flow and structure recovery in order to complement the performance of each other, using a number of calibrated cameras. Singh and Allen [9] employ the Best Linear Unbiased Estimator (BLUE) to fuse local motion. Comaniciu [10], [11] developed a method for motion estimation under multiple source models. Neumann et al. [12] present a method for establishing a hierarchy of cameras based upon the stability and complexity of structure and motion estimation. To the best of our knowledge, the work we present is the first approach using information fusion for multi-camera and multi-body structure and motion.
3
Fusing Multiple Cameras
Assume we have a multi-camera rig composed of N monocular cameras. A maxiN mum of camera pairs exist. The coordinate system of camera C0 is referred 2
Information Fusion for Multi-camera and Multi-body Structure and Motion
387
C3 C4
C5
C2
C0
C6
C1
C8 C7
(a)
(b)
(c)
(d)
Fig. 1. (a) Diagram of a hypothetical nine camera rig. (b) A five camera rig mounted on a mobile robotic platform. (c)A planar textured region we used in some of the experiments for structure and motion estimation at a depth of 300cm. (d)The region after a 20 degree rotation around the camera’s optical axis.
to as the basis coordinate system. By convention a vector’s superscript will denote the coordinate system with respect to which we are expressing the vector. The camera rig is calibrated and therefore, for each pair of cameras Ci , Cj , we know a rotation matrix Rij and translation vector Tij = (Tijx , Tijy , Tijz )T that describes the rotation and translation that aligns camera Ci ’s coordinate axes with camera Cj ’s coordinate axes. See Fig. 1(a),(b) for examples of camera rigs where Rij = I (the identity matrix) ∀i, j. For each pixel p0 in camera C0 , and for each camera pair (Cj , Ci ) such that i = 0, j = i, we can use a stereo correspondence algorithm, such as [13], to obtain estimates of the pixels pj , pi in cameras Cj , Ci respectively, corresponding to pixel p0 in basis camera C0 . Similarly, we can obtain motion flow estimates for each pixel pj , pi in Cj , Ci . With each such pair of image pixels pj , pi , we can associate a 6D vector C0 C0 C0 0 V(p0 , Cj , Ci ), containing the 3D coordinates XC 1 = (X1 , Y1 , Z1 ) of a point P that is imaged by pj , pi in camera pair (Cj , Ci ). We can also associate with V(p0 , Cj , Ci ) a 3D vector uC0 corresponding to the 3D displacement vector of P that was extracted using the camera pair (Cj , Ci ). The displacement vector might be due to camera movement, an independent motion of scene point P or a C0 0 combination of both. As we have indicated above, the superscript C0 in XC 1 ,u indicates that the vectors are expressed with respect to the coordinate system C0 C0 C0 0 of C0 . Let XC 2 = (X2 , Y2 , Z2 ) denote the coordinate of P with respect to camera’s C0 coordinate system, obtained after an arbitrary camera rig or scene motion. The context will always make it clear with respect to which camera pair C0 0 (Cj , Ci ) we estimated XC 1 , X2 . We can then obtain the 3D motion estimate C0 C0 C0 C0 C0 T 0 u for point P by u = X2 − XC 1 . Then V(p0 , Cj , Ci ) (X1 , u ) . Given a small neighborhood Δp0 of pixels around a pixel p0 in C0 – we use 3×3 N N pixel neighborhoods in this paper –, the set p∈Δp j=0 i=1,i>j V(p, Cj , Ci ) 0 contains estimates of scene structure and motion over all camera pairs. If we need to enforce a hard real-time constraint, we can select to process a subset of the camera pairs. For each camera pair Cj , Ci and each pixel p0 in C0 that we process, we assign the covariance matrix Cov(V(p0 , Cj , Ci )). In the next section we will show how to estimate this covariance matrix and how to use it to assign a weight of importance to each one of those vectors. We will also show how to use
388
A. Andreopoulos and J.K. Tsotsos
information fusion techniques to get a robust estimate of the true scene structure and motion. Notice that in the above mentioned set, mainly due to occlusions, V(p0 , Cj , Ci ) will not always contribute a vector for all p0 , Cj , Ci .
4
Fusing the Camera Data
We need to model the uncertainty in each of the 6D vectors V(p0 , Cj , Ci ) in order to obtain each vector’s 6 × 6 covariance matrix. These covariance matrices are used by the BLUE estimator to obtain a reliable estimate of the scene structure and motion. For example, an image pixel that is near the focus of expansion in one monocular camera needs to assign a high uncertainty to its motion elements and assign a 3D structure uncertainty that depends on the scene depth relative to the camera pair used. From a different stereo camera’s point of view, these uncertainties will differ. By combining bottom-up and top-down information related to the scene uncertainty we obtain the noise model used by our BLUE estimator. For notational simplicity, we initially assume a perspective projection camera model where all cameras have the same focal length f , the aspect ratio is 1, the skew is 0, and the principal point is set to (0,0). The camera set up similar to Fig. 1(a),(b) where Rij = I and Tijz = 0 ∀i, j. The extension to arbitrary camera setups is presented in Section 5. Every pair of cameras (Cj , Ci ) can be viewed as a stereo camera with a focal length f , such that the projection C0 C0 C0 0 of a point XC 1 = (X1 , Y1 , Z1 ) in camera Ci is given by: xr =
x )∗f (X1C0 − T0i , C0 Z1
yr =
y )∗f (Y1C0 − T0i C0 Z1
(1)
and the projection of the same point in camera Cj is: xl =
x (X1C0 − T0j )∗f
Z1C0
,
yl =
y )∗f (Y1C0 − T0j
Z1C0
.
(2)
y y x x If | − T0j + T0i | ≥ | − T0j + T0i |, we have: x x x x (−T0j + T0i ) (xr + xl ) T0j + T0i + 2 xl − xr 2 yr y x x = (−T0j + T0i ) + T0i xl − xr x x + T0i )f (−T0j = . xl − xr
X1C0 =
(3)
Y1C0
(4)
Z1C0
(5)
y y x x Conversely, if | − T0j + T0i | < | − T0j + T0i |, we have:
xr x + T0i yl − yr y y y y + T0i ) (yr + yl ) T0j + T0i (−T0j = + 2 yl − yr 2 y y + T0i )f (−T0j = . yl − yr
y y X1C0 = (−T0j + T0i )
(6)
Y1C0
(7)
Z1C0
(8)
Information Fusion for Multi-camera and Multi-body Structure and Motion
389
Notice that in Eqs.(3)-(5) and Eqs.(6)-(8), yl and xl respectively, are not used. 0 This provides a simple approximation for XC 1 when due to small errors (xl , yl ), (xr , yr ) are not corresponding pixels. The corresponding image coordinates in C C the next frame are given by (xl , yl ) = (xl , yl ) + (vx j , vy j ), (xr , yr ) = (xr , yr ) + C C j j (vxCi , vyCi ) where (vx , vy ), (vxCi , vyCi ) denote the motion flow vectors in cameras Cj , Ci respectively. We can use (xl , yl ),(xr , yr ), in conjunction with Eqs.(3)-(8), C0 C0 C0 0 to estimate XC 2 = (X2 , Y2 , Z2 ) and calculate V(p0 , Cj , Ci ). We now show how Eqs. (3)-(8) can be used to define a covariance matrix for x V(p0 , Cj , Ci ). We only describe the covariance matrix derivation for | − T0j + y y y y x x x T0i | ≥ | − T0j + T0i |, since the case | − T0j + T0i | < | − T0j + T0i | is similar. We model the error in the correspondences of the image points as (xr + nxr , yr + nyr ), (xl + nxl , yl + nyl ) where nxr ,nyr ,nxl ,nyl are zero mean Gaussian random variables. Their standard deviation can depend on how noisy the images are and on prior knowledge regarding the accuracy of the correspondences –e.g., the sample variance of the correspondences within Δp0 . In this paper we assume a variance of 12 pixel for each of the four random variables. We also assume that the random variables are independent. Furthermore, we notice that in Eqs. (3)-(8) we can view X1C0 , Y1C0 , Z1C0 as functions in terms of nxr ,nyr ,nxl ,nyl . We obtain first order Taylor expansions of X1C0 , Y1C0 , Z1C0 and we use these Taylor 0 expansions to obtain variance/covariance measures for vector XC 1 . It can be shown that within first order: x x x x + T0i ) xl )2 − T0i ) xr )2 ((−T0j ((T0j V ar(x ) + V ar(xl ) r ( xl − x r )4 ( xl − x r )4 x x 2 x x + T0i ) + T0i ) yr )2 (−T0j ((−T0j V ar(yr ) + V ar(xr ) + V ar(Y1C0 ) ≈ 2 4 ( xl − x r ) ( xl − x r ) x x ((T0j − T0i ) yr )2 V ar(xl ) ( xl − x r )4 x x x x + T0i )f )2 − T0i )f )2 ((−T0j ((T0j V ar(Z1C0 ) ≈ V ar(x ) + V ar(xl ) r ( xl − x r )4 ( xl − x r )4
V ar(X1C0 ) ≈
(9)
(10) (11)
r , yl and yr are estimated using a trimmed mean estimator, with where x l , x the top and bottom, 25% of the samples being rejected before calculating the mean. The samples used to estimate x l , x r , yl and yr are the pixels in Ci , Cj corresponding to the neighborhood Δp0 in C0 . For example, to estimate x r we use the stereo matching algorithm to find the pixels in Ci that correspond to the r . pixels Δp0 in C0 , and then apply the trimmed mean estimator to get x
⎞ ⎛ Var ( X 1C0 ) 0 0 0 0 0 ⎞ ⎛ Var ( X 1C0 ) 0 0 −Var ( X 1C0 ) 0 0 ⎟ ⎜ ⎟ ⎜ Var (Y1C0 ) 0 0 0 0 0 ⎟ ⎜ 0 Var (Y1C0 ) 0 0 −Var (Y1C0 ) 0 ⎟ ⎜ ⎟ ⎜ C0 ⎜ Var ( Z ) 0 0 0 0 0 Var ( Z1C0 ) 0 0 0 0 −Var ( Z1C0 ) ⎟ 1 ⎟ ⎜ + a Cov(V (p 0 , C j , Ci ) = (1 − a ) ⎜ ⎟ C0 ⎟ ⎜ Var ( X 2C0 − X 1C0 ) 0 0 0 0 0 0 0 Var ( X 2C0 − X 1C0 ) 0 0 ⎟ ⎜ −Var ( X 1 ) ⎟ ⎜ C0 C0 C0 ⎟ ⎜ C C −Var (Y1 ) Var (Y2 − Y1 ) 0 0 0 0 ⎟ ⎜ 0 0 0 0 Var (Y2 0 − Y1 0 ) 0 ⎟ ⎜ ⎟ ⎜ ⎜ −Var ( Z1C0 ) Var ( Z 2C0 − Z1C0 ) ⎟⎠ 0 0 0 0 C0 C0 ⎟ ⎜ ⎝ 0 0 0 0 0 Var ( Z 2 − Z1 ) ⎠ ⎝
Fig. 2. The covariance matrix encoding the uncertainties
390
A. Andreopoulos and J.K. Tsotsos
In order to obtain a covariance matrix for V(p0 , Cj , Ci ), we also need to obtain an estimate of the variance of the elements of uC0 . We know that for a physical point PCj that is moving with velocity SCj with respect to camera Cj and its coordinate frame, we can decompose the velocity as SCj = −TCj − C C C C C C ΩCj × PCj where TCj = (Tx j , Ty j , Tz j )T and ΩCj = (Ωx j , Ωy j , Ωz j )T denote the translational and angular velocity vectors of camera Cj that would cause the same apparent motion of the particle PCj with respect to camera Cj ’s coordinate frame. Then, the image velocity of the projection (xl , yl ) of PCj in camera Cj is given by C
vx j C vy j
= B Cj ΩCj + dCj ACj TCj
(12)
where dCj is the inverse of the scene depth with respect to camera Cj ’s coordinate system – it is estimated using Eqs.(3)-(8) and the current camera pair – and B
Cj
=
(f
xl yl f y2 + fl )
−(f +
x2l f )
− xlfyl
yl −xl
Cj
A
=
−f 0 xl 0 −f yl
.
(13)
Similar conditions hold for camera Ci . We use Eq.(12) to model the noise senC0 0 sitivity of XC 2 , as we did for X1 in Eqs.(9)-(11). This allows us to weigh the suitability of each camera for tracking a particular object. In the case of multibody structure and motion and due to the reasons mentioned in the introduction, it is quite feasible to end up with degenerate situations of objects whose motion estimation is ill-conditioned from a particular viewpoint. If we model TCj , ΩCj 0 as being corrupted by Gaussian noise, we can view XC 2 as a function of nxr , nyr , nxl , nyl , nTCj , nΩCj , nTCi , nΩCi , where nTCj , nΩCj , nTCi , nΩCi denote zero mean Gaussian noise vectors. For each camera pair (Cj , Ci ) and for each corresponding pixel pair pj , pi in the two cameras, we obtain approximations for TCj , ΩCj , TCi , ΩCi – de Cj , Ω Cj , T Ci , Ω Ci – and the variances of n Cj , n Cj ,nTCi , nΩCi as noted T T Ω follows: For each local image region centered at pα , α ∈ {i, j}, or for each image region containing pα and undergoing independent rigid motion – esti Cα , mated using any popular motion segmentation algorithm – we estimate T Cα Ω , the approximation of the camera’s translational and rotational velocity that would lead to the motion flow observed in that particular image region using camera pair (Cj , Ci ). For each such image region, we use a least squares pseudo-inverse based approach on a random subset of the estimated displacement vectors to approximate the translational and rotational velocity. We repeat this approach a number of times and the mean of the results is used as Cα , Ω Cα and their variance provides an estimate for the variance used in T the noise model described above. If we take the partial derivatives of X2C0 , Y2C0 , Z2C0 with respect to the above mentioned random variables and expand Cj , Ω Cj , T Ci , Ω Ci we obtain the desired expressions around x l , x r , yl , yr , T C0 C0 C0 for V ar(X2 ), V ar(Y2 ) and V ar(Z2 ). In the appendix we list the derived expressions for these variances. The above mentioned variances are referred to
Information Fusion for Multi-camera and Multi-body Structure and Motion
391
as the “top-down” information. Note that in our experiments, when modeling V ar(W2C0 − W1C0 ), we make the assumption of independence between W1C0 and W2C0 for all W ∈ {X, Y, Z}. Notice that V ar(X2C0 ), V ar(Y2C0 ) and V ar(Z2C0 ) are C C calculated using the derivatives of velocities vx j , vy j and might have very differC0 C0 ent magnitudes from V ar(X1 ), V ar(Y1 ), V ar(Z1C0 ). To guarantee the numerical stability of the covariance matrices, we perform two simple modifications to the top-down variances. We first set an upper bound maxvar to each of the variances by setting V ar(W1C0 ) ← min (V ar(W1C0 ), maxvar), V ar(W2C0 − W1C0 ) ← min (V ar(W2C0 − W1C0 ), maxvar). Secondly, for each W ∈ {X, Y, Z} and each C0 pixel in C0 , wescale the variances V ar(W 2 ) acquired across all camera pairs by C0 aW c · min( allpairs V ar(W1 ))/ min( allpairs V ar(W2C0 )) for some constant c (we set c = 2 in our experiments). For each pair (Cj , Ci ), we also estimate the sample variances V ar(X1C0 ), V ar(Y1C0 ), V ar(Z1C0 ), V ar(X2C0 − X1C0 ), V ar(Y2C0 − Y1C0 ) and V ar(Z2C0 − Z1C0 ) by using the samples in p∈Δp V(p, Cj , Ci ) and using the mean of the vectors 0 N N in p∈Δp j=0 i=1,i>j V(p, Cj , Ci ) as the sample mean. We refer to these 0 sample variances as the “bottom-up” information. We define the final covariance matrix corresponding to each vector V(p0 , Cj , Ci ) as a linear combination of their corresponding top-down and bottom-up variances. For each point p0 in camera C0 and by using the two cameras Cj ,Ci for depth estimation, we use the set p∈Δp V(p, Cj , Ci ) in conjunction with the variances defined above, to 0 model the covariance matrix of V(p0 , Cj , Ci ) as given by Fig.2, where 0 ≤ a ≤ 1. , where for each k ∈ {1, ..., n}, Assume we have n vectors Vi(1),j(1) ,...,Vi(n),j(n) Vi(k),j(k) is the average of all the vectors in p∈Δp V(p, Cj(k) , Ci(k) ). Also with 0 each of the vectors Vi(k),j(k) we associate a covariance matrix Nk indicating our confidence in this measure, as described in this section. If we ignore any potential cross-correlation between the n vectors, the Best Linear Unbiased Estimator (BLUE) [9] is the vector X that minimizes the sum of the Mahalanobis distances n T T = (Vi(1),j(1) N−1 1 + ... + k=1 D(X, Vi(k),j(k) , Nk ). It can be shown that X −1 T −1 −1 −1 Vi(n),j(n) Nn )(N1 + ... + Nn ) . In the next section we extend our approach to camera rigs with arbitrary intrinsic and extrinsic parameters.
5
Arbitrary Camera Rig Setup
Let us suppose that for a camera pair (Cj , Ci ) with intrinsic camera parameters (Kj , Ki ) and for pixels pj = (xl , yl )T , pi = (xr , yr )T imaging a common scene point P = (X1C0 , Y1C0 , Z1C0 )T , the following equations hold:
xl yl
⎛ ⎜ =⎝
C
C0 C0 y z −T0,j ))+Kj1,3 (R1,3 j,0 (Z1 −T0,j )) C0 z )) Kj3,3 (R3,3 (Z −T 1 j,0 0,j C0 C0 C0 2,2 2,2 y x z Kj2,1 (R2,1 −T0,j ))+Kj2,3 (R2,3 j,0 (X1 −T0,j ))+Kj (Rj,0 (Y1 j,0 (Z1 −T0,j )) C0 3,3 3,3 z Kj (Rj,0 (Z1 −T0,j ))
1,2 1,2 x 0 Kj1,1 (R1,1 j,0 (X1 −T0,j ))+Kj (Rj,0 (Y1
⎞ ⎟ ⎠ (14)
392
A. Andreopoulos and J.K. Tsotsos
xr yr
⎛ ⎜ =⎝
C
C0 C0 y z −T0,i ))+Ki1,3 (R1,3 i,0 (Z1 −T0,i )) C0 z )) Ki3,3 (R3,3 (Z −T 1 i,0 0,i C0 C0 C0 2,2 2,2 y x z Ki2,1 (R2,1 −T0,i ))+Ki2,3 (R2,3 i,0 (X1 −T0,i ))+Ki (Ri,0 (Y1 i,0 (Z1 −T0,i )) C0 3,3 3,3 z Ki (Ri,0 (Z1 −T0,i ))
1,2 1,2 x 0 Ki1,1 (R1,1 i,0 (X1 −T0,i ))+Ki (Ri,0 (Y1
⎞ ⎟ ⎠ (15)
m,n denote the m, nth entry of Kj /Rj0 . As we did in Eqs.(3)-(8), where Kjm,n /Rj,0 y y x x 0 + T0i |, we can express XC if | − T0j + T0i | ≥ | − T0j 1 in terms of xl , xr , yr . y y x x 0 Conversely, if | − T0j + T0i | < | − T0j + T0i | we can express XC 1 in terms of yl , C0 yr , xr Thus, we can define a function g(pj , pi ) X1 with respect to camera C0 ’s coordinate system. By using g(·) and the approach described in Section 4, we can obtain the desired variance approximations. We also need to redefine Eqs.(12)-(13) in order to obtain variance estimates for the motion error. We will only deal with the case of camera Cj , as the case of camera Ci is similar. As indicated in Section 4, SCj = −TCj − ΩCj × PCj . Then: 1,1 Cj 1,1 1,2 YCj 1,3 1,2 d YCj C d XCj K + K d Kj X vx j Cj + Kj Cj + Kj C C j j j j dt Z dt Z Z Z = = Cj C 2,3 2,2 d YCj dt vy j Kj2,2 Y K Cj + Kj Cj j dt Z Z (16) assuming Kj2,1 = 0, Kj3,3 = 1. The derivatives are taken with respect to time t, and by using the expression for SCj we can express Eq.(16) in terms of TCj and ΩCj . Then the variance derivation proceeds as described in Section 4. The derivatives can be determined analytically, or via common numerical methods such as finite differences.
6
Experiments
We present our camera setup and results in Figs. 1, 3, 4. We test our approach on a number of synthetic and non-synthetic datasets. Synthetic dataset (i) consists of a 30cm × 30cm planar surface on a black background (Fig. 1(c),(d)) centered at camera C0 , moving in depth, along the optical axis, by 15cm per frame. Synthetic dataset (ii) consists of the planar surface, rotated by 4 degrees around the optical axis between each frame. Syntethic dataset (iii) consists of a (2cm, 2cm) translation of the planar surface, parallel to the image plane, between each frame. The camera setup is similar to that of Fig. 1(a). All cameras Ci , i > 0 are radially distributed around camera C0 at a radius of 12cm, have a focal length of 4mm, and have corrupting Gaussian noise added to their images. We fuse all (C0 , Ci ) camera pairs and demonstrate the performance of the algorithm with an increasing object range from 300cm to 800cm by setting a = 0.5 in Fig.2. The stereo correspondence and optical flow algorithm used is described in [13] and is available by the authors online1 . Our results are illustrated in Fig. 3(a)-(f). We also test our algorithm using a five camera rig, as shown in Fig. 1(b). The corresponding results are presented in Fig. 4(a)-(h). In the synthetic data set we used the entire planar surface to estimate each TCα , ΩCα 1
http://www.cs.umd.edu/users/ogale/download/code.html
Information Fusion for Multi-camera and Multi-body Structure and Motion
Stereo reconstruction error of world coordinates Object shift:[0;0;15], SNR:4 dB, no calibration error
Stereo reconstruction error of world coordinates Object rotated by 4 degrees around optical axis, SNR:4 dB, no calibration error
18
393
Stereo reconstruction error of world coordinates Object shift:[2;2;0], SNR:4 dB, no calibration error 25
25
12 10 8 6 4
20
20
RMS error of 3D reconstruction
14 RMS error of 3D reconstruction
RMS error of 3D reconstruction
16
15
10
15
10
5
5
2 0
300
400 500 600 700 Distance of object from camera (cm)
0
800
300
0
800
300
(c)
(a)
400 500 600 700 Distance of object from camera (cm)
800
(e)
3D motion error Object rotated by 4 degrees around optical axis, SNR:4 dB, no calibration error
3D motion error Object shift:[0;0;15], SNR:4 dB, no calibration error
3D motion error Object shift:[2;2;0], SNR:4 dB, no calibration error 25
25
25
20
RMS error of 3D motion
15
10
15
10
5
5
0
0 300
400 500 600 700 Distance of object from camera (cm)
(b)
800
RMS error of 3D motion
20
20 RMS error of 3D motion
400 500 600 700 Distance of object from camera (cm)
15
10
5
300
400 500 600 700 Distance of object from camera (cm)
(d)
800
0
300
400 500 600 700 Distance of object from camera (cm)
800
(f)
Fig. 3. (a)-(f):The results of our tests on the synthetic dataset. The x-axes represent the depth of the object in cm, and the y-axes represent the RMS error of the stereo reconstructed coordinates and the 3D motion vector (in cm). The RMS error for a particular camera pair is calculated by estimating the error across all pixels in the base camera C0 that fall within the textured region. The solid/dashed lines correspond to the errors of our information fusion based approach using the BLUE/mean estimator, and the boxplots represent the distribution of the errors across each of the camera pairs used. The red crosses represent outliers. Note that in some figures the outliers are not displayed as they fall outside the vertical range of our error axes. (a),(b) correspond to the stereo reconstruction and 3D motion error respectively, when the planar object was translated by 15cm in depth along the optical axis. (c),(d) correspond to the stereo reconstruction and 3D motion error respectively, when the planar object was rotated by 4 degrees around the optical axis between frames. (e),(f) correspond to the stereo reconstruction and 3D motion error respectively when the translation occured parallel to the image plane. The object was translated by 2cm along the x and y axes of the world coordinate system. Notice how, even though gross outliers exist in most of the figures, the effect of those outliers on the estimated scene structure and motion is minimal in general. We also performed a number of experiments with modest errors in the external parameters’ calibration and similar observations were made. The mean RMS error of the stereo reconstruction using the information fusion/mean approach for all instances of the reconstructed planar surface is 2.05 ± 1.71/3.71 ± 3.87 cm respectively. The respective values for the motion data are 2.35 ± 1.43/2.56 ± 1.41 cm. In both cases the improvement compared to the mean approach is statistically significant using a paired-samples t-test (p ≈ 0.01).
394
A. Andreopoulos and J.K. Tsotsos
(a)
(c)
(e)
(g)
(b)
(d)
(f)
(h)
Fig. 4. (a)-(h):Experimental results from an image sequence showing a robotic wheelchair that is equipped with a 6-d.o.f. robotic arm. The robotic arm is moving diagonally towards the top left image corner. (a)-(b): Adjacent frames from the respective sequence (before correcting for radial/tangential distortions). (c): The reconstructed scene depth using a single pair of cameras to reconstruct each scene. Image regions in black denote pixels where the left-right consistency constraint could not be enforced. (d): The reconstructed scene depth of frames (a),(b) using the five camera rig setup shown in Fig. 1 in conjunction with our information fusion based algorithm. The colorbar depths of (c),(d) represent mm. Notice the significant decrease in occlusions. (e)-(f): Image motion of the sequence after projecting the estimated 3D motion on the image plane using a single camera pair in conjunction with our information fusion based algorithm. Image motion is represented in pixel units. (g)-(h): Image motion of the respective image sequences after using the five camera rig setup shown in Fig. 1 in conjunction with our information fusion based algorithm. (e),(g): The image motion component parallel to the horizontal axis and (f),(h): The image motion component parallel to the vertical axis.
(simulating perfect motion segmentation) and in the non-synthetic data set we used 21 × 21 pixel regions centered at the current pixel of interest. From Fig. 3, we observe that the multi-camera approach provides a significant decrease of the RMS error in both structure and motion estimation compared to the errors achieved using the stereo camera pairs. In almost all cases the quality of the results surpasses that obtained by any one of the camera pairs. As indicated in the caption of Fig. 3 the BLUE estimator provides better results than the results obtained by the mean vector across all cameras and their neighborhoods. For both the structure and motion data the improvement is judged statistically significant. In Fig. 3(a),(b) where the plane is moving along the z-axis and we are dealing with ill-conditioned motion estimation near the focus of expansion, we observe significant improvements. In Fig. 3(c)-(d) we present results after a pure rotation of the plane around the optical axis. It is interesting
Information Fusion for Multi-camera and Multi-body Structure and Motion
395
to notice however, that for depths 700cm, 800cm the optical flow estimation algorithm we used performs poorly on about half of our camera pairs, thus, resulting in a big RMS error, as the boxplots show. Our algorithm is capable of ignoring the erroneous data and gives us a relatively robust estimate of the 3D motion. This indicates that if we are using a multi-camera rig with cameras that break down quite often and provide gross outliers, our algorithm remains reliable. We observe that the mean estimator is severely affected by outliers at various depths, while the information fusion based algorithm is more robust in the presence of outliers. In Fig. 4(a)-(h) we compare the performance of our algorithm using a two camera rig versus a five camera rig (Fig. 1(b)). The five camera rig consists of two Point Grey Research Bumblebee stereo cameras and a Point Grey Research Flea camera. The coordinate system of the Flea camera is used as our basis coordinate system and represents camera C0 . The two camera rig is represented using the Flea camera and one of the four Bumblebee monocular cameras. The robotic wheelchair presented in Fig. 3 is equipped with a 6-d.o.f. robotic arm providing a number of independent rigid motions to test our algorithm. We used the algorithm described in [13] to determine the correspondences. We notice a dramatic increase in the number of pixels satisfying the left-right consistency constraint as the number of cameras in our rig increases.
7
Conclusion
We presented an algorithm for multi-camera and multi-body structure and motion. The algorithm combines top-down and bottom-up knowledge on scene structure and motion to model the respective uncertainties. An information fusion based algorithm uses these uncertainties to obtain competitive results demonstrating that our algorithm performs robustly in situations where a number of camera pairs provide severely degraded results. Such situations arise in practice due to hardware failures and poor environmental conditions. We are currently investigating the use of other information fusion algorithms for solving this problem [10]. Some potential application areas in future research are dynamic scene interpretation, vision based simultaneous localization and mapping (SLAM) and dynamic rendering. Acknowledgments. JKT holds the Canada Research Chair in Computational Vision and gratefully acknowledges its financial support. AA would also like to thank the Natural Sciences and Engineering Research Council of Canada (NSERC) for its financial support through the PGS-D scholarship program.
References 1. Schindler, K., Suter, D.: Two-view multibody structure and motion. In: Proc. Conf. Computer Vision and Pattern Recognition (2005) 2. Zhang, W., Kosecka, J.: Nonparametric estimation of multiple structures with outliers. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3954, Springer, Heidelberg (2006)
396
A. Andreopoulos and J.K. Tsotsos
3. Hanna, K.J., Okamoto, N.E.: Combining stereo and motion analysis for direct estimation of scene structure. In: Proc. Int. Conf. on Computer Vision (1993) 4. Richards, W.: Structure from stereo and motion. Journal of the Optical Society of America A. 2(2), 343–349 (1985) 5. Waxman, A., Duncan, J.: Binocular image flows. IEEE Trans. Patt. Anal. Mach. Intell. 8(6), 715–729 (1986) 6. Grosso, E., Tistarelli, M.: Active dynamic stereo vision. IEEE Trans. Patt. Anal. Mach. Intell. 17(11), 1117–1128 (1995) 7. Mandelbaum, R., Salgian, G., Sawhney, H.: Correlation-based estimation of egomotion and structure from motion and stereo. In: Proc. Int. Conf. on Computer Vision (1999) 8. Zhang, Y., Kambhamettu, C.: Integrated 3D scene flow and structure recovery from multiview image sequences. In: IEEE Conf. Computer Vision and Pattern Recognition, IEEE Computer Society Press, Los Alamitos (2000) 9. Singh, A., Allen, P.: Image-flow computation: An estimation-theoretic framework and a unified perspective. CVGIP: Image Understanding 56(2), 152–177 (1992) 10. Comaniciu, D.: Nonparametric information fusion for motion estimation. In: IEEE Conf. Computer Vision and Pattern Recognition, IEEE Computer Society Press, Los Alamitos (2003) 11. Comaniciu, D., Meer, P.: Mean shift: A robust approach toward feature space analysis. IEEE Trans. Patt. Anal. Mach. Intell. 24, 603–619 (2002) 12. Neumann, J., Fermuller, C., Aloimonos, Y.: A hierarchy of cameras for 3D photography. Computer Vision and Image Understanding 96, 274–293 (2004) 13. Ogale, A.S., Aloimonos, Y.: A roadmap to the integration of early visual modules. International Journal of Computer Vision: Special Issue on Early Cognitive Vision 72(1), 9–25 (2007)
Appendix In this section we derive expressions for V ar(X2C0 ), V ar(Y2C0 ), V ar(Z2C0 ) y y x x + T0i | ≥ | − T0j + T0i |. From Eqs.(9)-(11) we can for the case | − T0j derive the corresponding expressions for V ar(X2C0 ), V ar(Y2C0 ), V ar(Z2C0 ):
x x x x ((−T0j +T0i ) xl )2 ((T0j −T0i ) xr )2 V ar(x ) + V ar(xl ), V ar(Y2C0 ) ≈ 4 r ( xl − xr ) ( xl − x r )4 x x 2 x x 2 x x (−T0j +T0i ) ((−T0j +T0i ) yr ) ((T0j −T0i ) yr )2 V ar(xr ) + ( V ar(xl ), V ar(Z2C0 ) ( xl − xr )2 V ar(yr ) + ( xl − xr )4 xl − xr )4 x x x x ((−T0j +T0i )f )2 ((T0j −T0i )f )2 ≈ V ar(xr ) + ( V ar(xl ). We have previously noted that ( xl − xr )4 xl − xr )4 C C (xr , yr ) = (xr , yr ) + (vxCi , vyCi ), (xl , yl ) = (xl , yl ) + (vx j , vy j ). If we let a ∈ {x, y},
V ar(X2C0 ) ≈
b ∈ {r, l} and k = i/j if b = r/l we obtain the following approximations ∂a ∂a for V ar(xr ), V ar(xl ), V ar(yr ): V ar(ab ) ≈ ( ∂xbr )2 V ar(xr )+ ( ∂xbl )2 V ar(xl )+ ∂a
( ∂yrb )2 V ar(yr )+ ( ∂a
∂ab
C ∂Ωx k
)2 V ar(ΩxCk )+ ( ∂a
∂ab
C ∂Ωy k
)2 V ar(ΩyCk )+ (
∂ab C
∂Ωz k
)2 V ar(ΩzCk )+
( Cbk )2 V ar(TaCk ) + ( Cbk )2 V ar(TzCk ). By using Eqs. (12)-(13) we can derive ∂Ta ∂Tz the expressions for the partial derivatives. By expanding these expressions for Ci , Ω Ci , T Cj , Ω Cj , as appropriate, the partial derivatives around x l , x r , yl , yr , T we can obtain the desired expressions.
Task Scheduling in Large Camera Networks Ser-Nam Lim1, , Larry Davis2 , and Anurag Mittal3 2
1 Cognex Corp., Natick, MA, USA CS Dept., University of Maryland, College Park, Maryland, USA 3 CSE Dept., IIT, Madras, India
Abstract. Camera networks are increasingly being deployed for security. In most of these camera networks, video sequences are captured, transmitted and archived continuously from all cameras, creating enormous stress on available transmission bandwidth, storage space and computing facilities. We describe an intelligent control system for scheduling Pan-Tilt-Zoom cameras to capture video only when task-specific requirements can be satisfied. These videos are collected in real time during predicted temporal “windows of opportunity”. We present a scalable algorithm that constructs schedules in which multiple tasks can possibly be satisfied simultaneously by a given camera. We describe two scheduling algorithms: a greedy algorithm and another based on Dynamic Programming (DP). We analyze their approximation factors and present simulations that show that the DP method is advantageous for large camera networks in terms of task coverage. Results from a prototype real time active camera system however reveal that the greedy algorithm performs faster than the DP algorithm, making it more suitable for a real time system. The prototype system, built using existing low-level vision algorithms, also illustrates the applicability of our algorithms.
1 Introduction Large scale camera network systems are being increasingly deployed for purposes that include security, traffic monitoring, etc. These systems typically consist of a large number of cameras, which can either be active (specifically, Pan-Tilt-Zoom or PTZ cameras) or static, transmitting in real time video streams to processing and/or storage systems. Our interest is in controlling these cameras to acquire video segments that satisfy taskspecific constraints. For example, one may wish to acquire at least a few images of each person who enters a given region, capture video segments lasting k seconds and containing well-magnified facial images for facial recognition, or, capture k second long video segments of the side view of a person for gait modeling and recognition. By intelligently transmitting and storing only video segments satisfying task requirements, we can reduce the bandwidth requirements and storage space significantly and increase the efficiency and effectiveness with which the collected video segments can be processed. The control of the cameras to collect these video segments is a challenging problem. The system must detect and track moving objects both within and between cameras in a sensing stage, a problem which is not fully solved yet. Papers such as [1,2,3] deal
This research was funded in part by the U.S. Government VACE program.
Y. Yagi et al. (Eds.): ACCV 2007, Part I, LNCS 4843, pp. 397–407, 2007. c Springer-Verlag Berlin Heidelberg 2007
398
S.-N. Lim, L. Davis, and A. Mittal
with tracking under occlusions, and other papers such as [4,5] describe algorithms for tracking across non-overlapping views. A second challenge is to predict, given a set of tracked targets, the time intervals during which video segments meeting the requirements of the tasks can be collected from available cameras. These requirements include ensuring (1) that the associated object is unobstructed by other objects, (2) that it is moving in a direction suitable for the task, (3) that it can be captured in a field of view of the camera (PTZ) assigned to collect its video segments, and (4) that the collected video segments must satisfy task-specific minimum resolution and duration. For example, if the task is to collect facial images, then we must ensure that the video segments are collected only during time intervals when the person is predicted to be walking towards the camera and unobstructed by other moving objects.This can be done using the observed tracks of the person and other moving objects, predicting their trajectories into the future, and then identifying periods of crossings between the predicted trajectories with respect to each of the available cameras. The complements of these periods of crossings are visibility time intervals during which the person is unobstructed, and camera settings can be determined within these temporal visibility predictions to capture the person in a well-magnified frontal image or video sequence satisfying the four requirements above. This problem has been addressed in earlier work. [6] described the construction of so-called “Task Visibility Intervals” (TVIs) and “Multiple Task Visibility Intervals” (MTVIs), that represent timevarying camera setting ranges that can be used to collect video segments satisfying one (TVI) or multiple tasks simultaneously (MTVIs). A TVI is a 6-tuple: (1) (c, (T, o), [r, d], V alidψ,φ,f (t)), where c represents a PTZ camera, (T, o) is a (task, object) pair - T is the index of a task to be accomplished and o is the index of the object to which the task is to be applied, and [r, d] is a future time interval during which task requirements can be satisfied using camera c. Then, for each time instance t ∈ [r, d], V alidψ,φ,f (t) is the range of valid combinations of the pan angle (ψ), tilt angle (φ) and focal length (f ) settings that camera c can employ to capture object o at time t. The tasks themselves are 3-tuples: (p, α, β),
(2)
where p is the required duration of the task, α is the orientation of the object relative to the optical axis of the camera used to accomplish the task, and β is the minimum image resolution needed to accomplish the task. [6] also described the composition of TVIs into MTVIs, time intervals during which collections of tasks can be satisfied simultaneously by one camera. A set of n TVIs, each represented in the form: (c, (Ti , oi ), [ri , di ], V alidψi ,φi ,fi (t)), for TVI i [Eqn. 1] can be combined into a valid MTVI, represented as: (Ti , oi ), [ri , di ], V alidψi ,φi ,fi (t), (c, i=1...n
i=1...n
i=1...n
(3)
Task Scheduling in Large Camera Networks
such that:
[ri , di ] = ∅,
399
(4)
i=1...n
i.e., there is some common time interval during which they can be scheduled, and: [ri , di ] ≥ pmax , i=1...n
where pmax is the largest processing time among the tasks, and for all t ∈ V alidψi ,φi ,fi (t) = ∅,
i=1...n [ri , di ],
(5)
i=1...n
i.e., the tasks can be captured with common PTZ settings. Besides [6], other work that has focused on temporal analysis and planning for camera scheduling includes [7,8], which discuss a dynamic sensor planning system, called the MVP system. They were concerned with determining occlusion-free viewpoints of a target. This involves handling occlusions between the target and the different moving objects in a scene, each of which generates a swept volume in temporal space. Using a temporal interval search, they divide the temporal intervals into halves while searching for a viewpoint that is not occluded in time by these sweep volumes. This is then integrated with other constraints such as focus and field of view in [8]. The culmination of this work is found in [7], where the algorithms are applied to an active robot work cell. In this paper, we will address the problem of job scheduling given the TVIs and MTVIs generated as in [6]. In general, job scheduling problems are NP-hard, and approximation algorithms have to be employed. We first analyze the approximation factor of a greedy scheduling algorithm (as a function of the number of cameras), which reveals that its performance deteriorates significantly as the number of cameras increases. We then describe a Dynamic Programming (DP) approximation algorithm with an approximation factor that is much better than the greedy approach. The performance advantage of the DP algorithm is confirmed by simulations. Finally, we describe a prototype real time active camera system. A scheduler controls PTZ cameras in real time to capture video segments based on automatically constructed TVIs and MTVIs. While the prototype system includes only a small number of cameras due to limited resources, the results illustrate the applicability of the algorithms for large scale camera networks.
2 Single-Camera Scheduling We first study the scheduling problem when only a single camera is used. This will be extended to the problem of multiple cameras in the next section. Also, we will limit our analysis to non-preemptive schedules in this paper. We introduce the following theorems that make the single-camera scheduling problem tractable: + Theorem 1. Let the slack for the i-th task be δi = [t− δi , tδi ], and define δmax = max(|δi |) and pmin as the smallest processing time among all (M)TVIs for some camera. Then, if |δmax | < pmin , then any feasible schedule for the camera is ordered by the slacks’ start times.
400
S.-N. Lim, L. Davis, and A. Mittal
+ − + Proof. Consider that the slack δ1 = [t− δ1 , tδ1 ] precedes δ2 = [tδ2 , tδ2 ] in a schedule and − − − tδ1 > tδ2 . Let the processing time corresponding to δ1 be p1 . Then t− δ1 + p1 > tδ2 + p1 . − + We know that if tδ1 + p1 > tδ2 , then the schedule is infeasible. This happens if t+ δ2 ≤ − + − + − tδ2 + p1 - i.e., tδ2 − tδ2 ≤ p1 . Given that |δmax | < pmin , tδ2 − tδ2 ≤ p1 is true.
Theorem 1 implies that if |δmax | < pmin , we can limit our attention to feasible schedules that are ordered by the slacks’ start times. This is a reasonably close assumption in many cases since the time to move the cameras and capture an object is generally quite large compared to the slack times in crowded scenes, where such scheduling matters most. This assumption allows us to construct a Directed Acyclic Graph (DAG), where each (M)TVI is a node with an incoming edge from a common source node and outgoing edge to a common sink node, with the weights of the outgoing edges initialized to zero. An outgoing edge from one (M)TVI node to another exists iff the slack’s start time of the first node precedes that of the second (Theorem 1), which can however be removed if it makes the schedule infeasible. Consider the following theorem and corollary: + Theorem 2. A schedule - a sequence of n (M)TVIs each with slack δi = [t− δi , tδi ], where − i = 1...n represents the order of execution - is feasible if t+ δn − tδ1 ≥ ( i=1...n−1 pi ) − th ( i=1...n−1 |δi |), pi being the processing time of the i (M)TVI in the schedule. + Proof. For the schedule to be feasible the following must be true: t− δ1 + p1 ≤ tδ2 , + − + − − t− δ2 + p2 ≤ tδ3 , ... , tδn−1 + pn−1 ≤ tδn . Summing them up gives tδ1 + tδ2 + ... + − + + + + tδn−1 + i=1...n−1 pi ≤ tδ2 +tδ3 +...+tδn , which can then be simplified as tδn −t− δ1 ≥ − + − + ( i=1...n−1 pi ) − ( i=1...n−1 |δi |). The condition, tδ1 + p1 ≤ tδ2 , tδ2 + p2 ≤ tδ3 , ... , + t− δn−1 + pn−1 ≤ tδn , is however only a sufficient condition for a feasible schedule. + − + Corollary 1. Define a new operator , such that if δ1 (= [t− δ1 , tδ1 ]) δ2 (= [tδ2 , tδ2 ]), − + then tδ1 + p1 ≤ tδ2 . Consider a schedule of (M)TVIs with slacks δi...n . The condition: δ1 δ2 , δ2 δ3 , ..., δn−1 δn , is necessary for the schedule to be feasible. Conversely, if a schedule is feasible, then δ1 δ2 , δ2 δ3 , ..., δn−1 δn . Proof is omitted since it follows easily from Theorem 2.
Due to Corollary 1, an edge between two (M)TVI nodes can be removed if it violates the relationship, since it can never be part of a feasible schedule. Using such a DAG, a Dynamic Programming (DP) algorithm can be used to solve the single-camera scheduling problem. Consider the following set of (M)TVIs that have been constructed for a given camera, represented by the tasks (T1...6) they satisfy and sorted in order of their slacks’ start times:{node1 = {T1 , T2 }, node2 = {T2 , T3 }, node3 = {T3 , T4 }, node4 = {T5 , T6 }}, where the set of nodes in the DAG in Figure 1 is given as nodei=1...4 . DP is run by first initializing paths of length 1 starting from each of the (M)TVI nodes to the sink, all with “merit” 0. At each subsequent path length, the next node nodenext chosen for a given node nodecurr in the current iteration is: max |S (6) nodenext =n∈Sarg T asks(nodecurr )|, n curr2next
Task Scheduling in Large Camera Networks
node1
Source
401
00
0 Sink
node2
0 node3 0 node4
Fig. 1. Single-camera scheduling: DAG formed from the set {node1 = {T1 , T2 }, node2 = {T2 , T3 }, node3 = {T3 , T4 }, node4 = {T5 , T6 }}. The weights between (M)TVI nodes are determined on the fly during DP. Assume that, in this example, the relationship is satisfied for the edges between the (M)TVI nodes.
where Scurr2next is the set of nodes that have valid paths starting from them in the previous iteration and for which nodecurr has an outgoing edge to. Sn is defined as the set of tasks covered by the path (in the previous iteration) starting from n, and T asks() gives the set of tasks covered by the (M)TVI associated with nodecurr . So, for example, from node1 , paths of length 2 exist by moving on to either one of node2...4 , with the move to node2 , node3 and node4 covering {T1 , T2 , T3 } (merits=3), {T1 , T2 , T3 , T4 } (merits=4) and {T1 , T2 , T5 , T6 } (merits=4) respectively. We choose the path of length 2 from node1 to node3 . Iterations are terminated when there is only one path left that starts at the source node or a path starting at the source node that covers all the tasks. In our example, the optimal path becomes node1 → node3 → node4 , terminated at paths of length 4 from the sink when all the tasks are covered.
3 Multi-camera Scheduling While single-camera scheduling using DP is optimal and has polynomial running time, the multi-camera scheduling problem is unfortunately NP-hard. Consequently, computationally feasible solutions can only be obtained with approximation algorithms. We consider both a simple greedy algorithm and a branch and bound-like algorithm. 3.1 Greedy Algorithm The greedy algorithm iteratively picks the (M)TVI that covers the maximum number of uncovered tasks, subject to schedule feasibility as given by Theorem 2. Under such a greedy scheme, the following is true: Theorem 3. Given k cameras, the approximation factor for multi-camera scheduling using the greedy algorithm is 2 + kλμ, where the definitions of λ and μ are given in the proof.
402
S.-N. Lim, L. Davis, and A. Mittal
Proof. Let G = i=1...k Gi , where Gi is the set of (M)TVIs scheduled on camera i by the greedy algorithm, and let OP T = i=1...k OP Ti , where OP Ti is the set of (M)TVIs assigned to camera i in the optimal schedule. We further define (1) H1 = i=1...k H1,i , where H1,i is the set of (M)TVIs for camera i, that have been chosen by the optimal schedule but not the greedy algorithm and each of these (M)TVIs contains tasks that are not covered by the greedy algorithm in any of the cameras, (2) H2 = i=1...k H=2,i , where H2,i is the set of (M)TVIs for camera i, that have been chosen by the optimal schedule but not the greedy algorithm and each of these (M)TVIs contains tasks that are also covered by the greedy algorithm, and finally (3) OG = OP T G. Clearly, OP T = H1 H2 OG. Then, for hj=1...ni ∈ H1,i where ni is the number of (M)TVIs in H1,i , ∃gj=1...ni ∈ Gi such that hj and gj cannot be scheduled together based on the requirement given in Theorem 2, else hj should have been included by G. If T asks(hj ) T asks(gj ) = ∅, then hj contains only tasks that are not covered by G. In this case, |hj | ≤ |gj |, else G would have chosen hj instead of gi . Note that the cardinality is defined as the number of unique tasks covered. In the same manner, even if T asks(hj ) T asks(g j ) = ∅,hj could have replaced gj unless |hj | ≤ |gj |. Consequently, |H1,i | = |h1 h2 ... hni | ≤ |h1 | + |h2 | + ... + |hni | ≤ |g1 | + |g2 | + |g | ... + |gni |. Let βj = |Gji | and λi = max(βj ∗ ni ). This gives |H1,i | ≤ β1 |Gi | + ... + βni |Gi | ≤ λi |Gi |. Similarly, we know |H1 | ≤ λ1 |G1 | + ... + λk |Gk | ≤ λ(|G1 | + i| ... + |Gk |), where λ = max(λi ). Introducing a new term, γi = |G |G| and letting μ = max(γi ), we get |H1 | ≤ kλμ|G|. Since |H2 | ≤ |G| and |OG| ≤ |G|, |OP T | ≤ (2 + kλμ)|G|. 3.2 Branch and Bound Algorithm The branch and bound approach runs DP in a similar manner as single-camera scheduling, but on a DAG that consists of multiple source-sink pairs (one pair per camera), with the node of one camera’s sink node linked to another camera’s source node. An example is shown in Figure 2. Then, for a source node s, we define its “upper bounding set” Ss as: Ss = Sc , (7) c∈Slink
where Slink is the set of cameras for which paths starting from the corresponding sink nodes to s exist in the DAG, and Sc is the set of all tasks that are covered by some (M)TVIs belonging to camera c. Intuitively, such an approach aims to overcome the “shortsightedness” of the greedy algorithm by “looking forward” in addition to backtracking and using the tasks that can be covered by other cameras to influence the (M)TVI nodes chosen for a particular camera. Admittedly, better performance is possibly achievable if “better” upper bounding sets are used, as opposed to blindly using all the tasks that other cameras can cover without taking scheduling feasibility into consideration. The algorithm can be illustrated with the example shown in Figure 2, which shows two cameras, c1 and c2 , and the following sets of (M)TVIs that have been constructed for them, again ordered by the slacks’ start times and shown here by the tasks (T1...4 )
Task Scheduling in Large Camera Networks
403
node1
Source1 Source2
node2 node3
Sink1 0 Sink2 0
Sink
Fig. 2. Multi-camera scheduling: DAG formed from the set {node1 = {T1 , T2 , T3 }, node2 = {T3 , T4 }} for the first camera, and the set {node3 = {T1 , T2 , T3 }} for the second camera
they satisfy. For c1 , the set is {node1 = {T1 , T2 , T3 }, node2 = {T3 , T4 }} and for c2 , {node3 = {T1 , T2 , T3 }}. The DAG that is constructed has two source-sink pairs, one for each camera - (Source1 , Sink1 ) belongs to c1 and (Source2 , Sink2 ) to c2 . The camera sinks are connected to a final sink node as shown, with the weights of the edges initialized to zero. Weights between nodes in the constructed DAG are similarly determined on the fly like in the single-camera scheduling. Directed edges from Sink2 to Source1 connects c1 to c2 . The DP algorithm is run in almost the same manner as single-camera scheduling, except that at paths of length 3 from the final sink node, the link from Source1 to node2 , is chosen because the upper bounding set indicates that choosing the link potentially covers a larger number of tasks (i.e., the upper bounding set of Source1 , {T1 , T2 , T3 } combines with the tasks covered by node2 to form {T1 , T2 , T3 , T4 }). The branch and bound algorithm can be viewed as applying the single-camera DP algorithm, camera by camera in the order given in the corresponding DAG, with the schedule of one camera depending on its upper bounding set. This allows us to derive a potentially better approximation factor than the greedy algorithm as follow: Theorem 4. For k cameras, the approximation factor of the branch and bound algo (1+kμ(1+u))k ∗ ∗ rithm is (1+kμ(1+u)) k −(kμ(1+u))k . μ and u are defined as follow. Let G = i=1...k Gi , ∗ where Gi is the set of (M)TVIs assigned to camera i by the branch and bound algorithm. |G∗ | Then, μ = max( |Gi∗ | ) and u = max(ui ), where ui is the ratio of the cardinality of the upper bounding set of camera i to |G∗i |. Proof. Let α be the approximation factor of the branch and bound algorithm. Then, assuming that schedules for G∗1 , ..., G∗i−1 have been determined, |G∗i | ≥ α1 (|OP T | − i−1 ∗ i−1 ∗ j=1 |Gj |). Adding j=1 |Gj | to both sides gives: i j=1
α−1 ∗ OP T + |G |). α α j=1 j i−1
|G∗j | ≥
A proof by induction shows, after some manipulation: k αk |G∗ | ≥ |OP T |. αk − (α − 1)k j=1 j
404
S.-N. Lim, L. Davis, and A. Mittal Greedy − 10 cameras
Greedy − 50 cameras
30 20 10 0 1
Greedy − 100 cameras
200
400
150
300
Approx. factor
Approx. factor
Approx. factor
40
100 50 0 1
μ
3
2 1
0.5
3 0.5
2
μ
λ
0 0
100 0 1
3 0.5
200
1 0 0
2
μ
λ
1 0 0
λ
(a) Branch and Bound − 100 cameras
Branch and Bound − 50 cameras 5
4
4
4
3 2
Approx. factor
5
5
Approx. factor
Approx. sfactor
Branch and Bound − 10 cameras
3 2 1 3
1 3 1
2
u
0.5
1
μ
0 0
2 1 3
1
2
u
3
0.5
1 0 0
1
2
u
μ
0.5
1 0 0
μ
(b) Fig. 3. (a) The approximation factor for the greedy algorithm using 10, 50 and 100 cameras respectively. λ and μ here are as defined in Theorem 3. (b) The same plots for the branch and bound algorithm. Here, the approximation factor depends only on the distribution parameters and not on the number of cameras. u and μ are as defined in Theorem 4. Comparing DP and Greedy Branch and Bound
% of tasks covered
100 90 80 70
Greedy
60 50 200
100 150
No. of tasks
50 100
0
No. of cameras
Fig. 4. The DP algorithm consistently covers more tasks than the greedy algorithm
Let H = i=1...k Hi , Hi being the set of (M)TVIs chosen by the optimal schedule on camera i but not the branch and bound algorithm. The condition |Hi | ≤ |G∗i | + ui |G∗i | is true; otherwise, Hi would have been added to G∗ instead. Consequently, |H| ≤ (|G∗1 | + ... + |G∗k |) +(u1 |G∗1 | + ... + uk |G∗k |) ≤ kμ|G∗ | + kuμ|G∗ | ≤ kμ(1 + u)|G∗ |. Since OP T = OG H (Theorem 3), we get |OP T | ≤ 1 + kμ(1 + u)|G∗ |. Thus, α = 1 + kμ(1 + u).
Task Scheduling in Large Camera Networks
405
By expressing the approximation factors of the greedy and branch and bound algorithm as a function of the number of cameras, we see that the branch and bound algorithm theoretically outperforms the greedy algorithm substantially in terms of task coverage. This is illustrated in Figure 3, whereby the approximation factors of the greedy and branch and bound algorithm are plotted as the “distribution” parameters vary when different number of cameras are used. These distribution parameters refer to λ and μ in Theorem 3, and μ and u in Theorem 4. They represent how well the tasks are distributed among the cameras and (M)TVIs. The plots show that the greedy algorithm is highly sensitive to the number of cameras, with the approximation factor becoming prohibitively high when the tasks are unevenly distributed. On the other hand, the performance of the branch and bound algorithm depends only on the distribution parameters and is not affected by the number of cameras. Both the single-camera and branch and bound DP-based multi-camera algorithm have a computational complexity of O(N 3 ), N being the average number of (M)TVIs constructed for a given camera and used in the resulting DAG. On the other hand, the greedy algorithm takes only O(N 2 ) time, which could outweigh the benefits of better scheduling for very large camera networks.
4 Implementation and Experiments Although we have theoretically found the approximation factors for the scheduling algorithms, it would be interesting for practical purposes to investigate the performance of the greedy algorithm relative to the DP algorithm under “normal” circumstances where we would expect “reasonable” task distribution. For this purpose, we conducted simulations using a scene of size 200m × 200m, and generated moving objects in the scene by randomly assigning to them different starting positions in the scene, sizes and velocities. Cameras are also simulated with calibration data from real cameras. The objects are assumed to be moving in straight lines at constant speeds, and the (M)TVIs for each camera are then constructed and utilized by the scheduler. We conducted simulations for 20, 40, 60, and 80 cameras and 100, 120, 140, 160, 180, and 200 objects, and plot the percentage of the total number of tasks that were captured by both the greedy and DP algorithm. For each object, the task is to capture video segments in which the full body of the object is visible. Since there is only one task for each object, the total number of tasks equals the number of objects. The results are shown in Figure 4. The DP algorithm schedules more tasks than the greedy algorithm by a minimum of 13.55 percent and a maximum of 33.78 percent. Finally, we test our algorithms in a small-scale real time image analysis system. Due to limited resources, building a system with large number of cameras was not possible. We developed a prototype multi-camera system consisting of four PTZ cameras synchronized by a Matrox four-channel card. For running the experiments, one camera is kept static, so that it can be used for background subtraction and tracking in the sensing stage[9,1]. From the detection and tracking, the system recovers an approximate 3D size estimate of each detected object from the ground plane and camera calibration. This is followed by the planning stage, during which the observed tracks allow the system to predict the future locations of
406
S.-N. Lim, L. Davis, and A. Mittal
(a)
(b)
(c)
Fig. 5. (a) The robots are tracked (left and middle image), and the predicted tracks are used to construct the TVIs and MTVIs, which are then used by the scheduler to assign cameras to the (M)TVIs (annotated in the right image). Next, (b) camera 0 captures robot 3, and (c) camera 1 captures robots 0, 1 and 2 simultaneously.
the objects, and to use them for constructing (M)TVIs, which are then scheduled for capture. The predicted position of each detected object on the ground plane is mapped to the PTZ cameras, after which the 3D size estimate of the object is used to construct a rough 3D model of the object for the corresponding PTZ camera. Such a 3D model is utilized to determine valid ranges of PTZ settings during the construction of TVIs. The experiments confirm that the greedy algorithm performs faster than the DP algorithm. This makes the greedy algorithm more suitable for our prototype system. A real time experiment using the greedy scheduler is illustrated in Figures 5. Four remotecontrollable 12x14 inches robots move through the scene. Two PTZ cameras were needed to capture the four robots using a (one task) TVI and a three-task MTVI.
5 Conclusion This paper addressed scheduling algorithms for smart video capture in large camera networks. We developed approximation algorithms for scheduling using a greedy and a DP based approach. While both theoretically and experimentally, the DP algorithm gives very good results, it is computationally more expensive than the greedy algorithm. A suitable algorithm can thus be chosen depending on the application scenario and computational resources available.
Task Scheduling in Large Camera Networks
407
References 1. Isard, M., Blake, A.: Condensation - conditional density propagation for visual tracking. International journal of computer vision 29, 5–28 (1998) 2. Mittal, A., Davis, L.: M2tracker: A multi-view approach to segmenting and tracking people in a cluttered scene. In: European Conference on Computer Vision, Copenhagen, Denmark (2002) 3. Zhao, T., Nevatia, R.: Bayesian human segmentation in crowded situation. In: IEEE Conference on Computer Vision and Pattern Recognition, IEEE Computer Society Press, Los Alamitos (2003) 4. Kaucic, R., Perera, A.A., Brooksby, G., Kaufhold, J., Hoogs, A.: A unified framework for tracking through occlusions and across sensor gaps. In: IEEE Conference on Computer Vision and Pattern Recognition, San Diego, CA, IEEE Computer Society Press, Los Alamitos (2005) 5. Rahimi, A., Dunagan, B., Darrell, T.: Simultaneous calibration and tracking with a network of non-overlapping sensors. In: IEEE Conference on Computer Vision and Pattern Recognition, Washington DC, IEEE Computer Society Press, Los Alamitos (2004) 6. Lim, S.N., Davis, L.S., Mittal, A.: Constructing task visibility intervals for video surveillance. ACM Multimedia Systems (2006) 7. Abrams, S., Allen, P.K., Tarabanis, K.: Computing camera viewpoints in an active robot work cell. International Journal of Robotics Research 18 (1999) 8. Tarabanis, K., Tsai, R., Allen, P.: The mvp sensor planning system for robotic vision tasks. IEEE Transactions on Robotics and Automation 11, 72–85 (1995) 9. Grimson, W.E.L., Stauffer, C.: Adaptive background mixture models for real-time tracking. In: IEEE Conference on Computer Vision and Pattern Recognition, IEEE Computer Society Press, Los Alamitos (1999)
Constrained Optimization for Human Pose Estimation from Depth Sequences Youding Zhu1 and Kikuo Fujimura2 1
Computer Science and Engineering, The Ohio State University 2 Honda Research Institute USA [email protected], [email protected]
Abstract. A new 2-step method is presented for human upper-body pose estimation from depth sequences, in which coarse human part labeling takes place first, followed by more precise joint position estimation as the second phase. In the first step, a number of constraints are extracted from notable image features such as the head and torso. The problem of pose estimation is cast as that of label assignment with these constraints. Major parts of the human upper body are labeled by this process. The second step estimates joint positions optimally based on kinematic constraints using dense correspondences between depth profile and human model parts. The proposed framework is shown to overcome some issues of existing approaches for human pose tracking using similar types of data streams. Performance comparison with motion capture data is presented to demonstrate the accuracy of our approach.
1
Introduction
Markerless human motion capture is a research field concerned with obtaining large scale human motion data, such as head, torso, limbs, from image observation of human subjects. For the past decades, markerless human motion capture has been an active research field motivated by various applications such as action recognition, surveillance, and man-machine interaction. Despite substantial advances in related aspects including tracking, pose estimation, and recognition, challenging problems still remain due to the high degrees of freedom coming from the dynamic range of poses during human activities, the diversity of visual appearance caused by clothing, visual ambiguities from self-occlusion of non-rigid 3D object, and background clutters. There have been many attempts at solving this problem using various modalities including a single image, a sequence of images, and multiple streams (using multiple cameras) of images. In this paper, we propose a method for upper-body pose estimation from a stream of depth images. The region occupied by a human subject is easier to capture in depth image and it usually contains a stronger cue to distinguish a human subject from other objects. By taking advantage of this characteristic, an optimization approach is presented to generate the most plausible pose from various cues provided by depth image analysis. Y. Yagi et al. (Eds.): ACCV 2007, Part I, LNCS 4843, pp. 408–418, 2007. c Springer-Verlag Berlin Heidelberg 2007
Constrained Optimization for Human Pose Estimation
409
More concretely, coarse body part labeling is cast as a linear programming problem, where various small segments of the human body are labeled with constraints arisen from body parts detection and tracking. Furthermore, 3D human pose is optimally estimated from dense correspondences. Our algorithm is structured to work for a stream of point clouds from depth sensors. It also configured to function when a color stream is provided in addition to a depth stream. Our implementation of the algorithm runs at 5∼9 frames per second in online mode on a 3.00GHz HP desktop. The rest of the paper is organized as follows. After a brief review of related work in Section 2, our algorithm is described in Section 3. Experimental results are presented in Section 4 and Section 5 concludes the paper.
2
Related Work
A large number of pose estimation methods have been proposed in the literature. A thorough discussion of various pose estimation approaches is beyond the scope of this paper and the reader is referred to a recent survey [1] for a comprehensive comparison between various approaches. Lately, there have been approaches at making use of depth sequences due to its advantages over a single color image. For example, depth measurement provides necessary information to resolve depth ambiguity which is an issue with approaches using a single color image [2]. Grest et al. [4] adapt an Iterative Closest Point (ICP) approach to the articulated human model, where pose parameters are updated using inverse kinematics based on dense correspondences between sampled depth observations and model vertices that are found based on nearest neighbor association. Knoop et al. [3] also use ICP to update pose parameters by incorporating multiple input data from different sensors such as stereo depth, hands/face tracking from color camera, etc. Ziegler et al. [11] use an unscented kalman filter based on a set of correspondences between model vertices and observed stereo point cloud. ICP is often used as a method of choice when a 3D model is to be fitted to 3-dimensional data. A common issue with ICP approaches for human pose tracking is that the model may drift away from the data or get stuck in local minima. An initial configuration is critical for ICP to converge correctly. Our framework also uses the idea of closest point correspondence as a part of the solution, but it is less susceptible to the problem of local minima due to the coarse body part identification. We also use a grid acceleration data structure so as to achieve pose estimation at a high frame rate without loss of accuracy.
3
Algorithm
Our algorithm takes a depth image sequence representing human motion and outputs pose vectors of the upper body. Depth data is usually obtained by using stereo cameras, structured light sensors, or time-of-flight sensors. If other image modality, e.g., a color image sequence corresponding to the depth sequence, is
410
Y. Zhu and K. Fujimura
Fig. 1. Flow of the Algorithm. The left half of the figure illustrates Step 1, in which coarse body part labeling is determined. The right half illustrates the process of determining joint positions by fitting.
also available, our framework allows such data to be integrated to strengthen the result. The algorithm consists of two major modules, namely, (i) coarse body part labeling and (ii) model fitting (Fig. 1). In the first module, the region within the given image corresponding to the human body is partitioned into small homogeneous segments. Such segments are formed such that within each segment, depth of each pixel is similar and areas of these segments are of a small and similar size. Each segment is then assigned a body part label (e.g., head, left arm) by using a label assignment framework. At this point, coarse body part identification within each image is completed and passed to the second module. For the second module, a polygonal human upper body model attaching to the underlying kinematic skeleton structure is fitted to the depth observation using ICP for each body part. 3.1
Body Constraints
A few body constraints are extracted from depth images. 1. Head and torso constraints: The head and torso are tracked by specialized modules. The head is tracked based on circle fitting with predicted head contour points from depth, while the torso is tracked based on box fitting, where a box with 5 degrees of freedom (x, y, height, width, and orientation) is positioned so as to minimize the number of background pixels within the box. 2. Depth constraint. For certain frames, an arm shape is clearly separable in depth when it is in front of the torso. These cues are further used by body part labeling as described in the next.
Constrained Optimization for Human Pose Estimation
3.2
411
Coarse Part Labeling
For the next step, we form an adjacent graph G where a node represents a small cluster of pixels within the human body (see Fig. 1, top left) and an edge represents adjacency relationship between two pixel clusters. A recursive subdivision strategy is taken to partition the image into pixel clusters based on homogeneity of depth values and spatial positions. Starting from the root node representing all pixels, the k-mean clustering (with k=2) is used to subdivide clusters further until each cluster has the property that it is sufficiently small in size and that it has small depth variance. All the leaf nodes form segment set S to be labeled. Segments si (i = 1, 2, · · · , N ) are to be classified into major body parts p1 , p2 , · · · , pM . This is a labeling problem and formulated in the following optimization problem. A segment si in S is to be assigned a label by a function f . Each si has an estimate of its likelihood of having labeling f (si ). This comes from a heuristic in pose estimation. For example, if s is near the bottom of the image, its likelihood of being a part of the head is low. For this purpose, a non-negative cost function c(s, f (s)) is introduced to represent the likelihood. Further, we consider two neighboring segments si and sj to be related, in the sense that we would like si and sj to have the same label. Each edge e in graph G has a nonnegative weight we indicating the strength of the relation. Moreover, certain pairs of labels are more similar than others. Thus, we impose a distance d() on the label set. Larger distance values indicate less similarity. The total cost of a labeling f is given by: c(s, f (s)) + we d(f (si ), f (sj )) Q(f ) = s∈S
In our problem, the following table (Fig. 2) is to be completed, where binary variable Aij indicates if segment si belongs to body part pj . For c(i, j), the Euclidean distance from segment to the model body part (from the previous frame) is used. More specifically, the Euclidean distance from a segment si to a model body part pj is estimated as the sum of squared distances from a number of sampled pixels to their nearest model vertices in the model body part. table Aij segment:s1 segment:s2 ... s egment:sN
Head (p1 ) A11 A21
Torso LeftArm RightArm ... (p2 ) (p3 ) (p4 ) (pM ) A12 A13 A14 (A1M ) A22 A23 A24 (A2M )
AN1 AN2
AN3
AN4
(ANM )
Label Assignment Problem Aij = 1 ifsi belongs to pj A = 0 otherwise ijM j=1 Aij = 1 Nearby pixels have similar label
Fig. 2. Labeling problem
M Since each segment belongs to only one body part, j=1 Aij = 1 hold. In addition to this constraint, a number of related constraints are considered.
412
1. 2. 3. 4.
Y. Zhu and K. Fujimura
Neighboring segments should have a similar label. Head and torso constraint. Depth slicing constraint. Color constraint.
It turns out that this is an instance of the Uniform Labeling Problem, which can be expressed as the following integer programming by introducing auxiliary variables Ze for an edge e to express the distance between the labels and we use Zej to express the absolute value |Apj − Aqj |. Following Kleinberg and Tardos [6], we can rewrite our optimization problem as follows: min{
N M
c(i, j)Aij +
i=1 j=1
we Ze }
(1)
eE
subject to M
Aij = 1, i = 1, 2, 3, · · · , N
(2)
j=1
1 Zej , e ∈ E 2 j=1 M
Ze =
(3)
Zej ≥ Apj − Aqj , e = (p, q); j = 1, 2, · · · , M
(4)
Zej ≥ Aqj − Apj , e = (p, q); j = 1, 2, · · · , M
(5)
Aij ∈ {0, 1}, i = 1, 2, · · · , N ; j = 1, 2, · · · , M
(6)
Here, terms involving Ze and Zej come from constraint (1). The weight we is given by we = e−αde , where de is depth difference between two adjacent segments and α is selected based on experiments. In our study, we let M = 4 (head, torso, left arm, right arm). For constraint (2), if the segment si is outside the tracked head circle, an additional constraint Ai,1 = 0 is added. If segment si is outside of the tracked torso box, constraint Ai,2 = 0 is added. To apply constraint (3) for the detected arm segments, constraint:Ai,3 + Ai,4 = 1 is added (as it is not clear whether it is the right or left arm). Finally, if there are tracked hand positions based on skin color information, we can add constraint: Ai,3 + Ai,4 = 1. In general, solving an integer program optimally is NP-hard. However, we can relax the above problem to linear programming with Aij ≥ 0, and this can be solved efficiently by using a publicly available library, e.g., [7]. Kleinberg and Tardos [6] describe a method for rounding fractional solutions so that the expected objective function Q(f ) is within a factor of 2 from the optimal solution. In our experiments we find that this relaxed linear programming always returns an integer solution. (See an observation due to Anguelov et al. [10].) Figure 1 shows one example of this body part labeling result.
Constrained Optimization for Human Pose Estimation
3.3
413
Model Fitting
The human body model is represented as a hierarchy of joint link models with a skin mesh attached to it as in Lewis et al. [8]. For the upper body model used in this paper, a skin mesh and hierarchical skeleton structure are illustrated as in Fig. 1. Given a set of 3D data point P = {p1 , p2 , · · · , pm } as targets and their corresponding model vertices V = {v1 , v2 , · · · , vm }, the model pose vector q is estimated as qˆ = argminq ||P − V (q)||2
(7)
where q = (θ0 , · · · , θn )T is the pose parameter vector and vi ’s are visible vertices of the polygonal model. To solve this minimization problem efficiently and robustly, we use a variant of inverse kinematics (known as damped least square), which is inspired by the well-known ICP algorithm [9]. The formulation (see Fig. 7) minimizes ||JΔq − ΔE||2 + λ||Δq||2 where J is the Jacobian. The inverse kinematics with damped least square [5] has the benefit of avoiding singularities, thus making the process numerical stable. We use λ=0.1 based on our experiments. For articulated body pose estimation, the algorithm depends on the accuracy of finding correspondence pairs of data point and model vertices. Most recent works apply a nearest neighbor search between two point clouds: one contains all the observed 3D points from depth or other sensor; the other contains all the model vertices. Since iteration may be attracted to local minima as the algorithm is inspired by the well-known ICP algorithm [9], we apply the aforementioned body part labeling to limit the nearest neighbor search between a subset of observed 3D points and a subset of visible model vertices for each body part. Thus, this not only speeds up the nearest neighbor search, but more importantly, it achieves robust pose estimation even for long sequences containing large motions between two consecutive frames. In our implementation, the OpenGL depth buffer is utilized to decide model vertex visibility. For faster computation, we use a grid based spatial index data structure to speed up nearest neighbor search between point clouds. We partition the working volume into a set of 3D grid. Because only scene profiles are used, we partition the xy plane of the working volume. Depth points and visible model vertices are indexed into corresponding grids. To perform nearest neighbor search for a model vertex, we first find the grid it is located. The nearest neighbor depth point in this grid is found afterward. Then, we recursively propagate the nearest neighbor depth point search to the neighboring grids until the minimal distance from grid corners to the model vertex is greater than the current minimal distance from the model vertex to the depth points. As illustrated in Fig. 8, the method contributes speed-up by a factor of 6. When capturing a pose sequence, the subject is initially requested to take an open-arm posture (so-called “T-pose”), in which his arms and torso do not
414
Y. Zhu and K. Fujimura
overlap. At this initialization stage, body dimensions are measured and further used to scale the kinematic skeleton and polygonal human body model.
4
Experimental Results
The proposed pose estimation algorithm has been tested on many sequences collected from a few human subjects. Depth sequences and color sequences have been obtained using a calibrated hybrid pair of stereo (Swiss ranger SR3000 for depth and Sony DFWV500 for color) in a synchronous fashion. Furthermore, a motion capture system by PhaseSpace Inc. with 8 camera units has also been run synchronously with the hybrid camera system, taking coordinates for eight major joints of the subject for ground-truth reference. The subject wears markers for a motion capture purpose only and these markers are not used for the main algorithm. Test motion sequences include a complete semaphore flag signaling motion (A to Z), simple exercise movements, and TaiChi movements. The total number of frames collected is 4800 and each test sequence is about 400 frames long. All sequences have been tracked successfully at the frame rate of 5∼9Hz on a 3.00GHz HP desktop. Figure 3 contains tracked frames taken from full-length sequences, where the subject performs (a) a TaiChi motion and (b) an exercise motion, respectively. Pose estimation precision has also been compared against joint position data captured by a marker-based motion capture system. Figure 4 contains error in various joint positions for the TaiChi motion sequence. As seen here, the overall tracking error is approximately 5cm (where the subject stands 1.5m to 2m from the camera). Similar tracking results have been obtained for the other sequences, such as out-of-plane-rotation (Fig. 6) which is usually difficult to capture with single-color-camera-based pose estimation methods. To compare tracking stability, we tested an ICP-based approach using exactly the same sequences. The ICP and our method have similar performance (in terms of error amount from the ground-truth), when tracking runs successfully. However, a significant difference is that for some frames (where our method processes successfully), the ICP based tracking fails and never recovers (as shown in Fig. 5). Our method functions for all cases where the ICP method works. The ICP-based method is slower, because it has to do more iteration for convergence. This illustrates the advantage of the part labeling step in our framework. At this point, let us contrast our approach with a few other approaches using depth sequences. Ziegler et al. [11] use depth sequences obtained by four stereo cameras. Point correspondences are based on spatial proximity which may result in wrong correspondences when body parts are close to each other. Our method is less susceptible to such a problem due to the use of inverse kinematics. Demirdjian et al. [13] use efficient example-based matching to improve the tracking of a set of likelihood modes. Large errors can still occur when the test example is not close to training examples. Our coarse labeling step has a similar function and finds the likelihood mode with constraints from bottom-up observations. Grest et al. [4] introduce an ICP method for articulated body pose estimation,
Constrained Optimization for Human Pose Estimation
415
(a)
(b)
Fig. 3. Snapshots of the algorithm output (a) TaiChi sequence, (b) Simple exercise sequence
Model Joints
error (in millimeter) ΔX(μ, σ) ΔY (μ, σ) ΔZ(μ, σ) Right Hand (-15, 49) (-39, 58) (23,44) Right Elbow (-23, 34) (-70, 42) (-48,59) Right Shoulder (21, 57) (-43,19) (1,25) Waist (-24, 26) (-12, 15) (-19,14) Left Hand (16, 61) (-6, 86) (44,45) Left Elbow (30, 35) (-74, 39) (71,66) Left Shoulder (-23, 53) (-36, 30) (27,30) Head (-15, 26) (-18, 15) (-22,15) Overall (-4, 49) (-37, 50) (22,52) Fig. 4. Comparison table
416
Y. Zhu and K. Fujimura TaiChi
Color Image Our Method
ICP only
Frame:178 Fig. 5. Stability comparison between our method and standard ICP
Fig. 6. Examples of out-of-plane rotation up to 50 degree
while they do not address the robustness of ICP-based pose estimation. Knoop et al. [3] utilize skin color segmentation (hence face and hand feature trackers) to improve the ICP-based pose estimation. It is not clear how to handle temporarily invisible face or hands. Some other approaches use multiple sensors to obtain more surface data. Cheung et al. [12] use visual hull while Anguelov et al. [10] use 3D range scan data to reconstruct human skeletal structures. Accurate pose estimation might be obtained for these methods since body parts are visible in multiple views.
5
Concluding Remarks
We have presented a method for estimating human pose from depth sequences. Our method consists of two major components that cooperate to estimate and track human motion. The first module is body component identification which has been solved by reducing it to linear programming. If a certain application requires only rough body labeling in the image, the first module provides such a solution. If more accurate positions for major joints are required, then the second module is to be used. The second module is model fitting by using inverse kinematics based on dense correspondences between the image data and the human kinematic model. The result of the second component, in turn, is used to initiate the first component for the next frame. The algorithm tracks human upper-body movements over several minutes of pose sequences at a speed of a few Hz using a laptop PC (up to 10Hz when a desktop with 3GHz is used).
Constrained Optimization for Human Pose Estimation
417
model fitting(P :3D point,V :model vertex) 1. Form ⎤ ⎡ ⎤ Δe1 pxi − vix ⎢ Δe2 ⎥ ⎥ Δei = ⎣ pyi − viy ⎦ , ΔE = ⎢ ⎣ ... ⎦ z z pi − vi Δem ⎡
2. Solve JΔq = ΔE by damped least square where J is Jacobian of model vertices Δq = (J T J + λI)−1 J T ΔE q = q + Δq 3. Repeat until Δq is sufficiently small Fig. 7. Model fitting procedure
Fig. 8. Performance comparison with and without grid acceleration (on a 2.13GHz IBM Laptop)
We have also made a comparative study of marker-less pose tracking based on a commercial marker-based tracking system and shown that our joint positions are accurate to several cm. The algorithm has a number of extension possibilities such as severe occlusions coming from environmental objects. We leave these for our future work.
References 1. Moeslund, T.B., Hilton, A., Kruger, V.: A survey of advances in vision-based human motion capture and analysis. Computer Vision and Image Understanding 104(2,3), 90–126 (2005)
418
Y. Zhu and K. Fujimura
2. Sminchisescu, C., Triggs, B.: Kinematic jump processes for monocular 3D human tracking. In: Int. Conf. on Computer Vision and Pattern Recognition, pp. 18–20 (2003) 3. Knoop, S., Vacek, S., Rillman, R.: Sensor fusion for 3D human body tracking with an articulated 3D body model. In: Int. Conf. on Robotics and Automation, pp. 1686–1691 (2006) 4. Grest, D., et al.: Nonlinear body pose estimation from depth images. In: Kropatsch, W.G., Sablatnig, R., Hanbury, A. (eds.) DAGM 2005. LNCS, vol. 3663, Springer, Heidelberg (2005) 5. Buss, S., Kim, J.: Selectively damped least squares for inverse kinematics. Journal of Graphics Tools 10(3), 37–49 (2005) 6. Kleinberg, J., Tardos, E.: Approximation algorithms for classfication problems with pairwise relationships: Metric partitioning and Markov random fields. Journal of the ACM 49(5), 616–639 (2002) 7. LP solve reference guide, http://lpsolve.sourceforge.net/5/5 8. Lewis, J.P., Cordner, M., Fong, N.: Pose space deformations: A unified approach to shape interpolation and skeleton-driven deformation, Siggraph, pp. 165–172 (2000) 9. Besl, P., McKay, N.: A method for registration of 3-d shapes. IEEE Trans. on Pattern Analysis and Machine Intelligence 14(2), 239–256 (1992) 10. Anguelov, D., Koller, D., Pang, H., Srinivasan, P., Thrun, S.: Recovering articulated object models from 3D range data. In: Proc. of Uncertainty in Artificial Intelligence Conference, pp. 18–26 (2004) 11. Ziegler, J., Nickel, K., Stiefelhagen, R.: Tracking of the articulated upper body on multi-view stereo image sequences. In: Int. Conf. on Computer Vision and Pattern Recognition, pp. 774–781 (2006) 12. Cheung, K.M., Baker, S., Kanade, T.: Shape-from-silhouette of articulated objects and its use for human body kinematics estimation and motion capture. In: Int. Conf. on Computer Vision and Pattern Recognition, pp. 77–84 (2003) 13. Demirdjian, D., Taycher, L., Shakhnarovich, G., Grauman, K., Darrell, T.: Avoiding the Streetlight Effect: tracking by exploring likelihood modes. In: Int. Conf. on Computer Vision, pp. 357–364 (2005)
Generative Estimation of 3D Human Pose Using Shape Contexts Matching Xu Zhao and Yuncai Liu Institute of Image Processing & Pattern Recognition, Shanghai Jiao Tong University 200240, Shanghai, China
Abstract. We present a method for 3D pose estimation of human motion in generative framework. For the generalization of application scenario, the observation information we utilized comes from monocular silhouettes. We distill prior information of human motion by performing conventional PCA on single motion capture data sequence. In doing so, the aims for both reducing dimensionality and extracting the prior knowledge of human motion are achieved simultaneously. We adopt the shape contexts descriptor to construct the matching function, by which the validity and the robustness of the matching between image features and synthesized model features can be ensured. To explore the solution space eÆciently, we design the Annealed Genetic Algorithm (AGA) and Hierarchical Annealed Genetic Algorithm (HAGA) that searches the optimal solutions eectively by utilizing the characteristics of state space. Results of pose estimation on dierent motion sequences demonstrate that the novel generative method can achieves viewpoint invariant 3D pose estimation.
1 Introduction Capturing 3D human motion from visual cues has received increasing attention in recent years, due to the drive from a wide spectrum of potential applications such as behavior understanding, content-based image retrieval and visual surveillance. Although having been attacked by many researchers, this challenging problem is still long standing because of the diÆculties conduced mainly by complicated nature of 3D human motion and incomplete information of 2D images for 3D human motion analysis. In the context of graphical models, the state-of-art approaches of 3D human motion estimation can be classified as generative and discriminative [1]. Generative methods [2,3,4,5,6,7] follow the bottom-up Bayes’ rule and model the state posterior density using observation likelihood or cost function. Given an image observation and prior state distribution, the posterior likelihood is usually evaluated using Bayes’ rule. This approach has a sound framework of probabilistic support and can achieve significant success for recovering complex unknown motions by utilizing well-defined state constrains. However, it is generally computationally expensive because one had to perform complex search over the state space in order to locate the peaks of the observation likelihood. Moreover, prediction model and initialization are also the bottlenecks of the approach especially in tracking situation. Y. Yagi et al. (Eds.): ACCV 2007, Part I, LNCS 4843, pp. 419–429, 2007. c Springer-Verlag Berlin Heidelberg 2007
420
X. Zhao and Y. Liu
In this paper, we propose a novel generative approach in the framework of evolutionary computation, by which we try to widen the bottlenecks mentioned above with eective search strategy embedded in the extracted state subspace. Considering the generalization of application scenario, the observation information we utilized comes from an uncalibrated monocular camera. This makes the state estimation get into severe illconditioned problem. And, we have to confront the curse of dimensionality because there are more than forty degrees of freedom (DOFs) of full body joints in our 3D human model. Therefore, the process for searching optimal solutions should be performed in some compact state space by the search algorithms which suit for the characteristics of this space. In doing so, infeasible solutions, namely, the absurd poses can be avoided naturally. To this end, we consider to reduce the dimensionality of state space by principal component analysis (PCA) of motion capture data. Actually, the motion capture data embody the prior knowledge about human motion. By PCA, the aims at both reducing dimensionality and extracting the prior knowledge of human motion are achieved simultaneously. From the theoretical view, PCA is optimal in the sense of reconstruction because it allows the minimal information loss in the course of state transformation from the subspace to original state space. Dierent from the previous works [8,9], we perform the lengthways PCA, by which the subspace can be extracted from only single sequence of motion capture data. To explore the solution space eÆciently, we design the Annealed Genetic Algorithm (AGA) combining the ideas of simulated annealing and genetic algorithm [10]. As the promoted version of AGA, Hierarchical Annealed Genetic Algorithm (HAGA) searches the optimal solutions more eectively than AGA by utilizing the characteristics of state space. According to the theory of PCA, in our problem, the first principle component captures the most important part of human motion and the rest of principle components capture the detailed parts of this motion. In monocular uncalibrated camera situation, the fitness function (observation likelihood function) is very sensitive to the change of global motion. The HAGA performs hierarchical search automatically in the extracted state subspace by localizing priorly the state variables such as the global motions and the coordinate of the first principle component which dominate the topology of state space. We adopt the shape contexts descriptor [11] to construct the fitness function, by which the validity and the robust matching between image features and synthesized model features can be achieved. 1.1 Related Work There has been considerable previous work on capturing human motion from image information. The earlier work on this research topic had been reviewed comprehensively by the survey papers [12,13,14]. Generally speaking, to recover 3D human pose configuration, more information are required than image can provide especially in the monocular situation. Therefore, much work focus on using prior knowledge and experiential data in order to alleviate the ill-condition of this problem. Explicit body model embodies the most important prior knowledge about pose configuration and thus be widely used in human motion analysis. Another class of important prior knowledge comes from the experiential data such as motion capture data acquired by commercial
Generative Estimation of 3D Human Pose
421
motion capture system and other hand-labeled data. The combination of the both prior information can produces favorable techniques for solving this problem. Agarwal et al. [5] distill prior information (the motion model) of human motion from hand-labeled training sequences using PCA and clustering on the base of a simple 2D human body model. This method presents a good autoregressive based tracking scheme but has no description about pose initialization. In the framework of generative approach, the prior information is usually employed to constrain or reduce the search space. Urtasun et al. [15,9] construct a dierentiable objective function based on the PCA of motion capture data and then find the poses of all frames simultaneous by optimizing a function in low-dim space. Sidenbladh et al. [3,8] present similar methods in the framework of stochastic optimization. For a specific activity, such methods need many example sequences of motion capture to perform PCA and all of these sequences must keep same length and same phase by interpolating and aligning. Ning et al. [6] learn a motion model from semi-automatically acquired training examples which are aligned with correlation function, and then, some motion constrains are introduced to cut the search space. Unlike these methods, we extract the state subspace from only one example sequence of a specific activity using the lengthways PCA and thus have no use for interpolating or aligning. In addition, useful motion constraints are included naturally in the low-dim subspace. In recent years, particle filter [16] (also known as condensation algorithm) based optimization methods are used widely for recovering human pose in generative framework [2,3,4,5,6,7]. However, as a stochastic search algorithm, we think that particle filter is essentially similar with evolutionary algorithm (EA) if having no explicit temporal dynamic model. The EA can provide more flexible evolutionary mechanism such as crossover operator. This is the important motivation for us to solve this problem in the framework of EA. A noticeable example showing the relationship between particle filter and EA is the work of Deutscher et al. [17]. By introducing the crossover operator, the annealed particle filter proposed in theirs earlier work [2] get remarkable improvement. Comparing with previous generative methods, extracting the common characteristic of a special types of motion from prior information and represent them with some compact forms are of particular interests to us. At the same time, we ensure the motion individuality of the input sequences with eective evolutionary search strategy suiting for the characteristic of state subspace.
2 State Space Analysis The potential special interests motivate us to analyze the characteristics and structure of the state space. Such interests involve mainly modeling the human activities eectively in the extracted state subspace and eliminating the curse of dimension. 2.1 Pose Representation We use a explicit model that represent the articulated structure of the human body. Our fundamental 3D skeleton model (see Figure 1.a) is composed of 34 articulated rigid sticks. The pose is described by a 44 dimensional vector x xg x j , where 3D vector
422
X. Zhao and Y. Liu
(a )
(b )
(c )
Fig. 1. (a) The 3D human skeleton model. (b) The 3D human convolution surface model. (c) The 2D convolution curves.
xg represents the global rotations of human motion and 41D vector x j represents the joint angles. Figure 1.b shows the 3D convolution surface [18] human model which actually is an isosurface in a scalar field defined by convolving the 3D body skeleton with a kernel function. Similarly, the 2D convolution curves of human body as shown in Figure 1.c are the isocurves generated by convolving the 2D projection skeleton. As the synthetical model features, the curves are used to match with the edges of image silhouettes for constructing the likelihood function. 2.2 Subspace Extraction All of the 3D poses distribute in the state space X. The pose set which belongs to a special activity, such as walking, running, handshaking, etc., generally crowd in a subspace of X. We extract the subspace X s from motion capture data obtained from the CMU database (http:mocap.cs.cmu.edu). Assuming xt xt X is a given data sequence of motion capture corresponding to one motion type, where t is the time tag, the subspace X s is extracted by PCA as follows: 1. Centering the state vectors and assembling them into a matrix (by rows): X [(x1 c); (x2 c); ; (xT c)], where c is the mean vector. 2. Performing a singular value decomposition of the matrix to project out the dominant directions: X U D VT . 3. Projecting the state vectors into the dominant subspace: each state vector is represented as a reduced vector x s (x c) Um , where Um is the matrix consisting of first m columns of U, by which the m-D subspace X s is spanned. Therefore, the original state vector x can be reconstructed by: x c x s UTm
(1)
The dimensionality m of subspace X s is determined according to the cumulative sum of principal component variance percentage. With our experiences, the value of is set to be not smaller than 095; accordingly, the value of m is not greater than 6 generally.
Generative Estimation of 3D Human Pose
423
3 Fitness Function In generative framework, pose capturing can be formulated as Bayesian posterior distribution inference: (2) p(x s y) p(x s )p(y x s) The function p(y x s) represents the likelihood observing in image y, conditioned on a pose candidate x s . It is used to evaluate every pose candidate generated from p(x s ). In the context of evolutionary algorithm, the likelihood function is just the fitness function. We propose a fitness function on the basis of shape contexts matching [11]. We choose the image silhouette of subject as the observed image feature, which is extracted using statistical background subtraction. The shape context descriptor is used to describe the shape of image silhouette and convolution curves generated by the pose candidate (see Figure 1). Figure 2 illustrate the shape contexts [11] (histograms of local edge pixels into log-polar bins) of human shape. Our shape contexts contain 12 angular five radial bins, giving rise to 60-dimensional histograms as shown in Figure 2.b. In the matching process, the regularly spaced points on the edge of the silhouette are sampled as the query shape. The point set sampled from the convolution curves is viewed as the candidate shape. Before matching, the image shape and the candidate shape are normalized to same scale. We represent the query shape and the candidate shape as S query (y ) and S m (x s) respectively. To this end, the matching cost function is formulated as: F(S query (y) S m (x s ))
r
2 (S Cquery (y) S Cm (x s ) ) j
(3)
j 1
where S C is the shape context, r is the number of sample point on the edge of image j silhouette, and S Cm (x s ) argminu2 (S Cquery (y) S Cmu (x s)). Here, we use the 2 distance as the similarity measurement. In AGA, the optimization mechanism are designed for searching the maximal value of object function. Therefore, according to Eq. (3), the fitness function can be reformulated as: (S query (y)
S m (x s )) C exp(F(S query (y) S m (x s)))
(4)
where C is a constant for adjusting the value range of fitness function.
4 Pose Estimation Using HAGA In this section, we describe the key algorithms of the generative framework, namely, the AGA and HAGA, and theirs adaption for pose estimation from monocular silhouettes. 4.1 Hierarchical Annealed Genetic Algorithm Combining simulated annealing (SA) and genetic algorithm (GA), we design the annealed genetic algorithm, which actually is a hybrid (1 1) evolutionary strategy. In our algorithm, the local optimal solutions are avoided by introducing several genetic evolutionary principles. We represent chromosome by state vector as x [x1 x2 xn ],
424
X. Zhao and Y. Liu
(a)
(b )
Fig. 2. (a) The shape contexts computed on edge points of image silhouette (right) and sampled points of convolution curves (left). (b) The example shape contexts for reference samples showed in (a) of image silhouette (bottom) and convolution curves (top).
where the genes xi i 1 2 n are random numbers uniformly distributed in the interval (0 1). We use real encodings. The algorithm searching for optimal solutions with the AGA is described as follows: Parameter initialization set values for evolution control parameters: S t – stop criteria; Nt – termination condition; Et – times for searching a equation state; for st 1 to S t do: NonImproveNum 0; Generate the genes of x uniformly at random in the interval (0 1); Evaluate the fitness function (x) by mapping x onto the problem domain; while (NonImproveNum Nt ) do for et 1 to Et do: Evolution of x driven by the genetic operators; (see Table 1.) Evaluate (x); end for If the value of fitness function is improved, NonImproveNum 0, else NonImproveNum NonImproveNum 1; end while Record the optimal x; end for We design five genetic operators, which are executed orderly in AGA. The operators are introduced by evolving a example chromosome x [x1 x2 x3 x4 x5 x6 ]. The new chromosome generated by the operators is denoted as x . Assuming the positions generated randomly are number 2 and number 6 or number 3 ( for point mutation operator), for example, the five operators are illustrated in Table 1. (The new genes are represented as x ). On the basis of AGA, we develop a HAGA by utilizing the characteristics of state space X. In HAGA, the state space is decomposed automatically by computing the variances of state components which are generated in each annealing run. According to the variances of state components, the state space is partitioned by localizing down the important components to a small area of theirs range. It is explainable in theory because ¼
¼
Generative Estimation of 3D Human Pose
425
Table 1. The genetic operators in AGA Operators
Example
Exchange
x [x1 x2 x3 x4 x5 x6 ] x
¼
Segment reversion x [x1 x2 x3 x4 x5 x6 ] x
¼
Segment shift Point mutation
x [x1 x2 x3 x4 x5 x6 ] x
¼
x [x1 x2 x3 x4 x5 x6 ] x
¼
Segment mutation x [x1 x2 x3 x4 x5 x6 ] x
¼
[x1 [x1 [x1 [x1 [x1
x6 x3 x4 x5 x2 ] x6 x5 x4 x3 x2 ] x6 x2 x3 x4 x5 ] ¼
x2 x3 x4 x5 x6 ] ¼
x2 x3 ¼ x4 ¼ x5 ¼ x6 ¼ ]
the important state components dominate the topology of the state space and the little changes of theirs value can produce great eect whereas the values of other state components had little influence on whether they were selected or not. This theory is illustrated in Figure 3. Focusing only on one annealing run of sate evolution (st st 1), we describe the detailed HAGA as follows. 1. Generate initial chromosome x [x1 x2 xn ] at random, where xi i 1 2 n are random numbers uniformly distributed in the interval (0 1). Mapping it linearly into the variance domain: x xt
(min xt max xt )
(5)
In the first round of state evolution, (min x1 max x1 ) (0 1). Evaluating the fitness function (x). 2. Evolve the chromosome according to the state evolutionary mechanism of AGA. Before evaluating the fitness function, every new chromosome needs to be mapped onto the variance domain as formulated in Eq.5. 3. Store N best states (chromosomes) and computing the covariance matrix: Vt1
1 N
N
(xit1 xct1 )T (xit1 xct1 )
(6)
i 1
where xct1 is the mean vector, and the covariance matrix Vt1 is a diagonal matrix on the assumption that the state components are independent each other. To this end, the variance domain can be formulated as:
min xt1 max xt1
xct1 ct1 Vt1 xct1 ct1 Vt1
(7)
where ct1 [ct1 ct1 ct1 ] is used to adjust the variance domain and ct1 is a positive constant. 4. The variance domain (min xt1 max xt1 ) is used to cut down the state space in next state evolution. 4.2 Experiments We demonstrate our method by extracting subspaces for dierent classes of human motion and using them to estimate 3D body pose in unseen video sequences.
426
X. Zhao and Y. Liu
(a)
(b )
(c)
Fig. 3. Variance reduction contrast between principal state components and other state components. Graph (a) shows the variances of state set in which the chromosomes have not been evolved, displaying almost equal variances for each components. Graph (b) shows the variances of state set which have come through one round of state evolution, noticing that the variances of first four principal state components have been greatly reduced whereas the variances of other components have been reduced with a slighter extent. In graph (c), the variances of the principal components have been reduced to very small scopes indicating advanced localization after coming through two rounds of state evolution.
Walking motion: straight walk and turning walk. To extract the motion subspace of walking, a data set consisting of motion capture data of a single subject was used. The total frame number is 316. It was found that the dierent subject and dierent frame numbers can produce generally identical subspace. To keep the ratio of information loss lower than 0.05, the dimensionality of the subspace was choose to be 5. For the sequence of one subject walking in a straight line, the parameters of HAGA are set as S t 2 Nt 2 Et 5. The results are showed in Figure.4. It can be seen that the estimator is successful in determining the correct global motion as well as the 3D pose of the subject. The occlusion problem are tackled by searching the optimal pose in the extracted subspace because the prior knowledge about walking motion is contained in
Fig. 4. Results of recovering the poses of a subject walking straight. (the images are part of a sequence from www.nada.kth.se hedvigdata.html). The second pose demonstrated the left-right confusion in the silhouette.
Generative Estimation of 3D Human Pose
427
Fig. 5. Results of recovering the poses of a subject performing a turning walking motion
this space. The left-right confusion is mostly disambiguated, however, in few frames, the left-right confusion conduced by silhouette ambiguity still exist. This can be seen from Figure. 4. We test the generalization capability of our method in a turning walk sequence. In this sequence [19], a subject is performing continuing turning walking motion around a circle therefore the global motion is changed in a wide range. The parameters of HAGA are set as S t 2 Nt 2 Et 5. The results can be seen in Figure.5. Running motion. The subspace of running motion is extracted from motion capture data that consisted of 130 frames. This subspace is more compact than that of walking motion. Figure.6 shows the estimation results of 3D poses.
Fig. 6. Results of recovering the poses of a subject performing a running motion. The images are extracted from the video taken from the web site http:mocap.cs.cmu.edu.
428
X. Zhao and Y. Liu
5 Conclusion We have discussed a novel generative approach to estimating 3D human pose from a single camera. Our approach is a step towards describing motion characteristic of highdimensional data spaces by extracting its subspace. From motion capture data, we not only distilled the prior knowledge about human motion, but also reduced the dimensionality of problem. In the compact subspace, we perform eective search for finding the optimal poses. To explore the solution space eÆciently, we designed the AGA and HAGA, by which the optimal solutions can be searched eectively by utilizing the characteristics of state subspace. The robust shape contexts descriptor allows us using the silhouettes as image features. The approach was tested on dierent human motion sequences with good results, and allows the estimation of complex unseen motions in the presence of image ambiguities. In terms of future work, the more interior edge information need be added to disambiguate some challenging sequences. Including a wider range of motion capture data would allow the estimator to cover more types of human motions.
Acknowledgements This research is supported by the National Basic Research Program (973 Program) of China (No. 2006CB303103) and the National Natural Science Foundation of China (No. 60675017).
References 1. Sminchisescu, C., Kanaujia, A., Li, Z., Metaxas, D.: Discriminative density propagation for 3d human motion estimation. In: Proc. Conf. Computer Vision and Pattern Recognition, pp. 217–323 (2005) 2. Deutscher, J., Blake, A., Reid, I.: Articulated body motion capture by annealed particle filtering. In: Proceedings of the 2000 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 2, pp. 126–133 (2000) 3. Sidenbladh, H., Black, M., Fleet, D.: Stochastic tracking of 3D human figures using 2D image motion. In: European Conference on Computer Vision, vol. 2, pp. 702–718 (2000) 4. Sminchisescu, C., Triggs, B.: Covariance scaled sampling for monocular 3D body tracking. In: IEEE International Conference on Computer Vision and Pattern Recognition, vol. 1, pp. 447–454 (2001) 5. Agarwal, A., Triggs, B.: Tracking articulated motion using a mixture of autoregressive models. In: Pajdla, T., Matas, J. (eds.) ECCV 2004. LNCS, vol. 3023, pp. 54–65. Springer, Heidelberg (2004) 6. Ning, H., Tan, T., Wang, L., Hu, W.: People tracking based on motion model and motion constraints withautomatic initialization. Pattern Recognition 37(7), 1423–1440 (2004) 7. Mori, G., Malik, J.: Recovering 3 D Human Body Configurations Using Shape Contexts. IEEE Transactions on Pattern Analysis and Machine Intelligence 28(7), 1052–1062 (2006) 8. Sidenbladh, H., Black, M., Sigal, L.: Implicit Probabilistic Models of Human Motion for Synthesis and Tracking. In: European Conference on Computer Vision, vol. 1, pp. 784–800 (2002)
Generative Estimation of 3D Human Pose
429
9. Urtasun, R., Fleet, D., Fua, P.: Monocular 3-D Tracking of the Golf Swing. In: Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 2, IEEE Computer Society Press, Los Alamitos (2005) 10. Michalewicz, Z.: Genetic algorithms data structures evolution programs. Springer, Heidelberg (1996) 11. Belongie, S., Malik, J., Puzicha, J.: Shape matching and object recognition using shape contexts. IEEE Transactions on Pattern Analysis and Machine Intelligence 24(4), 509–522 (2002) 12. Aggarwal, J., Cai, Q.: Human motion analysis: A review. Computer Vision and Image Understanding 73(3), 428–440 (1999) 13. Gavrila, D.: Visual analysis of human movement: A survey. Computer Vision and Image Understanding 73(1), 82–98 (1999) 14. Moeslund, T., Granum, E.: A survey of computer vision-based human motion capture. Computer Vision and Image Understanding 81(3), 231–268 (2001) 15. Urtasun, R., Fua, P.: 3D Human Body Tracking using Deterministic Temporal Motion Models. In: European Conference on Computer Vision, vol. 3, pp. 92–106 (2004) 16. Arulampalam, M., Maskell, S., Gordon, N., Clapp, T., Sci, D., Organ, T., Adelaide, S.: A tutorial on particle filters for online nonlinearnon-GaussianBayesian tracking. IEEE Transactions on Signal Processing, IEEE Transactions on [see also Acoustics, Speech, and Signal Processing 50(2), 174–188 (2002) 17. Deutscher, J., Davison, A., Reid, I.: Automatic partitioning of high dimensional search spaces associated with articulated body motion capture. In: Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, IEEE Computer Society Press, Los Alamitos (2001) 18. Jin, X., Tai, C.: Convolution surfaces for arcs and quadratic curves with a varying kernel. The Visual Computer 18(8), 530–546 (2002) 19. Sigal, L., Black, M.J.: Humaneva: Synchronized video and motion capture dataset for evaluation of articulated human motion. Technical Report CS-06-08, Brown University (2006)
An Active Multi-camera Motion Capture for Face, Fingers and Whole Body Eng Hui Loke and Masanobu Yamamoto Graduate School of Science & Technology, and Department of Information Engineering, Niigata University, Ikarashi 2-nocho 8050, Nishi-ku, Niigata-city 950-2181, Japan [email protected]
Abstract. This paper explores a novel endeavor of deploying only four active-tracking cameras and fundamental vision-based technologies for 3D motion capture of a full human body figure, which includes facial expression, motion of fingers of both hands and a whole body. The proposed methods suggest alternatives to extract motion parameters of the mentioned body parts from four single-view image sequences. The proposed ellipsoidal model- and flow-based facial expression motion capture solution tackles both 3D head pose and non-rigid facial motion effectively and we observe that a set of 22 self-defined feature points suffice the expression representation. The body figure and fingers motion capture is solved with a combination of articulated model and flow-based methods.
1
Introduction
The human-based character animation technology has been growing in an incredible speed. Indirectly, this induces the growth in and demands towards motion capture. An abundance of work has been proposed in suggesting better and robust solutions for human figure [8], fingers or facial [10] motion capture. Despite these efforts, never has anyone in vision history proposed a solution on full human motion processing, that includes the motion of the body figure, facial expression and fingers simultaneously. In this paper, we propose a novel motion capture idea to implement an imagebased full human motion capture system where the simultaneous motion data estimated with our framework can easily be reconstructed in realizing the 3D character animation. This paper presents the idea: by simply adopting four autotracking function installed cameras and employing the knowledge in computer vision techniques, the 3D motion of a human figure, fingers and facial expression can be estimated from single-view image sequences recorded against a cluttered background. This first attempt of using only four cameras in image acquisition of the motion of the whole body eliminates the costing and performance area issues. Besides, in one system, we tackle the facial features recognition problem, 3D rigid head and non-rigid expression motion tracking based on the mixture vision techniques while treating human figure and fingers as articulated models problems for the 3D motion estimation. Y. Yagi et al. (Eds.): ACCV 2007, Part I, LNCS 4843, pp. 430–441, 2007. c Springer-Verlag Berlin Heidelberg 2007
An Active Multi-camera Motion Capture for Face, Fingers and Whole Body
431
Taking account of the fact that the characteristics of the facial expression and the other body parts are dissimilar, we treat them differently. To be precise, the facial expression is caused by muscles’ motion and thus, is changeable and induces the variant outlook and size of the features, that it is posed as a non-rigid-motion problem [1]. Meanwhile, body figure and fingers motion are caused by the bones and joints with respective limited D.O.F. [9]. Therefore, their size is invariant and appropriate to be represented as articulated model with rigid-motion. In brief, our framework consists of three main modules, image acquisition (section 2), vision-based motion estimation (sections 3 and 4), full model reconstruction and animation (sections 5 and 6).
2
Image Acquisition
Most of the image acquisition in the past works is done using VGA camera, with which it is either dealing solely with the movement of one small body part or tackling with the whole human figure motion without involving smaller body parts. However in our context, a VGA image, sized 640x480 pixels, has its limitation on the resolution and affects the image analysis quality on the smaller body parts such as face and fingers. To solve this, a high-vision camera with its high resolution of 1920x1080 pixels suits our requirement. Despite its high resolution, one non-compressive highvision camera with a full set of image capture devices can easily reach a cost that enfeebles its feasibility in most cases. We adopt an active multi-camera system consisting of four VGA color cameras (SONY, EVI-D30) which are connected parallel to the computer, where each automatically tracks the four different body parts: the face, the whole body, and the left and right hands. These cameras are able to auto-pan, tilt and zoom in tracking the movement of the target by color- and brightness-matching the current view with the user predefined area [11]. Once the starter is intrigued, the computer with multi-image capture board (Micro Vision, MV-34) starts fetching image data from these four cameras in parallel. This avoids the starting time delay that is common in a manual starting set-up and achieves a real definition of simultaneous motion tracking. The sequences of images are captured at the frame rate 30 fps, sized 320x240 pixels and in a limited designated range. Each camera is targeted on tracking the movement of different body parts, where the resulted images are shown in Figure 1. The hand and face images keep enough high resolution to capture the targets in motion while each camera can keep track of the target within the field of view.
3
Facial Expression Solutions
This section introduces a mixture of rigid and non-rigid solution framework, with which both the 3D head pose and facial features motion can be solved concurrently. We define a set of 22 feature points to represent the facial features as shown in Figure 2: one per nostril, three per eyebrow, four per eye and six
432
E.H. Loke and M. Yamamoto
Fig. 1. The 4 image sequences recorded using the active multi-camera system
Fig. 2. Facial feature points location. The numbering is the detection sequence.
for mouth. These feature points are assumed adhering to the surface of the ellipsoidal head model yet independently tracked on the image. The initial facial pose is required to be at its front view with no occlusion onto the facial features allowed. The following sub-section details the facial features localization, followed by the facial expression motion capture adopted for both rigid and non-rigid motion recovery. 3.1
Skin Color Region Detection and Facial Region Recognition
Chromatic color space is the traditional approach for skin color space analysis since it carries such characteristic where its two useful color components are in fact the result of the color elements with intensity normalization. However, this paper considers also the HSV space of the skin color based on two facts: (i) the evaluation done in [12] proves that a Gaussian mixture is more accurate than a single Gaussian model, and (ii) HSV color space performs the same clustering characteristic as chromatic’s with a broader range. A Gaussian model can approximate these clustering characteristics. Applying this Gaussian model onto the chromatic and HSV color space of the initial facial image, two grey level images are generated with the intensity of each pixel representing its probability of the skin color existence. We average the intensity values of these two images and apply an empirical threshold on them to get the binary skin-likelihood segments. Spurious pixels induced by problematic noise or color defects in the image are cancelled off with morphological closing operator and followed by median filter. The biggest skin-color blob is selected as the facial candidate-region while the other regions are discarded. With the assumption that the front-face-view is assured on the initial image, we use a grey scale eye-nose template as a detector identifying the exact location
An Active Multi-camera Motion Capture for Face, Fingers and Whole Body
433
of the facial features, eyes and nose. This is based on the conclusion deduced by Brunelli et al [2] that eyes perform the best matching result followed by the nose. A full frontal face template would be too time-consuming and the matching with different mouth-opening images is error-prone, that we settle down on this eye-nose template and it is rescaled only once to the updated width of the facial-feature area image. Template matching is carried out on the grey-level image of the updated facial features area. The Normalized Correlation Coefficients (NCC) of different y location are computed vertically. Normalization on the correlation coefficients subdues the illumination- and brightness-induced ambiguous matching results. The highest correlation value of the facial upper region reports the presence of the eyes and nose. 3.2
Extraction of Facial Feature Points
There are 22 feature points in a set to be localized after the position of the eyes and nose are confirmed. Various approaches are proposed in different works, to name a few, edge detection, integral projection or snake localization. The facial feature area recognized in the previous section is segmented into different blocks on the color image according to the scaled size of the eyes and nose on the template. The approaches for each feature point’s localization are detailed as followed. – Calculate the color histogram within the eye feature block and extract the highest value as the threshold, where pixels with lower luminance are extracted – Convert the extracted pixels block into a binary image – Get the Vertical/Horizontal Integral Projection values of the binary image to locate the hair region – Identify the upper rows and side columns at which the projection value is above-threshold. These regions are recognized for the existence of hair and thus are eliminated – The binary region is thus segmented as the eye region The two corners of an eye are located at the left and right bounds of this region. The Harris corner detector is further carried out within a fixed area centered at these two bounds to find good corner values. Following that, the y values of both upper and lower bounds of the eye region are taken as the opening values of the eye, albeit the x values of them are not determined yet. The initial try was done by calculating the center of the two corner points but it doesn’t end up a stable nor good solution. The final adopted approach is that by calculating the Horizontal Integral Projection of the eye region on the binary image. The x value with the highest integral projection value is adopted as the x coordinate of the opening points as plotted in Figure 3. It is proven in the detection result that this method works well as it indirectly identifies the gazing view of the person. In addition, this approach works stably even when the captured eye is closed because dark line forms at the edge of the eyelid.
434
E.H. Loke and M. Yamamoto
Fig. 4. Lips opening feature points detection
Fig. 3. Eyes feature points dDetection
Fig. 5. Eyebrow mid point detection
Detecting feature points on Eyes: In [4], eyes (pupils) are located using template matching but in [3,5], the eyes feature points are detected using integral projection or in others’, manually. However, before the two corners and two opening points of each left and right eye are detected, we have to eliminate the hair-likelihood region with the following steps: Detecting feature points on Nose: The approach taken in detecting nostrils is straightforward as it carries a special characteristic - nostrils are the darkest regions in the mid area of a face (between the vertical distance of the eyes and mouth). This system searches one dark region in each half of the nose feature block. The center point of each region is taken as the nostril feature point. Detecting feature points on Mouth: In this system, we define six feature points on the mouth where two are the corners and the rest represents the upper and lower lips. To calculate each of their location, we have to estimate the searching area on the image. This is done by fixing the searching boundary based
An Active Multi-camera Motion Capture for Face, Fingers and Whole Body
435
on the known eyes and nostrils coordinates, which is shown as the black segment in Figure 4. The luminance of the lips is the most easily recognized characteristic in the case of mouth detection. Thus, this informative color can be considered as the best choice for such a task on a color image instead of the edge extraction method adopted in [4], which is susceptible to noise that can lead to fallacious detection result. Our method applies HSV or RGB color space in the mentioned bounding box to extract the segment where lips-likelihood pixels exist. Easily, the two feature points at each upper and lower lip can be derived at the mid x value between the nostrils. The y values of each point is extracted from the lips-likelihood segment as illustrated in Figure 4. Similar to the searching method in detecting corners of the eyes, the two corner points of the lip are determined using Harris corner detector one at each half of the mouth feature block and they must satisfy certain geometric constraints. Detecting feature points on Eyebrows: Eyebrow is one of the difficult objects to be detected and tracked on the face for its uneven shape and thickness traits, besides the problem of being blocked or misrecognized as forelock. Not many facial features detection research has a clear solution for it. We are representing one eyebrow with three points where one at the middle and two at both of its ends. Figures 5 and 6 illustrate the detection mechanism explained below. First, the middle position of the eyebrow has to be determined. The searching boundary is estimated in between of the beginning y value of the skin-likelihood region, the ystart and the beginning of the eyes’ bounding box, the yend . This bounding area is gotten rid of noise with a Gaussian convolution method. Starting from the yend to ystart , a searching-for-edge function is run at the mid point of both corners of the eye. The first point encountered with a significant intensity changes (from bright to dark) would be taken as the lower edge of the eyebrow. Once this point is detected, we search for the upper edge of the eyebrow that would be the first point encountered with the characteristic of significant intensity changes (from dark to bright). Averaging the y value of these two points, we mark it as the mid feature point for the eyebrow. Both ends of an eyebrow are detected by carrying out the detection function starting from the known mid eyebrow point towards both ends horizontally. This
Fig. 6. Eyebrow corner points localization
436
E.H. Loke and M. Yamamoto
bounding area is converted to an edge-image. At every incremental/decremental x location, the upper and lower edges (y value) of the eyebrow have to be determined. The search continues by taking the previous average y value as the starting y location at next x value until it meets either of the two conditions: (i) the two upper and lower edges cling/are close to each other under certain threshold, or (ii) the edges seem too apart and the distance is larger than the thickness of the center point of the eyebrow. The head is represented as a 3D ellipsoidal model which is fitted onto the facial region in the image. The depth values of each facial feature points are calculated according to their x and y positions to be placed at the surface of the ellipsoidal head. 3.3
Facial Features Motion Capture
Facial expression, also the facial features motion, poses as a non-rigid motion issue only very recently as most of the earlier works either treat it as twodimensional affine solution or leave it untouched. Among the proposed solutions are inclusive of but not limited to, the application of either complicated or simple physically based models (polygonal or mesh) of the head, local parametric flow model and optical flow calculation with model-based solution. Aiming for a robust but less computation intensive approach, this system took the middle path between the simple template-based affine approximation approach and the intricate physical or muscle structure model-based tracking method. Adopting the similar idea proposed in [1], we tackle the rigid head motion and non-rigid facial expression tracking separately. Rigid Head Motion Estimation: The 2D image flow field in the head region between two successive frames (exclusive of the sparse motion field in facial features region of the eyebrows, eyes and mouth) can be calculated by the constant brightness constraint over time coupling with the Lucas-Kanade technique[6]. This 2D flow is then interpreted as the 3D rigid motion of the head using the depth constraint by referring to the current pose of the ellipsoidal head model. The 3D rigid motion updates current pose of the head at every frame. Non-Rigid Facial Features Motion Estimation: This module inherits the 3D motion parameter from the rigid tracking module to expect the facial feature points in the next frame. The facial feature points are also tracked individually using the NCC method. A displacement between the expected position and tracked one denotes the relative motion with respect to the head as reference coordinates. If the displacement is large, expression deformation occurs.
4
Articulated Model Solutions
The use of articulated model for human body and hand figures representation is common in model-based tracking system [8]. Fitting the model to image sequence can produce a pose sequence, i.e. motion, of the body and hand. In this paper, we
An Active Multi-camera Motion Capture for Face, Fingers and Whole Body
437
Head Neck Index F. Middle F. Ring F. Seg. 3 Seg. 3 Seg. 3
Little F. Seg. 3
Index F. Middle F. Ring F. Seg. 2 Seg. 2 Seg. 2
Little F. Seg. 2
Thumb. Index F. Middle F. Ring F. Seg. 2 Seg. 1 Seg. 1 Seg. 1
Little F. Seg. 1
Right Left Upper Upper Arm Chest Arm Right Forearm
Waist
Right Hand Right Thigh
Left Forearm
Left Left Hand Thigh
Right Shin
Left Shin
Right Foot
Left Foot
Thumb Seg. 1
Palm (Left)
Palm (Right)
Fig. 7. Left: Body ireframe model and the tree structure of the body parts, Right: Fingers wireframe model and its tree structure
inherit a motion capture method proposed in [9]. We briefly explain the method, but the details can be further referred in [9]. Our body model consists of 16 parts arranged in a tree structure with the waist model part as the root of the model, as shown in Figure 7 left. The similar concept in body figure modeling is extended to the fingers modeling. We define it with the similar structure with 16 parts as illustrated in Figure 7 right. The root of this model is placed at the palm. The tree structure in the model denotes connection from parent to child by the arrow direction. Each part has its own coordinates system of which one axis aligns along the body axis and the origin is located at the joint with its parent part (a center of gravity for waist and chest). We manually adjust the pose of the articulated model to fit it onto the initial frame image of each body and fingers images. The automatic model fitting makes further issue. It will be possible since we can utilize existing methods, e.g. [7], for detection of the human body in image and determining the 3D pose. A pose displacement of human body can be estimated from image subtraction between the successive frames. Therefore, after getting a pose at the initial frame by model fitting, pose at any frame can be obtained from accumulating the successive pose displacements onto the initial pose. In fact, the estimation of pose displacement is based on that (1) optical flow is constrained by temporal-spatial linear equation, (2) a 3D translation vector on the model is approximated as a sum of pose displacements weighted by the Jacobian matrix, and (3) a depth of human body can be obtained from the model fitted to the body. Chain substitutions based on the above (1), (2) and (3) can produce a system of linear equations with respect to the pose displacements as unknowns. Solving the system of linear equations and accumulating the obtained pose displacements onto the initial pose can result in the human motion. However, this approach causes a pose drift as an inherent drawback. To cope with this drift, additional
438
E.H. Loke and M. Yamamoto
model fitting is manually required at several key-frames, and the accumulated pose at in-between frame is corrected by propagation of the poses given at keyframes [9].
5
Animation of the Simultaneous Motion
A general humanoid polygonal model is deformed automatically according to the initial pose acquired from the earlier initialization module. Since this animation step highlights the reconstruction of a humanoid model coupling with the restoration and animation of the captured simultaneous motion from an actor, no human motion kinematics is applied. The fingers and facial features models are installed on the body figure model for the reconstruction of a full humanoid model. Nevertheless, the initial estimated size of each body parts calculated from each image sequence are different. Therefore, several steps are taken for the integration: (i) The system recognizes the parental/replacement part for each model : The head is the parental part for facial features as is the palm for the fingers model. The system locates the center of the ellipsoidal head on the body figure model, makes it the origin of the coordinate system for the facial feature points. As for the both hands, fingers models can just be considered the replacements for hand parts and be installed at the origin of their respective local coordinate systems, ΣLef tHand and ΣRightHand , on the body figure model. (ii) Model size scaling : Assuming the three axes of the initial ellipsoidal head size of the body figure and facial image sequences are Hbody = (x, y, z) and Hf ace = (x, y, z) respectively, we rescale each features position with Hbody /Hf ace at all the x, y and z axes. The similar calculation works on fingers model as well. 5.1
Reconstruction of the Full Human Model
By utilizing the camera-referenced motion data of all the targetted subjects acquired in the previous motion capture section, this system automatically constructs a full human model and realises the simultaneous motions of each body part on it. However, each facial features, fingers and body figure model consists of each model structure and poses in different camera coordinate systems. Thus, in constructing these different body parts into one complete human model structure, they need to be referenced in the same coordinate system, namely the world coordinate system. In this paper, the human body camera coordinate system is taken as the world coordinate system. The hand model shares a palm with the body model. The face model shares a head with the body model. Therefore, if the parent of the fingers changes from the palm of the hand model to the palm of the body model, and the parent of the facial features changes from the head of the face model to the head of the body model, a full body model can be established without further camera calibration. We draw the transformation from a local camera coordinates to the world coordinates by an example of the little fingertip. In order to assemble the hand models on the body model, we assume j Ti as the transformation from coordinates
An Active Multi-camera Motion Capture for Face, Fingers and Whole Body
439
Fig. 8. The tracking result of each face, body figure, right and left hand image sequence, at frame 56, 101 and 197
Σi of part i to Σj of part j. Let denote parts 1, 2, · · ·, 8 be waist, chest, upper arm, forearm, hand (alias palm), finger seg.1, finger seg.2, and finger seg.3 (alias fingertip), respectively, and Σ0 denotes the world coordinate. The body camera can capture the motion up to the palm. The transformation from the palm to the world coordinate system is given by 0
T1 1 T2 2 T3 3 T4 4 T5 .
(1)
Meanwhile, the hand camera captures the hand motion from the palm to fingertip. The transformation from the fingertip to the hand camera coordinate system is given by h T5 5 T6 6 T7 7 T8 (2) where h denotes the hand camera coordinate system. To represent a pose of the fingertip in the world coordinate, adding 5 T6 6 T7 7 T8 in eq. (2) to eq. (1), we have the transformation 0
T1 1 T2 2 T3 3 T4 4 T5 5 T6 6 T7 7 T8 .
Similarly, the face model is embedded into the full human model.
(3)
440
6
E.H. Loke and M. Yamamoto
Experimental Results
To prepare the motion image sequences for the system experiment, we carry out the image acquisition in the laboratory without constraint on the background scene and costumes of the actor. The actor is required to start with a front view position but free to move along the recording process of 200 frames with the conditions that facial features and fingers visibility is guaranteed and the motion has to be smooth. Two sets of motion were recorded with this multicamera system as sized 320x240 pixels bitmap image sequences,in which the actor (i) did simple movements on the hands and face, and (ii) performed airguitar while singing. The tracking result at different time-frame of the four image sequences on face features, whole body figure and both hands can be refered in Figure 8 while Figure 9 illustrates the reconstructed model’s appearance and the motion recovered at the same time-frame. The average processing time consumes about 30 seconds per frame with which the 200 frames motion capture and animation process takes less than 15 minutes to complete.
Fig. 9. Animation reconstructed on the humanoid model at frame 56, 101 and 197
An Active Multi-camera Motion Capture for Face, Fingers and Whole Body
7
441
Conclusion
In this paper, we have suggested a novel idea for full human motion capture, inclusive of the motion on the full human figure, fingers of both hands and facial features simultaneously, for which only 4 cameras are needed for the recording purpose. Our major contributions lie in several aspects. The main relates to the novel endeavor of using only 4 cameras to capture the full motion of an actor. We also successfully built a foundation platform for the concurrent motion estimation of the body figure, fingers and facial expression solely from the digital information on single-view image sequences and the motion reconstruction on a full humanoid model. We also proposed several alternatives for facial features detection and 3D rigid head pose and non-rigid features motion estimation. Through the animation result, we have demonstrated how our approaches provide a concise description of the human motion on different body parts and are feasible for humanoid model animation construction. Acknowledgments. We thank Hideaki Sasagawa for the hands tracking works.
References 1. Black, M.J., Yacoob, Y.: Recognizing Facial Expressions In Image Sequences Using Local Parameterized Models Of Image Motion. IJCV 25(1), 23–48 (1997) 2. Brunelli, R., Poggio, T.: Face Recognition: Features versus Templates. IEEE PAMI 15(10), 1042–1052 (1993) 3. Chuang, M.M., Chang, R.F., Huang, Y.L.: Automatic Facial Feature Extraction In Model-Based Coding. Journal of Information Science And Engineering 16, 447–458 (2000) 4. Feris, R.S., De Campos, T.E., Junior, R.M.C.: Detection and Tracking Of Facial Features In Video Sequences. In: Cair´ o, O., Cant´ u, F.J. (eds.) MICAI 2000. LNCS, vol. 1793, pp. 197–206. Springer, Heidelberg (2000) 5. Gu, H., Su, G.: Feature Points Extraction From Faces. Image and Vision Computing NZ (2003) 6. Lucas, B.D., Kanade, T.: An iterative image registration technique with an application to stereo vision. In: Proc. Imaging Understanding Workshop, pp. 121–130 (1981) 7. Mori, G., Malik, J.: Recovering 3D human body configurations using shape contexts. IEEE PAMI 7(28), 1052–1062 (2006) 8. Wang, J.J., Singh, S.: Video analysis of human dynamics - a survey. Real-Time Imaging 9(5), 321–346 (2003) 9. Yamamoto, M., Ohta, Y., Yamagiwa, T., Yagishita, K., Yamanaka, H., Ohkubo, H.: Human Action Tracking Guided by Key-Frames. FG2000, 354–361 (2000) 10. Zhao, W., Chellappa, R., Phillips, P.J., Rosenfeld, A.: Face recognition: A literature survey. ACM Computiong Surveys 35(4), 299–458 (2003) 11. http://www.Sony.Co.Jp/Products/ISP/Products/Model/Ptz/EVID30.Html 12. Evaluation of RGB and HSV models in Human Faces Detection, [Online] (2004) Available, http://www.cescg.org/CESCG-2004,/web/Sedlacek-Marian/
Tracking and Classifying of Human Motions with Gaussian Process Annealed Particle Filter Leonid Raskin, Michael Rudzsky, and Ehud Rivlin Computer Science Department, Technion—Israel Institute of Technology, Haifa, Israel, 32000 {raskinl,rudzsky,ehudr}@cs.technion.ac.il
Abstract. This paper presents a framework for 3D articulated human body tracking and action classification. The method is based on nonlinear dimensionality reduction of high dimensional data space to low dimensional latent space. Motion of human body is described by concatenation of low dimensional manifolds which characterize different motion types. We introduce a body pose tracker, which uses the learned mapping function from low dimensional latent space to high dimensional body pose space. The trajectories in the latent space provide low dimensional representations of body poses performed during motion. They are used to classify human actions. The approach was checked on HumanEva dataset as well as on our own one. The results and the comparison to other methods are presented.
1
Introduction
Human body pose estimation and tracking is a challenging task for several reasons. The main problem that has to be solved in order to achieve satisfactory results of pose tracking and understanding is large dimensionality of the human pose model, which complicates the examination of the entire subject and makes it harder to detect each body part separately. Despite the high dimensionality of the problem, many poses can be presented in a low dimensional space by dimensionality reduction. Human body motions can be displayed as curves in this space. This space can be obtained by learning different motion types [1,2]. This paper presents an approach to 3D people tracking and motion analysis. In this approach we apply a nonlinear dimensionality reduction using Gaussian Process Dynamical Model (GPDM) [3,4] and annealed particle filter [5]. GPDM is better able to capture properties of high dimensional motion data than linear methods such as PCA. This method generates a mapping function from the low dimensional latent space to the full data space based on learning from previously observed poses of different motion types. For the tracking we separate model state into two independent parts: one contains information about 3D location and orientation of the body and the second one describes the pose. We learn latent space that describes poses only. The tracking algorithm consists of two stages. Firstly the particles are generated in the latent space and are transformed into the data space by using learned a priori mapping function. Y. Yagi et al. (Eds.): ACCV 2007, Part I, LNCS 4843, pp. 442–451, 2007. c Springer-Verlag Berlin Heidelberg 2007
Tracking and Classifying of Human Motions
443
Secondly we add rotation and translation parameters to obtain valid poses. The likelihood function is calculated in order to evaluate how well a pose matches the visual data. The resulting tracker estimates the locations in the latent space that represents poses with the highest likelihood. As the latent space is learned from sequences of poses from different motion types, each action is represented by a curve in the latent space. The classification of the motion action is based on the comparison of the sequences of latent coordinates that the tracker has produced, to the ones that represent poses sequences of different motion types. We use a modified Fr`echet distance [6] in order to compare the pose sequences. This approach allows introducing different actions from the ones we have used for learning by exploiting the curve that represents it. We show that our tracking algorithm provides good results even for the low frame rate. An additional advantage of our tracking algorithm is the capability to recover after temporal loss of the target. We also show that the task of action classification, when performed in the latent space, is robust.
2
Related Works
One of the common approaches for tracking is the particle filter method. This method uses multiple predictions, obtained by drawing samples of pose and location prior and then propagating them using the dynamic model, which are refined by comparing them with the local image data, calculating the likelihood [7]. The prior is typically quite diffused (because motion can be fast) but the likelihood function may be very peaky, containing multiple local maxima which are hard to account for in detail [8]. Annealed particle filter [5,19] or local searches are the ways to attack this difficulty. An alternative is to apply a strong model of dynamics [9]. There exist several possible strategies for reducing the dimensionality of the configuration space. Firstly it is possible to restrict the range of movement of the subject [10]. Due to the restricting assumptions the resulting trackers are not capable of tracking general human poses. Another way to cope with highdimensional data space is to learn low-dimensional latent variable models [11]. However, methods like Isomap [12] and locally linear embedding (LLE) [13] do not provide a mapping between the latent space and the data space, and, therefore Urtasun et al. [14] proposed to use a form of probabilistic dimensionality reduction by GPDM [3,4] to formulate the tracking as a nonlinear least-squares optimization problem. During the last decade many different methods for behavior recognition and classification of human actions have been proposed. The popular methods are based on Hidden Markov Models (HMM), Finite State Automata (FSA), context-free grammar (SCFG) etc. Sato et al. [15] presented a method to use extraction of human trajectory patterns that identify the interactions. Park et al. [16] proposed a method using a nearest neighbor classifier for recognition of twoperson interactions such as hand-shaking, pointing, and standing hand-in-hand. Hongeng et al. [17] proposed probabilistic finite state automata for recognition
444
L. Raskin, M. Rudzsky, and E. Rivlin
of a sequential occurrence of several scenarios. Park et al. [18] presented a recognition method that combines model-based tracking and deterministic finite state automata. This paper is organized as follows. Section 3 describes the tracking algorithm. Section 4 describes the classification algorithm. Section 5 shows the experimental results for tracking and action classification of different data sets and motion types.
3 3.1
Tracking GPAPF Tracker
The drawback in the annealed particle filter [5] tracker is that a high dimensionality of the state space causes an increase in the number of particles that are needed to be generated in order to preserve the same density of particles. This makes this algorithm computationally ineffective for low frame rate videos (30 fps and lower). The other problem is that once a target is lost (i.e. the body pose was wrongly estimated, which can happen for the fast and not smooth movements) it becomes highly unlikely that the next pose will be estimated correctly in the following frames. In order to reduce the dimension of the space we propose to use Gaussian Process Annealed Particle Filter (GPAPF). Using Gaussian Process Dynamical Model (GPDM) [3,4] we embedded several types of poses into a low dimensional space. We used two and three dimensional spaces which was enough for robust tracking and classification. The poses are taken from different sequences, such as walking, running, punching and kicking. We divide our state into two independent parts. The first part contains the global 3D body rotation and translation, which is independent of the actual pose. The second part contains only information regarding the pose (26 DoF). We use GPDM to reduce the dimensionality of the second part. This way we construct a latent space (Fig. 1). This space has a significantly lower dimensionality (for example 2 or 3 DoF). The latent space includes solely pose information and is therefore rotation and translation invariant. For the tracking task we use a modified annealed particle filter tracker [5]. We are using a 2-stage algorithm. The first stage is generation of new particles in the latent space, which is the main modification of the tracking algorithm. Then we apply the learned mapping function that transforms latent coordinates to the data space. As a result, after adding the translation and rotation information, we construct 31 dimensional vectors that describe a valid data state, which includes location and pose information, in the data space. In order to estimate how well the pose matches the images the likelihood function is calculated [19]. Suppose we have M annealing layers. The state is defined as a pair Γ = {Λ, Ω}, where Λ is the location information and Ω is the pose information. We also define ω as a latent coordinates corresponding to the data vector Ω: Ω = ℘ (ω), where ℘ is the mapping function learned by the GPDM. Λn,m , Ωn,m and ωn,m are the location, pose vector and corresponding latent coordinates on the frame n and annealing layer m. For each 1 ≤ m ≤ M − 1 Λn,m and ωn,m
Tracking and Classifying of Human Motions
445
Fig. 1. The latent space that is learned from different motion types. (a) 2D latent space from 3 different motions: lifting an object (red), kicking with the left (green) and the right (magenta) legs. (b) 3D latent space from 3 different motions: hand waving (red), lifting an object (magenta), kicking (blue), sitting down (black), and punching (green).
are generated by adding multi-dimensional Gaussian random variable to Λn,m+1 and ωn,m+1 respectively. Then Ωn,m is calculated using ωn,m . Full body state Γn,m = {Λn,m , Ωn,m } is projected to the cameras and the likelihood πn,m is calculated using likelihood function. The main difficulty is that the latent space is not uniformly distributed and sequential poses may be not close on the latest space to each other. Therefore we use a dynamic model, as proposed by Wang et al. [4], in order to achieve smoothed transitions between sequential poses in the latent space. However, there are still some irregularities and discontinuities. Moreover, in the latent space each pose has a certain probability to occur and thus the probability to be drawn as a hypothesis should be dependent on it. For each location in the latent space the variance can be estimated that can be used for generation of the new particles. In Fig. 1(a) the lighter pixels represent lower variance, which depicts the regions of latent space that corresponds to more likely poses. The additional modification that has been done is in the way the optimal configuration is calculated. In the original annealed particle filter algorithm the optimal configuration is achieved by averaging over the particles in the last layer. However, as the latent space is not an Euclidian one, applying this method on ω will produce poor results. We propose to calculate the optimal configuration in the data space and then project it back to the latent space. At the first stage we apply the ℘ on all the particles to generate vectors in the data space. Then in the data space we calculate the average on these vectorsand project it back to N (i) (i) . π ℘ ω the latent space. It can be written as follows: ωn = ℘−1 n,0 i=1 n,0 The resulting tracker is capable of recovering after several frames of poor estimations. The reason for this is that particles generated in the latent space are representing valid poses more authentically. Furthermore because of its low dimensionality the latent space can be covered with a relatively small number of particles. Therefore, most of possible poses will be tested with emphasis on the pose that is close to the one that was retrieved in the previous frame. So if the
446
L. Raskin, M. Rudzsky, and E. Rivlin
Fig. 2. Losing and finding the tracked target despite the miss-tracking on the previous frame
pose was estimated correctly the tracker will be able to choose the most suitable one from the tested poses. At the same time, if the pose on the previous frame was miscalculated the tracker will still consider the poses that are quite different. As these poses are expected to get higher value of the weighting function the next layers of the annealed will generate many particles using these different poses. In this way the pose is likely to be estimated correctly, despite the misstracking on the previous frame as shown in Fig. 2. Another advantage of our approach is that the generated poses are, in most cases, natural. In case of CONDENSATION or annealed particle filter, the large variance in the data space, can cause generation of unnatural poses. Poses that are produced by the latent space that correspond to points with low variance are usually natural and therefore the number of the particles effectively used is higher, which enables more accurate tracking. 3.2
Obtaining Better Tracker
The problem with such a 2-staged approach is that Gaussian field is not capable to describe all possible poses. As we have mentioned above, this approach resembles using probabilistic PCA in order to reduce the data dimensionality. However, for tracking issues we are interesting to get pose estimation as close as possible to the actual one. Therefore, we add an additional annealing layer as the last step. This layer consists only from one stage. We are using data states, which were generated on the previous two staged annealing layer, in order to generate data states for the next layer. This is done with very low variances in all the dimensions, which practically are even for all actions, as the purpose of this layer is to make only the slight changes in the final estimated pose. Thus it does not depend on the actual frame rate, contrarily to original annealing particle tracker, where if the frame rate is changed one need to update the model parameters (the variances for each layer).
4
Action Classification
The classification of the actions is done based on the sequences of the poses that were detected by the tracker during the performed motion. We use Fr`echet distance [6] in order to determine the class of the motion, i.e. walking, kicking,
Tracking and Classifying of Human Motions
447
waving etc. The Fr`echet distance between two curves measures the resemblance of the curves taking into consideration their direction. This method is quite tolerant to position errors. Suppose there are K different motion types. Each type k is represented by a model Mk which is a sequence of the lk + 1 latent coordinates Mk = {μ0 , ..., μlk }. The GPAPF tracker generates a sequence of l + 1 latent coordinates: Γ = {ϕ0 , ..., ϕl }. We define a polygonal curve P E as a continuous and piecewise linear curve made of segments connecting vertexes E = {v0 , ..., vn }. The curve can be parameterized with a parameter α ∈ [0, n], where P E (α) refers to a given position on the curve, with P E (0) deE notes Mv0 and between two curves is defined as P (n) denotes vn . The distance Γ i F P , P l] , = minα,β f P Mi (α), P Γ (β) : α [0, 1] → [0, lk ] , β [0, 1] → [0, where f P Mi (α) , P Γ (β) = max P Mk (α (t)) − P Γ (β (t)) 2 : t ∈ [0, 1] and α (t) and β (t) represent sets of continuous and increasing functions with α (0) = 0, α (1) = lk , β (0) = 0, β (1) = l. The model with the smallest distance is chosen to represent the type of the action. While in general it is hard to calculate the Fr`echet distance, Alt et al. [6] has shown an efficient algorithm to calculate it between two piecewise linear curves.
5
Results
We have tested GPAPF tracking algorithm using HumanEva data set. The data set contains different activities, such as walking, boxing etc. and provides the correct 3D locations of the body joints, such as hips and knees, for evaluation of the results and comparison to other tracking algorithms. We have compared our results to the ones produced by the annealed particle filter body tracker [20] and compared the results with the ones produced by the GPAPF tracker. The error measures the average 3D distance between the locations of the joints that is provided by the MoCap system and by ones that were estimated the tracker [20]. Fig. 3 shows the actual poses that were estimated for this sequence. The poses are projected to the first and second cameras. The first two rows show the results of the GPAPF tracker. The last two rows show the results of the annealed particle filter. Fig. 4.a shows the error graphs, produced by GPAPF tracker (blue circles) and by the annealed particle filter (red crosses) for the walking sequence taken at 30 fps. The graph suggests that the GPAPF tracker produces more accurate estimation. We also compared the performance of the tracker with and without the additional annealed layer. We have used 5 double staged annealing layers in both cases. For the second tracker we have added additional single staged layer. The Fig. 4.b shows the errors of the GPAPF tracker version with the additional layer (blue circles) and without it (red crosses); Fig. 5 shows sample poses, projected on the cameras. The improvement is not dramatic. This is explained by the fact that the difference between the estimated pose using only the latent space annealing and the actual pose is not very big. That suggests that the latent space represents accurately the data space. We have also created a database, which contains videos with similar actions, produced by a different actor. The
448
L. Raskin, M. Rudzsky, and E. Rivlin
Fig. 3. Tracking results of annealed particle filter tracker and GPAPF tracker. Sample frames from the walking sequence. First row: GPAPF tracker, first camera. Second row: GPAPF tracker, second camera. Third row: annealed particle filter tracker, first camera. Forth row: annealed particle filter tracker, second camera.
Fig. 4. (a) The errors of the annealed tracker (red crosses) and GPAPF tracker (blue circles) for a walking sequence captured at 30 fps. (b) The errors GPAPF tracker with additional annealing layer (blue circles) and without it (red crosses) for a walking.
frame rate was 15 fps. We have manually marked some of the sequences in order to produce the needed training sets for GPDM. After the learning we validated the results on the other sequences containing same behavior. We have experimented with different number of particles. For the 100 particles per layer the computational cost was 30 sec per frame. Using the same number of particles and layers in the annealed particle filter algorithm takes 20 seconds per frame. However, the annealed particle filter algorithm was not capable of tracking the body pose with such a low number of particles for 30 fps and 15
Tracking and Classifying of Human Motions
449
Fig. 5. GPAPF algorithm with (a) and without (b) additional annealed layer
Fig. 6. Tracking results of annealed particle filter tracker and GPAPF tracker. Sample frames from the running, leg movements and object lifting sequences.
fps. Therefore, we had to increase the number of particles used in the annealed particle filter to 500. 5.1
Motion Classification
The classification algorithm was tested on two different data sets. The first set contained 3 different activities: (1) lifting an object, kicking with (2) the left and (3) the right leg. For each activity 5 different sequences were captured. We have used one sequence for each motion type in order to construct the models. The latent space was learned based on the poses in these models (Fig. 1.a). The latent space had a clear and very distinguishable separation between these 3 actions. Therefore, although the results of the tracker contained much noise as shown in Fig. 7, the algorithm was able to perform perfect classification. The second set contained 5 different activities: (1) hand waving, (2) lifting an object, (3) kicking, (4) sitting down, and (5) punching. Once again 5 different sequences were captured for each activity. The cross validation procedure was used to classify the sequences (see Fig. 1.b). The accuracies of the classification,
450
L. Raskin, M. Rudzsky, and E. Rivlin
Fig. 7. Tracking trajectories in the latent space for different activities: (a) lifting an object, kicking with (b) the left and (c). On each image the black lines represent incorrect activities, the red line represents the correct one, and other colored lines represent the trajectories produced by the GPAPF tracker.
Table 1. The accuracies of the classification for 5 different activities: hand waving, object lifting, kicking, sitting down, and punching. The rows represent the correct motion type; the columns represent the classification results. Hand waving Object lifting Kicking Sitting down Punching Hand waving Object lifting Kicking Sitting down Punching
15 0 0 0 6
0 17 0 3 0
0 0 20 1 20
0 3 0 16 0
5 0 0 0 14
as shown in Table 1, are 75, 85, 100, 80, 70 percent for the above interactions (1)-(5) respectively. The low classification rates of actions involving the hand gestures are due to the similarity of the native actions. The low classification rates of sitting down and object lifting actions are due to the high self occlusions, which caused the tracker to perform wrong estimations of the actual poses.
6
Conclusion and Future Work
In this paper we have introduced an approach for articulated body tracking and human motion classification using a low dimensional latent space. The latent space is constructed from pose samples from different motion types. The tracker generates trajectories in the latent space, which are classified using Fr`echet distance. The interesting problem that has not been solved yet is to perform classification of the interactions between multiple actors. The main problem is constructing a latent space. While a single persons poses can be described using a low dimensional space it may not be the case for multiple people.
Tracking and Classifying of Human Motions
451
References 1. Christoudias, C.M., Darrell, T.: On modelling nonlinear shape-and-texture appearance manifolds. In: Proc. CVPR, vol. 2, pp. 1067–1074 (2005) 2. Elgammal, A., Lee, C.: Inferring 3d body pose from silhouettes using activity manifold learning. In: Proc. CVPR, vol. 2, pp. 681–688 (2004) 3. Lawrence, N.: Gaussian process latent variable models for visualization of high dimensional data. In: NIPS. Information Processing Systems, vol. 16, pp. 329–336 (2004) 4. Wang, J., Fleet, D., Hetzmann, A.: Gaussian process dynamical models. In: NIPS. Information Processing Systems, pp. 1441–1448 (2005) 5. Deutscher, J., Blake, A., Reid, I.: Articulated body motion capture by annealed particle filtering. In: Proc. CVPR, pp. 2126–2133 (2000) 6. Alt, H., Knauer, C., Wenk, C.: Matching polygonal curves with respect to the fr`echet distance. In: Ferreira, A., Reichel, H. (eds.) STACS 2001. LNCS, vol. 2010, pp. 63–74. Springer, Heidelberg (2001) 7. Isard, M., Blake, A.: Condensation - conditional density propagation for visual tracking. International Journal of Computer Vision 29(1), 5–28 (1998) 8. Sidenbladh, H., Black, M., Fleet, D.: Stochastic tracking of 3d human figures using 2d image motion. In: Vernon, D. (ed.) ECCV 2000. LNCS, vol. 1842, pp. 702–718. Springer, Heidelberg (2000) 9. Mikolajczyk, K., Schmid, K., Zisserman, A.: Human detection based on a probabilistic assembly of robust part detectors. In: Proc. ECCV, vol. 1, pp. 69–82 (2003) 10. Rohr, K.: Human movement analysis based on explicit motion models. MotionBased Recognition 8, 171–198 (1997) 11. Wang, Q., Xu, G., Ai, H.: Learning object intrinsic structure for robust visual tracking. In: Proc. CVPR, vol. 2, pp. 227–233 (2003) 12. Tenenbaum, J., de Silva, V.: A global geometric framework for nonlinear dimensionality reduction. Science 290, 2319–2323 (2000) 13. Roweis, S., Saul, L.: Nonlinear dimensionality reduction by locally linear embedding. Science 290, 2323–2326 (2000) 14. Urtasun, R., Fleet, D., Fua, P.: 3d people tracking with gaussian process dynamical models. In: Proc. CVPR, vol. 1, pp. 238–245 (2006) 15. Sato, K., Aggarwal, J.: Recognizing two-person interactions in outdoor image sequences. In: IEEE Workshop on Multi-Object Tracking, IEEE Computer Society Press, Los Alamitos (2001) 16. Park, S., Aggrawal, J.: Recognition of human interactions using multiple features in a grayscale images. In: Proc. ICPR, vol. 1, pp. 51–54 (2000) 17. Hongeng, S., Bremond, F., Nevatia, R.: Representation and optimal recognition of human activities. In: Proc. CVPR, vol. 1, pp. 818–825 (2000) 18. Park, J., Park, S., Aggrawal, J.: Video retrieval of human interactions using modelbased motion tracking and multi-layer finite state automata. In: Bakker, E.M., Lew, M.S., Huang, T.S., Sebe, N., Zhou, X.S. (eds.) CIVR 2003. LNCS, vol. 2728, Springer, Heidelberg (2003) 19. Deutscher, J., Reid, I.: Articulated body motion capture by stochastic search. International Journal of Computer Vision 61(2), 185–205 (2004) 20. Balan, A., Sigal, L., Black, M.: A quantitative evaluation of video-based 3d person tracking. In: VS-PETS. IEEE Workshop on Visual Surveillance and Performance Evaluation of Tracking and Surveillance, pp. 349–356. IEEE Computer Society Press, Los Alamitos (2005)
Gait Identification Based on Multi-view Observations Using Omnidirectional Camera Kazushige Sugiura, Yasushi Makihara, and Yasushi Yagi Osaka University 8-1 Mihogaoka, Ibaraki, Osaka, 567-0047, Japan {sugiura,makihara,yagi}@am.sanken.osaka-u.ac.jp
Abstract. We propose a method of gait identification based on multiview gait images using an omnidirectional camera. We first transform omnidirectional silhouette images into panoramic ones and obtain a spatiotemporal Gait Silhouette Volume (GSV). Next, we extract frequencydomain features by Fourier analysis based on gait periods estimated by autocorrelation of the GSVs. Because the omnidirectional camera makes it possible to observe a straight-walking person from various views, multiview features can be extracted from the GSVs composed of multi-view images. In an identification phase, distance between a probe and a gallery feature of the same view is calculated, and then these for all views are integrated for matching. Experiments of gait identification including 15 subjects from 5 views demonstrate the effectiveness of the proposed method.
1
Introduction
There is a growing necessity in modern society to identify individuals in many situations, including, surveillance and access control. For personal identification, many biometrics-based authentication methods are proposed using a wide variety of cues; fingerprint, iris, face, and gait. Among these, gait identification has recently gained considerable attention because gait promises to enable surveillance systems to ascertain identity at a distance. Currently, many gait identification approaches are proposed by model base [1][2] and appearance base [3][4]. One of the difficulties facing those approches is an appearance change due to changes of viewing or walking direction. Yu et al. [5] discussed the effects of view angle variation on gait identification and reported a performance drop when view difference is large. To cope with view changes, Kale et al. [6] proposed a view transformation method based on perspective projection of the sagittal plane. The method does not, however, work well when view difference is large. Shakhnarovich et al. [7] proposed a visual hull-based method. However, the method needs multiple-view synchronized images for all subjects. As a training-based method, View Transformation Model (VTM) in the frequency domain was proposed [8]. Once the VTM is trained using sets of gait features of multiple views and subjects, a few-view reference can be transformed Y. Yagi et al. (Eds.): ACCV 2007, Part I, LNCS 4843, pp. 452–461, 2007. c Springer-Verlag Berlin Heidelberg 2007
Gait Identification Based on Multi-view Observations
453
into an arbitrary-view gallery so as to match a probe view. They also reported that verification rate increased as the number of reference views increased [9]. Moreover, a method of multi-view gait identification using walking direction changes in a sequence was proposed [10], and it was reported that the verification rate increased as the number of walking directions increased. It is, however, troublesome to capture gait images many times to acquire many references in registration phase. In addition, it is unreasonable to assume that subjects always change their walking directions enough for multi-view identification. Therefore, we propose a method of gait identification based on multi-view observations from an omnidirectional camera. Note that an omnidirectional camera makes it possible to observe multi-view gait images even if a subject walks straight. Observation views are estimated by azimuth angles of tracked person regions in the omnidirectional image and walking trajectory on the floor. Then, for each gallery and probe sequence, a silhouette-based gait features are extracted for multiple basis views which are common both for the gallery and the probe. Finally the extracted multiple gait features are matched for each same view and the matching results are integrated for better identification. The outline of this paper is as follows. First, construction of a Gait Silhouette Volume (GSV) is addressed with silhouette extraction and panoramic expansion in section 2. Next, extraction and matching of multi-view frequency-domain gait feature are described in section 3. Finally, experimental results for gait identification are presented with an analysis of the effect of view variations in section 4. Section 5 contains conclusions and discussion of further work in the area.
2 2.1
GSV Construction Extraction of Gait Silhouette Images
The first step in constructing a GSV is to extract gait silhouette images by background subtraction from omnidirectional images. First, background is modeled by average color vector u(x, y) and its covariance matrix Σ (x, y) at position (x, y) using background image sequence as follows. u (x, y) =
N 1 u(x, y, n) N n=1
(1)
1 u(x,y,n)u(x,y,n)T −u(x,y)u (x,y)T , N n=1
(2)
N
Σ (x,y) =
where u(x, y, n) is a background color vector at position (x, y) at nth frame, and N is the number of total frames for a background training sequence. Second, to extract foreground regions, Mahalanobis distance D(x, y, n) between an input image c(x, y, n) and the modeled background is calculated at each position (x, y) at each nth frame as d (x, y, n) = c(x, y, n) − u (x, y)
(3)
454
K. Sugiura, Y. Makihara, and Y. Yagi
D(x,y,n) =
d (x,y,n)T Σ (x,y)−1 d (x,y,n).
(4)
A foreground region is defined as a set of pixels whose Mahalanobis distance D(x, y, n) is larger than threshold value Dthresh . Here, the threshold Dthresh is set to be 12.0 empirically. Figure 1 shows an input image and a result of background subtraction. We can see that person region is extracted correctly. Background subtraction, however, sometimes fails because of cast shadows and illumination condition changes. To overcome such difficulties, a shadow removal is processed based on color vector angle between background and foreground. Moreover, morphological closing filter is applied to improve silhouette quality. 2.2
Panorama Extension
The second step is panorama extension of silhouettes in omnidirectional image [11]. Let P (X, Y, Z) be a point in world coordinate and p(x, y) be a point in an omnidirectional image projected from point P . Then, let ρ and Z be azimuth angle and vertical position in a cylindrical coordinate whose center axis passes through mirror focal point Om and camera center Oc , and whose radius is RP . Thus, the panorama extension is expressed as follows. tan ρ = Y /X = y/x Z = RP tan α + c where α = tan−1
(b2 +c2 ) sin γ−2bc (b2 −c2 ) cos γ
(5) (6)
and γ = tan−1 √
f x2 +y 2
are viewing directions
defined in Fig. 2 respectively, and b and c are mirror parameters. 2.3
Scaling and Registration of Silhouette Images
The third step is scaling and registration of the panoramic silhouettes to acquire normalized gait patterns.
(a) Input image with omnidirectional camera
(b) Background subtraction
Fig. 1. Result of background subtraction
Gait Identification Based on Multi-view Observations
455
Fig. 2. Projection to cylindrical surface and floor surface
(b) Panoramic image coordinate (a) Omnidirectional image coordinate Fig. 3. Definition of person region for scaling and registration
First, person regions are simply tracked in the omnidirectional image considering connected region’s area sizes, and position differences between adjacent frames. Next, in order to normalize the silhouette by person region height, the maximum radius (head point) rmax and minimum radius (foot point) rmin of the person region in polar coordinate (r, ρ) of the omnidirectional image are found (see Fig. 3(a)). Then, in order to register the horizontal position, median of azimuth angle ρmed of the person region is found (see Fig. 3(a)). Note that radius and azimuth angle are corresponds to vertical and horizontal positions in the panorama image respectively. As a result, head, foot position, and horizontal center in panorama image are represented by Zmax , Zmin , and ρmed as shown in Fig. 3(b). Second, silhouette images are scaled so that the height (Zmax − Zmin ) in panoramic image can be just 30 pixels, and so that the aspect ratio of each region can be kept. Then, we produce a 20 × 30 pixel-sized image in which the horizontal median ρmed corresponds to the horizontal center of the produced
456
K. Sugiura, Y. Makihara, and Y. Yagi
(a) front-oblique
(b) fronto-parallel
(c) rear-oblique
(d) Definition of observation view
Fig. 4. GSV examples for multiple observation views
image. A GSV is finally constructed by aligning the images on the temporal axis. Figure 4 shows GSV examples for multiple observation views. We can clearly see appearance changes in each view.
3 3.1
Multi-view Feature Extraction and Matching Frequency-Domain Feature Extraction
The second step in the proposed method is frequency-domain feature extraction from the constructed GSV. First, gait period Ngait is detected by maximizing the following normalized autocorrelation C(N ) T(N) x,y n=0g(x,y,n)g(x,y,n+N) , (7) C(N)= T(N) T(N) 2 2 g(x,y,n) g(x,y,n+N) x,y n=0 x,y n=0 of the GSV g(x, y, n) with the N frame shift for the temporal axis, where Ntotal and T (N ) = Ntotal − N − 1 is the number of total and overlapped frames in the sequence respectively. The domain of N is set to [25, 45] empirically for natural gait periods. This is because various gait types such as running, brisk walking, and ’ox walking’ are not within the scope of this paper. For the autocorrelation-based period detection, adjacent gait-period sequences need to be similar each other. We assume that a walker’s trajectory is smooth to some extent and that appearance changes between adjacent gait-period sequences are small. Next, the subsequences S ns is picked up from a complete sequence S . Note that the frame range of the subsequence S ns is [ns , ns + Ngait − 1]. A Discrete Fourier Transformation (DFT) Gns (x, y, k) for the temporal axis is then applied for the subsequence, and amplitude spectra Ans (x, y, k) are calculated as ns +Ngait −1
Gns (x, y, k) =
n=ns
g(x, y, n)e−jω0 kn
(8)
Gait Identification Based on Multi-view Observations
Ans (x, y, k) =
1 |Gns (x, y, k)|. Ngait
457
(9)
where ω0 is the base angular frequency for the gait period Ngait . In this paper, direct-current elements (k = 0) (averaged silhouette) and low-frequency elements (k = 1, 2) are chosen as experimental gait features. Let a be a feature vector composed of elements of the amplitude spectra A(x, y, k). As a results, the dimension of the feature vector a sums up to 20 × 30 × 3 = 1800. 3.2
Observation View Estimation
In this section, observation view estimation for multi-view feature extraction is addressed. The observation view θ is defined as θ = (180 − φ) + ρ
(10)
where ρ is a azimuth angle, and φ is a walking direction (see Fig. 4(d)). Azimuth angle ρ is simply defined as direction of vector (x, y), where (x, y) is a foot point in the omnidirectional image. Walking direction φ is estimated from a trajectory of subject’s foot points F (X, Y ) on a floor coordinate. Let (Rf , ρ) be polar coordinate on the floor. If the floor plane is regarded as image plane, the distance Hr from mirror focal point Om to the floor can be seen as focal length to the floor image plane. Then, radius Rf is calculated as follows [11]. Rf =
−(b2 − c2 )Hr rf (b2 + c2 )f − 2bc rf2 + f 2
(11)
Thus, walking trajectory on the floor is obtained as a time series of the above floor points (Rf , ρ). Next, walking direction φ is defined as tangential direction of the estimated walking trajectory. Let (Xn , Yn ) and (VXn , VYn ) be foot point’s position and velocity at nth frame. The velocity is introduced by central difference as follows. VXn =
Xn+Δn − Xn−Δn Yn+Δn − Yn−Δn , VYn = 2Δn 2Δn
(12)
Here, Δn is set to be 15 [frame] considering velocity smoothness. Finally, walking direction φn in nth frame is defined as direction of velocity vector (VXn , VYn ). 3.3
Multi-view Feature Extraction
In this section, multi-view feature extraction is introduced based on the estimated observation views. First, multiple basis views θi (i = 1, 2, . . .) are chosen from observation views. In this time, interval of the basis views is set to 15 deg empirically. Next, a basis frame nθi corresponding to a basis view θi is found from a complete sequence, and a subsequence is picked up as a set of Ngait frames around the basis frame nθi as shown in Fig. 5(a). Concretely speaking, the start frame s in eq. (9) is replaced by ns = nθi − Ngait /2.
458
K. Sugiura, Y. Makihara, and Y. Yagi
(a) Overview of multi-view feature extraction
(b) Multi-view features for each subject (every 15 deg)
Fig. 5. Multi-view feature extraction
Results of multi-view feature extraction for multiple subjects are shown in Fig. 5(b). In this figure, each block indicates each subject, and each row and column indicate observation view and frequency respectively. We can see individual differences, for example, swing motion difference of subject 2 and 4 in 2-times frequency of 270-deg features. In addition, we can also see view differences for each subject. Thus, by integrating the different type of features across views, gait identification performance should improve more than the case of a single-view feature. Next section gives how to match the multi-view features. 3.4
Matching Features
A matching measure between two subsequences must first be found if the proposed method is to work. Let S P and S G be complete sequences for probe and galley, respectively, and let SθPi , SθGi be their subsequences for basis angle θi , respectively. Also let a(S θi ) be feature vector for subsequence S θi . The matching measure d(S θi , S θi ) is simply chosen as the Euclidean distance as d(SθPi , SθGi ) = a(SθPi ) − a(SθGi ). Complete sequences have variations in general and may contain outliers. Because a measure candidate D(SP , SG ) can handle this noise, the median value of each subsequence result is used: D(S P , S G ) = Mediani {d (SθPi , SθGi )}
4 4.1
(13)
Experiment Datasets
A total of 60 gait sequences from 15 subjects were used for the experiments. Each sequence consisted of approximately 10 steps of a straight walk in front of the omnidirectional camera, and it included 5 basis views: 240, 255, 270,
Gait Identification Based on Multi-view Observations
459
285, and 300 deg. The camera was Sony Inc. DCR-VX2000, and images were captured by 720 × 480 pixel size at 30 fps. The hyperboloidal mirror and camera parameters were a = 13.722Cb = 11.708Cc = 18.038Cf = 427.944 (unit: mm). The dataset was taken for two days, that is, there were two sequences per day for each subject. A test set is composed of one gallery sequence of a day and two probe sequences of the other day, therefore totally four combinations of test sets were generated. 4.2
Results
The gait identification experiments were done for the above four combinations of datasets and average performance was evaluated by Receiver Operating Characteristics (ROC) curves [12]. The ROC curves shows relation between verification rate PV and false alarm rate PF when the receiver changes the acceptance thresholds. The ROC curves tilting toward left top corner in the graph indicates high performance because it means high verification rate at low false alarm rate. In addition, the effect of the number of observation views and combinations of views on performance are analyzed to validate the effectiveness of multi-view observations. First, ROC curves for each single view are illustrated in Fig. 6(a). The figure shows that performance varies greatly among basis views, and that it is difficult to gain enough performance when an arbitrary single-view feature is used for matching. Next, ROC curves for two-view combinations are illustrated in Fig. 6(b). Here, the best and the worst three combinations are shown. Note that the performance order is judged by Equal Error Rate (EER), that is, error rate when false alarm rate PF becomes equal to false rejection rate (1 − PV ). Focusing on the worst cases, view differences are small (within 15 deg except for the worst 2). On the other hand, focusing on the best cases, view differences are relatively large (more than 30 deg). As a result, it is clear that the combination with large view difference is effective for identification. Moreover, ROC curves for each number of observation views are shown in Fig. 6(c). In verification rates in this graph are averaged over all the combinations for each number of observation views. As a result, we can see that the performance becomes better as the number of observation views increase. Finally, verification rates at 3% false alarm rate are picked up for each number of observation views. Figure 6(d) shows of the best, the worst, and the averaged performance over all the combinations. As for the best combinations, the verification rate for two observation views reaches the highest performance. Thus a small number of observation views are enough when the combination can be specified. As for the worst combination, the verification rates make a steady progress as the number of observation views increase. In case of the worst, because combinations are usually composed of adjacent views as known from two views combination case, the increase of the number of observation views directly leads to observation views variation. In summary, it is validated that observation view variations greatly contribute to performance improvement.
460
K. Sugiura, Y. Makihara, and Y. Yagi
(a) ROC curves for single views
(b) ROC curves for two views
(c) ROC curves for each number of views
(d) Verification rate at 3% false alarm for each number of views
Fig. 6. Experimental results
5
Conclusion
This paper describes a method of gait identification based on multi-view gait images using an omnidirectional camera. The omnidirectional silhouette images first transformed into panoramic ones and a spatio-temporal Gait Silhouette Volume (GSV) is obtained. Next, frequency-domain features are extracted by Fourier analysis. Because the omnidirectional camera makes it possible to observe a person from various views, multi-view features can be extracted from the GSVs composed of multi-view images. In an identification phase, distance between a probe and a gallery feature of the same view is calculated, and then these for all views are integrated for matching. The effect of observation view variation on gait identification performance was analyzed through experiments including 15 subjects from 5 views. As a result, average performance increases from 82% (single view) to 93% (5 views), and it was clear that observation view variation contributes to gait identification performance. In this paper, basis views are chosen only from views common for a gallery and a probe. It is possible to use other-view features interpolated by View Transformation Model (VTM) [8] for better performance in a future work. Moreover, subjects in this experiment walked within 5m from the omnidirectional camera, and thus relatively high-resolution silhouettes (approximately 60 pixel height) are obtained. Therefore, effects of distance from the camera or silhouette resolution on identification performance should be analyzed. That also leads to analysis of the optimal alignment of the omnidirectional camera to capture multi-view
Gait Identification Based on Multi-view Observations
461
gait images effectively considering both silhouette resolution and observation view variation.
References 1. Urtasun, R., Fua, P.: 3d tracking for gait characterization and recognition. In: Proc. of the 6th IEEE Int. Conf. on Automatic Face and Gesture Recognition, pp. 17–22. IEEE Computer Society Press, Los Alamitos (2004) 2. Yam, C., Nixon, M., Carter, J.: Automated person recognition by walking and running via model-based approaches. Pattern Recognition 37(5), 1057–1072 (2004) 3. Sarkar, S., Phillips, J., Liu, Z., Vega, I., Grother, P., Bowyer, K.: The humanid gait challenge problem: Data sets, performance, and analysis. Trans. of Pattern Analysis and Machine Intelligence 27(2), 162–177 (2005) 4. Han, J., Bhanu, B.: Individual recognition using gait energy image. Trans. on Pattern Analysis and Machine Intelligence 28(2), 316–322 (2006) 5. Yu, S., Tan, D., Tan, T.: Modelling the effect of view angle variation on appearancebased gait recognition. In: Proc. of 7th Asian Conf. on Computer Vision, vol. 1, pp. 807–816 (2006) 6. Kale, A., Roy-Chowdhury, A., Chellappa, R.: Towards a view invariant gait recognition algorithm. In: Proc. of IEEE Conf. on Advanced Video and Signal Based Surveillance, pp. 143–150. IEEE Computer Society Press, Los Alamitos (2003) 7. Shakhnarovich, G., Lee, L., Darrell, T.: Integrated face and gait recognition from multiple views. In: Proc. of IEEE Conf. on Computer Vision and Pattern Recognition, vol. 1, pp. 439–446 (2001) 8. Makihara, Y., Sagawa, R., Mukaigawa, Y., Echigo, T., Yagi, Y.: Gait recognition using a view transformation model in the frequency domain. In: Proc. of the 9th European Conf. on Computer Vision, Graz, Austria, vol. 3, pp. 151–163 (2006) 9. Makihara, Y., Sagawa, R., Mukaigawa, Y., Echigo, T., Yagi, Y.: Which reference view is effective for gait identification using a view transformation model? In: Proc. of the IEEE Computer Society Workshop on Biometrics 2006, New York, USA (2006) 10. Makihara, Y., Sagawa, R., Mukaigawa, Y., Echigo, T., Yagi, Y.: Adaptation to walking direction changes for gait identification. In: Proc. of the 18th Int. Conf. on Pattern Recognition, Hong Kong, China, vol. 2, pp. 96–99 (2006) 11. Yamazawa, K., Yagi, Y., Yachida, M.: Hyperomni vision: Visual navigation with an omnidirectional image sensor. Systems and Computers in Japan 28(4), 36–47 (1997) 12. Phillips, P., Moon, H., Rizvi, S., Rauss, P.: The feret evaluation methodology for face-recognition algorithms. Trans. of Pattern Analysis and Machine Intelligence 22(10), 1090–1104 (2000)
Gender Classification Based on Fusion of Multi-view Gait Sequences Guochang Huang and Yunhong Wang Intelligent Recognition and Image Processing Lab, School of Computer Science and Engineering, Beihang University, Beijing 100083, China gc [email protected], [email protected]
Abstract. In this paper, we present a new method for gender classification based on fusion of multi-view gait sequences. For each silhouette of gait sequences, we first use a simple method to divide the silhouette into 7 (for 90 degree, i.e. fronto-parallel view) or 5 (for 0 and 180 degree, i.e. front view and back view) parts, and then fit ellipses to each of the regions. Next, the features are extracted from each sequence by computing the ellipse parameters. For each view angle, every subject’s features are normalized and combined as a feature vector. The combination of feature vector contains enough information to perform well on gender recognition. Sum rule and SVM are applied to fuse the similarity measures from 0o , 90o , and 180o . We carried our experiments on CASIA Gait Database, one of the largest gait databases as we know, and achieved the classification accuracy of 89.5%.
1
Introduction
Gait is an attractive biometric feature for human recognition and classification. In recent years, gait receives more and more attentions from computer vision and biometric researchers. Compared with other biometric features, such as fingerprint, face, and iris, gait has many good qualities such as non-invasive, capture at distance and non-perceivable. Gait analysis plays an important role in surveillance. Gait analysis mainly consists of two areas, one is gait recognition, which is to identify subject’s ID in the specific environment where the other biometric is very difficult to be captured. The other is gait classification, including gender recognition, action classification, and estimation of age. Meanwhile gender classification has attracted much attention recently [2,4], due to its widely potential application. Researching on gait recognition and gait classification have a long history, but most of the articles about gait analysis focus on human recognition, the articles that about how to use gait to classify gender is few. In complex real surveillance scenarios, describing the subject’s features, such as gender and age, is very important and necessary. Because in these specific environments, it is very difficult or even impossible to capture other available features to correctly identify the subject’s ID. Y. Yagi et al. (Eds.): ACCV 2007, Part I, LNCS 4843, pp. 462–471, 2007. c Springer-Verlag Berlin Heidelberg 2007
Gender Classification Based on Fusion of Multi-view Gait Sequences
463
The remainder of this paper is organized as follows, Section 2 summarizes the related work on the gender recognition. Gait representation and feature extraction are described in Section 3. Section 4 provides classification and fusion scheme. Experiments and results are presented in Section 5, followed by the conclusions in Section 6.
2
Related Work
Point-light display was an important method for walking manner researches and received much attention during the pass few decades. Kozlowski and Cutting [3] were the first researchers who began to study on gender recognition from human walking manner. They demonstrated that observers are able to recognize the point-light walkers’ gender. Barclay et al. [1] conducted a further research on gender recognition, they investigated the influenced of spatial and temporal factors to the correct rate. The result shows that at least two gait cycles are necessary for successful gender recognition, the speed of walkers also has great influence for on classification accuracy. In their experiments, the highest recognition accuracy is 68%. Most of the studies were based on the side-view presentation of walkers to observers, on the other hand some experiments were try to find the effect of view angle on gender recognition [1,8]. It was found that the front-view presentation contains more information than side-view for gender recognition [8]. In our method, we combine the front-view, back-view, and side-view presentation of gait, and thus conducts a higher correct rate then ever before. Troje [8] recently used linear pattern recognition technique to deal with the analysis of biological motion, and presented a two-stage PCA framework for recognizing gender. He reported 92.5% recognition rates. Davis and Gao [2] used three-mode PCA model basing point-light walkers to do gender recognition. In their experiments, there are 40 walkers, and the best recognition rate is 95.5%. But in real surveillance environments, it is very difficult to attached small point lights to the main joints of subject’s body, so weather this approach will wok at good performance on video-based gait database is unknown. Most of the aforementioned studies used point-light display to form the aspect of biological motion, as mentioned above, the point-light display has fatal limitation in surveillance environments. Lee and Grimson [5,4] recently proposed a computer vision algorithm to extract visual feature of gaits from image sequences for gender recognition. Their experiments performed on a database of 24 objects and the approach achieved the recognition rate of 84%.
3
Gait Representation
In our method, three view angles are chose,including 0o (front view), 90o (frontoparallel view), and 180o(back view). A lot of researches have demonstrated that the front- view presentation contains more information for gender classification, and some research[11] show that fusion of gait sequences with an angle difference
464
G. Huang and Y. Wang
greater than or equal to 90o can achieve better improvement than fusion of those with an acute angle. Therefore, 0o , 90o and 180o are chose to do gender classification in our method. First of all, we assume that silhouettes have been extracted from original video files. And then, the extracted silhouettes sequences are normalized to the same size, so all silhouettes have the same height.Horizontal center of normalized silhouettes are obtained and all the normalized silhouettes are aligned according to the horizontal center.
Fig. 1. Examples of normalized silhouettes in different view angle from the same subject. The top row shows the 0o normalized silhouettes, the middle row shows the 90o , and the bottom row shows the 180o .
The gait image representation method is first proposed by Lee [5]. but in Lee’s thesis, she only applied this representation method on 90o gait images, in this paper, we apply the method to divide the 0o and 180o images into five parts, and the experimental results demonstrate that the segmentation way of 0o and 180o image is effective in gender classification. For the gait images of 0o and 180o, we proportionally divide the silhouette into 5 parts as shown in Fig.2 (a). These 5 regions roughly correspond to: Z1 , head region; Z2 , left of torso; Z3 , right of torso; Z4 , left leg; Z5 , right leg. For the gait images of 90o , we proportionally divide the silhouette into 7 parts as shown in Fig.3 (a). These 7 regions roughly correspond to: N1 , head region; N2 , front of torso; N3 , back of torso; N4 , front thigh; N5 , back thigh; N6 , front foot; N7 , back foot. For each of the parts, we fit an ellipse to the foreground in this region, as shown in Fig.2 (b) and Fig.3 (b). The intuition behind the segmentation of 0o and 180o silhouette is that men show a larger extent of lateral sway of the upper body than women do, and the orientation of the major axis can reflect this phenomenon. The difference of the shoulder-hip ratio between men and women also can be reflected from the orientation of the major axis. For facilitating the expression of each part of the silhouette, we divide the 90o silhouettes into 7 regions, each region roughly correspond to one part of human body. For each ellipse fitted to these regions, we compute four parameters, including ¯ Y¯ ), the elongation of the ellipse (L), and the orientation of the the centroid (X,
Gender Classification Based on Fusion of Multi-view Gait Sequences
465
Fig. 2. Example of the 0 and 180 degree silhouette which is divided into 5 regions, and five ellipses are fitted to these regions
Fig. 3. Example of the 90 degree silhouette which is divided into 7 regions, and seven ellipses are fitted to these regions
major axis (α). The details about how to calculate these four parameters is completely included in [4], and we do not present it here. For each region there are 4 parameters, they form the region feature vector Ri ¯ i , Y¯i , Li , αi ) Ri = (X
(1)
Where i = 1, . . . , 5 (for 0o and 180o images)or 7(for 90o images).And for each image there are 20 parameters (5 regions × 4 parameters) or 28 parameters (7 regions × 4 parameters), these parameters form the image feature vector Ij ¯1 , Y¯1 , L1 , α1 , . . . , X ¯ 5(7) , Y¯5(7) , L5(7) , α5(7) )j Ij = (R1 , . . . , R5(or7) )j = (X
(2)
Where j = 1, . . . , n, n is the total number of the images in one gait sequence. By computing the mean value of image feature vectors in one sequence, we get the sequence feature vector Sp (k) Sp (k) = mean(I1 (k), . . . , In (k))p
(3)
For example ¯ 1 ) = mean(I1 (X ¯ 1 ), . . . , In (X ¯ 1 ))p = mean(I1 (1), . . . , In (1))p (4) Sp (1) = Sp (X Where p is the index of sequences, p = 1, . . . , total number of sequences, n is the total number of images from one sequences, and k is the index of features,
466
G. Huang and Y. Wang
k = 1, . . . , 20 (or 28), so there are totally 20 features for each 0o and 180o sequence and 28 features for each 90o sequence.
4 4.1
Gender Classification Similarity Measure
When the sequence feature vectors are obtained for each subject, the similarity measures are calculated next. Here, the calculation of similarity measures is applied on three view angles, respectively. We randomly choose some male and female subjects from the database to construct testing and training set. In both testing and training sets, the number of female subjects must equates to the number of male subjects, because if the number of female and male do not equate, the experimental results may be influenced by the separability of the larger class. According to the gender attribute, training set is further divided into female training subset and male training subset. Mean Euclidean distance from testing set to the female training subset or male training subset is calculated. Let M be the total number of female (or male) training sequences, and St (k) be the number k feature of number t testing sequence. The Mean Euclidean Distance of number k feature between testing sequence and female training set, DFt (k), is defined as 1 DFt (k) = Euclidean(St(k), Sn (k)) (5) M Where n = 1, . . . , M , Sn ∈ female training set, St ∈ testing set, and k = 1, . . . , 20 (or 28). And the Mean Euclidean Distance of number k feature between testing sequence and male training set, DMt (k), is defined as DMt (k) =
1 Euclidean(St (k), Sm (k)) M
(6)
Where m = 1, . . . , M , Sm ∈ male training set, St ∈ testing set, and k = 1, . . . , 20 (or 28). Both distance, DFt and DMt , are regarded as the similarity measures of number t sequence, they mean the similar degree between testing sequence and two subsets. DFt and DMt are 20 (or 28) dimensional vectors. 4.2
Fusion Scheme
Based on the similarity measure algorithm which is introduced in 4.1, we get two vectors, DFt and DMt , for each angle. So there are total three female vectors o o o o o o and three male vectors, DFt0 , DFt90 , DFt180 , DMt0 , DMt90 , DMt180 . We concatenate three female vectors to one vector and three male vectors to another vector, respectively.
Gender Classification Based on Fusion of Multi-view Gait Sequences o
o
o
CFt = concatenate(DFt0 , DFt90 , DFt180 ) o
o
o
CMt = concatenate(DMt0 , DMt90 , DMt180 ) o
o
467
(7) (8)
o
DFt0 and DFt180 are 20 dimensional vectors and DFt90 is 28 dimensional vector, so CFt is 68 dimensional vectors, and the same to CMt . Before fusion the similarity measures of all feature, we normalize these similarity measures to common a range [0, 1] by using the Min-Max normalization method [6] CFt (k) − min (9) CFt (k) = max − min CMt (k) − min (10) max − min Where max and min denote the maximum and the minimum value of number k feature in number t sequence,respectively. CFt (k) and CMt (k) denote the normalized gender similarity measures of number k feature in number t sequence. CMt (k) =
Sum Rule. Snelick et al. [7] found that the Min-Max normalization followed by the sum of scores fusion method outperform other schemes. So we adopt this fusion scheme. N CFt (k) (11) CFt = k=1
CMt =
N
CMt (k)
(12)
k=1
Where N denotes the dimension of CFt and CMt , N = 68. CFt and CMt are the similarity measures of gender. Comparing CFt and CMt , we can make the decision that whether testing sequence belong to female or male. Testing sequence is considered to have the same gender attribute with the minimum of CFt and CMt f emale, CFt < CMt gender = (13) male, CFt > CMt . Support Vector Machine (SVM). SVM method attempts to maximize the distance between the hyperplane and the closest training samples on either side of the hyperplane. It is a powerful technique for classification and in particular, for solving binary classification problems. We choose SVM to be the classifier. First, we construct feature vectors for training and testing SVM classifier. We concatenate the normalized vectors, CFt and CMt , into one vector, which will be used to represent the sequence. Gt = concatenate(CFt , CMt )
(14)
468
G. Huang and Y. Wang
We construct a feature vector G for each training sequence. And then, Gt (where t = 1, . . ., total number of training sequence) can be used as the input vector to train SVM classifier.
5 5.1
Experiments and Results Data
We carried our experiments on the CASIA Gait Database [10], one of the largest shared databases in current gait-research community. There are 124 subjects in the database, of which 93 were male and 31 were female. We only chose the normal sequences from 0 degree, 90 degree, and 180 degree to construct experimental data set. There are totally 2, 232 (124 subjects × 6 sequences × 3 angles) video sequences in our experimental data set. As mentioned above, if the numbers of female and male are unbalance in training and testing sets, the experimental results may be influenced by the separability of the larger class. In CASIA database, there are only 31 females, so, in our experiments, we randomly choose 25 males and 25 females from the database to form a training set and randomly choose 5 males and 5 females from the rest subjects to form a training set. So there are totally 600(50×6) sequences in training set and 60(10×6) sequences in testing set. If the subject is assigned to training set or testing set, all the sequences of this subject are assigned into corresponding set too. 5.2
Experimental Results
Since the training set and testing set are chose randomly, we repeat our experiments two hundred times and use the mean value of these experimental results to be the terminal recognition accuracy. So the recognition rates which are listed here can correctly reflect the performance of our method. First, we use Sum Rule scheme to fuse 20 features from 0 degree view angle, 28 features from 90 degree view angle, and 20 features from 180 degree view angle, respectively, in order to see what performance can be achieved before fusion all the three view angles. The recognition results are listed in Table 1. And then, we use Sum Rule scheme to fuse all the features from three view angles, total of 68 features, and achieve the recognition accuracy of 87.7%. Fig.4 Table 1. Results of respectively fuse the features of three view angles by using sum rule View angle (degree) Recognition rate (%) 0 83.0 90 85.5 180 85.5
Gender Classification Based on Fusion of Multi-view Gait Sequences
469
Fig. 4. Two-dimensional scatter-plots showing the testing data and decision boundary. Red ’x’ denotes male and green ’o’ demotes female. Table 2. Results of respectively fuse the features of three view angles by using Linear, Polynomial (d = 2), RBF (g = 1) kernels. (d = degree of polynomial, g = width of RBF network). View angle (degree) Kernel Recognition rate (%) 0 Linear 79.5 0 Polynomial 88.0 0 RBF 80.0 90 Linear 82.0 90 Polynomial 85.0 90 RBF 82.5 180 Linear 86.0 180 Polynomial 88.0 180 RBF 86.0 Table 3. Results of fuse all the features from three view angles by using using Linear, Polynomial (d = 2), RBF (g = 1) kernels Kernel Recognition rate (%) Linear 89.5 Polynomial 89.5 RBF 88.5
shows scatter plot of the test data and decision boundary. The x axis denotes the dissimilarity measure of male, in other words, it means the distance between testing subject and male; and y axis denotes the dissimilarity measure of female, similarly, it means the distance between testing subject and female. Three kernels, including Linear, 2nd degree Polynomial, and RBF (width of RBF network = 1) kernels, are used to train SVM. The results of respectively fuse the features of three view angles are shown in Table 2. And the results of fuse all the features from three view angles are shown in Table 3.
470
G. Huang and Y. Wang Table 4. Comparison of experimental results
Recognition rate(%) 63% 92.5% 84% 95.5% 89.5%
5.3
Authors Kozlowski and Cutting (1977) Troje (2002) Lee and Grimson (2002) Davis and Gao (2004) This paper
Database Representation View 6 subjects Point-light Side-view 40 subjects Point-light Mixed 24 subjects Video-based Side-view 40 subjects Point-light Front-view 124 subjects Video-based Mixed
Discussions
Based on the above results, the conclusions are draw as follows: using fusion schemes can improve the performance of gender recognition systems, especially by suing SVM fusion scheme the performance achieved 89.5%. Comparing the results of using Sum Rule and SVM scheme, we note that SVM scheme has more advantages than Sum Rule. And SVM scheme shows an improvement of up to 5% over Sum Rule. According to Table 2, we can find that 2nd degree Polynomial kernel performs better than other kernels, this leads us to believe that Polynomial kernel is more effective to gender recognition task. In Table 3, the performance of the three kernels show that the Linear kernel performed as well as the 2nd degree Polynomial kernels. Compared with other methods, the recognition rate of our method is higher than most of other methods. Table 4 shows the recognition accuracy of our method and other related methods. From Table 4 we note that although our recognition rate is a little lower than Davis’s and Troje’s , our experiments are applied on large database and based on video sequences. From this point, our method is more suitable to do gender recognition in surveillance environments. We also realize Lee and Grimson’s method and carry it on the same CASIA database, the recognition rate is 85%, which equates to the result of only using 90o sequences in our experiments. The rest methods in Table 4 are hard to be realized and ran on the same CASIA database, because they use point-light representation method. In this representation method, points of light must be attached to body joints in order to accurately locate the joints position, but in video sequences it is hard or even impossible to accurately locate the joints position. So it is meaningless to realize these methods and compare them with our method on the same CASIA database.
6
Conclusion
In this paper, we have presented a new gender classification scheme based on fusion the similarity measures from multi-view gait sequences. From the experimental results, we can see that our method achieved higher performance than most of other methods and more suitable for surveillance scenarios. The proposed fusion method can help improve the performance of gender classification.
Gender Classification Based on Fusion of Multi-view Gait Sequences
471
Acknowledgments This work was supported by the program of New Century Excellent Talents in University, National Nature Science Foundation of China (No. 60332010), Joint Project supported by Nation Science Foundation of China and Royal Society of UK (No. 60710059), and Hi-Tech Research and Development Program of China (No. 2006AA01Z133). Portions of the research in this paper use the CASIA Gait Database collected by Institute of Automation, Chinese Academy of Sciences.
References 1. Barclay, C.D., Cutting, J.E., Kozlowski, L.T.: Temporal and spatial actors in gait perception that influence gender recognition. Perception & Psychophysics 23(2), 145–152 (1978) 2. Davis, J.W., Gao, H.: Gender recognition from walking movements using adaptive three-mode pca. In: IEEE CVPR Workshop on Articu-lated and Nonrigid Motion, IEEE Computer Society Press, Los Alamitos (2004) 3. Kozlowski, L.T., Cutting, J.E.: Recognizing the sex of a walker from dynamic point-light display. Perception & Psychophysics 21(6), 575–580 (1977) 4. Lee, L.: Gait Analysis for Classification. Technical report, MIT AI Lab (2003) 5. Lee, L., Grimson, W.E.L.: Gait analysis for recognition and classification. In: FG. IEEE International Conference on Automatic Face Gesture Recognition, IEEE Computer Society Press, Los Alamitos (2002) 6. Nandakumar, K., Jain, A.K., Ross, A.A.: Score normalization in multimodal biometric systems. Pattern Recognition 38(1212), 2270–2285 (2005) 7. Snelick, R., Indovina, M., Yen, J., Mink, A.: Multimodal biometrics: Issues in design and testing. In: Proceedings of Fifth International Conference on Multimodal Interfaces. Vancouver (2003) 8. Troje, N.F.: Decomposing biological motion: A framework for analysis and synthesis of human gait patterns. Journal of Vision 2(5), 371–387 (2002) 9. Wang, Y., Yu, S., Wang, Y., Tan, T.: Gait recognition based on fusion of multi-view gait sequences. In: Zhang, D., Jain, A.K. (eds.) Advances in Biometrics. LNCS, vol. 3832, Springer, Heidelberg (2005) 10. Yu, S., Tan, D., Tan, T.: A framework for evaluating the effect of view angle, clothing and carrying condition on gait recognition. In: ICPR 2006. Proc. of the 18’th International Conference on Pattern Recognition, Hong Kong, China (2006) 11. CASIA Gait Database, http://www.sinobiometrics.com
MAPACo-Training: A Novel Online Learning Algorithm of Behavior Models Heping Li1,2 , Zhanyi Hu1 , Yihong Wu1 , and Fuchao Wu1 National Laboratory of Pattern Recognition Digital Content Technology Research Center, Institute of Automation,Chinese Academy of Sciences, P.O. 2728, Beijing 100080, P.R. China {hpli,huzy,yhwu,fcwu}@nlpr.ia.ac.cn 1
2
Abstract. The traditional co-training algorithm, which needs a great number of unlabeled examples in advance and then trains classifiers by iterative learning approach, is not suitable for online learning of classifiers. To overcome this barrier, we propose a novel semi-supervised learning algorithm, called MAPACo-Training, by combining the co-training with the principle of Maximum A Posteriori adaptation. This MAPACoTraining algorithm is an online multi-class learning algorithm, and has been successfully applied to online learning of behaviors modeled by Hidden Markov Model. The proposed algorithm is tested with the Li’s database as well as Schuldt’s dataset.
1
Introduction
Behavior modeling is driven by a wide range of applications, such as advanced user interface, visual surveillance, virtual reality and so on. The most existing works in this field focused on modeling the behaviors with manually labeling like [1,2,3,4]. For example, Li and Greenspan [1] built a multi-scale model from timevarying contours and Gong and Xiang [2] learned a Dynamically Multi-Linked Hidden Markov Model (DML-HMM). However, manual labeling of behavior patterns is laborious, impractical and error prone [5]. Recently, some behavior modeling methods based on semi-supervised/unsupervised learning [5,6,7,8] have been proposed. For instance, Xiang and Gong [5] discovered natural grouping of behavior patterns through unsupervised model selection and feature selection, and Zelnik-Manor and Irani [6] used the normalized-cut approach to automatically cluster the data and then build the statistical behavior model. Unfortunately, these methods need to get a great number of unlabeled examples beforehand, which are therefor unsuitable for online learning of behavior models and cannot automatically adjust the models’ parameters according to the circumstantial changes. The co-training approach proposed by Blum and Mitchell [9] is also a semisupervised learning method. Levin et al. [10] used the co-training framework in the context of boosted binary classifiers to build the automobile detectors. Y. Yagi et al. (Eds.): ACCV 2007, Part I, LNCS 4843, pp. 472–481, 2007. c Springer-Verlag Berlin Heidelberg 2007
MAPACo-Training: A Novel Online Learning Algorithm of Behavior Models
473
And Yan and Naphade [11] proposed a multi-view semi-supervised learning algorithm which avoids the requirement of the co-training approach about that each view of examples is sufficient for learning the target concepts. However, these methods belong to the off-line learning category. Javed et al. [12] combined the co-training approach and boosting to propose an algorithm for online detection and classification of moving objects, where behavior modeling is not considered. In this paper, we present a novel semi-supervised learning method called MAPACo-Training which combines the co-training approach and the principle of Maximum A Posteriori adaptation [8,13,16]. The proposed method can simultaneously train the parameters of multi-class models. We have successfully applied the method to online learning the parameters of behaviors modeled by Hidden Markov Model (HMM). Since it only needs a small labeled sample set beforehand, our method can alleviate the problem in the methods [1,2,3,4]. And unlike the approaches [5,6,7,8], the method can automatically adjust the parameters with the current example online. The remainder of this paper is organized as follows: Motion signature representation is outlined in Section 2. Section 3 is a detailed description of MAPACoTraining. Experimental results are reported in section 4. And conclusions as well as future research directions are listed in section 5.
2 2.1
Motion Signature Representation Feature Extraction
Background subtraction is used to detect foreground. In our approach, two types of features are considered: (1) shape feature; (2) optical flow feature [14]. The size of the foreground region varies with the distance of object to camera, camera parameters and the size of object. We therefore need to normalize the foreground region. Firstly, we equidistantly divided the bounded rectangle of the foreground into U × V non-overlapping sub-blocks. Then, the normalized value of each sub-block is calculated as follows: x1i = s− sub(i)/max,
i = 1, 2, . . . , num,
(1)
where num = U × V is the number of sub-blocks; s sub(i) is the number of the foreground pixels in the ith sub-block; max is the maximum value of {s sub(i), i = 1, 2, . . . , num}. The optical flow value of each sub-block is calculated as follows: xji = f− sub(i, j)/sum(i), i = 1, 2, . . . , num, j = 2, 3, (2) where f sub(i,j) with j = 2, 3 are respectively the sum of horizontal, vertical optical flow in the ith sub-block; sum(i) is the pixel number in the ith sub-block. Then, the feature vectors at frame t from shape and optical flows are as: odt = [xd1 , xd2 , . . . , xdnum ],
d = 1, 2, 3.
474
2.2
H. Li et al.
Motion Signature Representation
Given the observation feature sequences OTd = {od1 , od2 , · · · , odt , · · · , odT }, two different Hidden Markov Models (HMMs) are adopted to build the behavior models. The one is a single continuous HMM from shape, and the other is like a Parallel Hidden Markov Model (PHMM) with two continuous HMMs from optical flow. These HMM topologies are shown in Figure 1, where shade circles are as observation nodes and clear circles as hidden nodes. For optical flow model, the two HMMs are learned independently. The output probability density function of learning is the following Gaussian Mixture Model (GMM): p(odt |θ) =
K
αk pk (odt |μk , Σk )
(3)
k=1
where θ = {αk , μk , Σk , k = 1, 2, . . . , K} represents the parameters of GMM including weight αk , mean value μk and covariance matrix Σk of every mixture K component; αk = 1. k=1
(a)
(b)
Fig. 1. HMM topology: (a) shape model; (b) optical flow model
By the Forward procedure, we compute the observation probabilities P (OTd |λdc ), c = 1, 2, . . . , C for the observation feature sequences OTd , where C is the class number of behaviors and λdc is the HMM parameter set of the cth class behavior. Since the output probability density function is GMM, the probabilities are normalized [15] as follows: P¯ (OTd |λdi ) = P (OTd |λdi )/
C c=1
P (OTd |λdc ).
(4)
And for optical flow model (Figure 1(b)), the following operation is further performed as: P¯ (OT2 |λ2i )P¯ (OT3 |λ3i ) P¯ (OT2,3 |λ2,3 . i ) = C ¯ 2 2 ¯ 3 3 c=1 [P (OT |λc )P (OT |λc )]
(5)
The Bayes classifier is used as our base classifier. According to the Bayes rule, the posterior
MAPACo-Training: A Novel Online Learning Algorithm of Behavior Models
475
P (c|OTd ) = P¯ (OTd |λdc )P (c)/P (OTd ), where P (c) = 1/C, so P (c|OTd ) ∝ P¯ (OTd |λdc ). Thus if P¯ (OTd |λdc0 ) = max P¯ (OTd |λdc ), c
OTd belongs to the c0 –th behavior. In our proposed algorithm, we set f1i = P¯ (OT1 |λ1i ) and f2i = P¯ (OT2,3 |λ2,3 i ).
3
MAPACo-Training
In this section, we propose a new semi-supervised learning algorithm called Maximum A Posteriori Adaptation Co-Training (MAPACo-Training) which attempts to learn behavior models online. We first describe the principle of MAP adaptation, and then give the details of MAPACo-Training. 3.1
MAP Adaptation
MAP adaptation has widely been used in speaker and face verification [13]. Recently, Zhang et al. in [8,16] used it for unusual event detection and meeting event recognition. During the course of learning the parameters of GMM-based HMM in [8,16], the state-transition probabilities are kept fixed while the mean, variance and mixture weights are adapted as follows: (1) According to the existing parameters, new statistical values are computed: K d d αk pk (odt |μk , Σk ) (6) P (i|ot ) = αi pi (ot |μi , Σi ) k=1
αnew i = μnew i
T t=1
T Σinew =
=
t=1
T t=1
P (i|odt )
T
T odt P (i|odt )
t=1
(7) P (i|odt )
P (i|odt )(odt − μnew )(odt − μnew )T i i T d t=1 P (i|ot )
(8)
(9)
(2) New parameters are estimated as follows: + (1 − ρ) · αold α ˆ i = ρ · αnew i i
(10)
+ (1 − ρ) · μold μ ˆi = ρ · μnew i i
(11)
T ˆi = ρ · Σinew + (1 − ρ) · [Σiold + (ˆ Σ μi − μold μi − μold i )(ˆ i ) ]
(12)
where ρ(0 ≤ ρ ≤ 1) is the scale factor. We use the principle of MAP adaptation into our algorithm. More details about MAP adaptation can be found in [8,13,16].
476
3.2
H. Li et al.
MAPACo-Training Algorithm
The traditional co-training algorithm [9] needs to get a great number of unlabeled samples in advance and then train models by an approach of iterative learning. It is an off-line learning method. By combining the co-training and the MAP adaptation, we propose a novel online multi-class learning algorithm called MAPACo-Training as follows: Input: Labeled data L including a small training sample set Ltr and a small validation sample set Lv with two views V1 and V2 , threshold value Th > 1 and Tnum ≥ 1. Output: a classifier from the probabilitiesf 1 , f 2 , ..., f C . MAPACo-Training 1. Create f1i and f2i (i = 1, 2, . . . , C) using Ltr on V1 and V2 . Set the new training sample set of each one of the C classes Lib = φ (b=1,2); 2. For k = 1, 2, . . . , C (a) For current sample S, assume n = max{f1j ,j = 1, 2, . . . , C, j = k}, j
m = max{f2j , j = 1, 2, . . . , C, j = k}, j
i. if f1k /f1n ≥ Th , the view V2 of sample S is added into Lk2 as a new sample; ii. if f2k /f2m ≥ Th , the view V1 of sample S is added into Lk1 as a new sample; iii. if 1 < f1k /f1n < Th and 1 < f2k /f2m < Th , the view V1 of sample S is added into Lk1 as a new sample and the view V2 of sample S is added into Lk2 as a new sample. (b) if the sample number in Lkb equals Tnum , the parameters of model fbk are updated according to MAP equations (6)∼(12) with these samples in Lkb and the scale factor ρ is decided by validation sample set Lv . And then let Lkb = φ. 3. Combine f i = ω1 f1i + ω2 f2i (ω1 + ω2 = 1) using Lv . 4. Create a new classifier using f i according to the Bayes theory. Similar to co-training, two base classifiers of every class model need to be trained on separate features of the same sample. How to select samples to train the models? In this algorithm, we use a threshold Th to do it. The conditions (i)(ii) show if one base classifier can predict the label of the sample confidently, then we add this sample into the training set of the other base classifier of the corresponding class. The condition (iii) means that both base classifiers can get the same label according to the bayes rule, but neither of them is confident, which shows the sample is useful for improving the performance of the two classifiers. During the course of updating parameters by MAP adaptation equations (6)∼(12), we use validation set Lv to decide the scale factor ρ. If the class prediction for a sample from the conditions (i)∼(iii) is not correct, which means the sample is a noise, the sample is no longer used for further learning by setting
MAPACo-Training: A Novel Online Learning Algorithm of Behavior Models
477
ρ = 0 according to Lv . In our experiment, we assume the possible value of ρ is 0 or a constant¯ ρ (0 ≤ ρ¯ ≤ 0.5). Remark: From the equations (6)∼(12), we can see that the MAP adaptation only uses the current samples to calculate the new statistical values and then gets the new parameters by simple weighted estimation. It avoids to directly train the HMM parameters from a great number of samples by EM algorithm and improves the computational efficiency. The MAPACo-Training algorithm starts from a small label sample set Ltr and then updates the parameters by the MAP adaptation. So the algorithm is suitable for online multi-class learning.
4
Experiments
We test our method from two datasets: Li’s dataset [17] and Schuldt’s dataset [18]. In the experiments, U =9 and V =5 are used for dividing the bounded rectangle of foreground. To each type of features such as shape, horizontal optical flow and vertical optical flow, the Principal Component Analysis (PCA) is used to reduce the 45-dimensional features to the 8-dimensional ones. 4.1
Results on Li’s Dataset
38
24
8
36
22
7
34
20
32
18
30
16
28
14
26
12
24
10
22 0
500
1000
1500
2000
6
HTER(%)
HTER(%)
HTER(%)
We get a video consisting of five kinds of behaviors from Li’s dataset [17], of which each one is performed by 18 subjects. Image size is of 160 × 120 pixels and frame rate is of 6 frames/sec. The video totals 38120 frames including “box”
8 0
2500
5
4
3
2
500
number of samples
1000
1500
2000
1 0
2500
500
number of samples
(a)
(b)
22
1000
1500
2000
2500
number of samples
(c)
25
22
20 20 18
14
18
12 10
HTER(%)
20
HTER(%)
HTER(%)
16
15
16
14
8 12 6 4 0
500
1000
1500
number of samples
(d)
2000
2500
10 0
500
1000
1500
number of samples
(e)
2000
2500
10 0
500
1000
1500
2000
2500
number of samples
(f)
Fig. 2. The learning curves: (a) box; (b) kick; (c) lookround; (d) standup (e) wave; (f) average HTER
478
H. Li et al. Table 1. Initial confusion matrix box kick lookround standup wave
box 37.14 12.86 1.43 1.43 16.67
kick 34.29 72.86 1.90 33.80 2.38
lookround 7.62 1.90 92.86 0.48 14.76
standup 8.57 12.38 3.33 63.81 5.24
wave 12.38 0 0.48 0.48 60.95
Table 2. Final confusion matrix box kick lookround standup wave
box 54.76 0 0 0.95 10.95
kick 18.10 86.67 0 3.80 0.95
lookround 3.33 1.43 95.72 0.48 1.43
standup 7.14 11.42 3.33 94.29 5.24
wave 16.67 0.48 0.95 0.48 81.43
(8000 frames), “kick” (7600 frames), “lookround” (7820 frames), “standup” (7040 frames) and “wave” (7660 frames). We slice this video sequence into 3810 segments with the fixed time duration of 20 frames and the step length of 10 frames, where 25 segments in every class are selected for the small training sample setLtr , 12 segments for the validation sample set Lv , 210 segments for the test sample set and the remaining segments for online learning. Parameters in our algorithm are preset as: Th =1.5, Tnum =5 and ρ¯ = 0.2. MAP adaptation is only used to update the means. The proposed algorithm is implemented in Matlab 6.0 and tested on a 2.0 GHz Pentium 4 PC with 256MB memory. The average time per frame is about 0.228s. As a result, our algorithm at the correct implement could be used for those applications with a frame rate of 6∼10 frames/sec. Figure 2 gives the learning curves for behavior models of “box”, “kick”, “lookround”, “standup”, “wave” and average half-total error rate (HTER), where HTER=(FAR+FRR)/2 [8], FAR is false acceptance rate and FRR is false rejection rate. The horizontal axis shows the number of effective samples for estimating the parameters in the MAPACo-Training algorithm. The vertical axis shows the HTER. Figure 2(f) is the average HTER curve of all behaviors. From these curves, we can see the learning performance of behavior models can be markedly improved by MAPACo-Training, and after about 500 samples are used, the curves almost become stable. Table 1 gives the initial confusion matrix from the initial behavior models trained by the small training set Ltr , and Table 2 shows the final confusion matrix from the final behavior models by our algorithm. From these tables, we can see that when the initial recognition rate is low, those for “box”, “kick”, “standup” and “wave”, the final recognition rate is clearly improved. And when the initial recognition rate is high, that for “lookround”, the final recognition rate is still high.
MAPACo-Training: A Novel Online Learning Algorithm of Behavior Models
4.2
479
Results on Schuldt’s Dataset
We get a video sequence of 49813 frames from Schuldt’s dataset [18] including “box” (8370 frames), “clap” (8476 frames), “wave” (8275 frames), “run” (7945 frames), “jog” (8170 frames) and “walk” (8577 frames). We slice this video sequence into 4978 segments with the fix time duration of 25 frames and the step length of 10 frames, where 30 segments in each class are selected for the small training sample set Ltr , 16 segments for the validation sample set Lv , 240 segments for the test sample set, and the remaining segments for online learning. Parameters in our algorithm are preset as: Th =1.5, Tnum =5 and ρ¯ = 0.4. MAP adaptation is only used to update the means and variances. Figure 3 shows the learning curves. We can see that the learning results for all the behaviors except “run” are very good. For the behavior “run”, the main reason of poor performance is that running of some people is very similar to the jogging of the others in this dataset [18], which is difficult to distinguish. From the initial confusion matrix (Table 3) and the final confusion matrix (Table 4), 15
30
22 20
25 18
5
16
HTER(%)
HTER(%)
20
HTER(%)
10
15
10
14 12 10 8
5 6 0 0
500
1000
1500
2000
2500
0 0
3000
500
number of samples
1000
1500
2000
2500
4 0
3000
500
1000
number of samples
(a)
(b)
25
40
24
38
1500
2000
2500
3000
number of samples
(c) 34 32 30
36
23
28
21
HTER(%)
HTER(%)
HTER(%)
34 22
32 30
26 24 22 20
20
28 18
19
26
18 0
24 0
500
1000
1500
2000
2500
3000
number of samples
16 500
1000
1500
2000
2500
3000
14 0
500
1000
number of samples
(d)
1500
2000
2500
3000
number of samples
(e)
(f)
24
15 14
22
13 12
HTER(%)
HTER(%)
20
18
16
11 10 9 8 7
14
6 12 0
500
1000
1500
2000
number of samples
(g)
2500
3000
5 0
500
1000
1500
2000
2500
3000
number of samples
(h)
Fig. 3. The learning curves: (a) box; (b) clap; (c) wave; (d) run; (e) jog; (f) walk; (g) average HTER (h) run+jog
480
H. Li et al. Table 3. Initial confusion matrix box clap wave run jog walk
box 79.17 24.17 6.25 0.42 2.50 7.08
clap 2.50 53.75 8.75 0 4.58 1.67
wave 2.92 9.17 60.00 0 0 0
run 3.33 3.33 2.50 80.00 50.42 25.83
jog 3.33 2.08 18.75 14.16 31.67 8.75
walk 8.75 7.50 3.75 5.42 10.83 56.67
jog 0 0 0 30.42 59.58 20.00
walk 0.84 0 0 4.58 7.08 57.92
Table 4. Final confusion matrix box clap wave run jog walk
box 94.58 3.33 0 0.83 0.42 0.41
clap 2.50 94.17 7.50 0 0.42 0.42
wave 2.08 2.50 91.25 0 2.08 3.33
run 0 0 1.25 64.17 30.42 17.92
we can see the confusion values between a pair of behaviors other than “run” and “jog” are not high. But for “run” and “jog”, the HTER about “run” is only increased from 18.5% to 23% and nearly unchanged after about 2000 samples while the HTER about “jog” is declined from 38.5% to 24.5%. When we regard “run” and “jog” as one behavior “run+jog”, the result becomes quite satisfactory as shown in Figure 3(h).
5
Conclusion
In this paper, we proposed a semi-supervised learning algorithm called MAPACoTraining, which combines the traditional co-training algorithm and the principle of MAP adaptation. The algorithm is suitable for online learning of behaviors modeled by HMM. Experiments on two datasets also validate our method. In the future, we will explore a better way to train the models of similar behaviors like “run” and “jog”. Acknowledgment. This work was supported by the National Natural Science Foundation of China under grant Nos (60633070, 60475009) and by National Key Technology R&D Program under grant Nos (2006BAH02A03,2006BAH02A13).
References 1. Li, H., Greenspan, M.: Multi-scale gesture recognition from time-Varying contours. In: IEEE Int’l Conf. On Computer Vision, pp. 236–243. IEEE Computer Society Press, Los Alamitos (2005)
MAPACo-Training: A Novel Online Learning Algorithm of Behavior Models
481
2. Gong, S.G., Xiang, T.: Recognition of group activities using dynamic probabilistic networks. In: IEEE Int’l Conf. On Computer Vision, pp. 742–749. IEEE Computer Society Press, Los Alamitos (2003) 3. Bobick, A.F., Davis, J.W.: The recognition of human movement using temporal templates. IEEE Trans. on Pattern Analysis and Machine Intelligence 23, 257–267 (2001) 4. Laptev, I., Linderberg, T.: Space-time interest points. In: IEEE Int’l Conf. On Computer Vision, pp. 432–439. IEEE Computer Society Press, Los Alamitos (2003) 5. Xiang, T., Gong, S.G.: Video behaviour profiling and abnormality detection without manual labeling. In: IEEE Int’l Conf. On Computer Vision, pp. 1238–1245. IEEE Computer Society Press, Los Alamitos (2005) 6. Zelnik-Manor, L., Irani, M.: Event-based analysis of video. In: IEEE Conf. on Computer Vision and Pattern Recognition, pp. 123–130. IEEE Computer Society Press, Los Alamitos (2001) 7. Zhong, H., Shi, J., Visontai, M.: Detecting unusual activity in video. In: IEEE Conf. on Computer Vision and Pattern Recognition, pp. 819–826. IEEE Computer Society Press, Los Alamitos (2004) 8. Zhang, D., Gatica-Perez, D., Bengio, S., McCowan, I.: Semi-supervised adapted HMMs for unusual event detection. In: IEEE Conf. on Computer Vision and Pattern Recognition, pp. 611–618. IEEE Computer Society Press, Los Alamitos (2005) 9. Blum, A., Mitchell, T.: Combining labeled and unlabeled data with co-training. In: 11th Annual Conference on Computational Learning Theory (1998) 10. Levin, A., Viola, P., Freund, Y.: Unsupervised improvement of visual detectors using co-training. In: IEEE Int’l Conf. On Computer Vision, pp. 626–633. IEEE Computer Society Press, Los Alamitos (2003) 11. Yan, R., Naphade, M.: Semi-supervised cross feature learning for semantic concept detection in videos. In: IEEE Conf. on Computer Vision and Pattern Recognition, pp. 657–663. IEEE Computer Society Press, Los Alamitos (2005) 12. Javed, O., Ali, S., Shah, M.: Online detection and classification of moving objects using progressively improving detectors. In: IEEE Conf. on Computer Vision and Pattern Recognition, pp. 696–701. IEEE Computer Society Press, Los Alamitos (2005) 13. Reynolds, D.A., Quatieri, T.F., Dumn, R.B.: Speaker verification using adapted Gauusian mixture models. Digital Signal Processing 10, 19–41 (2000) 14. Lucas, B.D., Kanade, T.: An iterative image registration technique with an application to stereo vision. In: DARPA Image Understanding Workshop (April 1981) 15. Lv, F., Nevatia, R.: Recognition and segmentation of 3-D human action using HMM and multi-class adaboost. In: Proc. European Conference on Computer Vision, vol. IV, pp. 359–372 (2006) 16. Zhang, D., Gatica-Perez, D., Bengio, S.: Semi-supervised meeting event recognition with adapted HMMs. In: ICME. IEEE International Conference on Multimedia Expo (2005) 17. Li, H., Hu, Z., Wu, Y., Wu, F.: Behavior modeling and recognition based on spacetime image features. In: International Conference on Pattern Recognition, vol. 1, pp. 243–246 (2006) 18. Schuldt, C., Laptev, I., Caputo, B.: Recognizing human actions: a local SVM approach. In: International Conference on Pattern Recognition, vol. 3, pp. 32–36 (2004)
Optimal Learning High-Order Markov Random Fields Priors of Colour Image Ke Zhang1,2 , Huidong Jin1,2 , Zhouyu Fu1,2 , and Nianjun Liu1,2 Research School of Information Sciences and Engineering (RSISE) Australian National University 2 National ICT Australia (NICTA), Canberra Lab, ACT, Australia {ke.zhang,huidong.jin,zhouyu.fu,nianjun.liu}@rsise.anu.edu.au 1
Abstract. In this paper, we present an optimised learning algorithm for learning the parametric prior models for high-order Markov random fields (MRF) of colour images. Compared to the priors used by conventional low-order MRFs, the learned priors have richer expressive power and can capture the statistics of natural scenes. Our proposed optimal learning algorithm is achieved by simplifying the estimation of partition function without compromising the accuracy of the learned model. The parameters in MRF colour image priors are learned alternatively and iteratively in an EM-like fashion by maximising their likelihood. We demonstrate the capability of the proposed learning algorithm of highorder MRF colour image priors with the application of colour image denoising. Experimental results show the superior performance of our algorithm compared to the state–of–the–art of colour image priors in [1], although we use a much smaller training image set. Keywords: Markov random fields, image prior, colour image denoising.
1
Introduction
The need for prior models of image structure occurs in a lot of computer vision problems including stereo, optical flow, denoising, super-resolution, image-based rendering and to name a few. Whenever an observed“scene” must be inferred from noisy, degraded or loss partial image information, a natural image prior is required [2]. Modeling image priors is a challenging task, because of the highdimensionality of images, their non-Gaussian statistics and the need to model correlations in image structure over extended image neighbourhoods [3]. Some researchers attempted to use sparse coding approaches to address the modeling of complex image structure. Based on a variety of simple assumptions, they obtained sparse representations of local image structure in terms of the statistics of filters that are local in position, orientation, and scale [4][5]. However, these methods which focus on image patches provide no direct way of modeling the statistics of whole images [3]. Markov random fields (MRF) on the other hand have been widely used in computer vision but exhibit serious limitations. In particular, as MRF priors typically exploit handcrafted clique potentials and small neighbourhood systems, it is limited in the expressiveness of the models, and it only crudely captures the statistics Y. Yagi et al. (Eds.): ACCV 2007, Part I, LNCS 4843, pp. 482–491, 2007. c Springer-Verlag Berlin Heidelberg 2007
Optimal Learning High-Order MRF Priors of Colour Image
483
of natural images [6]. Since typical MRF models consider simple nearest neighbour relations and model first derivative filter response, extremely local (e.g. first order) priors employed by most MRF methods may not show any advantages, comparing with rich, patch based priors obtained by sparse coding methods [3]. However, Roth and Black [3] went beyond this limitation with the Fields of Experts (FoE) model, which is a generic MRF model of image priors over extended neighbourhoods. For availability of application, they represented the MRF potentials as a Product of Experts (PoE) [7]. As FoE takes the product over all neighbourhoods of each image patch, the number of parameters is only determined by the size of the maximal cliques in the MRF model and the number of filters defining the potential [3]. Furthermore, because of the homogeneity of the potential functions, the model does not have any restriction for the size of images [3]. As shown in their experiment, FoE can achieve the state–of–the– art performance for monochromatic image denoising and inpainting. Based on the work of Roth and Black [3], McAuley et al. [1] proposed an MRF colour image prior model by generalising the FoE model to capture the correlations between different colour channels. Their model was compared with the original FoE monochromatic prior for colour image denoising and evidenced performance improvements although their learning algorithm is clearly sub-optimal. In this paper we build on McAuley et al.’s [1] contribution to further improve the learning algorithm for colour image priors. The proposed model of colour image priors is akin to the one in [1], yet by improving the estimation of the model partition function, both high dimensional filters and their corresponding weights can be optimally learned by maximising the likelihood. The experimental results show improvements of results reported in McAuley et al. [1] on colour image denoising, although we use a much smaller training image set. The remainder of this paper is organised as follows. In Section 2, we briefly illustrate the MRF image prior models and their original learning approaches. Our optimised learning algorithm is introduced in Section 3. In Section 4, we demonstrate the performance of our learning algorithm and compare the denoising quality with three other methods (McAuley et al’s [1], Bilateral Filtering and Wavelet-based denoising approach). Finally, Section 5 concludes this paper.
2 2.1
MRF Prior Model Monochromatic MRF Prior Model
In [3], Roth and Black merged the ideas of learning in MRF and sparse image coding in order to develop a high-order MRF prior model where the cliques are square image patches [8][4]. According to the Hammersley–Clifford theorem, the joint probability distribution of an MRF with clique set C can be written as P (x) =
1 φc (xc ), Z(Θ) c∈C
where φc (xc ) is a potential function and Z(Θ) is the partition function.
(1)
484
K. Zhang et al.
The potential functions over these cliques are assumed to be Products of Experts [7], i.e., products of individual function φf (with a parameter αf ) given the response of a filter Jf to the image patch xc , φc (xc ; J, α) =
F
φf (xc ; Jf , αf ),
(2)
f =1
In the prior model, the potential functions are assumed to be stationary, i.e., every clique in the image has the same parameter set Θ = {Jf , αf : 1 ≤ f ≤ F }. The particular form they postulated for individual experts is related to the Student–T distribution, and is given by [3] 1 φf (xc ; Jf , αf ) = (1 + Jf , xc 2 )−αf . 2
(3)
The problem of learning MRF priors can then be recast in a parameter estimation setting. The optimal J’s and α’s are recovered by maximising the likelihood given by the joint probability in Equation 1. However, calculating the true partition function Z(Θ) is intractable due to its high complexity. Approximate procedures are often used here to get an estimation of the partition function, such as the contrastive divergence method used by Roth and Black [3]. 2.2
Colour Image MRF Prior Model
McAuley et al. [1] extended the monochromatic MRF prior to handle colour images. They proposed the “higher” order MRF prior model, i.e., using 3 × 3 × 3 clique instead of 3 × 3 clique, to represent the correlations of colour channels over the local neighbourhood. To deal with the dramatic increase in the computational load due to the significant rise of data dimensions in this colour model, they adopted a simple gradient-ascent-based learning algorithm rather than learning by maximising the likelihood. In their learning algorithm, they performed singular value decomposition (SVD) over the covariance matrix of training data to learn filters J’s, and only updated α’s along the gradient direction with fixed filters J’s. The estimation of the α’s in their model takes the following form: Let D = {X1 , X2 , · · · , XM } a set of training images, R = {Y1 , Y2 , · · · , YN } a set of random images, P (D|Θ) the likelihood of the training images given the model. Let Θ = {θ1 , θ2 , · · · , θF } where θf = (Jf , αf ) a set of filters and their corresponding weights. Then the likelihood of training images, P (D|Θ) is given by P (D|Θ) =
M i=1
1 φc (xic ; J, α), Z(Θ)
(4)
c∈C
where Z(Θ) is the partition function. They used the arithmetic mean of responses from the random images, Zˆam (Θ), to approximate the real value of partition function, given by
Optimal Learning High-Order MRF Priors of Colour Image
Z(Θ) ∝ Zˆam (Θ) =
N 1 φc (xic ; J, α). N i=1
485
(5)
c∈Yi
From Eq. 4, the gradient of the log-likelihood function with respect to αk is given by M ∂ ∂ log P (D|Θ) = ψc (Jk , xic ) − M log Z(Θ), ∂αk ∂α k i=1
(6)
c∈Xi
where ψ(a, b) = − log(1 + 12 a, b2 ). From Eq.5, the gradient of the log partition function is given by ∂ log Z(Θ) = ∂αk
N c∈Yi ψc (Jk , x)( c∈Yi φc (xc ; J, α))] i=1 [ , N c∈Yi φc (xc ; J, α) i=1
(7)
and the α’s are updated along the gradient ascent direction which can be obtained by Eq. 6 and Eq. 7. Note that the algorithm proposed by McAuley et al. [1] only updates the α’s with fixed values of filters. It is not optimal for several reasons: (1) the filters obtained from SVD are sub–optimal because they ideally should be learned by maximising the likelihood; (2) the α’s in their learning algorithm must be initialised to zero for numerical reasons [1], and the gradient ascent dose not work (absolute values of α’s are not convergent and their relative values remain the same) after the first iteration according to our implementation. Our aim is to learn a set of filters and their corresponding weights that maximises the likelihood. This can be achieved via standard gradient ascent method given initial estimates of the model parameters. However, as the Z(Θ) estimated in [1] is just proportional to the true Z(Θ), we can not obtain a reliable model likelihood by Eq.4. Furthermore, when we performed the partial derivative with respect to J and implemented it, we found that the J–α gradient iterations did not converge due to numerical instability in the estimation of Z(Θ).
3 3.1
An Optimised Learning Algorithm Estimation of Partition Function
Although the approximation of Z(Θ), Zˆam (Θ), has a clear physical meaning (when the images used to approximate Z(Θ) cover all possible images, this estimation represents the true form of Z(Θ)), the main problem is in the update of parameters, i.e. the complicated form of Eq. 7, which makes the gradient iteration error-prone due to numerical problems. For solving the problems described above, we use the geometric mean Zˆgm (Θ) with more robust performance in the case of non-Gaussian distributions [9], instead of arithmetic mean Zˆam (Θ) (Eq.5). The geometric mean of the partition function can be written as
486
K. Zhang et al.
Zˆgm (Θ) = (
N
1
φc (xic ; J, α)) N .
(8)
i=1 c∈Yi
According to Jensen’s inequality [10], we can obtain the upper and lower boundaries of Zˆam (Θ) and Zˆgm (Θ) N N N N fi (ε)2 1 1 1 1 −1 fi (ε) ≥ ( fi (ε)) N ≥ N ( (9) ( i=1 )2 ≥ ) , N N i=1 f (ε) i=1 i=1 i where fi (ε) = c∈Yi φc (xic ; J, α). As shown in Fig. 1, the log values of Zˆgm (Θ) and Zˆam (Θ) are very close along the increasing number of random images. Furthermore, we found that the standard deviations of log Zˆgm (Θ) is smaller than those of log Zˆam (Θ) over various amount of random images tests. Therefore, we can say that Zˆgm (Θ) is a robust approximation to the mean of the partition function, and can use a small set of random images to estimate the mean values of partition function in log form. In the calculation of the model likelihood given a set of parameters, ˆ we can use Z(Θ) = T × Zˆgm (Θ) to estimate the true value of Z(Θ). Here, T is the number of assignments for all the possible pixel values of the images patch, i.e., T = 2563×3×3 (3 × 3 clique size) or T = 2565×5×3 (5 × 5 clique size). Based ˆ on this observation, the approximation of log–partition function Z(Θ) can be rewritten as: ˆ log Z(Θ) = log T + log Zˆgm (Θ) = log T +
N F 1 αf ψf (Jf , xic ). N i=1
(10)
c∈Xi f =1
3.2
Proposed Learning Algorithm
The log–likelihood of MRF prior model (Eq.4) can be rewritten as follows: log P (D|Θ) =
M F
αf ψf (Jf , xic ) − M log T
i=1 c∈Xi f =1
−
N F M αf ψf (Jf , xic ), N i=1
(11)
c∈Yi f =1
where parameters have the same denotations in the Section 2.1. As we can expect to obtain a more accurate value of the log model likelihood than the one suggested by [1], Eq.11 can be an indicator that determines whether a given set of parameters has higher likelihood in our learning algorithm. Furthermore, based on Eq.11, the partial derivative with respect to both filters J’s and their corresponding weights α’s are significantly simplified: M N M −αk xic Jk , xic ∂ log P (D|Θ) −αk xic Jk , xic = − ,(12) ∂Jk N i=1 (1 + 12 Jk , xic 2 ) (1 + 12 Jk , x2c 2 ) i=1 c∈X c∈Y i
i
Optimal Learning High-Order MRF Priors of Colour Image M N ∂ log P (D|Θ) M = ψc (Jk , xic ) − ψc (Jk , xic ). ∂αk N i=1 i=1 c∈Xi
487
(13)
c∈Yi
Our learning algorithm is summarised as follows: 1. Initialise the filters (J’s) by performing SVD over training images. The initial values of α’s are randomly generated. 2. Update α’s by applying a line search in the gradient direction given by Eq. 13. The step size μα is chosen such that the highest log–likelihood in Eq. 11 is reached. ∂ α ← α + μα { log P (D|Θ)}. (14) ∂αi 3. Update J’s by applying a line search in the gradient direction given by Eq. 12. The step size μJ is, again, chosen by maximising the log–likelihood in Eq. 11. ∂ log P (D|Θ)}. (15) J ← J + μJ { ∂Ji 4. Repeat steps 2–3 until the log–likelihood of the model dose not change. As Eq. 14 and Eq. 15 indicate that both α’s and J’s are updated along the direction of maximising model likelihood, the proposed learning algorithm is optimal based on the model likelihood in Eq. 11. Since the update step sizes (μα and μJ ) are very sensitive to the input parameters, it is quite difficult to specify them with any constants. In our implementation, we employ back–tracking line search to find the optimal solution in each update step [11].
Fig. 1. The log values of two Z(Θ) estimation methods along the increasing number of random images
488
3.3
K. Zhang et al.
Inference
After we get the MRF prior model, in order to perform inference (i.e. denoising in our experiments), we adopted a standard gradient based approach, as used by McAuley et al [1]. Gradient ascent is a valid technique in the case of denoising, since the noisy image is “close to” the original image, meaning that a local maximum is likely to be a global one [1]. In the denoising problem, the purpose is to infer the most likely correction for the image given the image prior and the noise model. Thenoise model assumed in our experiments, as in [1], is i.i.d. Gaussian: P (y|x) ∝ j exp(− 2σ1 2 (yj − xj )2 ). Here, j ranges over all the pixels in the image, yj denotes the real colour value of the noisy image at pixel j, and xi denotes the colour to be estimated at pixel j; σ denotes the variance of Gaussian noise. Combining the noise model and the MRF prior (Eq.1), the gradient of the log–posterior becomes [1]: ∇x log P (x|y) =
F
αf Jf− ∗
f =1
(Jf ∗ x) λ + 2 (y − x) σ 1 + 12 (Jf ∗ x)2
(16)
where ∗ denotes matrix convolution, and the algebraic operations above are performed in an elementwise fashion on the corresponding convolution matrix. Jf− denotes the mirror image of Jf in two dimensions. λ is a critical parameter that gauges the relative importance of the prior and the image terms. The updated image is then simply computed by xt+1 = xt + δ
∂ log P (x|y) ∂x
(17)
where δ is the step size of the gradient ascent. We find it is not sensitive to the inference result and can be selected empirically.
4
Experimental Results and Comparison
In our experiments, to initialise our filters J’s, we randomly selected 8,000 3×3×3 and 5 × 5 × 3 patches, cropped from 200 images in the Berkeley Segmentation Database, and performed singular value decomposition (SVD) over their covariance matrices [12]. Thus, we obtained 27 and 75 filters for the two clique sizes, with 27 and 75 dimensions for each kind of filters respectively. α’s were initialised to be a set of random values with the same dimension as the number of filters. There was no constraint on the scale of the initial α’s, and they converged for both absolute and relative values after several steps of update. In the updating process, we randomly selected 2,000 training image patches and 2,000 random images patches from the same image database for each update step. The sizes of training/random image patches for 3 × 3 × 3 and 5 × 5 × 3 cliques were, respectively, 7 × 7 × 3 and 13 × 13 × 3.
Optimal Learning High-Order MRF Priors of Colour Image
489
Fig. 2. Typical denoising results. The first column displays the original image (up), the noisy image (middle) with σ = 75 (red), 25 (green), 15 (blue) (PSNR=14.97) and the result of bilateral filtering 5 × 5 window (down, PSNR=23.55); the second column shows the result using Wavelet-based approach 3 × 3 window (up, PSNR=23.32), the results of McAuley et al’s 3 × 3 prior (middle, PSNR=25.03) and our 3 × 3 prior (down, PSNR=25.90). the third column shows the result of 5 × 5 window Waveletbased approach (up, PSNR=23.84), McAuley et al’s 5 × 5 prior (middle, PSNR=25.99) and our 5 × 5 prior (down, PSNR=26.82).
In the inference, we did not need to eliminate the filter with highest variance since in our algorithm α’s are normalised (with range of [0,1]). The least important filter will be ignored automatically in the denoising process because its corresponding weight will be zero. In the selection of λ, we used images other than those involving in our experiments. This was done by denoising a test image with several candidate λ–value, and selecting whichever one yields the best results [1]. The step size δ has been chosen to grow linearly with the noise level, which was found to work well in practice. The denoising performances 2
I,J,K
(Ri,j,k −Oi,j,k )2
255 ), where M SE = i,j,k IJK ; were evaluated by PSNR (10 log10 ( MSE R denotes restored image and O denotes original image). In Figure 2 we show results obtained for denoising an image in which a different amount of noise is applied to each of the three channels, and compare these with the state-ofthe-art [1], simple bilateral filtering (using the MATLAB code from [13]), and
490
K. Zhang et al.
Table 1. (3 × 3 × 3 window) Average denoising performance over 50 testing images. Results are measured in PSNR. image/σ 5 Noisy image 34.10 McAuley et al1 36.19 Our algorithm 1 37.12∗ McAuley et al2 36.83 Bilateral filtering 28.11 Wavelet-based denoising 36.11 ∗
15 24.70 29.17 29.78∗ 29.74 27.18 28.99
25 20.15 26.04 26.98∗ 26.69 25.75 25.98
50 14.16 22.45 23.38∗ 23.15 21.87 22.41
Indicates significant difference in performance compared with the upper one.
Table 2. (5 × 5 × 3 window) Average denoising performance over 50 testing images. Results are measured in PSNR. image/σ 5 Noisy image 34.10 McAuley et al1 36.57 Our algorithm 1 37.56∗ McAuley et al2 37.28 Bilateral filtering 29.32 Wavelet-based denoising 36.41 ∗
15 24.70 29.59 30.11∗ 30.08 27.78 29.50
25 20.15 26.55 27.41∗ 27.16 25.82 26.32
50 14.16 22.79 23.69∗ 23.41 21.90 22.46
Indicates significant difference in performance compared with the upper one.
Wavelet-based denoising [14]. In the bilateral filtering and Wavelet-based denoising experiments, the RGB test images were converted into YCbCr format before processing, which has less correlation between colour channels. As we show by using different priors, the denoising performance of the priors learned by our algorithm has significant improvements comparing that of using McAuley et al.’s [1] and the other two denoising approaches. Tables 1 and 2 summarise results about the denoising 50 test colour images (from the Berkeley Segmentation Database) in which all three channels have been equally corrupted, and compare our algorithm with McAuley et al.’s priors and two well known methods in the 3 × 3 × 3 and 5 × 5 × 3 case, respectively. As results show, the performance of priors learned by our proposed algorithm for both model sizes is statistically significantly (paired T-test at the 0.05 level) superior to its counterparts which use the same training/random image set. Furthermore, the performance of our priors learned from 2,000/2,000 training images/random image patches is comparable with the priors learned in [1], which used 100,000/50,000 training/random image patches.
5
Conclusion
In this paper, we have proposed the learning algorithm of high-order MRF prior models for colour image denoising. By collecting a relatively small set of sample 1 2
Indicates priors learned from 2,000/2,000 training/random patches. Indicates priors learned from 100,000/50,000 training/random patches.
Optimal Learning High-Order MRF Priors of Colour Image
491
colour image patches from a standard colour image database, we have learned priors specified for colour images using several steps of gradient ascent update with the rule of maximum likelihood. Results comparing the colour prior models learned by our algorithm to a state–of–the-art colour image prior model [1] show performance improvements.
References 1. McAuley, J., Caetano, T., Smola, A., Franz, M.: Learning high-order MRF priors of color images. In: ICML. LNCS, vol. 4503, pp. 617–624. Springer, Heidelberg (2006) 2. Freeman, W., Pasztor, E., Carmichael, O.: Learning low-level vision. International Journal of Computer Vision 40, 25–47 (2000) 3. Roth, S., Black, M.: Fields of experts: A framework for learning image priors. In: ICCV, pp. 860–867 (2005) 4. Olshausen, B., Field, D.: Sparse coding with an overcomlete basis set: a strategy employed by v1? Vision Research 37, 3311–3325 (1997) 5. Welling, M., Hinton, G., Osindero, S.: Learning sparse topographic representations with products of Student-t distributions. In: NIPS, vol. 15, pp. 1359–1366 (2003) 6. Geman, S., Geman, D.: Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images. PAMI 6(6), 721–741 (1984) 7. Hinton, G.: Products of experts. In: 9th ICANN, pp. 1–6 (1999) 8. Zhu, S., Wu, Y., Mumford, D.: Filters, random fields and maximum entropy (frame): Towards a unified theory of texture modeling. International Journal of Computer Vision 27, 107–126 (1998) 9. Abramowitz, M., Stegun, I.E.: The process of the Arithmetic–Geometric mean. Handbook of Mathematical Functions with Formulas, Graphs, and Mathematical Tables, 9th printing, 571 (1972) 10. Krantz, S.: In: Handbook of Complex Variables. Boston, MA:Brikhauser, p.118 (1999) 11. Mor´e, J., Thuente, D.: Line search algorithms with guaranteed sufficient decrease. ACM Trans. Math. Software 20, 286–307 (1994) 12. Martin, D., Fowlkes, C., Tal, D., Malik, J.: A database of human segmented natural images and ites application to evaluating segmentation algorithms and measuring ecological statistics. In: 8th ICCV, pp. 416–423 (2001) 13. http://mesh.brown.edu/dlanman/photos/Bilateral 14. Portilla, J., Strela, V., Wainwright, M., Simoncelli, E.: Image denoising using scale mixtures of Gaussians in the wavelet domain. IEEE Trans. Image Processing. 12(11), 1338–1351 (2003)
Hierarchical Learning of Dominant Constellations for Object Class Recognition Nathan Mekuz and John K. Tsotsos Center for Vision Research (CVR) and Department of Computer Science and Engineering, York University, Toronto, Canada M3J 1P3 {mekuz,tsotsos}@cse.yorku.ca
Abstract. The importance of spatial configuration information for object class recognition is widely recognized. Single isolated local appearance codes are often ambiguous. On the other hand, object classes are often characterized by groups of local features appearing in a specific spatial structure. Learning these structures can provide additional discriminant cues and boost recognition performance. However, the problem of learning such features automatically from raw images remains largely uninvestigated. In contrast to previous approaches which require accurate localization and segmentation of objects to learn spatial information, we propose learning by hierarchical voting to identify frequently occurring spatial relationships among local features directly from raw images. The method is resistant to common geometric perturbations in both the training and test data. We describe a novel representation developed to this end and present experimental results that validate its efficacy by demonstrating the improvement in class recognition results realized by including the additional learned information.
1
Introduction
Humans are highly adept at classifying and recognizing objects with significant variations in shape and pose. However, the complexity and degree of variance involved make this task extremely challenging for machines. Current leading edge methods use a variety of tools including local features [1,2,3,4], global [5] and region [6] histograms, dominant colors [7], textons [8] and others, collecting features sparsely at detected key points, at random locations, or densely on a grid and at single or multiple scales. In practice, different types of features are often complementary and work well in different scenarios, and good results are often achieved by combining different classifiers. Of the above approaches, much focus has recently been dedicated to learning with local appearance descriptors, which have been shown to be extremely effective thanks to their discriminant qualities and high degree of resistance to geometric and photometric variations as well as partial occlusions. A very effective and widely-used technique that enables the use of efficient search methods borrowed from the text retrieval field is vector quantization, Y. Yagi et al. (Eds.): ACCV 2007, Part I, LNCS 4843, pp. 492–501, 2007. c Springer-Verlag Berlin Heidelberg 2007
Hierarchical Learning of Dominant Constellations
493
whereby each patch is associated with a label (visual word) from a vocabulary. The vocabulary is usually constructed offline by means of some clustering algorithm. To avoid aliasing effects arising from boundary conditions, soft voting is employed, whereby each vote is distributed into several nearby words using some kernel function. Finally, images are coded as histograms of their constituent visual words. While the importance of local features’ spatial configuration information for object class recognition is widely recognized, the basic scheme described above is typically employed on sets of isolated local appearance descriptors. However, for the most part, local appearance descriptors were designed to recognize local patches. When used for recognizing objects, the spatial layout that they appear in is of paramount importance. The SIFT algorithm [9], for example, represents local features in a way that is invariant to geometric perturbations. However, it also stores the parameters of the local geometry, and subsequently applies a Hough transform to select from potential hypotheses a model pose that conforms to the geometry associated with a large number of identified keys. Current systems that capture spatial information do so by learning and enforcing local relationships [10,11], global relationships [12,13], using dense sampling [2,1], or at multiple levels [14,15]. In [14], the system learns groups of local features that appear frequently in the training data, followed by global features composed of local groupings. In [11], spatial consistency is enforced by requiring a minimum number of features to co-occur in a feature neighborhood of fixed size. The authors of [16,12] demonstrate the benefit of learning the spatial relationships between various components in an image from a vocabulary of relative relationships. In [2], appearance models are built where clusters are learned around object centers and the object representation encodes the position and scale of local parts within each cluster. Significant performance gains are reported resulting from the inclusion of location distribution information. Fergus et al. [1] learn a scale-normalized shape descriptor for localized objects. However, the shapes are not normalized with respect to any anchor point. Consequently, some preprocessing of the input images is required. A boosting algorithm that combines local feature descriptors and global shape descriptors is presented in [13], however extracting global shape is extremely difficult under occlusion or cluttered background. We take a different approach and seek to learn object class-specific hierarchies of constellations, based on the following principles: Unsupervised learning. A clear tradeoff exists between the amount of training data required for effective learning, and the quality of its labeling. Given the high cost of manual annotation and segmentation, and the increased availability (e.g. on the internet) of images that are only globally annotated with a binary class label, a logical goal is the automatic learning of constellation information from images with minimal human intervention. Specifically, this precludes manual segmentation and localization of objects in the scene. Invariance to shift, scale and rotation. In order to be able to train with and recognize objects in various poses, we require a representation that
494
N. Mekuz and J.K. Tsotsos
captures spatial information, yet is resistant to common geometric perturbations. Robustness. In order to successfully learn in an unsupervised fashion, the algorithm must be robust to feature distortions and partial occlusion. A common approach for achieving robustness is voting. Learn with no spatial restrictions We would like to learn spatial relationships over the entire image, without restrictions of region or prior (e.g. Gestalt principles). This allows grouping discontinuous features, e.g. such that lie on the outline of an object with a variable texture interior. The main contribution of this paper is a novel representation that captures spatial relationship information in a scale and rotation invariant manner. The constellation descriptors are made invariant by anchoring with respect to one local feature descriptor, similar to the way the SIFT local descriptor anchors with respect to the dominant orientation. We present a framework for learning spatial configuration information by collecting inter-patch statistics hierarchically in an unsupervised manner. To tackle the combinatorial complexity problem, higher level histograms are constructed by successive pruning. The most frequently occurring constellations are learned and added to the vocabulary as new visual words. We also describe an efficient representation for matching learned constellations in novel images for the purpose of object class recognition or computing similarity. The remainder of this paper is organized as follows: in Section 2 we describe our proposed constellation representation. This is followed by implementation details of the voting scheme in Section 3, and the matching algorithm in Section 4. The results obtained on images of various categories are presented in Section 5, and finally, Section 6 concludes with a discussion.
2
Invariant Constellation Representation
The constellation representation captures the types and relative positions, orientations and scales of the constituent parts. An effective representation must
f2 f4
f1
f3
Fig. 1. An illustration of the constellation representation. Local features are represented with circles, with arrows emanating out of them to indicate dominant orientation. Feature f1 is selected as anchor, and the positions, scales and orientations of the remaining features are expressed relative to it.
Hierarchical Learning of Dominant Constellations
495
be resistant to minor distortions arising from changes in pose or artifacts of the local feature extraction process. Pose changes can have a significant effect on the coordinates of local features. Another key requirement is a consistent frame of reference. In the absence of models of localized objects, a frame of reference can be constructed as a function of the constituent features. One option is to use the average attributes of local features [17]. However, since our method uses local features that are quantized into discrete visual words, we opt for the simpler alternative of pivoting at the feature with the lowest vocabulary index. This results in a more compact representation (and in turn, computational complexity savings) by eliminating the need to store spatial information for the anchor feature. On the downside, the anchor feature may not lie close to the geometric center of the constellation, reducing the granularity of position information for the other features. Whatever method is used for selecting the pivot, detection of the constellation depends on the reliable recovery of the pivot feature. However, even if the pivot feature cannot be recovered (e.g. under occlusion), subsets of the constellation may still be detected. Our representation is illustrated graphically in Figure 1, with the local appearance features represented as circles, and their dominant orientation as arrows. Feature f1 is selected as anchor and the coordinate system representing the remaining constellation features is centered about it and rotated to align with its dominant orientation. More formally, given a set of local features Fi = Γi , ti , xi , yi , αi where Γi is the index of the visual word corresponding to feature Fi , ti is its scale, xi and yi its position in the image and αi its orientation, relative to the global image coordinate system, we select the anchor F∗ as F∗ = arg min Γi , and construct the constellation descriptor encoding the anchor feature’s type Γ∗ , as well as the following attributes for each remaining feature Fj : Type: Γj Scale ratio: tj /t∗ Relative orientation: αj − α∗ Relative position: atan2(yj − y∗ , xj − x∗ )− α∗ where atan2 is the quadrantsensitive arctangent function. This attribute ignores distances, and merely provides a measure of Fj ’s polar angle relative to F∗ , using F∗ ’s coordinate system. As an example, using this representation, a pair of local features {F1 , F2 } with Γ1 < Γ2 is represented as Γ1 , Γ2 , t2 /t1 , α2 − α1 , atan2(y2 − y1 , x2 − x1 ) − α1 . A constellation of m local features is represented as an n-tuple with n = 4m − 3 elements. To maintain consistent representation, the descriptor orders the local features by their vocabulary index. If the lowest vocabulary index is not unique to one local feature, we build multiple descriptors, just as the SIFT algorithm creates multiple descriptors at each keypoint where multiple dominant orientations exist.
496
3
N. Mekuz and J.K. Tsotsos
Voting by Successive Pruning
ī1
ī1
The learning phase performs histogram voting in order to identify the most frequently occurring constellations in each category, using the representation described above. Since the descriptor orders the local features by their type attribute, each resulting histogram takes the shape of a triangular hyper-prism, with the Γ (type) axes along the hyper-triangular bases and the other attributes forming the rectangular component of the prism. Spatial information is encoded in 8 × 8 × 8 bins. The relative orientation and relative position attributes encode the angle into one of eight bins, similar to way this is done in SIFT. Scale ratios are also placed into a bin according to log2 (tj /t∗ ) + 3. Co-occurrences with scale ratios outside the range [1/16, . . . , 32] are discarded. As in SIFT, in order to avoid aliasing effects due to boundary conditions, we use soft voting whereby for each attribute, each vote is hashed proportionally into two neighboring bins. For the type attribute, the vote is distributed into several nearby visual words using a kernel. We use a Gaussian kernel with σ set to the average cluster radius, although other weighting formulas are certainly possible. It is also possible to threshold by distance rather than fixing the number of neighbors. The exponential complexity of the problem, and in particular, the size of the voting space, call for an approximate solution. A simple method that has often been used successfully is successive pruning. In computer vision, good results have been reported in [18] although some human supervision was necessary at the highest levels of the hierarchy. In some sense, the successive pruning strategy can be viewed as a coarse-to-fine refinement process. Starting with coarse bins, the algorithm identifies areas of the search space with a high number of counts. It then iteratively discards bins with a low number of counts, re-divides the rest of the voting space into finer bins, and repeats the voting process. In the case of multi-dimensional histograms, coarse bins can also be created by collapsing dimensions. This latter approach is more convenient in our case since it fits
ī2
ī2
(a)
(b)
Fig. 2. (a) A depiction of a triangular histogram used for voting for the most frequently co-occurring pairs. (b) Histogram pruning: the bin with a low number of counts are discarded. Finer bins are allocated for each bin with a number of counts above a threshold.
Hierarchical Learning of Dominant Constellations
497
naturally with the notion of hierarchical learning, creating larger constellations from smaller ones. Also, collapsing the dimensions associated with spatial information offers computational advantages by allowing early termination of the voting in the discarded bins, since visual word indices for local features are available immediately. Figures 2(a) and 2(b) illustrate the structures used in the two-phase voting process to identify pairs of local features that appear frequently in a particular spatial configuration. In the first phase, local feature descriptors are extracted and cached from all images belonging to an object class, and a triangular histogram (Fig. 2(a)) is collected to count the number of times each pairs of local features co-occurs, regardless of geometry. In the second phase (Fig. 2(b)), subhistograms are allocated for the bins with a high number of counts, and the remaining bins are discarded. Each sub-histogram consists of 8 × 8 × 8 bins and captures spatial information for its associated bin in the triangular histogram. Finally, the vocabulary is augmented with new visual words corresponding to the most frequent constellations in each class identified in phase 2.
4
Indexing Constellation Descriptors for Efficient Matching
Given novel input images, the system compares constellations extracted from these images against the learned constellation stored in its vocabulary. In order to achieve this, an exhaustive search of all local feature combinations in the input images is not necessary. A more efficient search is possible by indexing the learned constellation information offline, as depicted in Figure 3. At the first level, the structure consists of a single array indexed by local feature type index. Given a moderate number of learned constellations, the resulting first level array is sparse. Local feature types for which learned constellations exist, have their array entries point to arrays of stored constellation descriptors, sorted lexicographically by the other type indices. The matching algorithm works by constructing an inverted file [19] of the local features in the image. A sparse inverted file containing only links to vocabulary
local feature words
sorted constellation descriptors
sorted constellation descriptors
Fig. 3. A depiction of an efficient indexing structure for fast lookup of constellation features
498
N. Mekuz and J.K. Tsotsos
entries that are in use suffices thanks to the sorted second-level arrays: a match is sought simply by traversing both lists simultaneously. Spatial relationship information is compared only when all type attributes in a constellation descriptor are matched in the inverted file.
5
Evaluation
We tested our technique by examining the effect of using spatial relationship information captured using our descriptors on object class recognition performance. In order to isolate the effect of our constellation learning algorithm on class detection, we limited our evaluation to still greyscale images. We used the SIFT detector and descriptor to extract and represent local appearance features. We constructed a vocabulary of 13,000 visual words by extracting features from the first 800 hits returned by Google Images for the keyword ‘the’ and clustering with k-means. The result is a neutral vocabulary, that is not tuned specifically for any object category. For training and test data, we used 600 images of faces, airplanes, watches, bonsai trees and motorbikes from the PASCAL 2006 data set [20], divided equally into training and test images. All images were converted to greyscale, but no other processing was performed. In the training phase, image descriptors were collected for each of the training images encoding histograms of their constituent visual words. We used our neutral vocabulary constructed as described above, and quantized each feature descriptor to its 15 nearest neighbors using a Gaussian kernel with σ set to the average radius covered by each vocabulary entry. In the testing phase, each image was matched and classified using a simple unweighted nearest neighbor classifier against the trained image descriptors. As is standard practice, we used
Faces Airplanes Watches Bonsai trees Motorcycles
Faces 88.3 53.3 45.0 21.7 70.0
Airplanes 3.3 36.7 11.7 1.7 5.0
Watches 5.0 0.0 26.7 1.7 1.7
Bonsai trees 1.7 8.3 16.7 71.7 8.3
Motorcycles 1.7 1.7 0.0 3.3 15.0
Bonsai trees 0.0 5.0 13.3 76.7 6.7
Motorcycles 0.0 1.7 1.7 1.7 21.7
(a)
Faces Airplanes Watches Bonsai trees Motorcycles
Faces 91.7 50.0 40.0 16.7 66.7
Airplanes 3.3 43.3 10.0 3.3 5.0
Watches 5.0 0.0 35.0 1.7 0.0 (b)
Fig. 4. Confusion matrices (a) using a vocabulary of only local appearance features. (b) using an augmented vocabulary with an additional 50 constellation words per category.
Hierarchical Learning of Dominant Constellations
499
100
No spatial words 10 Spatial words 20 Spatial words 50 Spatial words
90
% Correct classification
80
70
60
50
40
30
20
10
0 faces
airplanes
watches
Category
bonsai
motorcycles
Fig. 5. Object class categorization performance as a function of the number of constellation visual words used. The error bars represent a margin of 3 standard errors.
a stop list to discard the 2% most frequently occurring visual words. Although more elaborate classifiers (e.g. SVM) and weighting schemes (e.g. tf-idf [21]) are possible, we opted for the simple scheme described here in order to focus on the effect of the additional spatial information. We expect the tf-idf scheme to place increased weights on the constellation features, since they carry more class-specific discriminant information. We tested the effect of augmenting the vocabulary with 10, 20 and 50 constellation words per object class on class recognition performance. It is worth noting that the vocabulary used for constructing these additional constellation words was again our generic neutral vocabulary: the only class-specific information captured in the training phase was the most frequently co-occurring pairs and their spatial relationships in each class. A 2% stop list was again used on the visual words associated with the local features but not on the pairs. Figure 4 presents confusion matrices for the categorization tests (a) using no spatial information, and (b) using 50 additional constellation visual words. Perhaps surprisingly, poor performance is realized in the motorcycles category, where local feature-based methods typically excel. A likely explanation is that normally the vocabulary is constructed using images of the modeled class, and
500
N. Mekuz and J.K. Tsotsos
captures features such as wheels in the case of the motorcycle class, whereas in our experiments we used a neutral vocabulary that was not trained specifically for any class. More importantly, however, we note that the addition of a few visual words corresponding to learned spatial features clearly boosted recognition performance in all classes, with average gains of about 5% using 50 constellation words. Figure 5 shows correct recognition results (corresponding to the diagonal of the confusion matrices) with different numbers of constellation words. The general trend shows recognition performance improving as more constellation features are used. The error bars represent an interval of 3 standard errors.
6
Discussion
This paper has presented a novel approach for representing constellation information that is learned directly from raw image data in a hierarchical fashion. The method is capable of learning spatial configuration information from possibly cluttered images where objects appear in various poses and possibly partly occluded. Novel images are tested for the presence of learned configurations in a way that is robust to common geometric perturbations. Additionally, the paper presents implementation details for an efficient voting algorithm that allows collecting robust co-occurrence statistics in a computationally highly complex voting space, and efficient indexing structures that allow fast lookup in the matching phase. Our experimental results confirm the importance of spatial structure to the class recognition problem, and show that the proposed representation can provide significant benefit with constellations consisting of pairs. We are currently exploring richer constellation structures corresponding to higher levels of the hierarchy and looking at ways for visualizing the learned constellations.
Acknowledgments The authors are grateful to Erich Leung and Kosta Derpanis for many helpful discussions. This work was supported by OGSST and Precarn incorporated.
References 1. Fergus, R., Perona, P., Zisserman, A.: Object class recognition by unsupervised scale-invariant learning. In: IEEE Conference on Computer Vision and Pattern Recognition, vol. 2, pp. 264–271 (2003) 2. Leibe, B., Mikolajczyk, K., Schiele, B.: Efficient clustering and matching for object class recognition. In: British Machine Vision Conference, Edinburgh, England (2006) 3. Berg, A.C., Berg, T.L., Malik, J.: Shape matching and object recognition using low distortion correspondences. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 26–33. IEEE Computer Society Press, Los Alamitos (2005) 4. Dorko, G., Schmid, C.: Object class recognition using discriminative local features (2005)
Hierarchical Learning of Dominant Constellations
501
5. Ortega, M., Rui, Y., Chakrabarti, K., Mehrotra, S., Huang, T.S.: Supporting similarity queries in mars. In: ACM International Conference on Multimedia, pp. 403– 413. ACM Press, New York (1997) 6. Carson, C., Thomas, M., Belongie, S., Hellerstein, J., Malik, J.: Blobworld: a system for region-based image indexing and retrieval. Technical report, Berkeley, CA, USA (1999) 7. Mukherjea, S., Hirata, K., Hara, Y.: Amore: a world-wide web image retrieval engine. In: CHI 1999. Extended abstracts on human factors in computing systems, pp. 17–18. ACM Press, New York (1999) 8. Malik, J., Belongie, S., Shi, J., Leung, T.K.: Textons, contours and regions: Cue integration in image segmentation. In: IEEE International Conference on Computer Vision, pp. 918–925. IEEE Computer Society Press, Los Alamitos (1999) 9. Lowe, D.G.: Object recognition from local scale-invariant features. In: IEEE International Conference on Computer Vision, vol. 1150, IEEE Computer Society Press, Los Alamitos (1999) 10. Lazebnik, S., Schmid, C., Ponce, J.: Affine-invariant local descriptors and neighborhood statistics for texture recognition. In: IEEE International Conference on Computer Vision, vol. 649, IEEE Computer Society, Los Alamitos (2003) 11. Sivic, J., Zisserman, A.: Video Google: A text retrieval approach to object matching in videos. In: IEEE International Conference on Computer Vision, vol. 2, pp. 1470– 1477 (2003) 12. Lipson, P., Grimson, E., Sinha, P.: Configuration based scene classification and image indexing. In: IEEE Conference on Computer Vision and Pattern Recognition, vol. 1007, IEEE Computer Society, Los Alamitos (1997) 13. Zhang, W., Yu, B., Zelinsky, G.J., Samaras, D.: Object class recognition using multiple layer boosting with heterogeneous features. In: IEEE Conference on Computer Vision and Pattern Recognition 14. Amit, Y., Geman, D.: A computational model for visual selection. Neural Comput. 11, 1691–1715 (1999) 15. Agarwal, A., Triggs, W.: Hyperfeatures - multilevel local coding for visual recognition. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, Springer, Heidelberg (2006) 16. Sinha, P.: Image invariants for object recognition. Invest. Opth. & Vis. Sci. 34(6) (1994) 17. Shokoufandeh, A., Dickinson, S.J., J¨ onsson, C., Bretzner, L., Lindeberg, T.: On the representation and matching of qualitative shape at multiple scales. In: European Conference on Computer Vision, pp. 759–775. Springer, Heidelberg (2002) 18. Fidler, S., Berginc, G., Leonardis, A.: Hierarchical statistical learning of generic parts of object structure. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 182–189. IEEE Computer Society Press, Los Alamitos (2006) 19. Witten, I.H., Moffat, A., Bell, T.C.: Managing gigabytes: compressing and indexing documents and images, 2nd edn. Morgan Kaufmann Publishers Inc, San Francisco (1999) 20. Everingham, M., Zisserman, A., Williams, C.K.I., Van Gool, L.: The PASCAL Visual Object Classes Challenge. In: VOC2006 (2006) 21. Baeza-Yates, R., Ribeiro-Neto, B.: Modern Information Retrieval. Addison-Wesley, Reading (1999)
Multistrategical Approach in Visual Learning Hiroki Nomiya and Kuniaki Uehara Graduate School of Science and Technology, Kobe University [email protected], [email protected]
Abstract. In this paper, we propose a novel visual learning framework to develop flexible and accurate object recognition methods. Currently, most of visual learning based recognition methods adopt the monostrategy learning framework using a single feature. However, the real-world objects are so complex that it is quite difficult for monostrategy method to correctly classify them. Thus, utilizing a wide variety of features is required to precisely distinguish them. In order to utilize various features, we propose multistrategical visual learning by integrating multiple visual learners. In our method, multiple visual learners are collaboratively trained. Specifically, a visual learner L intensively learns the examples misclassified by the other visual learners. Instead, the other visual learners learn the examples misclassified by L. As a result, a powerful object recognition method can be developed by integrating various visual learners even if they have mediocre recognition performance.
1
Introduction
For the flexible and accurate recognition in computer vision, an effective framework called visual learning has been proposed by introducing machine learning frameworks into computer vision understanding. However, conventional visual learning methods adopt monostrategy learning. That is, they are based on a single learning strategy and able to utilize only a few features. Thus, they often fail to distinguish complex objects. Since an image is described with various features such as contour, color and texture, a wide variety of features are required for flexible recognition. Therefore, it is essential to integrate multiple features. To solve this problem, multistrategy learning is developed [1]. Since it can deal with multiple learning frameworks, it has a potentially better competence. For example, Nomiya et al. proposed a multistrategy visual learning method by integrating two types of object recognition methods (appearance-based and region-based) using decision trees and discriminant analysis [2]. But the integration method is so simple that the recognition performance is inadequate. Most of existing multistrategy learning methods simply integrates the learning results of all base learners using, for example, linear combination. Moreover, each learner is separately trained. Since the features are mutually interrelated, the visual learners should be trained collaboratively. Thus, we propose an effective learning framework that the visual learners can cooperate with each other. If a visual learner L frequently misclassifies some examples, it seems to be difficult for L to correctly discriminate the examples. Then, L never learns the Y. Yagi et al. (Eds.): ACCV 2007, Part I, LNCS 4843, pp. 502–511, 2007. c Springer-Verlag Berlin Heidelberg 2007
Multistrategical Approach in Visual Learning
503
examples and they are learned by the other visual learners. Instead, L learns the examples misclassified by the other learners. As a result, each visual learner is specialized in discriminating the objects which have particular features. For example, an appearance-based visual learner will be specialized in discriminating the objects whose shapes are unique while they have various colors and textures. Conversely, a region-based visual learner will be specialized in discriminating the objects whose colors and textures are unique while they have similar shapes. To classify an example (i.e. an image), the more accurate prediction can be obtained by determining the most suitable visual learner based on the learning result. Therefore, our method can be more efficient than the conventional multistrategy learning methods. In order to develop this learning scheme, we propose the intensive and collaborative learning framework in the following section.
2
Multistrategical Visual Learning
We propose an effective multistrategy learning model based on AdaBoost [3]. We improve its weighting algorithm so that each visual learner can be trained collaboratively. First, we extend AdaBoost to solve multiclass problems. AdaBoost.M1 [3] is a direct extension to the multiclass problems. But it has the same property as AdaBoost that the classification error of each weak learner must be less than 1 2 . This is a crucial constraint when the number of classes is large. AdaBoost.M2 [3] gives a solution to this problem by using pseudo-loss. A hypothesis whose pseudo-loss is lower than 12 is much more easily generated for multiclass problems. Thus, we introduce AdaBoost.M2 algorithm. It takes a training set {(x1 , y1 ), · · · , (xm , ym )} where m is the number of training examples. xi is an element of instance space X and yi is an element of label space Y = {1, · · · , C}, where C is the number of classes. In object recognition problems, an example corresponds to an image. A class label corresponds to the object in the image. AdaBoost.M2 can be extended to our multistrategy learning framework. l In our framework, the label weighting functions qtX are computed for each base Xl (i), which learner as follows, using the weight vectors of the l-th base learner wt,y is the i-th example’s weight for the y-th class (y = yi ) at the t-th round: l
l qtX (i, y)
l
X where w1,y (i) =
1 m(k−1)
=
X wt,y (i) l
WtX (i)
for all l and i, and l Xl WtX (i) = wt,y (i). y=yi l
Then, the weight distribution DtX for the l-th base learner at the t-th round can be computed based on the weight vectors of the other base learners as follows: n Xj l 1 W (i) DtX (i) = . (1) m t X j n−1 i=1 Wt (i) j=l
504
H. Nomiya and K. Uehara
Equation (1) represents the intensive and collaborative learning framework. It l is obvious that the weight distribution DtX depends on the learning result of the other base learners X j (j = l). If X j misclassifies an example, then X l ’s weight is increased. Thus, X l intensively learns the examples misclassified by X j . Conversely, if X l misclassifies some examples, X j intensively learns them. Consequently, each base learner is collaboratively trained by leaving the examples misclassified by itself to the other base learners and training the examples misclassified by the other base learners. This intensive and collaborative learning framework leads to the better performance than the conventional multistrategy learning which integrates only the learning results of each base learners. l of the l-th base learner at the t-th round is given by The pseudo-loss εX t ⎛ ⎞ m l l l l l 1 ⎠ DX (i) ⎝1 − [hX qtX (i, y)[hX εX t = t (xi ) = yi ] + t (xi ) = y] . 2 i=1 t y=yi
For any predicate π, we define that [π] = 1 if π holds, otherwise [π] = 0. At the next round, the new weights vectors can be calculated as
l
βtX Xl Xl Xl Xl wt+1,y (i) = wt,y (i) exp 1 + [ht (xi ) = yi ] − [ht (xi ) = y] 2 l
l
l
X where, βtX = log{(1 − εX t )/εt }. l We define the final hypothesis H X of the base learner X l as follows: l
l
l
H X (x) = argmax HTX (c, x); HtX (c, x) = c∈Y
t
l
l
βτX [hX τ (x) = c].
τ =1
where T is the number of rounds. To evaluate each hypothesis, we compute the class separability at each round. The class separability is a criterion calculated using a confusion matrix. If most of the training examples belonging to a class are correctly classified, then the class separability is high. Conversely, if most of the training examples are confused with the other class(es), then the class l l separability is low. We define the class separability sX t (c) of X for the c-th class at the t-th round as follows: ⎧ l Xl ⎨ sX t+ (c) if argmax Ht (y, x) = c l X y∈Y (2) st (c) = ⎩ sX l (c) otherwise. t− where,
C C
l
l sX t+ (c)
nX t,c,c
= C
i=1 l
l
nX t,c,i
,
l sX t− (c)
l
i=c
j=c
nX t,i,j
i=c
j=1
nX t,i,j
= C C
l
and nX t,i,j denotes the number of the examples whose class label is i and classified l
into the class j by HtX .
Multistrategical Approach in Visual Learning
505
As the learning proceeds, each base learner will be specialized in discriminating a kind of objects which are relatively easy to classify. As a result, the base learner can very precisely classify some kinds of objects but may sometimes misclassify the other objects. Thus, we estimate the confidence of the predictions of all the base learners to determine reliable base learners. We utilize the class separability of each weak hypothesis to estimate the confidence of the prediction of a base learner. Especially, the weak hypotheses generated at the beginning of the learning are more suitable because the weak hypotheses in the later stage of the learning are specialized. Thus, we define the confidence K based on the class separability of each weak hypothesis, emphasizing the suitable hypotheses. Xl (c) + KtXl (c) = Kt−1
t
l sˆX i (c)
(3)
i=1 X where K0X (c) = 0 for all c and sˆX t (c) is calculated by replacing Ht in equation X (2) with ht . The final hypothesis H is computed by integrating the learning results of all the base learners considering their confidence values as follows:
L X Xl l KT (c)HT (x) . (4) H(x) =argmax c
l=1
That is, the final hypothesis contains two prediction steps. The first step is to predict which base learner can correctly classify the example. The second step is to predict the class label of the example by the hypotheses of the base learners.
3
Base Learners
The appearance is an essential feature to recognize objects. Thus, we utilize a set of straight lines extracted from the contour. We call the lines contour fragments. Since a contour fragment is too simple, we discriminate objects by finding meaningful combinations of the contour fragments called patterns. In the learning process, we first extract contour fragments using stick growing method [4]. Next, we find meaningful combinations of contour fragments. We show the process in Figure 1. In Figure 1, (a) and (b) represent the original image and the extracted contour fragments respectively. P1 , P2 and P3 in (c) are the patterns found by searching mutually adjacent contour fragments.
Fig. 1. An example of the pattern extraction process
506
H. Nomiya and K. Uehara
A frequent pattern, which is common to the examples in a certain class is useful. We define a frequent pattern as the pattern which satisfies the condition nc that N > ρ, where, for the c-th class, nc is the number of examples which c nc contain the pattern. Nc is the number of examples in the c-th class. Thus, N c corresponds to the probability that the pattern is included in the c-th class. ρ is the frequency threshold. To find useful frequent patterns, we define a criterion to evaluate the usefulness U of a frequent pattern p for the c-th class as follows: nc . (5) Uc (p) = C i=1 ni where C is the number of classes. When a test example is given, the frequent patterns {pi } (i = 1, · · · , m) are extracted from the object, where m is the number of frequent patterns. Each pattern pi is compared with each useful frequent pattern qic (i = 1, · · · , mc ) for each class, where mc is the number of frequent patterns in the c-th class. The similarity σ(pi , qi ) between pi and qi is calculated for each frequent pattern. Then, the confidence Sc of the test example for the c-th class is calculated as follows. Sc corresponds to the possibility that the class label of the example is c. Sc =
M
σ(pi , qi ) where M = min{m, mc }.
(6)
i=1
If p is similar to q, σ(pi , qi ) is Uc (qi ), otherwise 0. In equation (6), pi and the corresponding frequent pattern qi are determined so that Sc is minimized. To determine whether a pattern p is similar to a pattern q, we define the following conditions for each contour fragment lip and liq in p and q: Condition 1: Condition 2: Condition 3:
np = nq = n. p q 1 r < |li |/|li | < r (i = 1, · · · , n). p q A(li , li ) < θ (i = 1, · · · , n).
where np and nq are the numbers of the contour fragments included in p and q. A(x, y) represents the angle between x and y. r and θ are the thresholds. When all the contour fragments in p and q satisfy these conditions, p is similar to q. The test example is classified into the class which has the highest confidence. The region component which represents the color and texture is a discriminative feature. We consider the region component in the minimum region that contains all contour fragments represented by the encircled regions in Figure 2. Figure 2 (a) is the minimum region in the original image. (b) is the corresponding contour fragments. We use the pixel intensity values in the minimum region. But there is the problem of the computational cost caused by a large amount of pixels. To solve this problem, we introduce Generic Fourier Descriptor (GFD) [5] and reduce the dimensionality of the feature vectors. GFD is a rotation-invariance descriptor derived by applying a 2D polar Fourier transform to the polar image as shown in Figure 2 (c). The transform is given by F (ρ, φ) =
−1 R−1 T r=0 i=0
2πi r φ) f (r, θi ) exp j2π( ρ + R T
(7)
Multistrategical Approach in Visual Learning
507
Fig. 2. An example of the minimum region and GFD
where R and T are the radial and angular resolutions and θi = 2πi T (0 ≤ i < T ). Figure 2 (d) represents the Fourier coefficients. We use the GFD feature vector as the feature vector calculated as follows:
GF D =
|F (0, n)| |F (m, n)| |F (0, 0)| ,···, ,···, area |F (0, 0)| |F (0, 0)|
(8)
where area represents the area of the polar image. m and n are the maximum numbers of the radial and angular frequencies respectively. GFD is calculated to obtain the feature vector of the minimum region. The similarity between two images is calculated as the distance between the two GFD feature vectors. The similarity S(x, t) between a test example x and a training example t is defined as the Euclidian distance between the GFD feature vectors as follows: S(x, t) =
d
− 12 {GF Di (x) − GF Di (t)}
2
(9)
i=1
where d is the number of dimensions of the GFD feature vectors. GF Di (x) and GF Di (t) are the i-th dimension’s values of the examples x and t respectively. The example x is classified into the class which has the highest average similarity.
4
Experiments
We carried out some experiments to verify the performance of our method with real-world images. The images are in the ETH-80 Image Set database [6]. This data set contains 8 different objects; apple, car, cow, cup, dog, horse, pear, and tomato. Each class contains 410 images (10 kinds of objects from 41 different directions). We used a total of 656 images as the training set and the other 2624 images as the test set. The number of rounds is experimentally set to 100. In this experiment, we used the following six base learners. The first and second learners are the appearance and region based methods proposed in this paper. The third learner is based on feature tree [2]. This method combines some predefined features into some decision trees called feature trees. The fourth learner is based on Scale Invariant Feature Transform (SIFT) [7]. This method generates the deformation-invariant descriptors by finding some keypoints in an object. The fifth learner is based on PCA-SIFT [8]. It can generate more distinctive and compact descriptors than SIFT by introducing Principal Component
508
H. Nomiya and K. Uehara
Analysis. The sixth learner is based on shape context method [9]. It discriminates an object using a set of points on its contour called shape context. 4.1
Comparison with Other Object Recognition Methods
We compare the recognition performance of our method with the following six object recognition methods. First, the shape context by Belongie et al. [9]1 . Second, an appearance-based method called multidimensional receptive histogram by Schiele et al. [10]. This method describes local shapes of objects using statistical representations to recognize the objects. Third, a region-based method based on color indexing by Swain et al. [11]. It discriminates an object using RGB histograms calculated from the pixels in the object. Fourth, a regionbased method using local invariant features by Grauman et al. [12]. It utilizes deformation-invariant local features generated by a gradient-based descriptor. Fifth, a region-based method based on boosting by Tu et al. [13]. In this method, probabilistic boosting-tree framework is introduced to construct discriminative models. Finally, a visual learning method by Mar´ee et al. [14]. In this method, an object is described by randomly extracted multi-scale subwindows in the image. The random decision tree ensembles are constructed using the subwindows. The recognition accuracy for each method is shown in Table 1. Table 1. The recognition accuracy (in %) of each method methods Swain accuracy(%) 64.85
Mar´ee 74.51
Tu 76
Schiele 79.79
Grauman 81
Belongie 81.06
proposed 84.49
Our method outperforms all the other recognition methods. This result reflects the effectiveness of the multistrategical learning. Although the base learner using shape context method greatly contributes the high accuracy, our multistrategy learning is fully effective because the recognition performance is considerably improved compared with the recognition accuracy of a single base learner. 4.2
Recognition Performance of Multistrategical Learning Methods
In order to verify the effectiveness of our multistrategy learning from the viewpoint of the number of base learners, we construct five object classifiers using two, three, four, five and six different base learners and compare them. We call these classifiers L2 , L3 , L4 , L5 and L6 respectively. Li contains a total of i base 1
It is reported in [6] that the shape context method achieved 86.40% accuracy. However, this recognition accuracy has been achieved using over 98% of examples in the training set. Since this method discriminates objects by matching with each object in the training set, the accuracy is proportional to the number of the training examples as shown in [9]. Thus, we show in Table 1 the recognition accuracy of the shape context method using 20% of training examples in the same way as our experiments.
Multistrategical Approach in Visual Learning
509
95% L2 L3 L4
90
L5 L6 shape context
Accuracy
85 80 75 70 65 60
apple
car
cow
cup
dog
horse
pear
tomato
total
Fig. 3. The recognition accuracy with multiple base learners 95% 90
Accuracy
85
L2 L3
L5 L6
L4
shape context
80 75 70 65 60
apple
car
cow
cup
dog
horse
pear
tomato
total
Fig. 4. The recognition accuracy with hard example elimination
learners. In addition, since we use the shape context method as the sixth base learner, we also show the recognition accuracy of the shape context method for comparison. The result of the experiment is shown in Figure 3. The recognition accuracy is proportional to the number of base learners. The third learner (feature tree) and the fourth learner (SIFT) give new features to L3 and L4 , so that more complex objects can be precisely described. The fifth learner (PCA-SIFT) is similar to the fourth learner because it is based on SIFT. Thus, the recognition performance is slightly improved in L5 . Introducing the sixth base learner (shape context) considerably improved the total recognition accuracy. Although this improvement is due to the high recognition performance of the shape context method, the recognition accuracy of L6 is significantly higher than that of the shape context method. In addition, the recognition accuracy of our method for apples does not degrade by introducing the shape context method, in spite of the low recognition accuracy of the shape context method. But there is room for improvement because the classification accuracy of our method for cars, cows and cups is lower than that of the shape context method.
510
H. Nomiya and K. Uehara
The main reason of this result is that our method is vulnerable to hard examples. Hard examples mean noisy examples such as deformed or occluded images. In our method, the examples misclassified by all the base learners are regarded as hard examples. In AdaBoost, it is a crucial problem that hard examples often cause overfittng and degrades the classification performance. In order to investigate the influence of hard examples, we performed an additive experiment by introducing NadaBoost [15], which can detect hard examples. We used NadaBoost instead of AdaBoost.M2 and eliminated hard examples detected by NadaBoost during learning process. The result of the experiment is shown in Fig 4. By eliminating hard examples, overall recognition performance of our method is improved and our method outperforms the shape context method in all the objects. From the result, we confirmed the influence of hard examples and the necessity to appropriately detect and eliminate hard examples. 4.3
Effectiveness of the Method to Integrate Visual Learners
To verify the effectiveness of our integration method, we compare our method with a voting method using the six base learners. The voting method separately trains the base learners and combines them using a voting method without weighting. The result using 5-fold cross validation is shown in Table 2. Table 2. The recognition accuracy (in %) of our method and the voting method apple car cow cup dog horse voting 84.51 85.06 69.45 82.38 70.07 70.43 proposed 90.30 89.88 76.10 87.01 78.23 79.27
pear 83.60 89.63
tomato 75.79 85.49
total 77.66 84.49
For all objects and the total recognition accuracy, the accuracy of our method is significantly better than the voting method using t-test with a significant level at 5%. The voting method equally treats the predictions of all the base learners even if the object is often misclassified by a base learner while the other base learner correctly classifies it. In addition, each base learner is separately trained. As a result, using the voting method deteriorates the total recognition performance. Our method can avoid this problem by selecting suitable base learner depending on the given object. This result shows the advantage of our integration method and our learning framework.
5
Conclusion and Future Work
In this paper, we proposed an effective object recognition method based on multistrategical visual learning. Since our method collaboratively trains and integrates multiple visual learners, the discrimination performance can be improved compared with the monostragety visual learning methods. Through the experiments, we verified the performance of our methods. The experimental result shows that complex objects can be correctly discriminated by integrating diverse visual learners. However, the recognition accuracy of our method is still
Multistrategical Approach in Visual Learning
511
inadequate in the presence of hard examples. But from the result of an additive experiment, we should make our method more robust over the hard examples by appropriately detecting and eliminating hard examples.
References 1. Levin, A., Viola, P., Freund, Y.: Unsupervised improvement of visual detectors using co-training. In: Proc. of the 9th IEEE International Conference on Computer Vision, pp. 626–633. IEEE Computer Society Press, Los Alamitos (2003) 2. Nomiya, H., Uehara, K.: Feature construction and feature integration in visual learning. In: Gama, J., Camacho, R., Brazdil, P.B., Jorge, A.M., Torgo, L. (eds.) ECML 2005. LNCS (LNAI), vol. 3720, pp. 86–95. Springer, Heidelberg (2005) 3. Freund, Y., Schapire, R.E.: A decision theoretic generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences 55(1), 119–139 (1997) 4. Nelson, R.C.: Finding line segments by stick growing. IEEE Transactions on Pattern Analysis and Machine Intelligence 16(5), 519–523 (1994) 5. Zhang, D., Lu, G.: Enhanced generic fourier descriptors for object-based image retrieval. In: Proc. of IEEE International Conference on Acoustics, Speech, and Signal Processing, pp. 3668–3671. IEEE Computer Society Press, Los Alamitos (2002) 6. Leibe, B., Schiele, B.: Analyzing appearance and contour based methods for object categorization. In: Proc. of International Conference on Computer Vision and Pattern Recognition, pp. 409–415 (2003) 7. Lowe, D.: Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision 60(2), 91–110 (2004) 8. Ke, Y., Sukthankar, R.: PCA-SIFT: A more distinctive representation for local image descriptors. Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 506–513 (2004) 9. Belongie, S., Malik, J., Puzicha, J.: Matching shapes. In: Proc. of the 8th IEEE International Conference on Computer Vision, pp. 454–463. IEEE Computer Society Press, Los Alamitos (2001) 10. Schiele, B., Crowley, J.L.: Recognition without correspondence using multidimensional receptive field histograms. International Journal of Computer Vision 36(1), 31–50 (2000) 11. Swain, M.J., Ballard, D.H.: Color indexing. International Journal of Computer Vision 7(1), 11–32 (1991) 12. Grauman, K., Darrell, T.: Efficient image matching with distributions of local invariant features. In: Proc. of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 2, pp. 627–634 (2005) 13. Tu, Z.: Probabilistic boosting-tree: Learning discriminative models for classification, recognition, and clustering. In: Proc. of the 10th IEEE International Conference on Computer Vision, vol. 2, pp. 1589–1596 (2005) 14. Mar´ee, R., Geurts, P., Piater, J., Wehenkel, L.: Random subwindows for robust image classification. In: Proc. of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 1, pp. 34–40 (2005) 15. Nakamura, M., Nomiya, H., Uehara, K.: Improvement of boosting algorithm by modifying the weighting rule. Annals of Mathematics and Artificial Intelligence 41, 95–109 (2004)
Cardiac Motion Estimation from Tagged MRI Using 3D-HARP and NURBS Volumetric Model Jia Liang1, Yuanquan Wang2, and Yunde Jia1 1
School of Computer Science&Technology, Beijing Institute of Technology, Beijing 100081, P.R. China 2 School of Computer Science, Tianjin University of Technology, Tianjin 300191, P.R. China {liangjia,yqwang,jiayunde}@bit.edu.cn
Abstract. Concerning analysis of tagged cardiac MR images, harmonic phase (HARP) is a promising technique with the largest potential for clinical use in terms of rapidity and automation without tags detection and tracking. However, it is usually applied to 2D images and only provides “apparent motion” information. In this paper, HARP is integrated with a nonuniform rational Bspline (NURBS) volumetric model to densely reconstruct 3D motion of left ventricle (LV). The NURBS model represents anatomy of LV compactly, and displacement information that HARP provides within short-axis and long-axis images drives the model to deform. After estimating the motion at each phase, we smooth the NURBS models temporally to achieve a 4D continuous timevarying representation of LV motion. Experimental results on in vivo data show that the proposed strategy could estimate 3D motion of LV rapidly and effectively benefiting from both HARP and NURBS model. Keywords: Tagged MRI, LV, motion estimation, HARP, NURBS model.
1 Introduction Tagged MRI [1,2] is a noninvasive technique that can be used for quantitative assessment of myocardial function and dynamic behavior of human heart, which is invaluable in the diagnosis of myocardial diseases [3]. In tagged MRI, tags move with underlying tissue during cardiac cycle, providing unsurpassed information about myocardium motion which can be used to calculate local strain and deformation indices from different myocardial regions. Hence, many researches have contributed to analyze deformation of tags to derive a motion model of the underlying tissue, such as [4,5,6]. These methods for tags detection and tracking are almost manual or semiautomatic with human interaction, which make cardiac motion analysis more time-consuming and more dependent on the validity of the detected tags. Therefore it is imperative to develop an automatic method for estimating heart motion. Recently, Osman et al. [7-9] have introduced a harmonic phase (HARP) technique for cardiac motion tracking without extracting tag features from an image. The approach treats harmonic phase which is computed from inverse Fourier transform of the first off-center isolated spectral peak in the Fourier domain of tagged MRI as a Y. Yagi et al. (Eds.): ACCV 2007, Part I, LNCS 4843, pp. 512–521, 2007. © Springer-Verlag Berlin Heidelberg 2007
Cardiac Motion Estimation from Tagged MRI
513
material property for tracking underlying motion and calculating myocardial strain. It is a promising technique with the largest potential for clinical use in terms of rapidity, simplicity, robustness and automation. However HARP technique is mostly applied to 2D images, typically on short-axis (SA) image planes and thus, only provides information about “apparent motion” while the true motion of heart is 3D. Some of work has been done to deal with this problem. Ryf et al. [10] presented a combined 3D tagging and imaging approach. Haber and Westin [11] constructed a FEM within LV wall and the HARP phase computed on a sparse collection of image planes. Pan et al. [12] developed a mesh model approximating the mid-wall of LV and then applied HARP to track the mesh. In this paper, we present a novel method of integrating HARP with a nonuniform rational B-spline (NURBS) volumetric model to densely reconstruct 3D motion of LV. First we extend HARP from 2D to 3D mathematically, making full use of information afforded by SA and LA images each with a grid tagging pattern to obtain three mutual orthogonal components of LV motion displacements. This process still remains the advantages of rapidity and automation of the HARP method. Then a NURBS volumetric model is employed to fit the complex anatomy of LV. The model has its trait of immediate and compact representation of wide variety of shapes, and can model the geometry of LV well only with few control points coming from the sparse image planes. Also the model interpolates the known sparse displacements to get a 3D dense motion field reflecting the natural continuity and smoothness of the three-dimensional tissue deformations. Finally, after smoothing the model temporally, we could estimate 3D motion of LV at any time, and consequently the local strain and deformation indices from different myocardial regions could be computed.
2 3D Extension of HARP Harmonic phase (HARP) is an image processing technique developed to analyze the motion of heart rapidly using MR tagging [7,8]. It is mostly applied to 2D images, typically on SA image planes and thus, only provides information about “apparent motion”. The actual motion of heart is not confined to the imaging plane but likely out of it, however, 2D-HARP motion computations do not yield the true motion. Therefore we extend this method to 3D-HARP that yields a more comprehensive description of the full 3D motion of myocardial tissues. This extension is also based on the principle that harmonic phase value of a material point is time-invariant which material property we could use to track the motion of material points of heart in threedimensional space. In tagged MRI, one tag pattern can only provide one component of the underlying motion. To achieve a full 3D tracking of any point, the information coming from different mutually orthogonal tagging patterns has to be combined and interpolated in space and time. Hence, we need acquire SA image planes with two orthogonal tag orientations to obtain in-plane motion and several LA image planes to capture the third directional component of tissue motion normal to the short axis image planes. In LA images arranged radially, tagging planes are usually applied orthogonal to the long axis appear as parallel lines measuring longitudinal compression. Since HARP
514
J. Liang, Y. Wang, and Y. Jia
method needs 2D tagging images, LA images should be added another orthogonal tag pattern when imaging. SA and LA images each with a grid tagging pattern are acquired for 3D-HARP. To detail the procedure of 3D-HARP analysis formally, we define a material point p ∈ IR 3 that lies in the cross-section of SA image and LA image. Points y S and y L are its projections onto the SA and LA. To extract useful information of the Fourier transform of the SA and LA, we use the band-pass filter [8] to isolate the off-center (non-dc) spectral peak in one tag direction and rotate this filter by 90 degree to isolate the spectral peak in the orthogonal tag direction. Then zero padding the rest of the Fourier transform and performing inverse Fourier transform yields a complex image whose calculated angle is called a harmonic phase image. The harmonic phase image gives a detailed picture of myocardial motion in corresponding direction. Harmonic
phases (φ1, φ 2 ) computing from the SA in x/y tag direction corresponding to y S and T
(φ3 , φ4 )T from
the LA in z/z’ (orthogonal to z) tag direction corresponding to y L remain time-invariant throughout the motion. Thus, we track the point that has the same phase value in the filtered image sequence. Here the phase value of p is denoted by a vector φ = (φ1 , φ 2 , φ 3 ) , where φ1 , φ 2 and φ 3 come from three mutually orthogonal tagging directions, φ 4 is just for tracking the point on LA with 2D-HARP. Assume a T
material point located at p t at time t . If p t +1 is the position of this point at time t + 1 , then
( )
(
)
φ n p t , t = φ n p t +1 , t + 1
n = 1,2,3 .
(1)
This relationship provides the basis for tracking p t from time t to time t + 1 . In practice, φ can not be calculated and visualized from the data directly, so its principle value produced by wrapping function often takes the place of it. Despite the wrapping artifact, the principle value is also the material property of tagged tissue and maintains time-invariant. The following work is to target y S with the same harmonic phase (φ1, φ 2 ) from SA and y L with the same harmonic phase (φ3 , φ 4 ) from LA. The T
T
(
displacement field u = u x ,u y ,u z
)T for all intersections could be received.
3 NonUniform Rational B-Splines Model 3.1 Principle
NURBS, an acronym for nonuniform rational B-splines, have become the de facto standard for computational geometric representation [13]. Their predominance lies on parametric continuity, local support, compact and unified mathematical representation and an extra degree of freedom, in the form of weights, apart from knot vector and control points, which can be used in designing wide variety of shapes.
Cardiac Motion Estimation from Tagged MRI
515
The NURBS volumetric model can be expressed as m
n
h
∑∑∑ ω
Di,j,f N i,k1 (u )N j,k2 (v )N f,k3 (w )
i,j,f
i =0 j =0 f =0 m n h
P(u,v,w) =
∑∑∑ ω
i,j,f
i =0 j =0 f =0
N i,k1 (u )N j,k 2 (v )N f,k3 (w)
,
(2)
where Di,j,f represents a control point, ωi,j,f is the corresponding weight. Quantities
u , v and w are location parameters. N (* ) is B-spline basis function. The variables k1 , k 2 and k 3 are orders (one more than degree) of model in the direction of u , v and w
respectively. After defining knot vectors U = {u1 ,… , u ms } , V = {v1 ,… , vns } and
W = {w1 ,… , whs } each to be a sequence of nondecreasing real numbers, called knots, a NURBS volume model can be uniquely defined. The following work is to solve control points and weights in a least-squares sense. Given a set of discrete points Γ = Pp ,q ,r = x p ,q ,r ,y p ,q ,r ,z p ,q ,r and a corresponding set
{
{
}
(
)}
of weights Λ = ξ p ,q ,r which quantifies the relative confidence in the measurement of each corresponding point, where p ∈ [0 , s1 ], q ∈ [0 , s 2 ], r ∈ [0 , s 3 ] . Then the weighted least squares error criterion Eξx for the x-coordinate is
⎛ ⎜ ⎜ Eξx = ξ p ,q ,r ⎜ x p ,q ,r − ⎜ p ,q ,r ⎜ ⎜ ⎝
∑
2
⎞ N i,k1 u p N j,k2 vq N f,k3 (wr ) ⎟ ⎟ i =0 j =0 f =0 ⎟ , m n h ⎟ ωi,j,f N i,k1 u p N j,k 2 vq N f,k 3 (wr ) ⎟ ⎟ i =0 j =0 f =0 ⎠ m
n
h
∑∑∑
ωi,j,f Dix, j , f
( )
( )
∑∑∑
( )
( )
(3)
where Dix, j , f is x component of control point Di,j,f . Other parameters are defined as previous. It is known that rational B-splines can be produced by projecting nonrational B-splines in hyperplane w = 1 . Therefore we substitute 3D control points
{ D ′ = {D
}
Di = Dix , Diy , Diz = {xi , yi , zi } in 3D space by 4D homogeneous coordinates wx i
i
, Diwy
squares error
Eξwx
}
, Diwz , Diw = {wi xi , wi yi , wi z i , wi } , and then the weighted least
criterion Eξwx
for the x-homogeneous coordinate is given by
m ⎛ = ξ p ,q ,r ⎜ x p ,q ,r D pw,q ,r − ⎜ p ,q ,r i =0 ⎝
∑
n
h
∑∑∑
k1 ,k 2 k3 Diwx , j , f Bi , j , f
j =0 f =0
2
⎞ u p , vq , wr ⎟ , ⎟ ⎠
(
)
(4)
where
(
)
( )
( )
Bik,1j,,kf2 k3 u p , vq , wr = N i,k1 u p N j,k 2 vq N f,k3 (wr ) ,
(5)
516
J. Liang, Y. Wang, and Y. Jia m
D pw,q ,r =
n
h
∑∑∑ ω
i,j,f
(
)
Bik,1j,,kf2 k3 u p , vq , wr .
i =0 j =0 f =0
(6)
Similar equations are used for y and z components. Then we follow the methodology given by Tustison and Amini [14] to solve the equations for the control points and the weights. 3.2 Anatomy of the LV
Image planes acquired by tagging MRI imaging technique are usually sparse 2D images while the actual anatomy of LV is very complex and looks like a prolate spheroid. In order to cover the full geometry we employ a NURBS volumetric model for it can be created with only few control points. From sparse SA and LA images, a set of finite discrete points locating on intersections of contours and images are observed. The phantom of LV anatomy and the parametric directions of model are shown in Fig.1(d). Applying the fitting algorithm in Section 3.1, the NURBS volumetric model is received. As the nature of parametric continuity and compact representation, the model could show any points of the 3D LV very well. 3.3 3D Motion Estimations
After building the NURBS volumetric model of the current phase, 3D-HARP is applied on the SA and LA images at the next phase to obtain the sparse 3D displacements. Then these displacements drive the model to deform. For the sake of the predominance of parametric continuity, local support and differentiability, we could gain the dense 3D motion displacement field by difference of the NURBS volumetric models. 3.4 Four-Dimensional (4D) NURBS Model
After the 3D NURBS volumetric model is generated across 3D space for each phase, a time variable dimension is added to smooth it along all continuous time points. Given orders of the model, 4D grid control points, corresponding weights, and knot vector sequence, the 4D NURBS model can be written as P(u,v,w, t) =
∑∑∑∑ N (u )N (v )N (w)N (t )ωD , ∑∑∑∑ N (u )N (v )N (w)N (t )ω
(7)
where t denotes the time instant. The meanings of other parameters are the same as defined previously, and the subscripts are omitted for concise writing.
4 Strain Analysis Strain is a dimensionless quantity measuring the percent change in length at different points to describe the internal deformation of a continuum body. It is an appealing tool to study and quantify myocardial deformation.
Cardiac Motion Estimation from Tagged MRI
517
Given the spatial coordinates of a point in the material, X at time t = 0 and x at time t > 0 , the deformation gradient tensor F includes both the rotation and deformation around a point in the material and can be calculated by
Fpq = ∂x p ∂X q ,
(8)
where the subscripts p and q range from 1 to 3 and denote one of the 3D Cartesian coordinates. The Lagrangian strain tensor E only includes the deformation of the material with respect to its initial configuration, and is related to F as follow:
(
)
E = 1/2 F T F − I ,
(9)
where the superscript T represents the matrix transpose and I represents the identity matrix. The Lagrangian strain E is used to describe systolic deformation in a region surrounding a point in the heart wall relative to its initial position at end-diastole.
5 Experiments on in vivo Data 5.1 Imaging Protocol
For the studies shown in this paper, the following imaging protocol was utilized on a 1.5T Sonata produced by the Manufacturer of Siemens. Two sequences consisting of 11 phases of 9 tagged MR SA images and 9 LA images each with a grid tagging plane, 256 × 208 acquisition matrix and 8 mm slice thickness were acquired throughout the cardiac cycle from a normal healthy volunteer. LA images were taken perpendicular to SA images and arranged radially every 20 degrees around the long axis of LV. Fig. 1 shows the spatial relative position of the SA and LA images. 5.2 Initial NURBS Volumetric Model
To construct the reference NURBS volumetric model of LV, we segment the endocardial and epicardial contours of LV from the end-diastolic images manually by experience. Then the initial reference NURBS (cubic spline) volumetric model of LV is built illustrated by the left top one of Fig. 2. Here color means nothing and the cyan pentagrams denote the points at the reference phase.
(a)
(b)
(c)
(d)
Fig. 1. (a) The spatial relative position of the SA and LA images of LV. Red circles denote the origins of parallel SA images. (b) LA images locations (denoted by red lines) on a sample SA image. (c) SA images locations (denoted by red lines) on a sample LA image. (d) Parametric directions of the NURBS volumetric model of LV.
518
J. Liang, Y. Wang, and Y. Jia
Fig. 2. The NURBS volumetric models of LV at six different instants and the displacements are illustrated. The cyan pentagrams denote the points at the reference phase and the blue pentagrams mean the corresponding positions at the current phase. The pink vectors represent the displacements roughly. The color bar denotes the length of displacements. Left top one shows the initial NURBS volumetric model of the real LV and the reference points.
5.3 3D Motion Reconstruction
Once the initial reference NURBS volumetric model of LV is achieved, the phases of the material points on the model are known. The following work is to track the 3D motion of this model according to the material property of phase-invariant using 3DHARP method. Fig. 2 shows the NURBS volumetric models and the displacement field during the cardiac systole. These results are similar to the work [15]. 5.4 4D NURBS Model and Strain Analysis
The 4D NURBS model is generated by smoothing the 3D NURBS volumetric models over time using cubic splines, which is exactly suitable for real heart well. The movement of each myocardial point over time can be captured accurately by assigning u , v and w to any fractional values. In addition, the shape of the LV at any time instant can be obtained by setting t to any desired value. Using this model the changes of displacement and strain over time can be obtained at all myocardial points with sub-pixel accuracy. Due to the LV geometry, it is appropriate to calculate the myocardial strains based on the radial, circumferential, and longitudinal directions. The basal and midcavity portions of it are each divided into six regions in the SA view: antero-septal, anterior, lateral, posterior, inferior, and infero-septal. Normal LV strains, i.e., average radial, circumferential, and longitudinal strains, are given in Fig. 3. These results are similar to the work [16]. The radial strains mostly remain positive indicative of the systolic thickening of the LV. Circumferential and longitudinal strains are negative denoting
Cardiac Motion Estimation from Tagged MRI
519
Fig. 3. Average Lagrangian normal strains are plotted for the six basal and midcavity regions of the left ventricle of a normal human volunteer. The different geometric shapes (star, diamond, and square), represent the radial, longitudinal, and circumferential strain values, respectively. The x axis marks the time point during systole.
520
J. Liang, Y. Wang, and Y. Jia
shortening in the circumferential direction and compression in the longitudinal direction during LV contraction.
6 Conclusion In this paper, we have proposed a novel method for dense 3D motion estimation of LV without tag detection and tracking. This method takes advantages of rapidity and automation of HARP technique. Also it benefits from NURBS properties, such as parametric continuity, local support, and compact and unified mathematical representation for wide variety of shapes. Under this framework, we have created the compact representation of the complex LV anatomy and reconstructed the motion of the LV on in vivo data, experimental results show that the dense 3D motion estimation and the local strains could be calculated rapidly and effectively. It is strongly felt that this tool will help take MR tagging from the ranks of a valuable scientific research tool into the ranks of a valuable diagnostic clinical tool. This method is also suitable for the right ventricle and even for the atria only with a different model initialization. Acknowledgments. We are grateful to Prof. Pheng Ann Heng in the Chinese University of Hong Kong for providing the in vivo human heart data. This work was supported by the Natural Science Foundation of China under grants 60602050 and 973 Program of China (No. 2006CB303105).
References 1. Zerhouni, E.A., Parish, D., Rogers, W., Yang, A., Shapiro, E.: Human heart: tagging with MR imaging — a method for non-invasive assessment of myocardial motion. J. Radiology 169, 59–63 (1988) 2. Axel, L., Dougherty, L.: MR imaging of motion with spatial modulation of magnetization. J. Radiology 171, 841–845 (1989) 3. Masood, S., Yang, G.-Z., Pennell, D.J., Firmin, D.N.: Investigating intrinsic myocardial mechanics: the role of MR tagging, velocity phase mapping, and diffusion imaging. J. Magn. Reson. Imag. 12, 873–883 (2000) 4. Guttman, M.A., Prince, J.L., McVeigh, E.R.: Tag and contour detection in tagged MR images of the left ventricle. IEEE Trans. Med. Imag. 13, 74–88 (1994) 5. Amini, A.A., Chen, Y., Curwen, R.W., Mani, V., Sun, J.: Coupled B-snake grids and constrained thin-plate splines for analysis of 2-D tissue deformations from tagged MRI. IEEE Trans. Med. Imag. 17, 344–356 (1998) 6. Young, A.: Model tags: direct 3D tracking of heart wall motion from tagged magnetic resonance images. In: Wells, W.M., Colchester, A.C.F., Delp, S.L. (eds.) MICCAI 1998. LNCS, vol. 1496, pp. 92–101. Springer, Heidelberg (1998) 7. Osman, N., Kerwin, W., McVeigh, E., Prince, J.: Cardiac motion tracking using CINE harmonic phase (HARP) magnetic resonance imaging. J. Magn. Reson. Med. 42, 1048– 1060 (1999) 8. Osman, N., McVeigh, E., Prince, J.: Imaging heart motion using harmonic phase MRI. IEEE Trans. Med. Imag. 19, 186–202 (2000)
Cardiac Motion Estimation from Tagged MRI
521
9. Osman, N.F., McVeigh, E.R., Prince, J.L.: Visualizing myocardial function using HARP MRI. J. Phys. in Med. and Biol. 45, 1665–1682 (2000) 10. Ryf, S., Spiegel, M.A., Gerber, M., Boesiger, P.: Myocardial tagging with 3D-SPAMM. J. Magn. Res. Imag. 16, 320–325 (2002) 11. Haber, I., Westin, C.F.: Model-based 3D tracking of cardiac motion in HARP images. In: Int. Soc. Mag. Reson. Med. Honolulu, HI (2002) 12. Li, P., Prince, L.J., Lima Jiao, A.C., Osman Nael, F.: Fast tracking of Cardiac Motion Using 3D-HARP. IEEE Trans. BioMed. Eng. 52, 1425–1435 (2005) 13. Piegl, L., Tiller, W.: The NURBS Book. Springer, Berlin (1997) 14. Tustison, N.J., Amini, A.A.: Biventricular myocardial kinematics based on tagged MRI from anatomical NURBS models. In: IEEE Conf. on Computer Vision and Pattern Recognition, vol. 2, pp. 514–519 (2004) 15. Luo, G., Heng, P.A.: LV Shape and Motion: B-Spline-Based Deformable Model and Sequential Motion Decomposition. IEEE Trans. Inform. Technol. BioMed. 9, 430–446 (2005) 16. Moore, C., Lugo-Olivieri, C., McVeigh, E., Zerhouni, E.: Three-dimensional systolic strain patterns in the normal human left ventricle: characterization with tagged MR imaging. J. Radiology 214, 453–466 (2000)
Fragments Based Parametric Tracking Prakash C, Balamanohar Paluri, Nalin Pradeep S, and Hitesh Shah Sarnoff Innovative Technologies Private Limited, Asha arch, Magrath Road, Bangalore-560025, India
Abstract. The paper proposes a parametric approach for color based tracking. The method fragments a multimodal color object into multiple homogeneous, unimodal, fragments. The fragmentation process consists of multi level thresholding of the object color space followed by an assembling. Each homogeneous region is then modelled using a single parametric distribution and the tracking is achieved by fusing the results of the multiple parametric distributions. The advantage of the method lies in tracking complex objects with partial occlusions and various deformations like non-rigid, orientation and scale changes. We evaluate the performance of the proposed approach on standard and challenging real world datasets.
1
Introduction
Two prominent components of a tracking system are: object descriptor and search mechanism. Object descriptor is the representation of the object to be tracked using a set of features that capture various properties of the object such as the appearance, shape, texture etc. Given an object descriptor, the search mechanism like [1,2], locates the region in a new image that best matches the object description. There are multiple methods suggested in the literature for object descriptors. Most of the successful methods for tracking employ non-parametric object descriptor like histogram [1,3,4,5,6,7], as it faithfully captures the variability in the features of the object to be tracked. However, with the increase in number of objects to be tracked or the features to be considered, the histogram size grows exponentially which is an undesired behavior. To address this issue, we propose a parametric object descriptor for color based tracking. An N-dimensional Gaussian distribution is employed as the object descriptor in the proposed approach. Such a descriptor can accurately model a unimodal object. But objects under consideration for tracking are generally multimodal in color space making N-d Gaussian descriptor insufficient. Hence, we need to convert multimodal objects to unimodal representation. Primarily, there are two ways to achieve this conversion: – By projecting the multimodal object into a space where it becomes unimodal – By representing each mode separately Y. Yagi et al. (Eds.): ACCV 2007, Part I, LNCS 4843, pp. 522–531, 2007. c Springer-Verlag Berlin Heidelberg 2007
Fragments Based Parametric Tracking
523
An approach similar to Collins et al. [5] can be used to find a linear transformation to project the multimodal color object into a unimodal space. Also, non-linear transformation as suggested by Larry et al. [7] can be used to the same effect. However, in both the cases, the search for finding such an optimal transformation is not exhaustive, as it is computationally expensive, over the entire space of possible transformations. Hence the obtained transformation is suboptimal. For representing a multimodal object, Gaussian mixture model (GMM) distributions can also be used. However, computation of GMM parameters are expensive and an aprior knowledge of number of modes is quintessential, thus rendering it is not applicable for object descriptors in tracking. Therefore, in this paper we propose a method based on fragmenting multimodal objects into multiple homogeneous models using Discriminant Analysis. The fragmentation process finds the fragments online as opposed to fragmenting the object into fixed sizes as suggested in [4]. Each fragment is then modelled using a single N-dimensional Gaussian distribution and tracked separately. These parametric distributions are used to generate a probability density function termed as strength image. The maximum likelihood(ML) framework proposed in [2] is used to estimate the location(mean) and shape(covariance) of the best matching region in the subsequent frame. The paper is organized as follows: Section 2 explains the proposed approach. Experimental results are presented in Section 3 to illustrate the performance of the tracker and Section 4 concludes the work by presenting the future work.
2
Proposed Method
The proposed tracking approach is color based, hence in modelling an object we use the color values of the pixel. Our initial step involves fragmenting based on the color values of the pixels. Prior work involved application of multi-level thresholding technique to segment an illumination/gray image [8]; but these techniques can’t be applied directly in our case as the our objective is to group regions similar in color rather than illumination (gray). Hence, multi-level thresholding is done in color space. The input template is in the RGB space. Multi level thresholding on the histogram generated using all three channels is not possible due to the immense size of the histogram (256 × 256 × 256). So, given the color template of the object in the RGB space, we first transform the input to HSV space. Since Hue represents the color component alone, multi-level thresholding on Hue gives the desired results. The grouped regions similar in color are then modelled using single parametric distribution. The Uni/Multi modal classification and the fragmentation processing use the Hue image for processing. 2.1
Fragmentation
The given region of interest (ROI) is initially divided into uniform blocks of size M × N . Each block(B) with mean(μ), variance(σ) and Hue histogram(H) is then classified as homogeneous if any of the following two criterions satisfy:
524
Prakash C. et al.
1. If the variance of the region is less than a certain pre-defined value. The variance of the region is given by 2 (Pi − μ) (1) σ2 = iB
where Pi is the hue value at i and the μ is the mean of the block. 2. If the block is divided into two classes C1 with values [1, . . . , t] and C2 with values [t + 1, . . . , L] using the optimal threshold given by eqn 2 and if the Separability factor(SF ) given by eqn 3 of the block is less than a certain pre-defined value. arg max t
N
2
Wi (μi − μ)
(2)
i=1
where N is the number of classes, Wi is the total number of pixels in the class i, μi is the mean of the it h class and μ is the mean of the block. BCV (3) TV where T V is the total variance of the block given by 1 and BCV is the between class variance given by: SF =
BCV =
N
Wi (μi − μ)2
(4)
i=1
The fragmentation process is applied only for non-homogeneous regions. The multi-level thresholding is carried until the SF of the block is less than a predefined valueT hSF . The T V of the region is constant and is used for normalizing purposes. The BCV will be high when the fragments of similar color are grouped and dissimilar colors are separated.The multi level thresholding is done in the following way: Each time the fragment/class with the maximum within class variance is selected (Initially, the entire block is started as one class), since high within class variance signifies that the class is non-homogeneous. The division is done by finding the Optimal threshold given by 2. The process is repeated till the SF of the block is less than T hSF . The class pool thus created needs to be assembled together based on the color similarity. The assembled regions will signify the multiple unimodal regions of the multimodal object. The assembling process is started with a new region which includes the first class of the first region. This is followed by a merging process which finds out the classes similar to this class. The criterion for similarity is the difference of the mean values of the two classes. The class with the least difference is identified and if the difference of the means is less than a pre-defined value T hmean the class is merged into the region. If none of the classes can be merged to any of the existing regions, a new region is created by picking up the class which
Fragments Based Parametric Tracking
(a)
(b)
(d)
(e)
525
(c)
(f)
Fig. 1. Parrot sequence: The input image (a) is fragmented into five parts. (b) represents the body of the parrot(green), (c) represents the forehead(white), (d) represents the hair(blue), (e) represents the cheeks(red) and (f) represents the beak(yellow).
has the largest difference in the mean value with the existing regions. Then the unclassified classes are again tried to merge to this new region. The process is repeated until all the classes are merged to the regions. The regions obtained thus form the unimodal fragments of the multimodal object. An example of the fragmentation is shown in Figure 1. 2.2
Modelling the Object
Each fragment obtained after the fragmentation process is modelled separately. Each region R ⊂ Regions is described by the color values {R, G, B} , thus t the feature descriptor at an image location x = x y is computed as f (x) = t R(I, x) G(I, x) B(I, x) . The region covariance, C of the feature descriptors in R is computed as C=
1 (f (x) − μ )(f (x) − μ )t |R|
(5)
x∈R
1 where μ = |R| x∈R f (x) is the mean feature descriptor in R and |R| is the number of pixels in the region R. A simple covariance matrix computed with color features contains the information needed to capture the appearance of the object. An estimate of the color distribution in the target region is the Gaussian distribution. The ML estimates of the parameters of the Gaussian, Θ=(μ, C) is the target model. The probability density function (PDF), also termed as the strength image, is computed over the new image. The value of each pixel in
526
Prakash C. et al.
the strength image signifies the probability with which the pixel belongs to the target model. In the remainder of this paper we denote this value as p(x|Θ ) where x is the pixel location. The PDF in this case is computed as : p(x|Θ ) ∝ exp(−(f (x) − μ )t C−1 (f (x) − μ ))
(6)
The PDF is calculated for each of the unimodal regions of the object obtained through the fragmentation process. The PDF will have high values for pixels which belong to the particular parametric distribution and vice-versa. In the next section, we show how the PDF computed for the image can be used to track the region accurately in presence of various deformations. 2.3
ML Framework
The region to be tracked R0 is represented by an ellipse in our case. The position and shape of the object are described by the mean M0 and covariance V0 of the pixels in the region. Given the target model Θ, the objective of the search mechanism is to find a region R in the new frame described by mean and covariance (M, V) that maximize the function: p(x|Θ )L(x|M, V) (7) J(M, V) = x∈R
where the term L(x|M, V) ∝ exp(−(x − M)t V−1 (x − M))
(8)
prevents pixel locations that are farther from the original region from distracting the tracker. As a pixel’s contribution falls off with the distance from the original region, this helps in both reducing the effect of outlier pixel on the search as well as preventing the tracker from drifting away from the object. As shown in [2,9,10], the maximum-likelihood estimates of M and V can be obtained via an EM-like iterative procedure. The key to the method is to assume a set of hidden variables w(x). Starting with an initial estimate M0 , V0 of R, the EM-iteration proceeds as below: – E-Step: Given current estimates Mk and Vk of the mean and covariance of the region in k th iteration, compute hidden variables wk (x): p(x|Θ )L(x|Mk , Vk ) k k x ∈R p(x |Θ )L(x |M , V )
wk (x) =
(9)
– M-Step: Using the hidden variables computed above, compute the next estimates of mean and covariance of the region, Mk+1 and Vk+1 of that maximize J(., .): wk (x)x (10) Mk+1 = x∈R
V
k+1
=
x∈R
wk (x)(x − Mk+1 )(x − Mk+1 )t
(11)
Fragments Based Parametric Tracking
527
The optimal values for M and V are obtained by iterating the above steps until convergence. Our experimental results demonstrate that the search mechanism described above is both efficient and robust to a wide variety of changes in the shape of the object. In Algorithm 1, we explain the complete tracking algorithm. Algorithm 1. Track(Video V , Region R0 ) 1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12: 13: 14: 15: 16:
3
I0 ←Initial Frame(V ) (M0 , V0 ) ← Fit Ellipse(R0 ) H ← HSV(R0 ) Class P ool ← Multi Level Thresholding(H) F ragments ← Assembling(Class P ool) Θ ← Region Covariance(R0 , F ragments) for each frame Ii in V do (Mi , Vi ) ← (Mi−1 , Vi−1 ) S ← Strength Image(I, Θ ) k←0 while not converged do compute weight wk using equation 9 update estimates (Mi , Vi ) using equations 10, 11 k ←k+1 end while end for
Experimental Results
The tracking algorithm was tested on various challenging datasets [11]. It was also tested on few low contrast videos taken from Internet. The tracker performance was encouraging when tested for its ability to handle the following aspects: Non-rigid deformations: Tracking non-rigid objects is a challenging problem in tracking. Couple of examples are highlighted in Figures 2, 4. In Cat sequence (Figure 2), the cat is tracked accurately under considerable deformations (sitting,jumping and running). Also, note that the contrast between the cat and the background is quite low. In case of Figure 4, the monkey is tracked successfully under extreme deformations. In both the cases, the tracked ellipse changes accurately to handle the non-rigid deformations of the object. Orientation: Change in the orientation of objects is a common scenario in tracking. Tracking the object with accurate orientation is possible in our case since we track the object using an ellipse. The mean of the ellipse characterizes the location and the covariance signifies the scale and orientation. In Figure 4(b,c), the monkey undergoes considerable changes in orientation. The orientation of the ellipse changes according to the orientation of the object. Figure 3 is another example where the fish is tracked accurately in the presence of rapid orientation changes.
528
Prakash C. et al.
(a)
(b)
(c)
Fig. 2. Cat sequence: The cat is tracked successfully in presence of changes in scale and non-rigid deformations
(a)
(b)
(c)
Fig. 3. Fish sequence: An example of low quality video containing partial occlusions and orientation changes. Note the other fishes in the tank with the similar color(but not pattern) to the object being tracked. Many existing trackers fail in such cases.
(a)
(b)
(c)
Fig. 4. Monkey sequence: Inspite of changes in orientation and non-rigid deformations, the monkey is tracked precisely
Scale: Earlier trackers relied on techniques such as searching through exhaustive search space [12] or using templates of the object at different scales [13]. In our case, the EM-like algorithm enables efficient handling of scale changes by estimating the covariance of the tracked ellipse. Figure 2, shows how the tracking handles scale changes. Partial Occlusion: Parital and full occlusions occur frequently in tracking scenearios and the tracker needs to handle these sucessfully. Even if an object is
Fragments Based Parametric Tracking
(a)
(b)
529
(c)
Fig. 5. Caviar sequence: The sequence shows the handling of partial occlusion of the person (blue ellipse) when he crosses two other people
(a)
(b)
(c)
Fig. 6. Parrot sequence: The multimodal object parrot is decomposed into multiple unimodals and tracked seperately
(a)
(b)
(c)
Fig. 7. Parrot sequence: The multimodal object is tracked sucessfully. Note that the ellipse completely fits the entire parrot enclosing all the homogenous regions.
completely occluded for considerable time, the tracker should be able to track the object on reappearance. A scenario with complete and partial occlusions is shown in the Figures 3, 5. Figure 3(b) shows the cases where the fish is partially occluded. In Figure 3(c), the fish reappears after being completely occluded and the tracker was able to relocate the object. On a standard dataset as in Figure 5, the ellipse fits the partially visible person where major portion of the person is occluded by two other people. Handling Multimodal: Many of the datasets to be tracked has multimodal objects. We handle multimodal objects by fusing information from each unimodal.
530
Prakash C. et al.
Figures 2, 3, 7 show tracking results on multimodal objects. For instance, Figure 7 shows the example of parrot, where each homogeneous region is extracted and modelled seperately as explained previously and Figure 6 shows the tracking of each unimodal region. Videos with low quality and contrast: The quality of multimedia data available on the web varies significantly owing to various compression and transmission techniques. Several tests using videos with low quality and contrast were carried out to test our technique. In case of Figures 3,4 taken from googlevideos, the quality is poor owing to compression. In these videos, background color merges more with the color of the object. The tracker performance is very good and insensitive to these variations in video.
4
Conclusion
We have proposed a fragment based tracking approach in which the multimodal objects were fragmented into homegenous regions based on hue. These unimodal regions are then tracked using a single parametric distribution and these distributions are fused to form the final tracking result of the entire object. The tracker proposed was also complimented with an efficient search mechanism to make the system robust to handle non-rigid deformations, occlusions, scale and orientation changes efficiently. For modelling the object more efficiently, the research is currently focused on combining other cues like motion,edge,texture with present color based tracker.
References 1. Comaniciu, D., Meer, P.: Mean shift analysis and applications. In: ICCV, vol. 2, pp. 1197–1203 (1999) 2. Zivkovic, Z., Krose, B.: An em-like algorithm for color-histogram-based object tracking. In: Proceedings of the International Conference on Computer Vision and Pattern Recognition, pp. 798–803 (2004) 3. Comaniciu, D., Ramesh, V., Meer, P.: Real-time tracking of non-rigid objects using mean shift. In: IEEE Conf. on Comp. Vis. and Pat. pp. 142–151. IEEE Computer Society Press, Los Alamitos (2000) 4. Adam, A., Rivlin, E., Shimshoni, I.: Robust fragments-based tracking using the integral histogram. In: CVPR 2006. Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 798–805. IEEE Computer Society Press, Los Alamitos (2006) 5. Leordeanu, M., Collins, R.T., Liu, Y. 6. Birchfield, S.T., Rangarajan, S.: Spatiograms versus histograms for region-based tracking. In: Proceedings of the Computer Vision and Pattern Recognition, vol. 2, pp. 1158–1163. IEEE Computer Society, Los Alamitos (2005) 7. Han, B., Davis, L.: Object tracking by adaptive feature extraction. In: In proceeding of International Conference on Image Processing (2004) 8. Liao, P.S., Chen, T.S., Chung, P.C.: A fast algorithm for multilevel thresholding.
Fragments Based Parametric Tracking
531
9. Dempster, A.P., Laird, N.M., Rubin, D.B.: Maximum likelihood from incomplete data via the em algorithm. Journal of the Royal Statistical Society. Series B (Methodological) 39(1), 1–38 (1977) 10. Neal, R.M., Hinton, G.E.: A new view of the EM algorithm that justifies incremental, sparse and other variants. In: Jordan, M.I. (ed.) Learning in Graphical Models, pp. 355–368. Kluwer Academic Publishers, Dordrecht (1998) 11. project/IST 2001 37540, E.F.C.: found (2004), at http://homepages.inf.ed. ac.uk/rbf/caviar/ 12. Porikli, F., Tuzel, O.: Covariance tracking using model update based on means on riemannian manifolds. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, IEEE Computer Society Press, Los Alamitos (2006) 13. Birchfield, S.: Elliptical head tracking using intensity gradients and color histograms. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 232, IEEE Computer Society, Los Alamitos (1998)
Spatiotemporal Oriented Energy Features for Visual Tracking Kevin Cannons and Richard Wildes York University Department of Computer Science and Engineering Toronto, Ontario, Canada {kcannons,wildes}@cse.yorku.ca Abstract. This paper presents a novel feature set for visual tracking that is derived from “oriented energies”. More specifically, energy measures are used to capture a target’s multiscale orientation structure across both space and time, yielding a rich description of its spatiotemporal characteristics. To illustrate utility with respect to a particular tracking mechanism, we show how to instantiate oriented energy features efficiently within the mean shift estimator. Empirical evaluations of the resulting algorithm illustrate that it excels in certain important situations, such as tracking in clutter with multiple similarly colored objects and environments with changing illumination. Many trackers fail when presented with these types of challenging video sequences.
1
Introduction
Target tracking is a critically important aspect to a wide range of computer vision applications, including surveillance, smart rooms, and human-computer interfaces. Significant contributions have been made to the field, but no generalpurpose tracker has been found that can operate effectively in every real-world setting [1]. Scenarios that are present in realistic sequences and challenge many trackers include changes in illumination, small targets, and significant clutter. In general, to facilitate accurate tracking, features must be selected that distinguish targets from the background and from one another, even while being robust to photometric and geometric distortions. In response to these requirements, many different proposals have been made; here, representative examples are provided. Perhaps the simplest approach is to make use of image intensitybased templates for feature definition [2,3,4]. To provide robustness to photometric distortions, consideration has been given to discrete features [5,6,7]. To encompass object outlines, methods have emerged that use contours and silhouettes [8,9,10]. Other features (e.g., color, texture) have been derived on a more regional basis [11,12,13]. Recovered motion also has been used in feature definitions [14,15,16]. Limited attention has been given to the integrated analysis of both the spatial and temporal domains when considering features for visual tracking. Potential benefits of a more integrated approach include the ability to combine static and dynamic target information in a natural fashion as well as simplicity of Y. Yagi et al. (Eds.): ACCV 2007, Part I, LNCS 4843, pp. 532–543, 2007. c Springer-Verlag Berlin Heidelberg 2007
Spatiotemporal Oriented Energy Features for Visual Tracking
533
design and implementation. In response to this observation, the present paper documents a novel feature set for visual tracking that uses energy measures to capture a target’s multiscale, spatiotemporal orientation structure. A considerable body of research has emerged on the use of orientation selective filters in the spatiotemporal domain for the purpose of analyzing motion [17,18,19]. However, it appears that no previous research has explored the use of multiscale, spatiotemporal oriented energies that uniformly encompass space and time as the basis for defining features in the service of visual tracking. To illustrate the use of the proposed oriented energy feature set, we make use of the mean shift tracking paradigm [13,20,21,22], a framework upon which these features readily map. Although, the energy features are also applicable to alternative paradigms, e.g., those that preserve within target spatial relationships, as the oriented energies are calculated locally. In light of previous research, the main contributions of this paper are as follows. (1) A novel oriented energy feature set is defined for visual tracking. This representation captures the spatiotemporal characteristics of a target in an integrated, compact fashion. (2) Oriented energy features are instantiated with respect to the mean shift estimator. (3) The performance of the resulting system is documented both qualitatively and quantitatively. Our algorithm outperforms a color-based mean shift implementation in three common, real-world situations: substantial clutter; multiple targets with similar color; and illumination changes.
2 2.1
Technical Approach Oriented Energy Features
Oriented Energy Computation. Events in a video sequence will generate diverse structures in the spatiotemporal domain. For instance, a textured, stationary object produces a much different signature in image space-time than if the same object were moving. One method of capturing the spatiotemporal characteristics of a video sequence is through the use of oriented energies [17]. These energies are derived using the filter responses of orientation selective bandpass filters when they are convolved with the spatiotemporal volume produced by a video stream. Responses of filters that are oriented parallel to the image plane are indicative of the spatial pattern of observed surfaces and objects (e.g., spatial texture); whereas, orientations that extend into the temporal dimension capture dynamic aspects (e.g., velocity and flicker). The basis of our approach is that energies computed at orientations which span the space-time domain can provide a rich description of a target for visual tracking. Here, multiscale processing is also important, as coarse scales capture gross spatial pattern and overall target motion while finer scales capture detailed spatial pattern and motion of individual parts (e.g., limbs). With regard to dynamic aspects, simple motion is captured (orientation along a single spatiotemporal diagonal) as well as more complex phenomena, e.g., multiple juxtaposed motions as limbs cross (multiple orientations in a spatiotemporal region). By encompassing both spatial and temporal target characteristics in an integrated fashion,
534
K. Cannons and R. Wildes
tracking is supported in the presence of significant clutter. Further, as detailed below, such representations can be made invariant to local image contrast to support tracking throughout substantial illumination changes. For this work, filtering was performed using broadly tuned, steerable, separable filters based on the second derivative of a Gaussian, G2 , and their corresponding Hilbert transforms, H2 [23], with responses pointwise rectified (squared) and summed. Filtering was executed across θ = (η, ξ) 3D orientations (η, ξ specifying polar angles) and σ scales using a Gaussian pyramid formulation. Hence, a measure of local energy, e, can be computed according to e (x; θ, σ) = [G2 (θ, σ) ∗ I (x)]2 + [H2 (θ, σ) ∗ I (x)]2 ,
(1)
where x = (x, y, t) corresponds to spatiotemporal image coordinates, I is the image sequence, and ∗ denotes convolution. This initial measure of local energy is dependent on image contrast. To attain a purer measure of the relative contribution of the orientations irrespective of contrast, (1) is normalized as e (x; θ, σ) eˆ (x; θ, σ) = , ˜σ e x; θ, ˜ + ˜ σ ˜ θ
(2)
where is a bias term to avoid instabilities when the energy content is small and the summations in the denominator cover all scale and orientation combinations. (In this paper, our convention is to superscript variables of summation with˜.) For illustrative purposes, Fig. 1 displays a subset of the energies that are computed for a single frame of a MERL traffic sequence [24]. Here, there is a white car moving to the left near the center of the frame. Notice how the energy channel that is tuned for leftward motion is very effective at distinguishing this car from the static background. Consideration of the channel tuned for horizontal structure shows how it captures the overall orientation structure of the white car. In contrast, while the channel tuned for vertical textures captures the outline of the crosswalks, it shows little response to the car, as it is largely devoid of vertical structure at the scales considered. Finally, note how the energies become more diffuse and capture more gross structure at the coarser scale. Given that the tracking problem is being considered, the goal is to locate the target’s position as precisely as possible. However, as seen in Fig. 1, the energies computed at coarser scales are diffuse due to the downsampling/upsampling that is employed in pyramid processing. Coarse energies are important because they provide information regarding the target’s gross shape and motion, but a method is required to improve their localization for accurate tracking. To that end, a set of weights are applied to the normalized energies of (2) according to ˆ (x; θ, σ) = eˆ (x; θ, σ) b (x; θ) , E
(3)
where b are pixel-wise weighting factors for a particular orientation channel, θ. The weighting factors for a specific orientation are computed by integrating the energies across all scales and applying a threshold, Tθ , according to b (x; θ) = eˆ (x; θ, σ ˜ ) > Tθ . (4) σ ˜
Spatiotemporal Oriented Energy Features for Visual Tracking
535
Fig. 1. Frame 29 of the MERL traffic video sequence with select corresponding energy channels. Finer and coarser scales are shown in rows two and three, resp. From left to right, the energy channels roughly correspond to horizontal structure, vertical structure, and leftward motion.
When computing the weights, summing across scales allows the better localized fine scales to sharpen the coarse scales, while the coarse scales help to smooth the responses of the fine scales. Furthermore, by calculating weights separately for each orientation, we avoid being prejudiced toward any particular type of oriented structure (e.g., static vs. dynamic). Two significant advantages of the proposed oriented energy feature set must be further highlighted. First, normalized energy, as defined by (1) and (2), captures local spatiotemporal structure at a particular orientation and scale with a degree of robustness to scene illumination: By virtue of the bandpass filtering, (1), invariance will be had to changes that are manifest in the image as additive offsets to image brightness; by virtue of the normalization, (2), invariance will be had to changes that are manifest in the image as multiplicative offsets. Second, the calculation of the defined normalized oriented energies requires nothing more than 3D separable convolution and pointwise nonlinear operations, and is thereby amenable to compact, efficient implementation [25]. Histogram Representation. As defined, oriented energies provide local characterization of image structure. Therefore, the energy measurements could be used to provide pointwise descriptors for target tracking (e.g., in conjunction
536
K. Cannons and R. Wildes
with spatial template-based matching). Alternatively, the pointwise measurements can be aggregated over target support to provide region-based descriptors (e.g., in conjunction with mean shift tracking). Here, we pursue the second option and demonstrate the efficacy of the features as regional descriptors. With an eye to mean shift tracking, we collapse the spatial information in our initial energy measurements and represent the target as a histogram. Each histogram bin corresponds to the weighted energy content of the target at a particular scale and orientation. Specifically, the template histogram that defines the target in the first frame is given by qˆu = C
n ˆ (x∗ ; φu ) , k x∗i 2 E i
(5)
i=1
where k is the profile of the tracking kernel, C is a normalization constant to ensure the histogram sums to unity, x∗i = (x∗ , y ∗ ) is a single target pixel at some temporal instant, i ranges so that x∗i covers the template support, and φu is the scale and orientation combination which corresponds to bin u of the histogram. When tracking a target, it may be necessary to evaluate several target candidates for the current frame. Candidate histograms are defined as
nh y − x∗i 2 E ˆ (x∗i ; φu ) , k (6) pˆu (y) = Ch h i=1 where y is the center of the target candidate’s tracking window, h is the bandwidth of the tracking kernel and i ranges so that x∗i covers the candidate support. A sample energy histogram for the target region shown in Fig. 1 (represented by the white box) is shown in Fig. 2. The bin corresponding most closely to leftward motion at the finest scale (bin 5) has by far the most energy. The next two high energy counts are found in bins 2 and 9 which are tuned to combinations of dynamic and static structure, with an emphasis on leftward motion and spatial orientation similar to that of the target. The overall horizontal structure of the car is captured by the energy in bins 1 and 4. In contrast, bins 3 and 6, which roughly represent static, vertical structure, do not have strong responses, given the nature of the car target. The histogram also shows that the oriented energies for the highest frequency structures have the strongest response, as the target is fairly small and dominated by relatively finer scale structure. 2.2
Oriented Energy Features in the Mean Shift Framework
Target Position Estimation. Under the mean shift framework, tracking an object involves locating the candidate position in the current frame that produces the histogram that is most similar to the template. Thus, a measure of similarity between two histograms is required. For histogram comparisons we utilize the Bhattacharyya coefficient, the sample estimate of which can be computed using ˆ] = ρ [ˆ p (y) , q
m pˆu (y) qˆu , u=1
(7)
Spatiotemporal Oriented Energy Features for Visual Tracking
537
0.25
Weighted Energy
0.2
High Frequency Energies Mid Frequency Energies Low Frequency Energies
0.15
0.1
0.05
0 0
5
10
15
20
25
30
Scale and Orientation
Fig. 2. Oriented energy histogram for the target region in Fig. 1
ˆ (y) and q ˆ are histograms with m bins apiece. Due to the definition of where p the Bhattacharyya coefficient, in order to minimize the distance between two histograms, (7) must be maximized with respect to the target position, y. The Bhattacharyya coefficient can be maximized via mean shift iterations [20]. The specific mean shift vector that can be used for this maximization is ⎡ ⎤ nh ∗ yˆ 0 −x∗i 2 x w g ⎥ m i h ⎢ i=1 i qˆu ∗ ˆ (x ; φu ) ˆ1 = ⎢ y E ,(8) ⎥ i ⎣ n ⎦ , where wi = ∗ 2 p ˆ (ˆ y0 ) ˆ y −x u 0 h i u=1 i=1 wi g h g (x) = k (x) is the derivative of the tracking kernel profile, k, with respect to x, ˆ 0 is the current target position. The Epanechnikov kernel has been shown and y to be effective [20] and is the most commonly used kernel for mean shift tracking. Thus, the position of the target in the current frame is estimated as follows. Starting from the target’s position in the previous frame, the mean shift vector is computed and the target candidate is moved to the position indicated by the mean shift vector. These steps are repeated until convergence has been reached or a fixed number of iterations have been executed. Template and Scale Updates. When tracking an object through a long video sequence, it is common that its characteristics will change. To combat the changes a target may incur over time (e.g., due to alterations in velocity or rotation), our tracker includes a simple template update mechanism defined as ˆ (yi ) , ˆ i+1 = απˆ qi + (1 − α) (1 − π) p q
(9)
ˆ i is the temwhere α is a weighting factor the speed of the updates, q to control i ˆ (yi ) , q ˆ is the Bhattacharyya coefficient between plate at frame i, and π = ρ p the current template and the optimal candidate found in the ith frame. Empirically, α was set to 0.85. Following each application of (9), the resulting template is renormalized and thereby remains consistent with our overall formulation. Owing to dependence on the Bhattacharyya coefficient, the template update rule
538
K. Cannons and R. Wildes
indicates that if the template and the optimal candidate are well-matched, the update to the template will be minimal. The size of a target may change during a video sequence as well. Although there are more effective methods of dealing with changes of object scale in the mean shift framework [21,22], in the current implementation we employ a simple approach, similar to that taken in [20]. In particular, our system performs mean shift optimization three times per frame using three different bandwidth values, h. Unless stated otherwise, h values of ±5% are used. We obtain the new bandwidth, hnew , by combining the best of the three bandwidths evaluated at the current frame, hopt , with the previous target size, hprev , according to hnew = γhopt + (1 − γ) hprev .
(10)
Empirically, we set γ = 0.15.
3
Empirical Evaluation
The performance of the oriented energy-based mean shift tracker has been evaluated on an illustrative set of test sequences. For comparative purposes, a mean shift tracker based on RGB color space was also developed and tested. Other than the use of different histograms, the two trackers were identical. The colorbased tracker was implemented in a similar manner to [20], whereby each color channel was quantized into 16 levels (yielding a histogram with 163 bins). In our current implementation of the energy-based tracker, energies were computed at 3 scales with 10 different spatiotemporal orientations per scale. Hence, the energy-based histograms contained 30 bins. For the oriented energy feature set, 10 orientations were selected because they span the space of 3D orientations for the highest order filters that we use (H2 ) [23]; in particular, the selected orientations correspond to the normals to the faces of an icosahedron with antipodal directions counted once, which provides a uniform tessellation of a sphere. For all results in this paper, an Epanechnikov kernel, K, was used. The thresholds for (4) were empirically set as 2.75× the mean energy for each orientation channel. The color and energy-based trackers were hand-initialized with identical target regions in the first frame of each video. Figure 3 illustrates the effectiveness of oriented energy-based features in dealing with illumination changes. An individual starts walking in a poorly lit area;
Fig. 3. Video sequence (x × y × t = 360 × 240 × 60) of a man walking through shadows. From left to right, frames 4, 18, 31, and 55 are shown. Tracked regions are highlighted with white boxes.
Spatiotemporal Oriented Energy Features for Visual Tracking
539
Fig. 4. Video sequence (x × y × t = 360 × 240 × 50) of people walking through a room with similar colored clothing. From left to right, frames 6, 18, 32, and 50 are shown. Tracked regions are highlighted with white boxes.
Fig. 5. MERL traffic video sequence (x × y × t = 368 × 240 × 64) where a white car is tracked as it travels through an intersection. From left to right frames 13, 24, 38, and 58 are shown. Tracked regions are highlighted with white boxes.
then, he travels into and out of the bright region as he walks across the room. Using our proposed feature set, the tracker appeared to be relatively unaffected by the changes in illumination. This robustness arises from the normalization performed in (2). In comparison, our color-based mean shift tracker completely lost track of the target after only a few frames, even when histograms created using normalized RG-space [20] were utilized. Figure 4 shows a case where two persons with similar colored clothing walk in opposite directions and the individual starting on the right side is being tracked. Despite the full occlusion that occurs for several frames, the tracker using energy features is capable of following the true target throughout the video. The different texture patterns and velocities of the walkers were sufficient cues for the energy-based tracker to achieve success, as the representation spans the spatiotemporal domain. In comparison, our color-based tracker became distracted by the other walker as the individuals have near-identical color distributions. Figure 5 shows a real-life, grayscale video sequence of a cluttered traffic scene that was obtained from MERL [24] (a portion also used in Fig. 1). As the figure shows, our proposed system experiences some slight difficulty when tracking the vehicle as it passes over the crosswalk (e.g. notice off-centered tracking in frames 13 and 24). This performance decrease occurs because the lack of contrast (essentially uniform white on white) between the car and the crosswalk yields little energy for the involved portions of the car. Nevertheless, the tracker never loses the target; indeed, the frames shown are representative of the worst case performance in this video. Our feature set was also successfully used when tracking people and vehicles in videos obtained from the PETS2001 dataset [26]. Figure 6 shows an example
540
K. Cannons and R. Wildes
Fig. 6. PETS2001 video sequence (x × y × t = 384 × 288 × 85) where a cyclist is being tracked. From left to right frames 18, 32, and 73 are displayed. Tracked regions are highlighted with white boxes.
Fig. 7. Video sequence (x × y × t = 360 × 240 × 100) showing an individual walking in an erratic pattern. From left to right frames 22, 74, 86, and 100 are displayed. Tracked regions are highlighted with white boxes.
of our results on this dataset where a cyclist is tracked. The tracker that utilizes oriented energy features is successful despite the fact that the cyclist is partially occluded by another individual near the beginning of the sequence. The results on this data sequence are impressive given that the video accurately reflects real-world surveillance settings where targets of interest are often small and of low-resolution. In contrast, our implementation of the color-based mean shift tracker drifted off the target after only a few frames. In Fig. 7 an individual is shown walking erratically, making sudden changes in direction and moving at a wide variety of speeds. Since the oriented energy features encompass both spatial and temporal information, tracking of the target continues throughout each change in velocity. In particular, at instances where the target motion changes radically, the spatially-based components of the representation keep the tracker on target. Subsequently, template updates, (9), incorporate changes to adapt the model for further tracking. Figure 8 shows footage that one might obtain from overhead surveillance cameras in public areas. The oriented energy-based tracker follows the target of interest even though there are multiple similar walkers with little texture, cast shadows, and complex reflectance effects, as the video was recorded through a window. Using the oriented energy feature set, the target is not lost, even during the partial occlusion. The tracker does lag behind the target for a few frames immediately following the occlusion; however, it ultimately follows the correct person. Indeed, frame 39 is representative of its worst-case performance for this
Spatiotemporal Oriented Energy Features for Visual Tracking
541
Fig. 8. Video sequence (x × y × t = 320 × 240 × 70) showing multiple people in motion that are similar in appearance. From left to right frames 9, 31, 39, and 59 are displayed. Tracked regions are highlighted with white boxes.
Bhattacharyya Coefficient
1.1
MERL
1.05
PETS 1 0.95 0.9 0.85 0.8 0.75 0
20
40
60
80
Frame Number
Fig. 9. Bhattacharyya coefficients over the entire video sequence for the MERL and PETS2001 videos
video. In comparison, our color-based implementation was only able to follow the true target for approximately 30 frames. Quantitative performance analysis was performed for the video sequences that are publicly available — MERL and PETS2001. Specifically, Fig. 9 shows the Bhattacharyya coefficient vs. frame number for these two sequences. The Bhattacharyya coefficient is a measure of the system’s confidence in the target found in each frame, with 1 being the largest possible value. For the MERL video, the decreased level of performance at the crosswalks that was qualitatively observed is also indicated quantitatively. In particular, Fig. 9 shows two slight decreases in the Bhattacharyya coefficient at frames 12 and 58 — precisely the frames when the vehicle is passing over the crosswalks. For the PETS video sequence, the significant deviation the Bhattacharyya coefficient experiences is a result of the partial occlusion of the cyclist by the walker (approximately frames 15 - 34). The other, less substantial decreases are a result of the significant background clutter (e.g., parked cars). Also of note is that an average of 3 mean shift iterations were required to reach convergence for these two videos. Twenty iterations, the maximum we allow, was observed only three times.
4
Summary
Spatiotemporal oriented energy features provide a rich, yet compact representation of a target’s characteristic structure across both space and time. In particular,
542
K. Cannons and R. Wildes
by encompassing a range of orientations and scales, the proposed feature set provides a natural integration of the static (e.g., spatial texture) and dynamic (e.g., motion) aspects of a target. To illustrate their usefulness with respect to a particular tracking mechanism, we provide an instantiation with respect to the mean shift estimator. In our experiments over a wide range of video sequences, the energybased tracker was considered to perform as well or better than an identical algorithm that used color histograms. Of primary interest in our work were surveillance-inspired video sequences that included challenges such as substantial background clutter, targets that contained similar colors to other objects in the scene, and changes in illumination. Tracking with the use of oriented energy features was shown to be robust to these challenges. Acknowledgments. Portions of this work were funded by an Ontario Graduate Scholarship to K. Cannons and an NSERC Discovery Grant to R. Wildes.
References 1. Yilmaz, A., Javed, O., Shah, M.: Object tracking: A survey. Comp. Surv. 38(4), 1–45 (2006) 2. Lucas, B., Kanade, T.: An iterative image registration technique with application to stereo vision. In: DARPA IUW, pp. 121–130 (1981) 3. Anandan, P.: A computational framework and an algorithm for the measurement of visual motion. IJCV 2(3), 283–310 (1989) 4. Shi, J., Tomasi, C.: Good features to track. CVPR 1, 593–600 (1994) 5. Sethi, I., Jain, R.: Finding trajectories of feature points in monocular images. PAMI 9(1), 56–73 (1987) 6. Deriche, R., Faugeras, O.: Tracking line segments. IVC 8(4), 261–270 (1991) 7. Rangarajan, K., Shah, M.: Establishing motion correspondence. CVGIP 54(1), 56– 73 (1991) 8. Terzopoulos, D., Szeliski, R.: Tracking with kalman snakes. In: Blake, A., Yuille, A. (eds.) Active Vision, pp. 553–556. MIT Press, Cambridge (1992) 9. Isard, M., Blake, A.: Contour tracking by stochastic propagation of conditional density. In: Buxton, B.F., Cipolla, R. (eds.) ECCV 1996. LNCS, vol. 1064, pp. 343–354. Springer, Heidelberg (1996) 10. Haritaoglu, L., Harwood, D., Davis, L.: W4: Real-time surveillance of people and their activities. PAMI 22(8), 809–830 (2000) 11. Birchfield, S.: Elliptic head tracking with intensity gradients and color histograms. CVPR 1, 232–237 (1998) 12. Sigal, L., Sclaroff, S., Athitsos, V.: Estimation and prediction of evolving color distributions for skin segmentation under varying illumination. CVPR 2, 152–159 (2000) 13. Elgammal, A., Duraiswami, R., Davis, L.: Probabilistic tracking in joint featurespatial spaces. CVPR 1, 781–788 (2003) 14. Bolgomolov, Y., Dror, G., Lapchev, S., Rivlin, E., Rudzsky, M.: Classification of moving targets based on motion and appearance. In: BMVC, pp. 142–149 (2003) 15. Cremers, D., Schnorr, C.: Statistical shape knowledge in variational motion segmentation. IVC 21(1), 77–86 (2003)
Spatiotemporal Oriented Energy Features for Visual Tracking
543
16. Sato, K., Aggarwal, J.: Temporal spatio-velocity transformation and its application to tracking and interaction. CVIU 96(2), 100–128 (2004) 17. Adelson, E., Bergen, J.: Spatiotemporal energy models for the perception of motion. JOSA 2(2), 284–299 (1985) 18. Heeger, D.: Optical flow from spatiotemporal filters. IJCV 1(4), 297–302 (1988) 19. Enzweiler, M., Wildes, R., Herpers, R.: Unified target detection and tracking using motion coherence. Wrkshp. Motion & Video Comp. 2, 66–71 (2005) 20. Comaniciu, D., Ramesh, V., Meer, P.: Kernel-based object tracking. IEEE PAMI 25(5), 564–575 (2003) 21. Collins, R.: Mean-shift blob tracking through scale space. CVPR 2, 234–240 (2003) 22. Zivkovic, Z., Krose, B.: An EM-like algorithm for color-histogram tracking. CVPR 1, 798–803 (2004) 23. Freeman, W., Adelson, E.: The design and use of steerable filters. IEEE PAMI 13(9), 891–906 (1991) 24. Brand, M., Kettnaker, V.: Discovery and segmentation of activities in video. IEEE PAMI 22(8), 844–851 (2000) 25. Derpanis, K., Gryn, J.: Three-dimensional nth derivative of Gaussian separable steerable filters. ICIP 3, 553–556 (2005) 26. PETS (2006), http://peipa.essex.ac.uk/ipa/pix/pets/
Synchronized Ego-Motion Recovery of Two Face-to-Face Cameras Jinshi Cui1, Yasushi Yagi2, Hongbin Zha1, Yasuhiro Mukaigawa2, and Kazuaki Kondo2 1
State Key Lab on Machine Perception, Peking University, China {cjs,zha}@cis.pku.edu.cn 2 Department of Intelligent Media, Osaka University, Japan {yagi,mukaigawa,kondo}@am.sanken.osaka-u.ac.jp
Abstract. A movie captured by a wearable camera affixed to an actor’s body gives audiences the sense of “immerse in the movie”. The raw movie captured by wearable camera needs stabilization with jitters due to ego-motion. However, conventional approaches often fail in accurate ego-motion estimation when there are moving objects in the image and no sufficient feature pairs provided by background region. To address this problem, we proposed a new approach that utilizes an additional synchronized video captured by the camera attached on the foreground object (another actor). Formally we configure above sensor system as two face-to-face moving cameras. Then we derived the relations between four views including two consecutive views from each camera. The proposed solution has two steps. Firstly we calibrate the extrinsic relationship of two cameras with an AX=XB formulation, and secondly estimate the motion using calibration matrix. Experiments verify that this approach can recover from failures of conventional approach and provide acceptable stabilization results for real data. Keywords: Wearable camera, synchronized ego-motion estimation, stabilization, two face-to-face cameras, extrinsic calibration.
1 Introduction The goal of this work is to recover ego-motion of two face-to-face moving cameras simultaneously. This work aims at some situations where ego-motion with only one camera may fail and use another camera to provide additional information. Ego-motion estimation of a moving camera is the task of recovering camera motion trajectory given a set of 2D image frames. It has many applications like stabilization in our application. Most existing methods take one of the following two cases. For the case of static scenes, the problem of fitting a 3D scene compatible with the images is well understood and essentially solved [1, 2]. The second case deals with dynamic scenes, where the segmentation into independently moving objects and the motion estimation for each object have to be solved simultaneously [3-4]. Above methods may fail in camera ego-motion estimation if: (1) the foreground occupies too much space in the image, (2) there are insufficient features in the background Y. Yagi et al. (Eds.): ACCV 2007, Part I, LNCS 4843, pp. 544–554, 2007. © Springer-Verlag Berlin Heidelberg 2007
Synchronized Ego-Motion Recovery of Two Face-to-Face Cameras
545
Fig. 1. Two image pairs captured by one wearable camera with a moving foreground. Left image pair: Camera motion can be computed using background region with enough feature point matches.Right image pair: There are very few feature matches in background region. It’s impossible to estimate ego-motion without additional information. And motion of the foreground point matches is related with both camera motion and person’s motion. If we know foreground person’s motion, camera ego-motion can be estimated.
Fig. 2. Two face-to-face cameras in our application of “Dive into Movie”. One camera is attached to the body of each person.
region of image pair, or (3) there is too much repeated structure for features to get a good match. Fig. 1 showed the situation that almost the whole image is covered by moving foreground. It’s impossible to estimate ego-motion in this case. Additional information can be utilized, such as inertial data [6] or synchronized image frames from another camera. In the case of using another camera, there can be two cases. One is that the additional camera is fixed somewhere watching person1 or person2. In the case of watching person1, the motion of camera is directly estimated by pose estimation. In the case of watching person2, first the motion of person 2 is estimated, and then camera motion is estimated by eliminating the person’s motion from the foreground motion of the camera. In both cases, it’s necessary to make the fixed camera always watching the moving person. The other one is that the additional camera is just the one attached on the foreground object (i.e. another person’s body). This configuration is very natural in our application (see Fig. 2). Motivation for the above work is from a new application of computer vision technology in entertainment, so called “Dive into Movie”. In this application, a movie captured by a wearable camera attached to the actor’s body can give audiences the sense of “immerse in the movie”. The raw movie captured by wearable camera needs to be stabilized due to jitters and ego-motion of the actor. And accurate ego-motion estimation of a moving camera is not easy when there are moving objects in the
546
J. Cui et al.
Fig. 3. Overview of the proposed approach at time k
image. In this application, there are at least two face-to-face interacting actors in a scene. The audience can choose anyone of the actors to see the movie from different views. One camera is attached to each actor. For simplicity, in this paper, we only consider the case of two actors in the scene. Then our goal is to recover ego-motion of two face-to-face cameras using information from both cameras. To address this problem, we first configure the sensor system as two face-to-face moving cameras. And then we derived the relationship between four views that consist of two consecutive views of each camera. In estimation stage, two cameras are calibrated first, and then ego-motion is estimated using calibration result. The calibration problem is formed as AX=XB and refer to the solutions in traditional robotics hand-eye calibration [6-9]. Compared with the consistent motion of hand/eye in traditional hand-eye calibration, we deal with two independently moving cameras. To our knowledge, there is no other work reported on this problem. In [10], a similar configuration of stereo camera is proposed, which used two face-to-face static cameras. The epipolar geometry for these mutual cameras is studied and used to improve the performance of structure from motion approach. In contrast to [10], our approach tries to estimate the ego-motion of two moving face-to-face cameras. The flowchart of the proposed system is showed in Fig. 3. Firstly input videos are pre-processed to segment out the background region and object region which moves consistently with the opposing camera. SIFT features are extracted and matched within consecutive two image pairs for background region and object respectively. If there are enough reliable point matches in background region, ego-motion is estimated and output stabilized frame. Above steps are processed for both cameras. Secondly, if estimation fails with background region, go to the synchronized estimation step, which includes two stages. Extrinsic parameters are calibrated in first stage for two cameras. Here, it’s necessary to get at least three consecutive images from each camera. And then, ego-motions are estimated with calibration result. The following section provides the two-camera geometry. Section III describes estimation procedure. Finally, the evaluation of the experiments is given in Section IV.
Synchronized Ego-Motion Recovery of Two Face-to-Face Cameras
547
2 Two-Camera Geometry Our application of the ego-motion estimation is stabilization for “Dive into Movie”. Cameras are affixed to the actors’ body and move consistently with person (see Fig. 2). First of all, it is convenient to assign frames of reference.
W : a fixed frame of reference; C1 (k): the camera1 frame located at the optical center of camera1 with positive z axis along the optical axis at time k, attached on person 1, watching camera2, varying with camera1’s motion; C2 (k): the camera2 frame located at the optical center of camera2 with positive z axis along the optical axis at time k, attached on person 2, watching camera1, varying with camera2 motion. The relation between any two coordinates is represented by the rotation matrix Ra −> b ∈ SO (3) and a translation vector ta −> b ∈ℜ3 . Ta −> b = [ Ra −> b ta −> b ; 03×3 1] is
the transformation from coordinates a to coordinates b. We express a point X a with respect to the reference a, then X b = Ta −>b X a . We assume that the internal parameters of cameras are initialized as known. Given enough correct feature matches in two views (with static scenes) captured by the same camera, camera ego-motion can be computed easily. In the following, we first recall two-view geometry of conventional static scene. And then foreground motion is taken into account. Finally, four-view (two from each camera) geometry is derived by 3D motion analysis on two moving cameras. 2.1 Two-View Geometry: Epipolar Constraint and Essential Matrix
As well known, the Essential matrix constrains the motion of points between two views from one camera. It encodes the epipolar constraint and motion matrix. The set of homogeneous image points {x i }, i = 1,..., n in the first image is related to the set
{x′i }, i = 1,..., n in the second image by Essential matrix with the following equation: ˆ , Tˆ = [0 −t xi′Exi = 0, E = TR 3
t2 ; t3
0 −t1 ; −t2
t1 0]
(1)
From above equation, given feature matches in two-view, Essential matrix can be determined, and then rotation matrix and translation vector can be computed up to a universal scale. We used RANSAC [1] for transformation matrix estimation. 2.2 Two-View Geometry with Moving Foreground
Let a set of homogeneous 3D space points { X F ,i (k )}, i = 1,..., n be positions of foreground points at time k in the view of camera1 with a rigid motion independently from camera1’s motion. Then, motion of these points in C1 (k) can be represented as
548
J. Cui et al.
X F1 ,C1 (k ) = TF1 ,C1 (k ) X F1 ,C1 (k − 1) = TC−11 (k )TC1 <−W ( k − 1)TF1 ,W ( k )TW <− C1 (k − 1) X F1 ,C1 ( k − 1)
(2)
where TF ,C1 ( k ) represents 3D foreground motion in C1 ’s coordinates from time k-1 to k. TC1 (k ) is C1 ’s motion and TF1 ,W (k ) is foreground motion in world coordinates. TF ,C1 ( k ) and TC1 (k ) can be computed with two-view geometry described in Section 2.1 using feature matches in foreground region and background region respectively. If there is no enough background feature matches for TC1 (k ) , and if TF1 ,W (k ) is given by some other way, TC1 (k ) can be computed using Equation (2). 2.3 Four-View Geometry of Two Face-to-Face Cameras
In this case (see Fig.2), motion of camera1’s foreground points F1 in C2 (k ) coordinates is the same as C2 ’s motion TC2 ( k ) , i.e. TF1 ,C2 (k ) = TC2 (k ) . Then TF1 ,W ( k ) = TW <− C2 ( k − 1)TF1 ,C2 ( k )TC2 <−W = TW <− C2 (k − 1)TC2 ( k )TC2 <−W ( k − 1)
(3)
Now, let’s derive relations among four 3D motion transformation matrices: TF1 ,C1 (k ) , TC2 ( k ) , TC1 (k ) and TF2 ,C2 ( k ) . With the relations, given any three matrices of these four, remaining unknown matrix can be computed. TC2 ( k ) and TC1 (k ) are the target matrices in this paper. From Equation (2) and (3), we have TF1 ,C1 (k ) = TC−11 (k )TC1 <−W ( k − 1)TW <− C2 ( k − 1)TC2 (k )TW <− C1 (k − 1)TC2 <−W (k − 1) = TC−11 (k )TC1 <− C2 (k − 1)TC2 ( k )TC−11<− C2 ( k − 1) If we let TC2 <− C1 = TC 2 −1 for simplicity, then we have TF1 ,C1 (k ) = TC−11 (k )TC1− 2 (k − 1)TC2 ( k )TC−11− 2 (k − 1)
(4)
Similarly, considering foreground points of Camera2, we can obtain: TF2 ,C2 (k ) = TC−21 (k )TC 2 −1 ( k − 1)TC1 (k )TC−21−1 ( k − 1)
(5)
Now let check relations between above matrices and image observations. a) TF1 ,C1 (k ) : motion of foreground points (belong to person2) in camera1; b) TF2 ,C2 ( k ) : motion of foreground points (belong to person1) in camera2, c) TC2 ( k ) : computed from motion of background points in camera2 d) TC1 (k ) : computed from motion of background points in camera1; e) TC 2 −1 (k − 1) : Extrinsic calibration matrix between camera1 and camera2. a)-d) can be computed using two-view relations described in Section 2.1. e) can not be directly computed, and to be determined in Section 3.1.
Synchronized Ego-Motion Recovery of Two Face-to-Face Cameras
549
3 Synchronized Estimation Recall the overview of the algorithm in Fig. 3. Synchronized estimation stage is divided into two steps: extrinsic calibration using frames at time k-3, k-2 and k-1 and motion estimation using frames at time k-1 and k. 3.1 Extrinsic Calibration of Two Face-to-Face Cameras
First we present an outline of our calibration procedure, and then the details of each step will be presented. The extrinsic calibration of two cameras is broken down into the following steps: a) for time k-3, k-2 and k-1, compute TF1 ,C1 , TF2 ,C2 , TC2 and TC1 with steps in Section 2.3 ; b) compute TC 2 −1 (k − 1) using Equation (6) in the following soon. To get a unique solution, at least three views from one camera is necessary [6], with avoiding special configurations of view angles. In the following equation, matrices with underline denote that they can be calculated as known. Using Equation (4) ⎧TF1 ,C1 ( k − 1) = TC−11 (k − 1)TC 1− 2 ( k − 2)TC2 (k − 1)TC−11− 2 ( k − 2) ⎪ ⇒ ⎨ −1 ⎪⎩TC 1− 2 (k − 1) = TC 1 (k − 1)TC 1− 2 (k − 2)T C2 (k − 1) TF1 ,C1 (k − 1) = TC1− 2 (k − 1)TC2 (k − 1)TC−11− 2 ( k − 1)TC−11 ( k − 1)
(6)
TF1 ,C1 (k − 1)TC 1 (k − 1)TC 1− 2 (k − 1) = TC1− 2 (k − 1)TC2 (k − 1)
(7)
From Equation (6), ⎧TF1 ,C1 (k − 2) = TC 1− 2 (k − 2)TC2 (k − 2)TC−11− 2 ( k − 2)TC−11 (k − 2) ⎪ ⇒ ⎨ −1 ⎪⎩TC 1− 2 (k − 2) = TC 1 (k − 1)TC 1− 2 (k − 1)TC2 (k − 1) TF1 ,C1 (k − 2) = TC 1 (k − 1)TC 1− 2 (k − 1)TC−21 (k − 1)TC2 ( k − 2)TC2 ( k − 1)TC−11− 2 ( k − 1)TC−11 ( k − 1)TC−11 ( k − 2) TC 1 (k − 1)TF1 ,C1 (k − 2)TC 1 ( k − 2)TC 1 (k − 1)TC 1− 2 (k − 1) = TC1− 2 ( k − 1)TC−21 ( k − 1)TC2 (k − 2)TC2 (k − 1)
(8)
(9)
In estimation of extrinsic motion, we decompose T into R and t. Then problem can be simplified as compute X that satisfies AX = XB in Equation (7) and (9) for X = RC1− 2 ( k − 1) . tC1− 2 ( k − 1) can be easily obtained from RC1− 2 ( k − 1) and Equation(7), (9). Here, both A and B are known, and X is unknown and has to be solved. While solutions to this question have been studied when A and B are general n n × n matrices, here we need solutions that belong to Euclidean group. In the context of robot sensor calibration, [6] first motivate this equation, and provide a closed-form solution. Their approach is based on geometric interpretations
550
J. Cui et al.
of the eigen-values and eigenvectors of a rotation matrix. Both translation and orientation values are calculated simultaneously using least-square fitting. [7] used this formulation of the problem and developed a non-linear optimization technique to solve it. Martin and Park [8] derive a closed form solution as a linear least squares fit . [9] formulated the problem using canonical coordinates of the rotation group, which enables a particularly simple closed form solution. In [6], conditions for uniqueness of solutions are discussed. It is concluded that the solution can not be found with only one measurement, and the parameters can be uniquely estimated with two camera positions, but the orientations of the camera cannot be zero or π value. In this paper, we used the approach described in [8]. 3.2 Ego-Motion Estimation
Given TF1 ,C1 (k ) - motion of foreground point in view of camera1 and TC2 ( k ) - motion of background points in view of camera2, and TC1− 2 (k − 1) obtained in Section 3.2,
TC1 (k ) is computed using Equation (4). TF1 ,C1 (k ) = TC−11 (k )TC1− 2 (k − 1)TC2 ( k )TC−11− 2 (k − 1)
(10)
4 Evaluations Both simulated data and real data are used for evaluations. Using synthesis data, we check the accuracy of the approach and the sensitivity to various levels of noise. Using real data, the procedure outlined in Fig. 3. is implemented along with the proposed calibration and estimation approach. Furthermore, stabilization results using estimated ego-motion matrices are shown to prove the feasibility and accuracy of the approach. 4.1 Evaluations with Simulated Data
The simulated data was created using a set of known 3D points and transformations. Transformations between two cameras and ego-motions of both cameras are constructed with random rotation axis, angle and translation vector. Additionally, in order to analysis the influence of noise, the data sets were defined by the radius of the Gaussian noise in the 2D pixel points. We create data sets with three different levels of noise. The resulting error in the calibration transformation is plotted in Fig.4. The error in the final estimation of transformation matrix, or residual error, is defined as Ttrue − Testimated F , where ⋅ F is the Frobenious norm of the matrix. Referring to the results (Fig.4 and 5), some interesting observations are made. The proposed approach produces results with low error. In Fig.5, we also showed the residual error resulted from noise in calibration matrix. We can find that the noise in calibration matrix doesn’t impact the final estimation result much. This is because the error in calibration stage might be eliminated by an inverse computation.
Synchronized Ego-Motion Recovery of Two Face-to-Face Cameras
551
Gaussian noise with var 1 pixel Gaussian noise with var 2 pixel Gaussian noise with var 4 pixel
0.15
Residual error
0.10
0.05
0.00 0
5
10
15
20
25
30
35
Date No.
Fig. 4. Residual error of calibration result with different level of noise added to 2d image pixels 0.20
5 degrees of rotation angle error 10 degrees of rotation angle error 20 degrees of rotation angle error 45 degrees of rotation angle error
Residual error
0.15
0.10
0.05
0.00 0
5
10
15
20
25
30
35
Data No.
Fig. 5. Residual error of estimated matrices with different level of noise on calibration matrix
4.2 Experimental Results with Real Data
For real experiments, recall the overview of the algorithm in Fig. 3. We used a real video data with cameras affixed on the body. Before estimation, synchronized input videos from both cameras are pre-processed to segment out the background region and object region. In this step, color distribution based mean-shift region tracking [11] is implemented for object region. SIFT features [12] are extracted and matched within consecutive two image pairs for background region and object respectively. In Fig.6, we showed the result of feature matches with SIFT features. The synchronized estimation step has two stages. Extrinsic parameters are calibrated in first stage for two cameras with totally four image pairs (two for each camera using data at three time steps) using method in Section 3.1. Two-view transformation matrices for foreground and background region are computed using RANSAC [1]. Calibration matrix is computed using the approach described in [8]. Ego-motion is estimated using method in Section 3.2.
552
J. Cui et al.
Fig. 6. One set of data for transformation computation (A and B) in calibration stage. Left column: Up) Point matches in background region of video2; Middle) Point matches in foreground region of video2; Down) Point matches in background region of video1. Right column: Up) stabilization result using background region in video2. Middle) stabilization result using foreground region in video2. Down) Stabilization result using background region in video1.
Fig. 7. Stabilization results. Left-top: point matches in background region of video2. No enough features and the conventional approach failed in this case. Right-top: original image before motion. Down: stabilization result using the proposed approach. Compared with original image on the right-top, we can see our approach can provide acceptable stabilization result.
Synchronized Ego-Motion Recovery of Two Face-to-Face Cameras
553
Finally, a 2D affine transformation is derived from motion matrix for stabilization only considering effect of rotation： x ′ = sRx ， where we set s = 1 R33 for simplicity. x and x′ are homogeneous image points before and after motion. Since the main purpose of this paper is ego-motion recovery, stabilization has not been carefully considered, which can be our future work. In Fig. 7, we showed the stabilization result on background region and foreground region, they all failed. Stabilization result with the proposed approach is given. Compared with original image, we can see it provides acceptable result.
5 Conclusion Accurate estimation of ego-motion is not easy when there is moving foreground. Especially in some special situations it’s almost impossible. To address the problem, we proposed a new approach that utilizes additional video captured by the camera attached on the foreground object (i.e. another actor in our application). We first configure the sensor system as two face-to-face moving cameras. And then we derived the relationship between four views from two cameras. In estimation stage, two cameras are calibrated firstly, and then ego-motion is estimated. We calibrate the extrinsic relationship of two cameras with an AX=XB formulation. Experiments with simulated data and real data verify that this approach can provide acceptable ego-motion estimation and stabilization results.
Acknowledgment This work was supported in part by the NKBRPC (No. 2006CB303100), NSFC Grant (No. 60333010), NSFC Grant (No. 60605001) and Key grant Project of Chinese Ministry of Education (No. 103001).
References 1. Hartley, R.I., Zisserman, A.: Multiple View Geometry in Computer Vision. Cambridge University Press, Cambridge (2000) 2. Faugeras, O., Luong, Q.T., Papadopoulo, T.: The geometry of multiple images. MIT Press, Cambridge (2001) 3. Schindler, K., Suter, D.: Two-view multibody structure-and-motion with outliers through model selection. IEEE T-PAMI 28(6), 983–995 (2006) 4. Wolf, L., Shashua, A.: Two-body segmentation from two perspective views. In: Proc. CVPR, pp. 263–270 (2001) 5. Makadia, A., Daniilidis, K.: Correspondenceless Ego-Motion Estimation Using an IMU. In: Proceedings of the IEEE International Conference on Robotics and Automation (2005) 6. Shiu, Y.C., Ahmad, S.: Calibration of wrist-mounted robotic sensors by solving homogenous transform equations of the form AX = XB. IEEE Transactions on Robotics and Automation 5(1), 16–29 (1989) 7. Li, M.: Kinematic calibration of an active head-eye system. IEEE Transactions on Robotics and Automation 14(1), 153–157 (1998)
554
J. Cui et al.
8. Park, F.C., Martin, B.J.: Robot sensor calibration: Solving AX = XB on the Euclidean group. IEEE T-RA 10(5), 717–721 (1994) 9. Neubert, J., Ferrier, N.J.: Robust active stereo calibration. In: Proceedings of the IEEE International Conference on Robotics and Automation, vol. 3, pp. 2525–2531 (2002) 10. Sato, J.: Recovering Multiple View Geometry from Mutual Projections of Multiple Cameras. Int. J. Comput. Vision 66(2), 123–140 (2006) 11. Comaniciu, D., Ramesh, V., Meer, P.: Kernel-Based Object Tracking. IEEE Trans. Pattern Analysis Machine Intell. 25(5), 564–575 (2003) 12. Lowe, D.G.: Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision 60(2), 91–110 (2004)
Optical Flow–Driven Motion Model with Automatic Variance Adjustment for Adaptive Tracking Kazuhiko Kawamoto Kyushu Institute of Technology, 1-1 Sensui-cho, Tobata-ku, Kitakyushu 804-8550, Japan [email protected]
Abstract. We propose a statistical motion model for sequential Bayesian tracking, called the optical flow–driven motion model, and show an adaptive particle filter algorithm with the motion model. It predicts the current state with the help of optical flows, i.e., it explores the state space with information based on the current and previous images of an image sequence. In addition, we introduce an automatic method for adjusting the variance of the motion model, which parameter is manually determined in most particle filters. In experiments with synthetic and real image sequences, we compare the proposed motion model with a random walk model, which is a widely used model for tracking, and show the proposed model outperform the random walk model in terms of accuracy even though their execution times are almost the same.
1
Introduction
Particle filters [1] have proven to be a powerful and popular tool for visual tracking. One strength of particle filters is the ability to deal with a wide range of statistical models in sequential Bayesian estimation. In particle filters, a filtering distribution is approximately represented by a finite number of weighted samples, referred to as particles, and is updated by propagating particles through time. The most common probability distribution used for propagation may be a prior model [2,3,4], which describes the state dynamics. However this often gives a poor estimate if unexpected motions happen, because the model explores the state space without any additional information on the current state. For adaptive particle propagation, we propose a statistical motion model which predicts the current state with the help of sparse optical flows. We call it optical flow–driven motion model. Due to recent computer power, the real–time computation of sparse optical flows has become possible even if sophisticated methods, such as robust estimation and hierarchical search, are employed. Hence a construction of the motion model is not expensive. In addition we introduce a method for adjusting the variance of the motion model. The variance affects the robustness to unexpected motions and the accuracy of Monte Carlo approximation. This adjustment is based on an error propagation technique [5] and is fully automatic, i.e., the manual setting of the variance is not required. Y. Yagi et al. (Eds.): ACCV 2007, Part I, LNCS 4843, pp. 555–564, 2007. c Springer-Verlag Berlin Heidelberg 2007
556
K. Kawamoto
This motion model becomes more effective in combination with global image feature–based observation models such as color histogram based models [6], because a combination of optical flow and such a global image feature is complementary. In experiments, we implement the particle filter algorithm using a color histogram based observation model. This paper is organized as follow. In section 2, we review particle filters and related works. In section 3, we propose the optical flow–driven motion model and show how to construct the model. In section 4, we show experimental results with synthetic and real image sequences.
2
Related Works
Visual tracking can be formulated as the problem of estimating recursively in time the filtering distribution p(xt |y 1:t ) of the state xt , given the sequence of observations y 1:t ≡ {yk |k = 1, 2, . . . , t}. In the context of visual tracking, xt represents the state of a target, such as the position, the rotational angle, and the velocity, and y t might be intensity pattern, feature points, or histogram. The states xk , k = 1, 2, . . . , t, are assumed to be Markovian given an initial distribution p(x0 ) and a transition distribution p(xt |xt−1 ). The observations y k , k = 1, 2, . . . , t, are conditionally independent of distribution p(y k |xk ) given the state xk . A recursive estimation of the filtering distribution p(xt |y 1:t ) can be achieved by iterating two steps: (1) prediction : p(xt |y 1:t−1 ) = p(xt |xt−1 )p(xt−1 |y 1:t−1 )dxt−1 , filtering : p(xt |y 1:t ) ∝ p(y t |xt )p(xt |y 1:t−1 ).
(2)
For linear–Gaussian state space models, the Kalman filter gives an analytical solution to this recursive estimation. However, for more general models including the model studied in the paper, the analytical evaluation is impossible. Particle filters give a numerical solution to the recursive estimation with Monte Carlo methods. The basic idea of particle filtering is that one approximately represents the filtering distribution in the pointwise form p(xt |y 1:t ) ≈ N (i) (i) N (i) = 1, where δ(·) denotes the delta-Dirac funci=1 wt δ(xt − xt ), i=1 wt (i) tion, and xt is an independent and identically distributed ith random sample at (i) time t, called particle, drawn from p(xt |y 1:t ), and wt is the normalized weight (i) associated with xk . There are some sampling methods for the prediction step in the recursive estimation. One of the most widely used sampling methods might be to draw (i) particles xt , i = 1, . . . , N, from a prior model p(xt |xt−1 ) [2,3,4]. As a prior model, “smooth” motion models such as random walk constant velocity model
xt+1 = xt + v t
(3)
xt+1 = 2xt − xt−1 + v t
(4)
Title Suppressed Due to Excessive Length
557
are often used for visual tracking, where v t is a white Gaussian noise. Although these motion models in eqs. (3) and (4) works well in many situations, they cannot deal with the rapid motion change of a target because of its nature. Thus, if the rapid motion change happens, the particles drawn from such a smooth model can give a poor Monte Carlo approximation of the filtering distribution. To improve the Monte Carlo approximation, adaptive sampling methods might be useful. The auxiliary particle (AP) filter [7], the self–organizing state space model, and the sequential importance sampling (SIS) filter [8,9] (and their variants) include the mechanism of adaptive sampling. The AP filter [7] draws particles from (i)
(i)
(i)
q(xt , i|y1:t ) ∝ wt−1 p(y t |μt )p(xt |xt−1 ), (i)
(5)
(i)
where μt is some characterization of xt , given xt−1 . This distribution is conditioned on the current observation y t . Thus the distribution adaptively changes according to y t . This adaptation leads to a more efficient sampling method, but, (i) if p(xt |xt−1 ) is far away from the true one, q(xt , i|y 1:t ) can also be poor. The self–organizing state space model [10,11] builds an augmented state vector ¯ x t ≡ (xt θ t ), where θ t is unknown parameters of the system, called hyper– parameter. The dynamics of the augmented state can be decomposed into xt−1 ) = p(xt |xt−1 , θ t )p(θ t |θt−1 ) p(¯ xt |¯
(6)
assuming p(xt |xt−1 , θt , θ t−1 ) = p(xt |xt−1 , θt ) and p(θt |xt−1 , θt−1 ) = p(θ t |θt−1 ). Particles are therefore drawn from the distribution conditioned on the hyper– parameter θt . Specifically θt can be taken as the variance of the prior model. In (i) (i) this case, each particle xt is diffused based on its own variance θt . Since the particles with large variances are expected to widely diffuse in the state space, such particles are likely to capture the rapid motion change. As a result, the self–organizing state space model is robust to unexpected motions. Although the model is a flexible approach for adaptive sampling, it renders the dimensionality of the state space twofold. This leads to inefficiency of sampling because more particles are required. The SIS filter [8,9] draws particles from a proposal distribution q(xt |x1:t−1 , y 1:t ). This distribution includes the prior model p(xt |xt−1 ) and eq. (5) as a special case. The optimal proposal distribution is (i)
(i)
p(xt |xt−1 , y t ) ∝ p(y t |xt )p(xt |xt−1 ).
(7)
In visual tracking, this optimal distribution is often not available because p(y t |xt ) is unknown in most cases. Hence an alternative proposal distribution is necessary. ICondensation [12] construct a proposal distribution by detecting skin color regions in the image for hand tracking. However such specific knowledge about objects is not always available for a wide range of objects.
558
3
K. Kawamoto
Optical Flow-Driven Motion Model
We propose a statistical motion model, formally expressed by xt = f (xt−1 , ut , v t ),
v t ∼ N (0, Σ v ),
(8)
where v t is a white Gaussian noise with mean 0 and covariance matrix Σ v . Unlike the smooth models in eqs. (3) and (4), this model includes the term ut which is estimated from sparse optical flows between successive two images. The term ut helps the particles to capture the object of interest by guiding the particles based on the current and previous images. We call this optical flow– driven motion model, because the state is mainly driven by optical flows. In experiments the object of interest is represented by its bounding box (more general object models are also available). The reference bounding box to be tracked is specified by the user at time 0. The bounding box is modeled by (w, h, tx , ty ), where w and h are the width and the length of the bounding box, respectively, and (tx , ty ) is the center of the bounding box. The state vector xt at time t is therefore defined by xt = (wt , ht , txt , tyt ) . The covariance matrix 2 2 2 2 , σv,h , σv,t , σv,t ) in what Σ v is assumed to be a diagonal matrix Σ v = diag(σv,w x y follows. In this setting, eq. (8) can be specified by ⎛ ⎞ (uwt + vwt )wt−1 ⎜ (uht + vht )ht−1 ⎟ ⎟ xt = ⎜ (9) ⎝ txt−1 + utx t + vtx t ⎠ tyt−1 + uty t + vty t where v t = (vwt vht vtx t vty t ) and ut = (uwt uht utx t uty t ) which is estimated from optical flows. The underlying motion in eq. (9) is anisotropic similar transformations, which include planar rigid transformations. Since the elements of ut changes at every time, the model in eq.(9) is adaptively updated. 3.1
Robust Estimation of u t from Optical Flows
The estimation of ut from optical flows is a common problem in computer vision. Let r αt = (xαt , yαt ) , α = 1, . . . , M, 1 denote the αth feature point in the image at time t and assume that r αt is a feature point of the object. The geometrical relation between r αt−1 and r αt is written by
uwt 0 utx t rαt = Dt r αt−1 + tt , where D t = and tt = . (10) 0 uht uty t Therefore, if more than a pair of corresponding points, i.e., r αt ↔ r αt−1 , α = 1, 2, . . ., are available, the least–squares estimate of ut is obtained by solving the optimization problem M
r αt − (Dt rαt−1 + tt )2 → min .
(11)
α=1 1
The number of feature points M may vary from time to time, but the time subscript is suppressed for the sake of brevity.
Title Suppressed Due to Excessive Length
The solution to eq. (11) is calculated by
M M M 1 uwt = M xαt−1 xαt − xαt−1 xαt , Δx α=1 α=1 α=1
M M M 1 uht = M yαt−1 yαt − yαt−1 yαt , Δy α=1 α=1 α=1
M M M M 1 2 utx t = x xαt − xαt−1 xαt−1 xαt , Δx α=1 αt−1 α=1 α=1 α=1
M M M M 1 2 uty t = y yαt − yαt−1 yαt−1 yαt , Δy α=1 αt−1 α=1 α=1 α=1
559
(12)
where Δx = M
M α=1
x2αt−1 − (
M
α=1
xαt−1 )2 , Δy = M
M α=1
2 yαt−1 −(
M
yαt−1 )2 . (13)
α=1
In practice a proportion of feature points, r αt , α = 1, . . . , M, may be outliers, i.e., some of them may be feature points of background or other objects. Since the least-squares estimate to eq. (12) is sensitive to outliers, a robust method for estimating ut is necessary. In order to remove outliers we employ the RANSAC (Random Sample Consensus) algorithm [13]. After removing outliers, we can obtain the least-squares estimate recalculated from only inliers. For RANSAC, an important parameter is the distance threshold d which is used to classify a given data into inliers and outliers. If the measurement error deviation σr , a of rαt is isotropic and Gaussian with zero mean and standard √ reasonable choice of the threshold is d = χ22 (0.95) σr ≈ 5.99 σr , where χ2m (α) is the α × 100 percentile of the χ2 distribution with m degrees of freedom. In practice σr is empirically determined because it is usually unknown (we set σr = 3 in the experiments). In what follows we assume the outliers are removed and all of r αt , α = 1, . . . , M, are inliers for simple notation. 3.2
Automatic Variance Adjustment by Error Propagation
The variance Σ v of the stochastic term v t in eq. (9) is important to an appropriate diffusion of particles. If the variance is too large, most of the particles may not contribute to the approximation of the filtering distribution. This situation results in inefficient sampling. If the variance is too small, the particles may not accurately capture characteristics of the filtering distribution because of a loss of diversity of particles. In order to adjust Σ v automatically, we assume the uncertainty of the model in eq. (8) is almost the same as that of ut , i.e., Σ v ≈ Σ u , where Σ u is the covariance matrix of ut and is assumed to be a diagonal matrix Σ u = 2 2 2 2 diag(σu,w , σu,h , σu,t , σu,t ). In face we take the variance of v t as that of ut . We x y
560
K. Kawamoto
estimate the variance of ut from optical flows using an error propagation technique [5]. If optical flows are inaccurate, the variance of ut increases accordingly, and vice versa. The error propagation is generally based on the relation
2 ∂f 2 σθ = σx2 (14) ∂x where x and θ are related by θ = f (x) and σx2 , σθ2 are the variances of x and θ. From eqs. (12) and (14), we estimate the variances of ut by
M M 2 2 σ ˆu,w = ˆu,h = σr2 , σ σr2 , Δx Δy
M M 1 2 1 2 2 2 2 x ˆu,ty = y (15) σr , σ σr2 . σ ˆu,tx = Δx α=1 αt−1 Δy α=1 αt−1 The variance σr2 of r αt is usually unknown, but the unbiased estimate of σr2 is calculated by 2 (16) σ ˆr2 = 2M − 2 where 2 is the sum of squares of the residuals to eq. (11). We therefore obtain the 2 2 2 2 ,σ ˆu,h ,σ ˆu,t ,σ ˆu,t by substituting σ ˆr2 in eq. (16) for σr2 in eq. (15). variances σ ˆu,w x y 3.3
Redetection of Feature Points
The number of feature points r αt , α = 1, . . . , M, decreases over time, because tracking some feature points may fail and RANSAC may remove some feature points as outliers. With a small number of feature points, the estimates in eq. (12) may be inaccurate. In the worst case, no feature points may exist in images, and thus it is impossible to estimate ut . This means the proposed tracking algorithm will no longer work well. Hence the redetection of feature points is necessary. To this end, feature points are redetected within the bounding box corresponding to the mode of the filtering (i) distribution, which is calculated by xt = arg maxx(i) p(y t |xt ), if the number of t feature points is less than a threshold (in the experiments we set it to 10).
4
Experiments with Synthetic and Real Image Sequences
We compare the performance of the proposed motion model with a prior model using synthetic and real image sequences. Specifically the random walk model in eq. (3) is used as a prior model in both experiments. The elements of noise v in the random walk model are assumed to be independent and white Gaussian 2 2 2 2 = 0.01w0 , σv,h = 0.01h0 , σv,t = σv,t = 3.02 (pixel), noises with variances σv,w x y where w0 and h0 are the width and length of the reference bounding box, respectively.
Title Suppressed Due to Excessive Length
561
The observations are taken to be the normalized histograms of the bounding box in the RGB channels, denoted by y = (hR , hG , hB ). Then the observation model is defined as
R G G B B 2 2 , h ) + B (h , h ) + B (h , h ) B 2 (hR r r r p(y t |xt ) ∝ exp , (17) 2σ 2 G B where (hR r , hr , hr ) is the normalized histograms of the reference bounding box and B(h, h ) is the Bhattacharyya distance between the normalized histograms Nh with Nh bins, defined as B 2 (h, h ) = (1 − i=1 hi hi ). In both experiments, the parameter σ in eq. (17) is set to σ = 0.001. The number of particles is set to 200. The two particle filters are implemented on a computer with Pentium 4 (3.4GHz) and 2 GB main memory.
4.1
Synthetic Example
The purpose of this example is to quantitatively evaluate the performance difference between the proposed model and the prior model. To this end, the ground truth is necessary. Then we generate the synthetic image sequence by transforming the “Lena” image in Fig. 1 (left) with the motion parameters in Fig. 1 (right). The white bounding box in Fig. 1 (left) is a manually selected region at time 0 in size 90 × 100 pixels and the feature points within the box are used for tracking the region by the proposed model. The feature points are detected by the Harris detector [14] and they are tracked by the Lucas–Kanade–Tomasi tracker [15] with pyramidal implementation [16]. We perform the 100 simulations and evaluate the root mean squared errors ¯t = (RMSE) between the mean estimates of the state, which is calculated by x N (i) 1/N i=1 xt , and the ground truth. In Fig. 2 the RMSEs with the two models are present. The results show the RMSEs by the prior model sharply increase at time 6. This increase is caused by the large displacement of the target region from time 5 to 6 ((305, 214) → (320, 200)), as shown in Fig. 1 (right), i.e., the prior model can not follow the target region because of the unexpected motion. On the other hand, the proposed model provides more accurate and stable estimates. The average execution times with the proposed model and the prior model are 39
t wt ht tx ty
0 90 100 300 220
1 90 100 301 219
2 90 100 302 218
3 90 100 303 217
4 90 100 304 216
5 90 100 305 215
6 90 100 320 200
7 90 100 321 199
8 90 100 322 198
9 90 100 323 197
10 90 100 324 196
Fig. 1. The “Lena” image (left) and the state true parameters (bottom) used for generating the synthetic image sequence
562
K. Kawamoto 18
SIR proposed
4
16
3.5
14
3
12 height(pixel)
width(pixel)
4.5
2.5 2
10 8
1.5
6
1
4
0.5
SIR proposed
2
0
0 0
2
4
6
8
10
0
2
4
time(frame)
(a) width wt 20
10
8
10
SIR proposed
18
16
16
14
14 y-translation(pixel)
x-translation(pixel)
8
(b) height ht 20
SIR proposed
18
6 time(frame)
12 10 8
12 10 8
6
6
4
4
2
2
0
0 0
2
4
6
8
time(frame)
(a) x–directional translation txt
10
0
2
4
6 time(frame)
(b) y–directional translation tyt
Fig. 2. Root mean squared errors (RMSE) between the estimated mean of the state and the ground truth for the synthetic image sequence
msec/frame and 36 msec/frame, respectively. Consequently the proposed model outperforms the prior model in terms of accuracy even though the execution times are almost the same. 4.2
Real Example
The real image sequence consists of 240 frames (8 sec) at 320 × 240 pixels resolution, in which a target object in size 80 × 100 pixels is moved by hand. Figures 3 show mean estimates (the white bounding box) provided by the prior model (upper) and the proposed model (lower) at time 126, 128, 130, 137, 139, 141. These results are selected as typical tracking examples to show the difference between the two models. The object being rapidly moved at time 126, the prior model loses track of the object, as shown at time 128 and 130. Similarly, after time 137, the prior model fails to track the object and captures a false object with relatively similar color pattern (the can on the desk), as shown at time 141. In contrast, the proposed model successfully tracks the object through the image sequence. The execution times with the proposed model and the prior model are 14 msec/frame and 12 msec/frame, respectively.
Title Suppressed Due to Excessive Length #126
#128
#130
#137
#139
#141
563
Fig. 3. Mean estimate of the state with the three proposal distributions
5
Conclusion
We propose an adaptive statistical motion model, called the optical flow–driven motion model, for particle filter based tracking. This motion model explores the state space with the help of sparse optical flows. This exploration can be carried out fast because optical flow can be estimated by a gradient method, which is a local search and is done fast. Furthermore, we introduce a variance adjustment method for the motion model. This adjustment is derived from the error propagation technique and fully automatic. The determination of the variance of motion model are important because the variance affects tracking accuracy and efficiency. It is however manually determined in most particle filters. The experimental results with the synthetic and real image sequences show the proposed model provides better performance than the random walk model.
564
K. Kawamoto
Acknowledgements This work is supported by the Ministry of Education, Culture, Sports, Science and Technology, Japan, under a Grant-in-Aid (No.19700174).
References 1. Doucet, A., de Freitas, N., Gordon, N.J.: Sequential Monte Carlo Methods in Practice. Springer, Heidelberg (2001) 2. Gordon, N.J., Salmond, D.J., Smith, A.F.M.: Novel approach to nonlinear/nonGaussian Bayesian state estimation. IEE Proc.–F 140(2), 107–113 (1993) 3. Kitagawa, G.: Monte Carlo filter and smoother for non-Gaussian nonlinear state space models. J. Comput. Graph. Stat. 5(1), 1–25 (1996) 4. Isard, M., Blake, A.: Condensation – Conditional density propagation for visual tracking. Int. J. Computer Vision 29(1), 5–28 (1998) 5. Haralick, R.M.: Propagating Covariance in Computer Vision. Int. J. Pattern Recognition and Artificial Intelligence 10(5), 561–572 (1996) 6. Nummiaro, K., Koller-Meier, E., Gool, L.V.: An adaptive color-based particle filter. Image and Vision Computing 21, 99–110 (2003) 7. Pitt, M.K., Shephard, N.: Filtering Via Simulation: Auxiliary Particle Filters. J. the American Statistical Association 94, 590–599 (1999) 8. Liu, J.S., Chen, R.: Sequential Monte Carlo methods for dynamical systems. J. the American Statistical Association 93(443), 1032–1044 (1998) 9. Doucet, A., Godsill, S., Andrieu, C.: On sequential Monte Carlo sampling methods for Bayesian filtering. Statistics and Computing 10, 197–208 (2000) 10. Kitagawa, G.: Self–organizing state space model. J. the American Statistical Association 93(443), 1203–1215 (1998) 11. Ichimura, N.: Stochstic Filtering for Motion Trajectory in Image Sequences Using a Monte Carlo Filter with Estimation of Hyper-Parameters. Proc. Int. Conf. on Pattern Recognition IV, 68–73 (2002) 12. Isard, M., Blake, A.: ICondensation: Unifying low-level and high-level tracking in a stochastic framework. Proc. European Conf. Computer Vision 1, 893–908 (1998) 13. Fischer, M.A., Bolles, R.C.: Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Comm. ACM 24(6), 381–395 (1981) 14. Harris, C., Stephens, M.: A combined corner and edge detector. In: Proc. 4th Alvey Vision Conf., pp.147–151 (August 1988) 15. Shi, J., Tomasi, C.: Good Features to Track, Proc. Computer Vision Pattern Recognition, pp.593–600 (1994) 16. Bouguet, J.Y.: Pyramidal Implementation of the Lucas Kanade Feature Tracker, Intel Corporation, Microprocessor Research Labs (2000)
A Noise-Insensitive Object Tracking Algorithm Chunsheng Hua1,2 , Qian Chen1 , Haiyuan Wu1 , and Toshikazu Wada1 1
2
Graduate School of Systems Enigneering, Wakayama University, 930 Sakaedani, Wakayama, 640-8510, Japan Institute of Scientific and Industrial Research, Osaka University, 8-1 Mihogaoka, Ibaraki, Osaka, 567-0047, Japan
Abstract. In this paper, we brought out a noise-insensitive pixel-wise object tracking algorithm whose kernel is a new reliable data grouping algorithm that introduces the reliability evaluation into the existing K-means clustering (called as RK-means clustering). The RK-means clustering concentrates on two problems of the existing K-mean clustering algorithm: 1) the unreliable clustering result when the noise data exists; 2) the bad/wrong clustering result caused by the incorrectly assumed number of clusters. The first problem is solved by evaluating the reliability of classifying an unknown data vector according to the triangular relationship among it and its two nearest cluster centers. Noise data will be ignored by being assigned low reliability. The second problem is solved by introducing a new group merging method that can delete pairs of ”too near” data groups by checking their variance and average reliability, and then combining them together. We developed a video-rate object tracking system (called as RK-means tracker) with the proposed algorithm. The extensive experiments of tracking various objects in cluttered environments confirmed its effectiveness and advantages.
1
Introduction
Object tracking has drawn the attention of more and more researchers since the last decade, and numerous powerful tracking algorithms have been brought out, such as background subtraction [16], optical flow [17], CONDENSATION [8], template matching [15,19], mean shift [11], EM algorithm [10], dynamic Bayesian network [9], iterative clustering[6], Kalman filter [7], etc. The common feature of these algorithms is that: their success depends on checking the similarity between the target model and an unknown region/pixel. To measure such similarity, a threshold is usually applied into these algorithms. Since the target object may move under the cluttered condition, it is difficult to select the proper threshold to work stably under all conditions. Furthermore, there is no guarantee that if the object with the maximum similarity is really the target one or not. Collins [13] et al. bring out an idea that: while object tracking, the most important thing is the ability to discriminate the target object from its surrounding background. Not only the target feature but also the background feature should be processed while object tracking. They propose a method to switch the meanshift tracking algorithm among the different linear combination of the RGB Y. Yagi et al. (Eds.): ACCV 2007, Part I, LNCS 4843, pp. 565–575, 2007. c Springer-Verlag Berlin Heidelberg 2007
566
C. Hua et al.
colors which can select the very features that distinguish the object most from the surrounding background. However, the color histogram has little identification power, and in the case of high dimensional features like textures, the large number of combination of colors (in fact, each image contains 49 such color combinations) will prevent their method from achieving the real-time performance. Similar ideas have been applied by Nguyen [14] and Zhang [18]. The performance of [18] will be unstable if the target object contains apertures, because they assume the target object to be solid. In [14], although the more powerful Gabor filter is used to discriminate the target from the background, the performance of [14] in the long term is suspected because the target is assumed to be solid. When the target is non-rigid, there is no guarantee that the update of target template is correct, and the multi-scale problem is also remained in [14]. Therefore, obviously almost all the mentioned tracking algorithms share one common problem: when the target object is non-rigid and/or contains apertures, background pixels will be mixed into the target object; when the target object moves under the cluttered background condition, the continuously mixed background pixels will greatly degrade the purity of the target feature (such as color or texture). In this paper, this phenomenon is called as background interfusion. In order to solve the background interfusion problem, Hua et al.[12] proposed a pixel-wise tracking algorithm called as “K-means tracker” which is based on applying K-means clustering into both the target and background samples to remove the mixed background pixels from the target object. However, two problems degrade the performance of K-means tracker: 1) K-means clustering will wrongly classify the noise data into some pre-defined clusters; 2) the wrongly assumed number of clusters sometimes leads to the wrong clustering result. Although some efforts [1,2,3,4,5] were made to K-means clustering, while object tracking, such problems still affect the performance of K-means tracker. In this paper, we solve these two problems by introducing the reliability estimation into K-means clustering. On considering the triangular relationship among an unknown data and its two nearest cluster centers, each data will be given a reliability value, and the noise data will be ignored by being assigned low reliability. With a new merging method which is based on the average reliability and variance of each cluster, the second problem is also solved.
2 2.1
The RK-Means Clustering Reliability Evaluation
Because the noise data is usually distant from any cluster center, the distance from a noise data to any cluster center is always longer than that of a normal data. Thus, such distance can be regarded as the feature that tells noise data from the normal data. However, since it is difficult to get a proper threshold to examine such feature (as we claimed before), we prefer the triangular relationship among an unknown data and its two nearest cluster centers to measure if it is reliable or not to classify this data into some clusters. High reliability will be sent to the normal data and low reliability to the noise data.
A Noise-Insensitive Object Tracking Algorithm
567
Fig. 1. The relationship between data vectors and their two closest cluster centers
As shown in Fig.1, w1 and w2 are the cluster centers, x1 and x2 are the unknown data vectors. Only according to d11 and d21 , we can only say that x2 is closer to w1 than x1 , but not judge x1 and x2 to be the noise data vector or not. However, according to the shape of triangle x1 w1 w2 and x2 w1 w2 , we can judge that, compared with x2 , x1 is more like to be a noise data vector. While clustering an arbitrary data vector xk of data set X ( {xk ; k = 1, . . . , n}), the reliability value of xk is defined as the ratio of the distance between its two closest cluster centers to the sum of the distance from xk to the two cluster centers: wf (xk ) − ws(xk ) R(xk ) = , (1) dkf + dks dkf = xk − wf (xk ) , f (xk ) = argmin (xk − wi ) ,
dks = xk − ws(xk ) . s(xk ) =
i=1,...,c
argmin
(xk − wi ) .
(2)
i=1,...,c,i=f (xk )
f (xk ) and s(xk ) are the subscript of the closest and the secondly closest cluster centers to xk . The degree (μkf ) of a data vector xk belonging to its closest cluster f (xk ) is computed from dkf and dks : μkf =
dks . dkf + dks
(3)
Since R(xk ) indicates how reliable that xk can be classified, the probability that xk belongs to its closest cluster can be computed as: tkf = R(xk ) ∗ μkf
(4)
tkf denotes that: under the reliability R(xk ), the probability that xk belongs to its closest cluster. Given the number of clusters and the initial cluster centers, the data grouping in RK-means clustering algorithm is carried out by two steps: 1). For each data vector xk , compute its probability of belonging to its nearest cluster center as Eq(4); 2). Update the clusters by minimizing the following objective function: Jrkm (w) =
n k=1
tkf xk − wf (xk ) 2 .
(5)
568
C. Hua et al.
Mistake area
(a) Input image
(b) K-means clustering (c) RK-means clustering
Fig. 2. RK-means clustering can detect the outliers with the reliability evaluation
The cluster centers w are obtained by solving the equation ∂Jrkm (w) = 0. ∂w
(6)
The existence of the solution to Eq.(6) can be proved easily if the Euclidean distance is assumed. The cluster centers w can be separately updated as: n δj (xk )tkf xk 1 if j = f (xk ) k=1 wj = n , δj (xk ) = . (7) 0 otherwise δ (x )t k kf k=1 j The output of Eq.(7) will be the initial value for step one. The step one and two are performed iteratively until w converges. 2.2
Redundant Cluster Deletion
When the assumed number of clusters is greater than that real number of a dataset, there will be some redundant clusters. Such redundant clusters will scramble for the data vectors that should belong to one cluster, thus one cluster may be divided into two or more clusters forcibly, which makes the clustering process become unstable and unreliable. To solve this problem, we merge two redundant clusters into one cluster according to their variance and average reliability. (A) of Fig.3 illustrates a reliability field in a two-dimension space around two cluster centers. The data vectors located on the line connecting the two
(A) Reliability field
(B) Initial state
(C) Iteration: 5
(D) Iteration: 9
Fig. 3. The changes of two cluster centers in one crowd of data vectors during updating
A Noise-Insensitive Object Tracking Algorithm
569
cluster centers will have higher reliability than the others. Such data vectors will attempt to attract the two cluster centers together, and such attraction will become stronger when the two cluster centers really get closer. (B) ∼ (D) of Fig.3 show an example of clustering one crowd of data vectors with RK-means clustering, when given two initial clusters. Because the two cluster centers (red circle) get closer as the iteration increases, the average reliability of the two clusters will decrease continuously according to Eq.(1). Therefore, we considered it is possible to judge if two clusters should be merged or not by checking their average reliability and variance. Here we describe the dispersion of cluster i as: 2 xk ∈cluster(i) (xk − wi ) , (8) v(i) = N where N is the number of data vectors in cluster i. The average reliability of cluster i can be computed by: xk ∈cluster(i) R(xk ) r(i) = . (9) N In Fig.4, Case 1∼4 show the clustering results of data sets with RK-means clustering under different distribution. In order to check if two clusters should be merged or not, we compare the dispersion and the average reliability of two clusters before and after merging. The result of this comparison is obtained as: Rv (i, j) =
v(i) + v(j) , v(i ∪ j)
Rr (i, j) =
r(i) + r(j) , r(i ∪ j)
(10)
where i ∪ j indicates the merged cluster from cluster i and j. The graph in the right part of Fig.4 shows the results of Rv and Rr of the two clusters. By analyzing these results, we discovered that the ratio of Rr to Rv can be used to judge if two clusters should be merged: M erge(i, j) = Pm (i, j) =
Rr (i, j) . Rv (i, j)
Fig. 4. Investigation on the possibility of merging two clusters
(11)
570
C. Hua et al.
If Pm (i, j) ≤ 1 then the cluster i and the cluster j should be merged. This procedure should be executed after each update of cluster centers. Therefore, we add the redundant cluster deletion procedure as the third step into the process of data grouping described in subsection 2.2. The RK-means clustering algorithm is summarized as: 1) Initialization: give the number of clusters c and the initial value to each cluster center wi , i = 1, . . . , c. 2) Iteration of data grouping: i) calculate f (xk ) and s(xk ) for each xk . ii) update wi , i = 1, . . . , c by solving Eq.(6). iii) delete redundant cluster with the method described in subsection 2.2
3
Object Tracking Using RK-means Algorithm
While object tracking, we solve the background interfusion by applying the RKmeans clustering to target and background samples. To describe the image feature properly, we assume that: each object in the image is composed of one or several regions, all the pixels within each region should : 1) contain similar colors; 2) be close to each other in the 2D space. Therefore, according to this assumption, the image feature is described by a constructed color-position 5D feature vector f . In fact, any kind of color system can be applied in Eq(12) (such as RGB, Y CbCr and Y U V color system). ⎤⎡ ⎤ ⎡ ⎤ ⎡ Y 100 0 0 f1 ⎢ f2 ⎥ ⎢ 0 1 0 0 0 ⎥ ⎢ U ⎥ ⎥⎢ ⎥ ⎢ ⎥ ⎢ ⎥⎢ ⎥ ⎥ ⎢ (12) f =⎢ ⎢ f3 ⎥ = ⎢ 0 0 1 0 0 ⎥ ⎢ V ⎥ , ⎣ f4 ⎦ ⎣ 0 0 0 α 0 ⎦ ⎣ x ⎦ y 000 0 α f5 where α is weight factor for adjusting the importance of the position information relative to the color information. Then, as shown in Fig.5, the minimum dissimilarities from an unknown data vector fu to the target clusters and to the surrounding background samples (selected from the ellipse contour) will be calculated as: dT = min fu − fT (i), i=1∼K
dB = min fu − fB (j), j=1∼m
(13)
correspondently, we can get the nearest target cluster center fT (i) and nearest background center fB (j) to fu . The dissimilarity between fT (i) and fB (j) is calculated as follows: dT B = fT (i) − fB (j). (14) The reliability of fu is estimated as: R(fu ) =
dT B , dT + dB
(15)
A Noise-Insensitive Object Tracking Algorithm Cross point
Unknown pixel
Target centers
Nearest point
dB
571
dTB
dT
Fig. 5. RK-means clustering on multi-color object with target and background samples
Then the probability that fu belongs to target cluster i is calculated as: (i)
μT (fu ) = R(fu ) ∗
dB . dT + dB
(16)
According to the Eq(7), the target centers will be smoothly updated in both the color and position space. How to select the background samples and update the search area, the tracking failure detection and recovery, as well as the initialization process have been reported in [12] in details.
4
Experiment and Discussion
4.1
Evaluating the Effectiveness of RK-Means
To confirm the effectiveness of the RK-means clustering (RKM), we compare it with the existing K-means (EKM) and fuzzy K-means (FKM) clustering. All the algorithms are given the same condition (e.g. initial cluster centers and iteration). The upper row of Fig.6 shows a comparative experiment where four clusters exist but the initial number of clusters is three. Here, the sky blue “” denotes the initial cluster centers and the yellow “•” shows the clustering result during each iteration and the final results are pointed by arrows. The EKM only succeeds in setting one cluster center correctly. The FKM also fails to give good clustering result. Our RKM successfully classifies three clusters according to the initial cluster centers because the data vectors of the fourth redundant cluster are given extremely low reliability by RKM. This means they can not be classified reliably and should not be assigned to any of given clusters. The lower row of Fig.6 shows another case that the real number of clusters is smaller than the assumed one. The hollow “” indicates the initial cluster center, and the solid sky blue “” shows the resulted cluster center. The EKM and FKM algorithm brutally divided the data vectors of group 2 into two separated clusters. Meanwhile, the RKM successfully classifies the three clusters by reducing one redundant assumed cluster as the way described in Section 2.2. In order to confirm the convergence of RKM, we applied it to the IRIS dataset1 as shown in (A) of Fig.7. The pink curve shows the convergence of RKM and blue 1
http://www.ics.uci.edu/˜mlearn/databases/iris/iris.data
572
C. Hua et al. 300
300
300
"Four_cluster1.dat" "Four_cluster2.dat" "Four_cluster3.dat" "Four_cluster4.dat" "Initial_Value.dat" "Four_cluster_CKM.dat"
250
"Four_cluster1.dat" "Four_cluster2.dat" "Four_cluster3.dat" "Four_cluster4.dat" "Initial_Value.dat" "Four_cluster_FKM.dat"
250
200
200
200
150
150
150
100
100
100
50
50
50
0
0
0
50
100
EKM
150
200
250
300
350
400
0 0
450
50
100
150
200
250
300
350
400
450
300 "Three_cluster1.dat" "Three_cluster2.dat" "Three_cluster3.dat" "Three_cluster_Initial_Value.dat" "Three_cluster_CKM.dat"
250
150
200
250
300
350
400
450
100
50
50
Group 3
Group 2 150
200
250
300
350
400
Group 3
Group 2
0 450
Group 1
150
100
50
"Three_cluster1.dat" "Three_cluster2.dat" "Three_cluster3.dat" "Three_cluster_Initial_Value.dat" "Three_cluster_RKM.dat"
200
Group 1
150
100
EKM
100
250
200
Group 1
100
50
300
"Three_cluster1.dat" "Three_cluster2.dat" "Three_cluster3.dat" "Three_cluster_Initial_Value.dat" "Three_cluster_FKM.dat"
250
200
50
0
FKM RKM Real number of clusters is larger than the assumed number.
300
150
"Four_cluster1.dat" "Four_cluster2.dat" "Four_cluster3.dat" "Four_cluster4.dat" "Initial_Value.dat" "Four_cluster_RKM.dat"
250
Group 3
Group 2 0
0 50
100
150
200
250
300
350
400
450
50
100
150
200
250
300
350
400
450
RKM FKM Real number of clusters is smaller than the assumed number. Fig. 6. Simulation experiment under different conditions
(A)convergence
(B)input
(C)iteration 1 (D)iteration 2 (E)iteration 3
Fig. 7. (A): Comparing the convergence of RKM and EKM. (B) ∼ (E): Image segmentation with redundant cluster deletion.
curve for EKM. We confirm that the convergence of the RKM is as good as the EKM, and its converging speed is not slower than the EKM. The (B) ∼ (E) of Fig.7 show an image segmentation experiment of the RKM with the redundant cluster deletion. (B) is extracted from the lower row (frame 285) of Fig.8. The yellow cross indicates the target cluster centers. Although three initial cluster centers are given, according to the redundant cluster deletion described in Section 2.2, the RKM correctly merge them into one cluster. 4.2
Object Tracking with RK-Means Tracker
Because Hua[12] have compared2 K-means tracker with some famous tracking algorithms, here we only compare our RK-means tracker with the K-means tracker. 2
http://vrl.sys.wakayama-u.ac.jp/VRL/studyresult/study result 3 en.html
A Noise-Insensitive Object Tracking Algorithm
573
Results of the K-means tracker.
Frame 220
Results of the RK-means tracker. Frame 258 Frame 285
Frame 305
Fig. 8. Results of comparative experiment with K-means tracker[12] and RK-means tracker under complex scenes
Frame 055
Frame 075
Frame 107
Frame 117
Frame 125
Frame 138
Frame 142
Frame 168
Fig. 9. Performance of RK-means tracker on PETS2004 where two people are fighting
Fig.8 shows a sequence of hand tracking. The K-means tracker fails to work since frame 285. In frame 285, since the color of some surrounding background parts (e.g. a corrugated carton) is similar to the skin color for some degree, K-means tracker mistakenly takes them as the target pixels. This causes the incorrect update of the search area and leads to the tracking failure. As for our RK-means tracker, in frame 285, the influence of such background parts is effectively repressed through the reliability evaluation. Therefore, the RK-means tracker can detect the target area (e.g. hand) and update the search area correctly. Fig.9 shows the performance of RK-means tracker on the public database PETS2004. In this sequence, although two persons are fighting and get entangled
574
C. Hua et al.
in each other, the RK-means tracker successfully treats the mixed person as noise data by giving him low reliability and ignores those noise data. All the experiments were performed with a desktop PC with a 3.06GHZ Intel XEON CPU, and the image size was 640 × 480 pixels. When the target size varied from 100 × 100 ∼ 200 × 200 pixels, target colors varied from 1 ∼ 6, the processing time of our algorithm was about 9 ∼ 15ms/frame.
5
Conclusion
In this paper, we have proposed a robust pixel-wise object tracking algorithm which is based on a new reliability-based K-means clustering algorithm (called as RK-means tracker). By considering the triangular relationship among an unknown data and its two nearest cluster centers, each data is assigned with a reliability value, and the noise data will be ignored by being given the low reliability value. When the number of clusters is incorrectly assumed, by checking the variance and average reliability of each cluster, a new group merging method is brought out to delete the redundant clusters. Through the extensive experiments, we have confirmed that the proposed RK-means tracker can work more robustly than other algorithms when the background is cluttered. Besides object tracking, we also confirm that the proposed RK-means clustering algorithm can be applied to image segmentation. Acknowledgement. This research is partially supported by the Ministry of Education, Culture, Sports, Science and Technology, Grant-in-Aid for Scientific Research (A)(2), 16200014 and (C)18500131, and (C)19500150.
References 1. Bezdek, J.C.: Pattern Recognition with Fuzzy Objective Function Algorithm. Plenum Press, New York (1981) 2. Krishnapuram, R., Keller, J.M.: A Possibilistic Approach to Clustering. IEEE Trans. Fuz. Sys. 1(2), 98–110 (1993) 3. Chintalapudi, K.K., Kam, M.: A noise-resistant fuzzy C means algorithm for clustering. FUZZ-IEEE 2, 1458–1463 (1998) 4. Jolion, J.M., et al.: Robust Clustering with Applications in Computer Vision. PAMI 13(8) (1991) 5. Zass, R., Shashua, A.: A Unifying approach to Hard and Probabilistic Clustering. ICCV 1, 294–301 (2005) 6. Hartigan, J., Wong, M.: Algorithm AS136: A K-means clustering algorithm. Applied Statistics 28, 100–108 (1979) 7. Peterfreund, N.: Robust tracking of position and velocity with Kalman snakes. PAMI 22, 564–569 (2000) 8. Isard, M., Blake, A.: CONDENSATION-Conditional density propagation for visual tracking. IJCV 29(1), 5–28 (1998) 9. Toyama, K., Blake, A.: Probabilistic Tracking in a Metric Space. ICCV 2, 50–57 (2001)
A Noise-Insensitive Object Tracking Algorithm
575
10. Japson, A.D., et al.: Robust Online Appearance Model for Visual Tracking. PAMI 25(10) (2003) 11. Comaniciu, D., Ramesh, V., Meer, P.: Kernel-Based Object Tracking. PAMI 25(5), 564–577 (2003) 12. Hua, C., Wu, H., Wada, T., Chen, Q.: K-means Tracking with Variable Ellipse Model. IPSJ Transactions on CVIM 46(Sig 15 CVIM12), 59–68 (2005) 13. Collins, R., Liu, Y.: On-line Selection of Discriminative Tracking Feature. ICCV 2, 346–352 (2003) 14. Nguyen, H.T., Semeulders, A.: Tracking aspects of the foreground against the background. In: Pajdla, T., Matas, J. (eds.) ECCV 2004. LNCS, vol. 3022, pp. 446–456. Springer, Heidelberg (2004) 15. Rosenfeld, A., Kak, A.C.: Digital Picture Processing, Computer Science and Applied Mathematics. Academic Press, New York (1976) 16. Stauffer, C., Grimson, W.E.L.: Adaptive Background Mixture Models for Real-time Tracking. CVPR, pp. 246–252 (1999) 17. Barron, J., Fleet, D., Beauchemin, S.: Performance of optical flow techniques. IJCV 2(1), 42–77 (1994) 18. Zhang, C., Rui, Y.: Robust Visual Tracking via Pixels Classification and Integration. ICPR, 37–42 (2006) 19. Gr¨ aβl, C., et al.: Illumination Insensitive Template Matching with Hyperplanes. In: Michaelis, B., Krell, G. (eds.) DAGM 2003. LNCS, vol. 2781, pp. 273–280. Springer, Heidelberg (2003)
Discriminative Mean Shift Tracking with Auxiliary Particles Junqiu Wang and Yasushi Yagi The Institute of Scientific and Industrial Research, OSAKA University 8-1 Mihogaoka, Ibaraki, Osaka, Japan [email protected]
Abstract. We present a new approach towards efficient and robust tracking by incorporating the efficiency of the mean shift algorithm with the robustness of the particle filtering. The mean shift tracking algorithm is robust and effective when the representation of a target is sufficiently discriminative, the target does not jump beyond the bandwidth, and no serious distractions exist. In case of sudden motion, the particle filtering outperforms the mean shift algorithm at the expense of using a large particle set. In our approach, the mean shift algorithm is used as long as it provides reasonable performance. Auxiliary particles are introduced to conquer the distraction and sudden motion problems when such threats are detected. Moreover, discriminative features are selected according to the separation of the foreground and background distributions. We demonstrate the performance of our approach by comparing it with other trackers on challenging image sequences.
1
Introduction
Tracking objects through image sequences is one of the fundamental problems in computer vision. Among the algorithms developed in the pursuit of robust and efficient tracking, two major successful approaches are the mean shift algorithm [1][5], which focuses on Target Representation and Localization, and particle filtering [7][9], which is developed based on Filtering and Data Association. Both of them have their respective advantages and drawbacks. This paper aims at developing a robust and efficient tracker that incorporates the efficiency of the mean shift algorithm with the multi-hypothesis characteristics of the particle filtering. The mean shift algorithm is a robust non-parametric probability density estimation method. Comaniciu et al. [5] define a spatially-smooth similarity function and reduce the state estimation problem to a search of the basin of attraction of this function. Since the similarity function is smooth, a gradient optimization method leading to fast localization is applied. Despite its efficiency and robustness, the mean shift algorithm is not good at coping with quick motions. The distractions in the neighborhood of the target are threats to successful tracking. In addition, the basic mean-shift algorithm assumes that the target representation is sufficiently discriminative against the background. This assumption is Y. Yagi et al. (Eds.): ACCV 2007, Part I, LNCS 4843, pp. 576–585, 2007. c Springer-Verlag Berlin Heidelberg 2007
Discriminative Mean Shift Tracking with Auxiliary Particles
577
not always true especially when tracking is carried out in a dynamic background such as surveillance with a moving camera. We introduce particles to deal with the first two problems because they are able to provide multiple hypothesis. Adaptive tracking is one possible solution to alleviate the third problem [3]. We update the target model according to the separation of the foreground and background distributions. Particle filtering stands out in filtering-based techniques due to its ability to represent multi-modal probability distributions using a weighted sample set S = {(s(n) , π (n) )|n = 1, . . . , N } that keeps multiple hypothesis of the states of targets [7] [9]. When the tracking is performed in a cluttered environment where multiple objects similar to the target can present, particle filters are able to find the target by validation and association of the measurements. However, since the number of particles can be large, a potential drawback of particle filtering is the high computational cost. Moreover, the particle set can degenerate and diffuse in a long sequence. Only few particles with high weights are useful after the tracking in certain frames. Accurate models of shape and motion learned from examples have been used to deal with these problems [9]. One of the drawbacks of this method though is that the construction of explicit models sometimes is hardly achievable because of viewpoint changes. Blake et al. [10] proposed the ICONDENSATION algorithm in which high and low-level information are combined using importance sampling. However, it is complicated to model the dynamic characteristics accurately in an uncontrolled environment. Sullivan and Rittscher [14] noticed the advantages of the mean shift and particle filter algorithms. They proposed a particle filter-based tracking guided by deterministic search based on a SSD type cost function. The size of particle set is adjusted according to the difficulty of the problem at hand, which is indicated by motion. Deterministic search using mean-shift has also been applied in a hand tracking algorithm by embedding the mean-shift optimization into particle filtering to move particles to local peaks in the likelihood, which improves the sampling efficiency [13]. Although the mean-shift and particle filters have been combined in various ways in previous works, none of them deal with occlusions and distractions explicitly. Cai et al. [2] embed the meanshift algorithm into the particle filter framework to stabilize the trajectories of the targets. It is necessary to learn classifiers for the targets in their work, which is not always possible in tracking applications. The mean shift tracking algorithm outperforms the particle filter when the representation of a target is discriminative enough, the target does not jump beyond the bandwidth, and no serious distractions exist. Although it seems that these conditions are too strict, we observed that they can be met in a large percentage of real image sequences captured for surveillance or other applications. In this work, the mean-shift algorithm is adopted as the main tracker as long as these conditions are met. In other words, only one particle driven by the mean-shift searching is used to estimate the state of the target. Auxiliary particles are introduced when sudden motion or distractions are detected. We compute log likelihood ratios of class conditional sample densities of the target
578
J. Wang and Y. Yagi
and its background. These ratios are applied in feature selection and distraction detection. The target model is updated according to feature selection results. Sudden motions are estimated using the efficient motion filters [16]. The proposed method offers several advantages. It achieves high efficiency when the target moves smoothly. When sudden motions or distractions are detected, auxiliary particles are initialized to support the mean shift tracker. The help from particle filtering partially solves the problems resulted from sudden motions or distractions. The remainder of the paper is organized as follows. Section 2 gives a brief introduction of the target model. Section 3 describes the feature selection and model updating methods. Section 4 introduces motion estimation and distraction detection. Section 5 discusses the use of auxiliary particles. The performance of the proposed method is evaluated in Section 6 and conclusions are given in Section 7.
2
Target Modeling
The target model should be as discriminative as possible to distinguish between complex target and background. We use an adaptive target model represented by the best features selected from shape-texture and color cues [17]. Color histograms are computed in three color spaces: RGB, HSV and normalized rg. There are 7 color features (R, G, B, H, S, r, g) in the candidate feature set. These color channels are quantized into 12 bins respectively. A color histogram is calculated using a weighting scheme in which the Epanechnikov kernel is applied [5]. A shape-texture cue is described by an orientation histogram, which is computed based on image derivatives. The orientations are also quantized into 12 bins. Each orientation is weighted and assigned to one of two adjacent bins according to its distance from the bin centers. The similarity between the model and its candidates is measured by Bhattacharya distance [5].
3 3.1
Feature Selection and Model Updating Log-Likelihood Ratio Images
To determine the descriptive ability of different features, we compute loglikelihood ratio images [3] [15] based on the histograms of the target and its background. Log-likelihood ratio images are also employed in detecting possible threats to the target. The likelihood ratio produces a function that maps feature values associated with the target to positive values and those associated with the background to negative values. The frequency of the pixels that appear in a histogram bin (b ) (b ) (b ) (b ) (p(bin ) ) is calculated as ζf in = pf in /nf and ζb in = pb in /nb , where nf is the pixel number of the target region and nb the pixel number of the background.
Discriminative Mean Shift Tracking with Auxiliary Particles
579
The log-likelihood ratio of a feature value is given by L(bin ) = max(−1, min(1, log
(bin )
, δL )
(bin )
, δL )
max(ζf max(ζb
)),
(1)
where δL is a very small number (δL is set to 0.001 in this work). The likelihood image for each feature is created by back-projecting the ratio into each pixel in the image. 3.2
Feature Selection
Given md features for tracking, the purpose of the feature selection module is to find the best subset feature of size mm , and mm < md . Feature selection can help minimize the tracking error and maximize the descriptive ability of the feature set. We find the features with the largest corresponding variances. Following the method in [3], based on the equality var(x) = E[x2 ] − (E[x])2 , the variance of Equation(1) is computed as var(L; p) = E[(Lbin )2 ] − (E[Lbin ])2 . The variance ratio of the likelihood function is defined as [3]: VR = 3.3
var(L; (pf + pb )/2) var(B ∪ F ) = . var(F ) + var(B) var(L; pf ) + var(L; pb )
(2)
Updating the Target Model
It is necessary to update the target model due to the fact that the appearance of a target tends to change during a tracking process. Unfortunately, updating the target model adaptively may lead to tracking drift because of the imperfect classification of the target and background. To reliably update the target model, we propose an approach based on similarities between the initial and current appearance of the target. Similarity θ is measured by a simple correlation based template matching performed between the initial and current frames. The updating is done according to the similarity θ: Hm = (1 − θ)Hi + θHc ,
(3)
where the Hi is the histogram computed on the initial target; the Hc the histogram of the target current appearance, the Hm the updated histogram of the target. Template matching is performed between the initial model and the current candidates. Since we do not use the search window that is necessary in template matching-based tracking, the matching process is efficient and brings little computational cost to our algorithm. In unstable tracking period (When sudden motions or distractions are detected), the classification of the target and background is not reliable. It is difficult to reliably update the target model at these moments. Thus the model is updated when the tracker is in stable states.
580
4 4.1
J. Wang and Y. Yagi
Motion Estimation and Distraction Detection Motion Estimation
The number of particles is adjusted according to motion information of the target. Discriminative mean shift tracking is sufficient to determine the position of a target when it moves smoothly and slowly. More particles are necessary to estimate the correct position of the target when it moves quickly. We use the efficient motion filters that have been applied in pedestrian detection [16]. We estimate the motion of foreground and background region simultaneously and partially solve the problem brought by dynamic background. There are five motion filters computed on 5 image pairs: 1 τi |It (x) − It+1 (x)|, (4) Δi = nRg x∈Rg where It and It+1 are consequential images, nRg is the number of pixels in a specific region, and τi ∈ {, ←, →, ↑, ↓} which are image shift operators denoting no shift, shift left, shift right, shift up, and shift down for one pixel respectively. The motion filters are computed on the target and its background region respectively. The results of the last four motion filters (Δi , i ∈ {1, 2, 3, 4}) are compared with the absolute differences Δ0 : Mif = |Δfi − Δf0 |, Mib = |Δbi − Δb0 |
(5)
Mi represent the likelihood that a particular region is moving in a given direction. We compute the maximum motion likelihood to determine the number of particles for the tracking: Mmax = max(|Mif − Mib |)i=1,2,3,4. .
(6)
Given the high efficiency of the estimation method, it is performed in each frame before tracking is carried out. 4.2
Distraction Detection
Distractions in the neighborhood of the target have similar appearance to the target. They are possible threats to successful tracking. When the similarity between the target model and its candidate is less than a certain value (ρT ), distraction detection is performed using spatial reasoning [3] to find peaks besides the target in the log-likelihood ratio images. Note that the log-likelihood ratio images here are back-projection results of the conditional distributions based on selected features. Assuming that the region RT actually contains the target and the region RD is a possible distraction, we want to find the region that have maximum strength of threat to the target. A certain region where the sum of its log-likelihood ratios
Discriminative Mean Shift Tracking with Auxiliary Particles
581
has minimum difference with that in the target region is the distraction we want to find: min(| L(bin ) − L(bin ) |) (7) RD
RX
where RX is a region in the neighborhood of the target. It is too expensive to compare the sums of log-likelihood ratio in all the possible regions with that in the target region. The searching process can be accelerated using a Gaussian kernel [3]. The value at each pixel in the convolved log-likelihood ratio image with a Gaussian kernel is a weighted sum of the log-likelihood ratios in a circular region surrounding it, normalized by the total weight pixels in that region. First, the log-likelihood image is convolved using a Gaussian kernel. The peak DT which represents the target region can be found in the convolved image. Second, the target region in the log-likelihood image is removed and the result is convolved using a Gaussian kernel again. The most dangerous distraction is detected by searching for the peak DD in the convolved image. The difference between the two peaks represents the threat strength of the distraction: ρ = |DD − DT |, (8) The distraction may attract the mean shift tracker to the incorrect position if it is strong enough. We initialize a auxiliary particle set to track the distraction region when ρ is less than the given threshold ρT .
5
Auxiliary Particle Filtering
Particle filtering implements recursive Bayesian filter by Monte Carlo simulations. In the implementation, the posterior density is approximated by a weighted (n) (n) (n) (n) particle set {st , πt }n=1,···,J , where πt = p(zt |xt = st ). We initialize auxiliary particles when sudden motion or distraction are detected. Different strategies are adopted for the generation of particles under these two circumstances. 5.1
Particle Filtering for Sudden Motion
When a sudden motion is detected Np particles are generated using a stochastic motion model. The number of particles is determined from to the motion computed: (9) JS = max(min(J0 Mmax , Jmax ), Jmin ), where J0 is the coefficient; Jmin is the smallest number of particles and Jmax the largest number of particles to maintain reasonable particles. The motion model is a normal density centered on the previous pose with a constant shift vector: (10) xjt = xt−1 + xc + ujt ; where ujt is a standard normal random vector and xc a constant shift vector from the previous position according to the motion estimation results (it is set to one pixel to the motion direction).
582
5.2
J. Wang and Y. Yagi
Particle Filtering for Distraction
After distractions are detected, a joint particle filter with an MRF motion model is initialized [12]. The motion interaction between the target and the distraction ψ(Xit , Xjt ) is described by the Gibbs distribution ψ(Xit , Xjt ) ∝ exp(−g(Xit , Xjt ), where g(Xit , Xjt ) is a penalty function approximated by the distance between the target and the distraction. The posterior on the joint state Xt is approximated as a set of J weighted samples: (J) (J) ψ(Xit , Xjt ) πt−1 P (Xit |Xi(t−1) ), P (Xt |Z t ) ≈ kP (Zt |Xt ) ij∈E
J
i
where the samples are drawn from the joint proposal distribution; k is a normalizing constant that does not depend on the state variables; E is edges in the MRF model; the samples are weighted according to the factored likelihood function: 2 (s) (s) (s) (s) P (Zit |Xit ) ψ(Xit , Xjt ). πt = i
ij∈E
where Zit are measurement nodes. 5.3
Algorithm Summary
In summary, the detailed steps of the proposed tracking algorithm are: Algorithm: Discriminative Mean-Shift Tracking with Auxiliary Particles Input:
t video frames I1 , . . . , It ; Initial target region given in the first frame I1 target regions in I2 , . . . , It
Output: Initialization in I1 1. Save the initial target appearance for model updating; 2. Compute the similarity (S1 ) between the target model and the candidate. For each new frame Ij : Estimate the motion (Mj ) on the consequential frames; IF Mj > MT THEN initialize particles according to the motion estimated. ELSE IF the similarity is less than a given threshold (Sj−1 < S T ) THEN detect distractions in the neighborhood of the target If Distraction is detected (ρ < ρT ) Initialize MRF particles; Else Update the target model. End If End If End If Estimate the position of the target. Compute the similarity Sj for next frame. End For
Discriminative Mean Shift Tracking with Auxiliary Particles
2 Particle filtering
90
90
80 70 60 50
50 30
20
20
10
10
0
0
3
4
5
25
60
30
Tracking approaches
20 15 10 5
1
2
4
3
0
5
Tracking approaches
90
30 25
14 12 10
20
8
15
6
10
4
5
2
0
0
3
4
5
Dataset tracked(%)
100
16
Dataset tracked(%)
18
35
Tracking approaches
(d)
4
3
5
(c)
40
2
2
Tracking approaches
45
1
1
(b)
(a)
Dataset tracked(%)
5 The proposed tracker
Peak difference
70
40
2
4
80
40
1
3 Variance ratio
Dataset tracked(%)
100
100
Dataset tracked(%)
Dataset tracked(%)
1 Basic mean-shift
583
80 70 60 50 40 30 20 10
1
2
4
3
5
Tracking approaches
(e)
0
1
2
4
3
5
Tracking approaches
(f)
Fig. 1. Tracking results using different tracking approaches. Tests are performed on (a) EgTest01; (b) EgTest02; (c) EgTest03; (d) EgTest04; (e) EgTest05; and (f) Redteam.
6
Experimental Results
To illustrate the performance of the proposed tracker, we have implemented and tested it on a wide variety of challenging image sequences in different environments and applications. Due to space limitation, we only show the results on the public CMU datasets with ground truth [4]. The datasets include 6 sequences: EgTest01, EgTest02, EgTest03, EgTest04, EgTest05 and Redteam. There are different factors that make the tracking challenging: different viewpoints (these sequences are captured by moving cameras); similar objects nearby; sudden motions; illumination changes; reflectance variations of the targets; and partial occlusions. The tracking results are compared with the basic mean shift and particle filtering trackers. Since the proposed tracker updates the target model based on feature selection, it is reasonable to compare it with those adaptive trackers. The variance ratio and peak difference [3] trackers are included for this purpose. In the particle filtering tracker, the target model is represented by 12 × 12 × 12-bin RGB histograms. There are 100 samples in the sample set. RGB histograms are also adopted in the basic mean shift algorithm. The similarity measure is Bhattacharya distance between the model and its candidate. The most important criterion for the comparison is the percentage of dataset tracked, which is the number of the tracked frames divided by the total number of frames. The track is considered to be lost if the bounding box does not overlap the ground truth. The tracking success rates achieved by each tracker are compared and the results are shown in Fig. 1. The proposed tracker gives the
584
J. Wang and Y. Yagi
(b) f=516
(a) f=1
(c) f=1300
(d) Target appearance changes
Fig. 2. Tracking results of the EgTest02 sequence
best results (or has same results with another tracker) in all the test sequences. Those comparisons demonstrate that the proposed tracking algorithm has better performance than other trackers. In Fig. 2, the tracking results for EgTest02 are shown. Despite the distractions and sudden motions in the sequence, the proposed tracker completes the tracking successfully. Fig. 2.(d) illustrates how the appearance of the target changes over time. There are sudden motions and image blur in the EgTest04, which leads to the failure of the basic mean-shift tracker. The proposed tracker detects this motion successfully and initializes auxiliary particles. These particles help the proposed tracker to conquer the problem brought by the sudden motion. The running time of the proposed tracker depends on the difficulty level of the image sequence being tracked. If sudden motions or distractions happen frequently, its efficiency is low. Otherwise it has high efficiency because the mean shift algorithm is adopted in most cases. The current implementation ran 16 frames per second (average speed) on a Intel Centrino 1.6GHz laptop with 1G RAM when applied to images of size 640×480. The average running time includes time to do the main tracking algorithm, to read image file from a USB disk, and to display color images with the object bounding box overlaid.
7
Conclusion and Future Work
We describe a discriminative mean shift tracking algorithm with auxiliary particles in the pursuit of robust and efficient tracking. The arrangement of the particle filtering and the mean shift algorithm is based on the difficulty of the tracking which is indicated by sudden motions and distractions. The model updating strategy in our tracker can effectively deal with appearance changes of targets. The proposed approach provides better performance than those of the mean shift, particle filtering and other trackers.
Discriminative Mean Shift Tracking with Auxiliary Particles
585
We are going to investigate how to extend the proposed method to multitarget tracking, in which multiple mean shift searching is necessary.
References 1. Bradski, G.R.: Computer Vision Face Tracking as a Component of a Perceptural User Interface. In: Proc. of the IEEE Workshop Applications of Computer Vision, pp. 214–219 (1998) 2. Cai, Y., Freitas, N., Little, J.: Robust Visual Tracking for Multiple Targets. In: Little, J. (ed.) Proc. of 6th Europearn Conf. on Computer Vision, pp. 893–908 (2006) 3. Collins, R.T., Liu, Y.: On-line Selection of Discriminative Tracking Features. IEEE Trans. on Pattern Analysis and Machine Intelligence 27(10), 1631–1643 (2005) 4. Collins, R.T., Zhou, X., Teh, S.K.: An Open Source Tracking Testbed and Evaluation Web Site. In: PETS 2005. IEEE Int’l Workshop on Performance Evaluation of Tracking and Surveillance (January 2005) 5. Comaniciu, D., Ramesh, V., Meer, P.: Kernel-based Object Tracking. IEEE Trans. Pattern Analysis Machine Intelligence 25(5), 564–577 (2003) 6. Cover, T.M., Thomas, J.A.: Elements of Information Theory. John Wiley and Sons Press, Chichester (1991) 7. Gordon, N., Salmond, D., Smith, A.: Novel approach to nonlinear/non-Gaussain Bayesian state estimation. IEEE Proc. 140(2), 107–113 (1993) 8. Gevers, T., Smeulders, A.W.M.: Color based object recognition. Pattern Recognition 32(3), 453–464 (1999) 9. Isard, M., Blake, A.: Condensation - conditional density propagation for tracking. Int’l Journal of Computer Vision 29(1), 2–28 (1998) 10. Isard, M., Blake, A.: ICONDENSATION: unifying low-level and high-level tracking in a stochastic framework. In: Proc. of 5th Europearn Conf. on Computer Vision, vol. I, pp. 893–908 (1998) 11. Jhne, B., Scharr, H., Krkel, S.: Handbook of Computer Vision and Applications. In: Jhne, B., Hauecker, H., Geiler, P. (eds.), vol. 2, pp. 125–151. Academic Press, London (1999) 12. Khan, Z., Balch, T., Dellaert, F.: An MCMC-based particle filter for tracking multiple interacting targets. In: Proc. of 5th Europearn Conf. on Computer Vision, vol. I, pp. 893–908 (2004) 13. Shan, C., Tan, T., Wei, Y.: Real-time hand tracking using a mean shift embedded particle filter. Pattern Recognition 40(7), 1958–1970 (2007) 14. Sullivan, J., Rittscher, J.: Guiding Random Particles by Deterministic Search. In: Proc. of Eighth IEEE Int’l Conf. on Computer Vision, vol. I, pp. 323–330 (2001) 15. Swain, M., Ballard, D.: Color Indexing. Int’l Journal of Computer Vision 7, 11–32 (1991) 16. Viola, P., Jones, M.J., Snow, D.: Detecting pedestrians using patterns of motion and appearance. Int’l Journal of Computer Vision 63(2), 153–161 (2005) 17. Wang, J., Yagi, Y.: Integrating Shape and Color Features for Adaptive Real-time Object Tracking. In: IEEE Int’l Conf. on Robotics and Biomimetics 2006 (2006)
Efficient Search in Document Image Collections Anand Kumar1 , C.V. Jawahar1, and R. Manmatha2 1 Center for Visual Information Technology International Institute of Information Technology Hyderabad, India - 500032 [email protected], [email protected] 2 Department of Computer Science University of Massachusetts Amherst, MA 01003, USA [email protected]
Abstract. This paper presents an efficient indexing and retrieval scheme for searching in document image databases. In many non-European languages, optical character recognizers are not very accurate. Word spotting - word image matching - may instead be used to retrieve word images in response to a word image query. The approaches used for word spotting so far, dynamic time warping and/or nearest neighbor search, tend to be slow. Here indexing is done using locality sensitive hashing (LSH) - a technique which computes multiple hashes - using word image features computed at word level. Efficiency and scalability is achieved by content-sensitive hashing implemented through approximate nearest neighbor computation. We demonstrate that the technique achieves high precision and recall (in the 90% range), using a large image corpus consisting of seven Kalidasa’s (a well known Indian poet of antiquity) books in the Telugu language. The accuracy is comparable to using dynamic time warping and nearest neighbor search while the speed is orders of magnitude better - 20000 word images can be searched in milliseconds.
1
Introduction
Many document image collections are now being scanned and made available over the Internet or in digital libraries. Effective access to such information sources is limited by the lack of efficient retrieval schemes. The use of text search methods requires efficient and robust optical character recognizers(OCR), which are presently unavailable for Indian languages [1]. Another possibility is to search in the image domain using word spotting [2,3,4]. Direct matching of images is inefficient due to the complexity of matching and thus impractical for large databases. We solve this problem by directly hashing word image representations. We present an efficient mechanism for indexing and retrieval in large document image collections. First, words are automatically segmented. Then features are computed at word level and indexed. Word retrieval is done very efficiently by using an approximate nearest neighbor retrieval technique called locality sensitive hashing (LSH). Word images are hashed into multiple tables with features computed at word level. Content-sensitive hash functions are used to hash words Y. Yagi et al. (Eds.): ACCV 2007, Part I, LNCS 4843, pp. 586–595, 2007. c Springer-Verlag Berlin Heidelberg 2007
Efficient Search in Document Image Collections
587
such that the probability of grouping similar words in the same index of the hash table is high. The sub-linear time content-sensitive hashing scheme makes the search very fast without degrading the accuracy. Experiments on a collection of Kalidasa’s - the classical Indian poet of antiquity - books in Telugu demonstrate that 20,000 word images may be searched in a few milliseconds. The approach thus makes searching large document image collections practical. There are essentially three classes of techniques to search document image collections. The first is based on using a recognizer to convert an image to text and then searching the results using a text search engine. An example is the gHMM approach of Chan et al. [5], suggested for printed and handwritten Arabic documents. It uses gHMMs with a bi-gram letter transition model, and KPCA/LDA for letter discrimination. In this approach segmentation and recognition go hand in hand. The words are modeled at letter level, where the likelihood of a word given a segment is used for discriminating words. The Byblos system [6] also uses a similar approach to recognize documents where a line is first segmented out and then divided in to image strips. Each line is then recognized using an HMM and a bi-gram letter transition model. The second used by Rath et al. [7], involves the automatic annotation of word images with a lexicon and probabilities using a relevance-based language model. Here, words are segmented out and then each word image is annotated using a statistical model with the entire lexicon and probabilities. A language model retrieval approach is then used to search the documents. The technique was successfully used to build a 1000 page demonstration for George Washington’s handwritten manuscripts. The third approach proposed by Rath and Manmatha [2,3] involves using what is called word spotting, where word images are matched with each other and then clustered. Each cluster is then annotated by a person. Alternatively, Jawahar et al. [4] showed that in the case of printed books one can synthesize the query image from a textual query for making the system more usable. Word spotting has been tried for many different kinds of documents both handwritten and print. Rath and Manmatha [2] used dynamic time warping (DTW) to compute image similarities for handwriting. The word similarities are then used for clustering using K-means or agglomerative clustering techniques. This approach was adopted in Jawahar et al. [4] for printed Indian language document images. To simplify the process of querying, a word image is generated for each query and the cluster corresponding to this word is identified. In such methods, efficiency is achieved by significant offline computation. Ataer and Duygulu [8] tried word spotting for handwritten Ottoman documents where they use successive pruning stages to eliminate irrelevant words. Gatos et al. [9] used word spotting for old Greek typewritten manuscripts for which OCRs did not work. One advantage of word spotting over traditional OCR methods is that they take advantage of the fact that within corpora such as books the word images are likely to be much more similar, which traditional OCRs do not do. In addition, techniques that work at the symbol level of word images, like [5], are very sensitive to segmentation errors. Segmentation of Indian language document images at symbol level is very difficult due to the complexity of the scripts.
588
A. Kumar, C.V. Jawahar, and R. Manmatha
Many of these techniques (for example DTW) are computationally expensive and do not scale very well. Inspite of this, Sankar et al. [10] successfully indexed 500 books in Indian languages using this approach by doing virtually all the computation off-line. Avoiding DTW, Rath et al. [3] demonstrated the use of direct clustering of word image features on historical handwritten manuscripts. However, clustering is itself an expensive operation. Image matching often involves offline nearest neighbor computations and storage for efficient access. These nearest neighbor techniques are expensive in high dimensions even when computed off-line. Indyk and Motwani [11] proposed an approximate nearest neighbor search technique called locality sensitive hashing (LSH) which is much more efficient. LSH has been applied to a number of problems including some in computer vision. For example, LSH is used to efficiently index high dimensional pose examples by Shakhnarovich et al. [12]. Matei et al. [13] use LSH for 3D object indexing. LSH is different from the geometric hashing approaches used in model based recognition of 3-D objects in occluded scenes from 2-D gray scale images [14] and also for finding documents from a set of camera-based document images [15].
Fig. 1. Sample document images from Kalidasa’s books in Telugu
Our work mainly aims at addressing some of the issues involved in effective and efficient retrieval in document images with effective representations of the word images. We demonstrate efficient retrieval through content-sensitive hashing on a collection of Kalidasa’s writings. Sample pages from the Kalidasa collection are shown in Figure 1.
2
Content Sensitive Hashing
In the proposed retrieval technique, the index is built by hashing word level features of document images. The features are hashed using content sensitive hash functions, such that the probability of finding words with similar content in the same bucket is high. The same content sensitive hash functions are used to
Efficient Search in Document Image Collections Input
589
Output
11 00 00 11 Document Images Textual Query
Relevant Documents
Pre−processing Hashed Words
Word Rendering
Segementation and Word detection
Feature Extraction
Hashing
Online Process
Feature Extraction Offline Process
Fig. 2. Hashing Method: Word image hashing for efficient search. Showing offline preprocessing and on-line query processing stages.
query similar words during the search. The major challenges in efficient indexing and retrieval are the preprocessing and word matching times. We overcome these challenges with the use of hashing. A conceptual block diagram of the technique is shown in Figure 2. Books are scanned and processed to index the document pages. The textual word query is first converted to an image by rendering, features are extracted from these images and then search is carried out to retrieve relevant word images. To facilitate searching, scanned document images are preprocessed and segmented at word level. A set of features are extracted as representatives of word images to be indexed. Content-sensitive hash functions are used to hash the features such that similar word images are grouped in the same index of the hash table. 2.1
Word Image Representation
We employ a combination of scalar, profile, structural and transform domain feature extraction methods as used in [2,3,4]. Scalar features include the number of ascenders, descenders and the aspect ratio. The profile and structural features include: projection profiles, background to ink transitions and upper and lower word profiles. Fixed length description of the features are obtained by computing lower order coefficients of a DFT (Discrete Fourier Transform) - discarding noisy high order coefficients makes the representation more robust. We use 84 Fourier coefficients of the segmented profiles and ink transition features to represent the word images. Finding similar word images is now equivalent to the nearest neighbor search (NNS) problem: Given a set of n points P = p1 , . . . , pn in some metric space X, we preprocess P so as to efficiently answer queries, which require finding the
590
A. Kumar, C.V. Jawahar, and R. Manmatha
point in P closest to a query point q ∈ X. Traditional data structures for similarity search suffer from the curse of dimensionality. Locality sensitive hashing (LSH) is a state-of-the-art technique introduced by Indyk and Motwani [11] to alleviate the problem of high dimensional similarity search in large databases. The main idea in LSH is to hash points into bins based on a probability of collision. Thus, points that are far in the parameter space will have a high probability of landing into different bins, while close points will go into the same bucket. It has been shown that LSH out-performs tree-based structures such as the Sphere/Rectangle-tree (SR-tree) by at least an order of magnitude. 2.2
Hashing Technique
Let P = {x1 , x2 , . . . , xn } be the words in the document image collection. A word is represented by a feature vector x = {f1 , . . . , fD }, represented as a point x ∈ RD in feature space, where fj is computed by extracting features that describe the content of the word images. The extracted features satisfy the following assumptions. 1. A distance function d is given which measures the content level similarity of the words, and a radius R in the feature space is given such that x1 , x2 are considered similar iff d(x1 , x2 ) < R. 2. For a randomly chosen word image, there exists a word image with high probability and similar feature values in the collection. 3. There are no significant variations in feature vectors of the words with similar content, or the feature extraction process is unbiased. The distance function and the similarity threshold are dependent on the particular task, and often reflect perceptual similarities between the words. The last assumption implies that there are no significant sources of variation in the word features for words that are similar in content. The content similarity search is done by efficient nearest neighbor searching with content-sensitive hashing algorithm. The content-sensitive hashing is achieved by hashing words using a number of hash functions from a family H = {h : S → U } of functions. H is called content-sensitive if for any q, the function p(t) = P rH [h(q) = h(x) : ||q − v|| = t]
(1)
is strictly decreasing in t. That is, the probability of collision of points q and x is decreasing with content dissimilarity (distance) between them. We concatenate several hash functions h ∈ H. In particular define a function family G = {g : S → U k } such that, g(x) = (h1 (x), . . . , hk (x)). For an integer L, the algorithm chooses L functions g1 , . . . , gL from G, independently and uniformly at random. During preprocessing, the algorithm stores each input point in buckets gj (x), for all j = 1, . . . , L. Since the total number of buckets may be large, the algorithm retains only the non-empty buckets by resorting to hashing.
Efficient Search in Document Image Collections
591
Algorithm 1. Content Sensitive Hashing Input: Word Images Wj , j = 1, . . . , n Output: Hash Tables Ti , i = 1, . . . , l 1: for each i = 1, . . . , l do 2: Initialize hash table Ti with hash functions gi 3: end for 4: for each i = 1, . . . , l do 5: for each j = 1, . . . , n do 6: Pre-process word image Wj (noise removal etc). 7: Extract features Fj of word image Wj . 8: Compute hash bucket I = gi (Fj ) 9: Store word image Wj on bucket I of hash table Ti 10: end for 11: end for
A D dimensional word feature x is mapped onto a set of integers by each hash function ha,b (x). Each hash function in the family is indexed by a choice of random a and b, where a is a D dimensional vector with entries chosen independently from a p-stable distribution and b is a real number chosen uniformly from the range[0, w]. For a fixed a, b the hash function ha,b is given by, ha,b (x) =
a · x + b w
(2)
Generally w = 4. The dot product a · x projects each vector onto a real line. The real line is chopped into equi-width segments of appropriate size w and hash values are assigned to vectors based on which segment they project onto. The value of k is chosen such that tc + tg is minimal, where tc is the mean query time and tg is the time to compute the hash functions in L hash tables. The values of k is determined by estimating the times on a sample data set S ⊆ P . The details about such parameter settings and the hash functions are presented in [11,16]. Algorithm 1 summarizes the major steps of content-sensitive hashing. Given a query word image, it is represented with the set of features q. The first level k hash functions are calculated and concatenated to get bucket id’s gi (q), i = 1, . . . , L in L hash tables. Then all the features, and the corresponding words, in the buckets of L tables are retrieved as the query results. Thus the problem of finding nearest neighbor boils down to searching only the vectors in the bucket that have the same hash index value as the query. Algorithm 2 summarizes the major steps of querying. The hash based search in a collection of document images is faster as compared to other approaches, like exhaustive search with DTW and nearest neighbor techniques. Approaches presented in the literature take a long time in building the index and retrieval due to preprocessing and complex matching procedures. This computational time can be reduced with the elimination of costly processes, like clustering. We achieve this by employing faster content-sensitive hashing technique. We achieve interactive retrieval with retrieval speed in milliseconds. Time
592
A. Kumar, C.V. Jawahar, and R. Manmatha
Algorithm 2. Word Retrieval Input: Query Word Image w Output: Similar word images 1: O ← φ 2: for each i = 1, . . . , l do 3: Pre-process word image w (noise removal etc). 4: Extract features Fw of word image w. 5: Compute hash function I = gi (Fw ) 6: O ← O ∪ {points found in index I of Ti } 7: end for 8: Return similar words O by linear search.
inefficient offline processing of the data is not required for creating the index. The hashing technique avoids complex image matching methods and searches in sub-linear time.
3
Results and Discussions
We evaluated the proposed hash based retrieval scheme on word image data sets obtained from a collection of 7 Kalidasa books. The books are printed in Telugu, an Indian language. The document images were scanned and preprocessed to get segmented words with little manual effort to remove segmentation errors. Then, the words were represented by a set of features. Around 20 words were annotated in each book for experimentation and performance evaluation purpose. Given a textual query word, an image is rendered (generated). Features are extracted from the query image and hashed to search and retrieve the relevant words. The book-wise performance measured using precision, recall and F-score values are shown in Table 1. The query image and example search results are shown in Figure 3. The first two rows show correct results. Sometimes other words may also appear somewhat visually similar and the last column in the last two rows shows examples of such words being retrieved. Table 1. Search Performance: Precision, recall and F-score values for retrieval experiments conducted on each book from Kalidasa collection Book # Pages Maalavikaagnimitra 292 Vikramuurvashiyam 286 Abhijnanasakuntalam 312 Ritusamhara 142 Kumarasambhava 282 Raghuvamsha 300 Meghaduta 238
# Words Precision Recall F-score 22,500 100.00 91.72 95.68 23, 600 100.00 95.58 97.74 22,500 96.79 91.27 93.96 11,000 94.65 93.67 94.16 56,100 92.37 90.21 91.27 36,000 93.23 92.6 92.91 44,000 96.15 93.53 94.82
Efficient Search in Document Image Collections
Query Image
593
Some of the Retrieved Images
Fig. 3. Results: Example (Telugu) words searched for input queries
Fig. 4. Results: Words with small variations in style and size are retrieved
Examples of queries containing words of different sizes and style types are shown in Figure 4. Such results are obtained by querying the same word in multiple books of the collection. Using the same query across two different books of the collection retrieves words which are content-wise similar. Indian language words have small form variations. For example, the same word may have different case endings. Such words are also searched correctly using the proposed solution. Example results of such queries are shown in Figure 5 (row 2). The retrieved words have the same stem, which is due to the similarity in image content. There are limits to the font variations that can be handled by the proposed retrieval technique. Experiments show that we cannot use combinations of different font words but such combinations are very unlikely to occur in books. The proposed hashed based search is sub-linear and much faster than exhaustive nearest neighbor search. The plot in Figure 6(a) shows the time efficiency of our hash based search versus nearest neighbor search. The experiments were conducted on data sets of increasing size (by 5,000 words) in each iteration. The maximum number of words used were around 45,000. With the use of the maximum size data set, the maximum time to search relevant words was of the order of milliseconds. The experiments were conducted on an AMD Athlon 64 bit processor using 512 MB memory. The precision and recall values are controlled by the query radius (distance) value. Experiments were conducted on synthetic images of Telugu language to see
594
A. Kumar, C.V. Jawahar, and R. Manmatha
Query Image
Some of the Retrieved Images
Fig. 5. Results: Words with small form variations are retrieved as relevant
100 Precision Recall
% Precision & Recall
Time (Sec)
0.2
Exhaustive NNS Hash based search
0.15 0.1
0.05
95
90
85
1
1.5
2
2.5 3 Data Size
(a)
3.5
4 4
x 10
0.2
0.25 Query Radius
0.3
0.35
(b)
Fig. 6. Performance comparison: (a) Hashing and exhaustive nearest neighbor search. (b) Effect of distance: Precision and recall change with the query radius.
the effect of radius on the performance. Around 6,000 synthetic word images of Telugu language were used for the experiments. Each word was repeated around 4-10 times in the whole collection. Figure 6(b) shows the change in precision and recall values with the radius. The degradation in performance with the increase in radius indicates that many irrelevant words are added to group of similar words. Similar results were obtained with different font datasets for Telugu language. Therefore query distance has to be determined experimentally. Table 2 compares this approach to one based on using the DTW score as a similarity measure. It shows that our method is much faster than the DTW based exhaustive matching and search procedure while the accuracy is similar. Table 2. Performance: DTW based exhaustive search is much slower. Accuracy of the proposed method similar to the DTW matching. Book
Hash Based Search DTW Based NNS Precision Recall Time(sec) Precision Recall Time(sec) Abhijnanasakuntalam 96.79 91.27 0.005 95.27 93.71 650 Ritusamhara 94.65 93.67 0.003 93.33 96.63 216
Efficient Search in Document Image Collections
4
595
Conclusion and Future Work
We presented an efficient indexing and retrieval scheme for searching in large document image databases. Efficiency and scalability along with high precision and recall values are achieved by content-sensitive hashing. The retrieval speed is orders of magnitude better - the technique can search 20,000 word images in milliseconds. We have demonstrated that this technique is practical for searching printed documents rapidly. Future improvements could include feature selection using machine learning techniques to include multiple fonts and styles.
References 1. Pal, U., Chaudhuri, B.: Indian script character recognition: A survey. Pattern Recognition 37, 1887–1899 (2004) 2. Rath, T.M., Manmatha, R.: Word image matching using dynamic time warping. In: Conference on Computer Vision and Pattern Recognition, vol. (2), pp. 521–527 (2003) 3. Rath, T.M., Manmatha, R.: Word spotting for historical documents. IJDAR 9(2), 139–152 (2007) 4. Balasubramanian, A., Meshesha, M., Jawahar, C.V.: Retrieval from document image collections. In: Bunke, H., Spitz, A.L. (eds.) DAS 2006. LNCS, vol. 3872, pp. 1–12. Springer, Heidelberg (2006) 5. Chan, J., Ziftci, C., Forsyth, D.A.: Searching off-line arabic documents. In: CVPR. Conference on Computer Vision and Pattern Recognition, vol. (2), pp. 1455–1462 (2006) 6. Lu, Z., Schwartz, R., Natarajan, P., Bazzi, I., Makhoul, J.: Advances in the bbn byblos ocr system. In: ICDAR, pp. 337–340 (1999) 7. Rath, T.M., Manmatha, R., Lavrenko, V.: A search engine for historical manuscript images. In: SIGIR, pp. 369–376 (2004) 8. Ataer, E., Duygulu, P.: Retrieval of ottoman documents. In: Multimedia Information Retrieval (MIR) workshop, pp. 155–162 (2006) 9. Konidaris, T., Gatos, B., Ntzios, K., Pratikakis, I., Theodoridis, S., Perantonis, S.J.: Keyword-guided word spotting in historical printed documents using synthetic data and user feedback. IJDAR 9(2), 167–177 (2007) 10. Sankar, K.P., Jawahar, C.V.: Probabilistic reverse annotation for large scale image retrieval. In: Conference on Computer Vision and Pattern Recognition, pp. 1–6 (2007) 11. Indyk, P., Motwani, R.: Approximate nearest neighbors: towards removing the curse of dimensionality. In: SOTC, pp. 604–613 (1998) 12. Shakhnarovich, G., Viola, P., Darrell, T.: Fast pose estimation with parametersensitive hashing. In: Sebe, N., Lew, M.S., Huang, T.S. (eds.) Computer Vision in Human-Computer Interaction. LNCS, vol. 3766, pp. 750–757. Springer, Heidelberg (2005) 13. Matei, B., Shan, Y., Sawhney, H., Tan, Y., Kumar, R., Huber, D., Hebert, M.: Rapid object indexing using locality sensitive hashing and joint 3D-signature space estimation. IEEE Trans. PAMI 28(7), 1111–1126 (2006) 14. Lamdan, Y., Wolfson, H.: Geometric hashing: A general and efficient model-based recognition scheme. In: ICCV, pp. 238–249 (1988) 15. Nakai, T., Kise, K., Iwamura, M.: Use of affine invariants in locally likely arrangement hashing for camera-based document image retrieval. In: Bunke, H., Spitz, A.L. (eds.) DAS 2006. LNCS, vol. 3872, pp. 541–552. Springer, Heidelberg (2006) 16. Gionis, A., Indyk, P., Motwani, R.: Similarity search in high dimensions via hashing. In: Proceedings of the 25th VLDB conference, pp. 518–529 (1999)
Hand Posture Estimation in Complex Backgrounds by Considering Mis-match of Model Akihiro Imai1 , Nobutaka Shimada2 , and Yoshiaki Shirai2
2
1 Dept.of Computer-Controlled Mechanical Systems, Osaka University Yamadaoka, Suita, Osaka 565-0871, Japan Dept.of Human and Computer Intelligence, Ritumeikan University, Nojihigashi, Kusatsu, Shiga 525-8577, Japan
Abstract. This paper proposes a novel method of estimating 3-D hand posture from images observed in complex backgrounds. Conventional methods often cause mistakes by mis-matches of local image features. Our method considers possibility of the mis-match between each posture model appearance and the other model appearances in a Baysian stochastic estimation form by introducing a novel likelihood concept “Mistakenly Matching Likelihood (MML)“. The correct posture model is discriminated from mis-matches by MML-based posture candidate evaluation. The method is applied to hand tracking problem in complex backgrounds and its effectiveness is shown.
1
Introduction
Precise hand-finger shape estimation methods using visual cues have been developed [1][2][3][4][5][6] in order to implement the gestural interfaces in a touch-less manner which are utilized in interaction with virtual environments and automatic sign-language translation. One of the difficulties of implementing the interfaces based on the hand shape estimation exists in its situations where the interfaces are needed: its complex backgrounds like colorful and textured clothes, skin-colored region as human face and some desktops on which various tools and objects are scattered. Since the hand shape even in simple backgrounds is a tough problem due to its great varieties of posture, shape estimation with simultaneous segmentation is still a challenging problem. To solve the problem with feasible computing resources, some trials were reported from the following two viewpoints: 1) how to reduce the number of posture candidates to consider (i.e. how to predict the posture), 2) how to evaluate the matching degree between the posture candidate and observed image features. From the viewpoint 1), Active Shape Model [7] is proposed, which learns acceptable shape deformations and tracks the region contour or texture assuming smooth deformation and motion. Non-smooth deformation can be treated by introducing Switching Linear Dynamics [8]. 3-D model-based shape prediction and tracking, not based on appearance learning, is also proposed [5]. Most of Y. Yagi et al. (Eds.): ACCV 2007, Part I, LNCS 4843, pp. 596–607, 2007. c Springer-Verlag Berlin Heidelberg 2007
Hand Posture Estimation in Complex Backgrounds
(a) Estimation result
(b) Edges of estimation result put on input image
(c) Shape similar to input image
(d) Edges of similar shape put on input image
Fig. 1. Mistake of conventional method
597
Fig. 2. Corresponding of edges
those methods employ a parallel search scheme in tracking, like beam-search or particle filter for robustness against temporal mis-estimation and tracking failure [5][9][10][11][12][13][14]. While many improvements from the first viewpoint are reported, those from the second viewpoint, concerned with the evaluation of matching degree, are comparatively few and most implementations employ a simple feature corresponding and evaluating method: chamfer matching [15][16]. Chamfer matching makes correspondences between the features with the least distance in the image, and evaluates the matching degree by the sum of the distances (chamfer distance). This simple matching scheme, of course, causes often a wrong shape estimate on the complicated backgrounds. Fig. 1 is an example of wrong estimate of hand posture caused by chamfer matching. Because many edge textures are observed in the hand region, the finger tips of the posture model are mistakenly corresponding to the inner edges (see Fig. 2) and as a result its chamfer distance is evaluated too small. While this problem is hard to avoid as long as using chamfer matching, no more appropriate matching method is found other than chamfer matching. Therefore the matching degree should be evaluated under the consideration of that such a wrong matching often happens. Nevertheless, the existence of the mismatch caused by chamfer matching does not directly mean its uselessness. Embedding approach [15] evaluates the matching degree between an input image feature and not only one posture model but also several other reference models. For example of Fig. 1, in addition to the correct posture candidate (c), the candidate (a) also has so high matching
598
A. Imai, N. Shimada, and Y. Shirai
degree that (a) is picked up as the estimate. However, if only (a) has the high matching degree when (a) should be the correct matching, these two cases can be discriminated by evaluating the matching degrees with both reference models (a) and (c). Since Embedding approach only uses an ad-hoc way by evaluating the squared sum of the matching degrees of all reference models, its estimate is not the optimum in Bayesian point of view. This paper mathematically derives the Baysian form of Embedding approach. In its derivation, a novel concept of likelihood is introduced: Mistakenly Matching Likelihood (MML), which predicts the high evaluation caused by wrong matching and gives the ability to discriminate the true estimate from the false match in a stochastic way. The derived MML-based candidate evaluation is applied to hand tracking problem in complex backgrounds and its effectiveness is experimentally shown.
2
Acquisition of Typical Hand Posture Images
3-D Hand model used in our research is originally wireframe model. The model is modified into a shaded model. The joint bending angles are denoted by θb,t,1 , θb,t,2 , θb,t,3 , θb,i,1 , θb,i,2 , θb,i,3 , . . . and opening angles at a base joint of the fingers are denoted by θo,t , θo,i , . . . (shown in Fig.3). The posture of the whole hand model is represented by translation tx , ty , tz and Ritalin θr,x , θr,y , θr,z . As a whole, the shape of the hand model has 26 degrees of freedom θ = (θb,t,1 , . . . , θr,z ).
(1)
CG images of typical hand models are shown in Fig. 4. The finger joints dependently moves in natural actions [5]. In index, middle, ring, pinky fingers, adjacent joint angles are usually similar. This kind of joint constraints reduce
θo,r
θo,m θo,i
θb,i,3 θb,i,2
θo,p θo,t
θb,i,1 t y θy t z θz (a) Wireframe model
t x θx
(b) Model after shading Fig. 3. Hand model
Hand Posture Estimation in Complex Backgrounds
(a)
(b)
(c)
(d)
599
(e)
Fig. 4. CG images of typical hand postures
Fig. 5. Edge images of hand model
Table 1. Quantization width from search center c Δθbend [◦ ] c Δθopen [◦ ] c Δθo,m [◦ ] c Δθr,x [◦ ] c Δθr,y [◦ ] c Δθr,z [◦ ] c c Δtx , Δty [mm] Δtcz [mm]
−ζ -6 -30 -15 -6 -8 -40
0 0 0 0 0 0 0 0
ζ 6 30 15 6 8 40
the number of possible postures. Fig.5 shows the edge images generated from the CG images of the typical hand postures under the constraints. The posture parameters θ to be estimated are quantized. The change is shown c , that of θo other than in Table 1 (The change of θb is represented by Δθbend c middle is represented by Δθopen ). Each parameter of θb is assumed to change 0 or ±ζ. Each parameter has own ζ between 9 deg to 15 deg.
3
Matching Method
The system has several hand models with various dimensions: i.e. lengths and widths of the palm and fingers. Since input image sequences are assumed to start from a predefined simple shape and an initial position, the dimensions are easily initialized at the first frame. The posture parameters to be estimated are around the posture estimate at the previous image frame. When each input image is obtained, the best-matched model is determined by the maximum likelihood criterion. Let I denote edges and skin regions extracted from an input image and Θj denote a quantized parameter vector of the j th model. The criterion is ˆ = arg max P (I|Θj ) Θ (2) Θj
where P (I|Θj ) denotes the likelihood of the j th model for the input. The likelihood is defined based on the difference of the image I and the the appearance of the shape model.
600
3.1
A. Imai, N. Shimada, and Y. Shirai
Difference of Image and Appearance of Shape Model
In this paper, edge image I (e) and skin-color region image I (s) are used as the image features of the input I. The difference between the silhouette of a typ(s) ical hand model and that of an image is computed. Let Aθ be the silhouette generated from a typical hand model θ. The Difference of the silhouette, (s) (s) fskin (Aθ ; I (s) ), is defined as the area of Aθ which does not overlap with I (s) . (e) (e) The difference between the I (e) and the edge appearance Aθ , fdist (Aθ ; I (e) ), is computed by a modified chamfer matching, in which the edge points are classified by gradient direction and the edges is matched by the original chamfer matching in each direction class[13][18]. The distance is weighted by the edge contrast and as a result fdist is defined as follows: (e) fdist (Aθ ; I (e) ) = wθ,j min (||xθ,j − xI (e) ,k || + fI (e) ,k + g(j, k)) (3) j
k
where xθ,j and xI (e) ,k denotes the j th edge point of the model and the k th edge point of an input edge image. || · || is 2-dimensional Euclidean norm, wθ,j is a weight constant, fI (e) ,k is a penalty for an edge with a low contrast, dθ,j wθ,j = . l dθ,l
(4)
fI (e) ,k = −wd dI (e) ,k
(5)
where wd represents weight constant. The difference of gradient direction g(j, k) is defined in terms of gradient direction φθ,j of the model edge and gradient direction φI (e) ,k as (6) g(j, k) = wφ ||φθ,j − φI (e) ,k ||. where wφ represents weight constant. All weights are experimentally determined. The modified chamfer matching is computed by using distance transformation as fast as the original chamfer matching. 3.2
Discrimination Principle
The example of mis-matching by a conventional method has been shown as Fig. 1 in section 1. Let Θa and Θc respectively denote the posture parameter of model (a) (fist shape) and (c) (flat shape). Suppose one case that the true hand posture should be Θc . In this section, we describe the principle to discriminate the true match from the wrong matches due to the complicated skin textures and the backgrounds and finally introduce its stochastic forms giving the discriminating criterion. Then the probability of that the appearance AΘc is matched to the input image, p(AΘc |Θc ), should be large enough because the true posture is the same as the one which generates the appearance, Θc . On the other hand, the probability of AΘa (small fist shape), p(AΘa |Θc ) can be also large in spite of the posture difference between Θa and Θc because
Hand Posture Estimation in Complex Backgrounds
601
p(AΘa |Θc ) ∼ = p(AΘa |Θa ) AΘa
hand with Θa
AΘc
hand with Θc p(AΘc |Θa ) < p(AΘc |Θc ) Fig. 6. Likelihood for edge images
almost all of the area of AΘa is included and the inner texture edges can be wrongly matched to the finger contours. Therefore the conventional likelihood maximization can often choose Θa mistakenly for flat hand shapes like Θc due to the image capture noise, inaccuracy of the 3-D shape model and quantization errors of the posture parameters. In order to solve the mis-matches, we carefully analyze the behaviours of two more probabilities: p(AΘa |Θa ) and p(AΘc |Θa ). p(AΘa |Θa ) should be large and p(AΘc |Θa ) should be small because AΘc protrudes from the area of Θa . The four probabilities take the behaviours as follows: while p(AΘa |Θc ), p(AΘc |Θc ) are large together for the posture Θc , p(AΘa |Θa ) is large and p(AΘc |Θa ) is small for the posture Θa (see Fig. 6). Therefore when the appearance AΘa and AΘc are observed together, that posture should be estimated as Θc . When AΘa is observed alone, that posture should be Θa . When the likelihood of an appearance AΘk to a model Θj , p(AΘk |Θj ), is obtained for each possible combination of k and j in advance, the appropriate model can be chosen by taking the all p(AΘk |Θj ) values into account like the above discussion. When k and j are identical, p(AΘk |Θj ) is equivalent to the conventional likelihood function. Otherwise, it means the likelihood of that an appearance AΘk comes from a mistakenly chosen model Θj . We call the likelihood as ”mistakenly matching likelihood” (MML). 3.3
Model Selection using Mistakenly Matching Likelihood
We introduce the stochastic discrimination criterion from the principle described in the previous section by employing Bayesian estimation framework. Let AΘ1 , AΘ2 , . . . denote appearances of typical hand models. Assuming AΘ1 , AΘ2 , . . . are exclusive under each Θj , the likelihood of Θj for I can be expanded as below: p(I|Θj ) = k p(I, AΘk |Θj ) = k p(I|AΘk , Θj )p(AΘk |Θj ). (7)
602
A. Imai, N. Shimada, and Y. Shirai
Assuming the appearance AΘk has all information to generate the observed image I, condition Θj can be removed, p(I|Θj ) = k p(I|AΘk )p(AΘk |Θj ). (8) In the conventional maximum likelihood estimation method, the likelihood for the case of k = j is considered alone. In contrast, we additionally consider the MML for the case of k = j. Assuming that I (e) and I (s) are independent when a certain Θj is specified, p(I|Θj ) = p(I (e) |Θj )p(I (s) |Θj )
(9)
is derived as the discrimination criterion in stochastic form. The likelihood p(I (e) |Θj ) and p(I (s) |Θj ) are respectively derived from the following equations: (e) (e) (10) p(I (e) |Θj ) = k p(I (e) |AΘk )p(AΘk |Θj ) p(I (s) |Θj ) =
(s)
k
(s)
p(I (s) |AΘk )p(AΘk |Θj )
(e)
(11)
(e)
The probabilistic distributions p(AΘk |Θj ) and p(I (e) |AΘk ) for edge images is (s)
introduced the following sections. Those for skin color silhouette, p(AΘk |Θj ) (s)
and p(I (s) |AΘk ) can be introduced in the same manner of those for edge image. 3.4
Likelihood of Typical Hand Models for Appearances
The likelihood of typical hand model is obtained as the following form because of quantization errors of Θj . (e) (e) p(AΘk |Θj ) = Θj p(AΘk , θj |Θj )dθj (e) = Θj p(AΘk |θj , Θj )p(θj |Θj )dθj (12) (e) = Θj p(AΘk |θj )p(θj |Θj )dθj . The sampling distribution of p(θj |Θj ) can be assumed as a uniform distribu(e) tion under each Θj . Assuming p(AΘk |θj ) is constant for each j since the the quantization interval of Θj is small enough, p(AΘk |θj ) is reduced to p(AΘk |θj∗ ), (e)
(e)
where θj∗ is the mean value of the interval Θj . p(AΘk |θj∗ ) is derived as follows from the definition of fdist in sec.3.1 and assuming that fdist obeys a gaussian distributionfdist [12][13][17]: (e)
p(AΘk |θj∗ ) = αθ∗ exp(−(dM (k, j))2 ) (e)
(e)
(e)
j
(13)
(e)
where Ir (θ) is the edge image rendered from the posture θ, and (e)
dM (k, j) =
fdist (AΘ ;Ir(e) (θj∗ )) (e)
k (e)
σM
.
(14)
Hand Posture Estimation in Complex Backgrounds (e) 2
σM
603
is the variance of the value of fdist (AΘk ; Ir (θj∗ )). σM is experimentally (e)
(e)
(e)
(e)
determined. αθ∗ is normalization constant, j
(e)
αθ∗ =
k
j
−1 (e) exp(−(dM (k, j))2 ) .
(15)
In the same manner as the above, p(AΘk |θj∗ ) = αθ∗ exp(−(dM (k, j))2 ) (s)
(s)
(s)
(16)
j
(s)
where Ir (θ) is the silhouette generated from θ, and fskin (AΘ ;Ir(s) (θj∗ )) (s)
(s)
dM (k, j) =
k (s)
σM
(s) 2
(17)
.
σM is the variance of the value of fskin (AΘk ; Ir (θj∗ )). αθ∗ is normalization j constant, −1 (s) (s) 2 (18) αθ∗ = . k exp(−(dM (k, j)) ) (s)
(s)
(s)
j
3.5
Likelihood of Appearance
In this section, we explain the evaluation of the likelihood of an appearance (e) p(I (e) |AΘk ). The likelihood is defined based on the definition of fdist as p(I (e) 2
where, σI
(e)
(e) |AΘk )
=
(e) βΘk exp
2 (e) (fdist (AΘ ;I (e) )) k − . (e) 2 σI
(19)
(e)
is the variance of (fdist (AΘk ; I (e) )). It is experimentally determined.
The normalization constant abilistic distributions:
(e) βΘk
is derived from the integral condition of prob(e)
p(i(e) |AΘk )di(e) = 1
(20)
(e)
Assuming that p(i(e) |AΘk ) can be large value only for i(e) of hand images and is 0 for most of other i(e) , (e) (e) (e) p(i(e) |AΘk )di(e) ≈ p(Ir (θl )|AΘk )dθl (e) ∗ (e) ≈ l p(Ir (θl )|AΘk ) · δ (21) (e) (e) = βΘk l exp(−(dM (k, l))2 ) · δ ≡1 where δ is the range of the quantization of Θ. −1 (e) (e) 2 βΘk = l exp(−(dM (k, l)) ) · δ
(22)
604
A. Imai, N. Shimada, and Y. Shirai (e)
(e)
(e)
When AΘk wrongly matches to many of Ir (θl )(l = k), βΘk becomes small. On (e) βΘk
becomes large. It means that ambiguous the other hand when a few of those, appearance model, which is easy to mis-match to other posture’s appearances, are automatically low evaluated.
4
Estimation of More Accurate Posture Parameters
Posture parameters of the best-matched model are slightly different from that of the hand of an input image due to quantization errors of posture parameters. Thus, more accurate parameters must be estimated. The wireframe CG model of the hand is deformed so that the model is matched to an input image, and the 3-D hand shape is reconstructed from the deformed model[19]. In this method, while the curved surface shape of the hand is reconstructed, posture parameters are not estimated. We deform the CG model so that the appearance of the model is matched to those of the input image by using this method. The accurate posture parameters are estimated from coordinates of the vertices of the triangle patches of the deformed wireframe model. Parameters are estimated by the following steps of a procedure. 1. We make correspondences of edges of the best-matched model to those of the input image. 2. The change of the appearance is evaluated from the correspondences so that the edges of the model move toward those of the input image. 3. In order to reduce the huge search region of posture parameters due to the high DOF of human hand, available deformations of surface mesh of the CG model are learned by PCA in advance for each of typical postures, and then the best approximated mesh deformation is estimated by the projection to the PCA subspace. 4. Return to 1. We make correspondences of edges of the CG model deformed at 3. to those of the input image. CG model is deformed by the change of appearance evaluated from the correspondence, again. Repeat these processes. 5. We evaluate the 2-dimensional Euclidean norm of the vertices of the triangle patches between the deformed CG model and CG model generated by the posture parameters. The sum of the norms is minimized using steepest descent method. The posture parameters with minimized sum of the norms are the posture estimate.
5
Experiment
We did the experiment of posture tracking for 250 hand images. The resolution of the images is 320 × 240. The images are captured by 30 fps. In the conventional (e) (s) method, where p(I|Θj ) = p(I|AΘj )p(I|AΘj ) is used as a matching criterion, 70.4% images are correctly matched. In our method, 82.0% images are correctly matched. The success rates show effectiveness of our method.
Hand Posture Estimation in Complex Backgrounds
605
Fig. 7. Experimental result
Fig. 9. Experimental result
Fig. 8. Experimental result
Fig. 10. Experimental result
The example of the image which is correctly matched in our method while mis-matched in the conventional method, is shown in Fig. 7. While the wrong fist hand shape is matched in conventional method, the correct flat shape is matched in our method. Fig. 8 shows the results of an image sequence. In the input images, fingers are partially occluded and the edges of the background are confusingly appeared near the fingers. Such cases causing mismatches are correctly matched in our method. Fig. 9 and Fig. 10 show the tracking results for other hand shapes. These images are also correctly matched in our method.
6
Conclusion and Discussion
The paper introduces a Bayesian form of evaluation of posture candidates for hand tracking in complex backgrounds. The novel concept of Mistakenly
606
A. Imai, N. Shimada, and Y. Shirai
Matching Likelihood (MML) enables to discriminate the true posture candidate from other confusing ones when the mismatch of image features frequently occurs. Experimental results for tracking of the real human hand show the effectiveness of this evaluation method. Additional image features like optical flows or range other than edges and silhouette should be considered on this framework as future work.
Acknowledgment This work is supported in part by Grant-in-Aid for Scientific Research from Ministry of Education, Science, Sports, and Culture, Japanese Government, No.15300058. The 3-D model of the real human hand was provided by courtesy of Prof. F. Kishino and Prof. Y. Kitamura, Osaka University.
References 1. Liu, X., Fujimura, K.: Hand Gesture Recognition using Depth Data. In: Proc. 6th Int. Conf. on Automatic Face and Gesture Recognition, pp. 529–534 (2004) 2. Iwai, Y., Yagi, Y., Yachida, M.: Estimation of Hand Motion and Position from Monocular Image Sequence. In: Li, S., Teoh, E.K., Mital, D., Wang, H. (eds.) ACCV1995. LNCS, vol. 1035, pp. 230–234. Springer, Heidelberg (1996) 3. Lee, S.U., Cohen, I.: 3D Hand Reconstruction from a Monocular View. In: Proc. 17th Int. Conf. on Pattern Recognition, vol. 3, pp. 310–313 (1995) 4. Kameda, Y., Minoh, M., Ikeda, K.: Three Dimensional Pose Estimation of an Articulated Object from its Silhouette Image. In: ACCV 1993, pp. 612–615 (1993) 5. Shimada, N., Kimura, K., Shirai, Y.: Real-time 3-D Hand Posture Estimation based on 2-D Appearance Retrieval Using Monocular Camera. In: Proc. Int. Workshop on RATFG-RTS, pp. 23–30 (2001) 6. Imai, A., Shimada, N., Shirai, Y.: 3-D Hand Posture Recognition by Training Contour Variation. In: Proc. of The 6th Int. Conf. on Automatic Face and Gesture Recognition, pp. 895–900 (2004) 7. Cootes, T.F., Taylor, C.J., Cooper, D.H., Graham, J.: Active Shape Models-Their Training and Application. COMPUTER VISION AND IMAGE UNDERSTANDING 61(1), 38–59 (1995) 8. Jeong, M., Kuno, Y., Shimada, N., Shirai, Y.: Recognition of shape-changing hand gestures. IEICE Trans. inf.Syst. E85-D(10), 1678–1687 (2002) 9. Isard, M., Blake, A.: Visual tracking by stochastic propagation of conditional density. In: Proc. European Conf. Computer Vision, pp. 343–356 (1996) 10. Isard, M., Blake, A.: ICONDENSATION:Unifying low-level and high-level tracking in a stochastic framework. In: Proc. European Conf. Computer Vision, pp. 767–781 (1996) 11. Heap, T., Hogg, D.: Wormholes in Shape Space:Tracking through Discontinuous Changes in Shape. In: 6th Int. Conf. on Computer Vision, pp. 344–349 (1998) 12. Zhou, H., Huand, T.S.: Tracking Articulated Hand Motion with Eigen Dynamics Analysis. 9th Int. Conf. on Computer Vision 2, 1102–1109 (2003)
Hand Posture Estimation in Complex Backgrounds
607
13. Stenger, B., Thayananthan, A., Torr, P.H.S., Cipolla, R.: Model-Based Hand Tracking Using a Hierarchical Bayesian Filter. IEEE Transactions on Pattern Analysis and Machine Intelligence 28(9), 1372–1384 (2006) 14. Wu, Y., Lin, J., Huang, T.S.: Analyzing and Capturing Articulated Hand Motion in Image Sequences. IEEE TRANS. ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 27(12), 1910–1922 (2005) 15. Athitsos, V., Sclaroff, S.: Estimating 3D Hand Pose from a Cluttered Image. In: IEEE Conf. on Computer Vision and Pattern Recognition, vol. II, pp. 432–439. IEEE Computer Society Press, Los Alamitos (2003) 16. Barrow, H.G., Tenenbaum, J.M., Bolles, R.C., Wolf, H.C.: Parametric correspondence and chamfer matching: Two new techniques for image matching. In: Proc. 5th Int. Joint Conf. Artificial Intelligence, pp. 659–663 (1977) 17. Blake, A., Isard, M.: Active Contours. Springer, Heidelberg (1998) 18. Navaratnam, R., Thayananthan, A., Torr, P.H.S., Cipolla, R.: Hierarchical PartBased Human Body Pose Estimation. In: Proc. British machine Vision Conference (2005) 19. Heap, T., Hogg, D.: Towards 3D Hand Tracking using a Deformable Model. In: 2nd Int. Conf. on Automatic Face and Gesture Recognition, pp. 140–145 (1996)
Learning Generative Models for Monocular Body Pose Estimation Tobias Jaeggli1 , Esther Koller-Meier1 , and Luc Van Gool1,2 1
2
ETH Zurich, D-ITET/BIWI, CH-8092 Zurich Katholieke Universiteit Leuven, ESAT/VISICS, B-3001 Leuven [email protected]
Abstract. We consider the problem of monocular 3d body pose tracking from video sequences. This task is inherently ambiguous. We propose to learn a generative model of the relationship of body pose and image appearance using a sparse kernel regressor. Within a particle filtering framework, the potentially multimodal posterior probability distributions can then be inferred. The 2d bounding box location of the person in the image is estimated along with its body pose. Body poses are modelled on a low-dimensional manifold, obtained by LLE dimensionality reduction. In addition to the appearance model, we learn a prior model of likely body poses and a nonlinear dynamical model, making both pose and bounding box estimation more robust. The approach is evaluated on a number of challenging video sequences, showing the ability of the approach to deal with low-resolution images and noise.
1 Introduction Monocular body pose estimation is difficult, because a certain input image can often be interpreted in different ways. Image features computed from the silhouette of the tracked figure hold rich information about the body pose, but silhouettes are inherently ambiguous, e.g. due to the Necker reversal. Through the use of prior models this problem can be alleviated to a certain degree, but in many cases the interpretation is ambiguous and multi-valued throughout the sequence. Several approaches have been proposed to tackle this problem, they can be divided into discriminative and generative methods. Discriminative approaches directly infer body poses given an appearance descriptor, whereas generative approaches provide a mechanism to predict the appearance features given a pose hypothesis, which is then used in a generative inference framework such as particle filtering or numerical optimisation. Recently, statistical methods have been introduced that can learn the relationship of pose and appearance from a training data set. They often follow a discriminative approach and have to deal explicitly with the nonfunctional nature of the multi-valued mapping from appearance to pose [1,2,3,4]. Generative approaches on the other hand typically use hand crafted geometric body models to predict image appearances (e.g. [5], see [6,7] for an overview). We propose to combine the generative methodology with a learning based statistical approach. The mapping from pose to appearance is single-valued and can thus be seen Y. Yagi et al. (Eds.): ACCV 2007, Part I, LNCS 4843, pp. 608–617, 2007. c Springer-Verlag Berlin Heidelberg 2007
Learning Generative Models for Monocular Body Pose Estimation
609
as a nonlinear regression problem. We approximate the mapping with a RVM kernel regressor [8] that is efficient due to its sparsity. The human body has many degrees of freedom, leading to high dimensional pose parametrisations. In oder to avoid the difficulties of high dimensionality in both the learning and the inference stage, we apply a nonlinear dimensionality reduction algorithm [9] to a set of motion capture data containing walking and running movements. 1.1 Related Work Statistical approaches to the monocular pose estimation problem include [1,2,3,4,10,11]. In [10] the focus lies on the appearance descriptor, and the discriminative mapping from appearance to pose is assumed to be single-valued and thus modelled with a single linear regressor. The one-to-many discriminative mapping is explicitly addressed in [1,2,3,4] by learning multiple mappings in parallel as a mixture of regressors. In order to choose between the different hypotheses that the different regressors deliver, [1,2] use a geometric model that is projected into the image to verify the hypotheses. Inference is performed for each frame independently in [1]. In [2] a temporal model is included using a bank of Kalman filters. In [3,4] gating functions are learned along with the regressors in order to pick the right regressor(s) for a given appearance descriptor. The distribution is propagated analytically in [3], and temporal aspects are included in the learned discriminative mapping, whereas [4] adopts a generative sampling-based tracking algorithm with a first-order autoregressive dynamic model. These discriminative approaches work in a bottom-up fashion, starting with the computation of the image descriptor, which requires the location of the figure in the images to be known beforehand. When including 2d bounding box estimation in the tracking problem, a learned dynamical model might help the bounding box tracking, and avoid loosing the subject when it is temporarily occluded. To this end, [12] learns a subjectspecific dynamic appearance model from a small set of initial frames, consisting of a low-dimensional embedding of the appearances and a motion model. This model is used to predict location and appearance of the figure in future frames, within a CONDENSATION tracking framework. Similarly, low-dimensional embeddings of appearance (silhouette) manifolds are found using LLE in [11], where additionally the mapping from the appearance manifold to 3d pose in body joint space is learned using RBF interpolants, allowing for pose inference from sequences of silhouettes. Instead of modelling manifolds in appearance space, [13,14,15] work with low dimensional embeddings of body poses. In [13], the low-dimensional pose representation, its dynamics, and the mapping back to the original pose space are learned in a unified framework. This approach does not include statistical models of image appearance. In a similar fashion, we also chose to model manifolds in pose space rather than appearance space, because the pose manifold has fewer self-intersections than the appearance manifold, making the dynamics and tracking less ambiguous. In contrast to [13,14,15], our model includes a learned generative likelihood model. When compared to [1,2,3,4,10,11], our approach can simultaneously estimate pose and bounding box, and learning a single regressor is more easily manageable than a mixture of regressors. The paper is structured as follows. Section 2 and 3 introduce our learned models and the inference algorithm, and in Section 4 we show experimental results.
610
T. Jaeggli, E. Koller-Meier, and L. Van Gool
2 Learning Figure 1 a) shows an overview of the tracking framework. Central element is the lowdimensional body pose parametrisation, with learned mappings back to the original pose space and into the appearance space. In this section all elements of the framework will be described in detail. Our models were trained on real motion capture data sets of different subjects, running and walking at different speeds. 2.1 Pose and Motion Prior Representations for the full body pose configuration are high dimensional by nature; our current representation is based on 3d joint locations of 20 body locations such as hips, knees and ankles, but any other representation (e.g. based on relative orientations between neighbouring limbs) can easily be plugged into the framework. To alleviate the difficulties of high dimensionality in both the learning and inference stages, a dimensionality reduction step identifies a low dimensional embedding of the body pose representations. We use Locally Linear Embedding (LLE) [9], which approximately maintains the local neighbourhood relationships of each data point and allows for global deformations (e.g. unrolling) of the dataset/manifold. LLE dimensionality reduction is performed on all poses in the data set and expresses each data point in a space of desired low dimensionality. We currently use a 4-dimensional embedding. However, LLE does
Body Pose
X : Body Pose (high dim.)
learn LLE dim. red.
reconstruct pose, eq. (1)
x : Body Pose (low dim.)
generative mapping
dynamic prior eq. (4)
Y : Image (high dim.)
BPCA reconstruction eq. (5)
(b) eq. (6)
y : Appearance Descriptor: (low dim.)
learn BPCA dim. red.
Appearance
(a)
(c)
Fig. 1. a) An overview of the tracking framework. Solid arrows represent signal flow during inference, the dashed arrow stands for LLE resp. BPCA dimensionality reduction during training. The figure refers to equations in Section 2. b) Body pose representation as a number of 3d joint locations. c) Corresponding synthetically generated silhouette, as used for training the appearance model.
Learning Generative Models for Monocular Body Pose Estimation
611
not provide explicit mappings between the high-dimensional and the low-dimensional space, that would allow to project new data points (that were not contained in the original data set) between the two spaces. Therefore, we model the reconstruction projection from the low-dimensional LLE space to the original pose space with a kernel regressor. X = fp (x) = Wp Φp (x)
(1)
Here, X and x are the body pose representations in original resp. LLE-reduced spaces, Φp is a vector of kernel functions, and Wp is a sparse matrix of weights, which are learned with a Relevance Vector Machine (RVM). We use Gaussian kernel functions, computed at the training data locations. The training examples form a periodic twisted ’ring’ in LLE space, with a curvature that varies with the phase within the periodic movement. A linear dynamical model, as often used in tracking applications, is not suitable to predict future poses on this curved manifold. We view the nonlinear dynamics as a regression problem, and model it using another RVM regressor, yielding the following dynamic prior, pd (xt |xt−1 ) = N (xt ; xt−1 + fd (xt−1 )ΔT , Σd ),
(2)
where fd (xt−1 ) = Wd Φd (xt−1 ) is the nonlinear mapping from poses to local velocities in LLE pose space, ΔT is the time interval between the subsequent discrete timesteps t − 1 and t, and Σd is the variance of the prediction errors of the mapping, computed on a hold-out data set that was not used for the estimation of the mapping itself. Not all body poses that can be expressed using the LLE pose parameterisation do correspond to valid body configurations that can be reached with a human body. The motion model described so far does only include information about the temporal evolution of the pose, but no information about how likely a certain body pose is to occur in general. In other words, it does not yet provide any means to restrict our tracking to feasible body poses. Worse, the learned regressors can produce erroneous outputs when they are applied to unfeasible input poses, since the extrapolation capabilities of kernel regressors to regions without any training data is limited. The additional prior knowledge about feasible body poses is introduced as a static prior that is modelled with a Gaussian Mixture Model (GMM). ps (x) =
C
pc N (x; μc , Σc ),
(3)
c
with C the number of mixture components. We obtain the following formulation for the temporal prior by combination with the dynamic prior pd (xt |xt−1 ). p(xt |xt−1 ) ∝ pd (xt |xt−1 ) ps (xt )
(4)
2.2 Likelihood Model The representation of the subject’s image appearance is based on a rough figure-ground segmentation. Under realistic imaging conditions, it is not possible to get a clean silhouette, therefore the image descriptor has to be robust to noisy segmentations to a certain degree. In order to obtain a compact representation of the appearance of a person, we apply Binary PCA [16] to the binary foreground images. The descriptors are
612
T. Jaeggli, E. Koller-Meier, and L. Van Gool
computed from the content of a bounding box around the centroid of the figure, and 10 to 20 BPCA components are kept to yield good reconstructions. The projection of a new bounding box into the BPCA subspace is done in an iterative fashion, as described in [16]. Since we model appearance in a generative top-down fashion, we also consider the inverse operation that projects the low-dimensional image descriptors y back into high dimensional pixel space and transforms it into binary images or foreground probability maps. By linearly projecting y back to the high-dimensional space using the mean μ and basis vectors V of the Binary PCA, we obtain a continuous representation Yc that is then converted back into a binary image by looking at its signs, or into a foreground probability map via the sigmoid function σ(Yc ). p(Y = f g|y) ∝ σ(V T y + μ)
(5)
Now we will look how the image appearance is linked to the LLE body pose representation x. We model the generative mapping fa from pose x to image descriptors y that allows to predict image appearance given pose hypotheses and fits well into generative inference algorithms such as particle filtering. In addition to the local body pose x, the appearance depends on the global body orientation ω relative to the camera, around the vertical axis. First, we map the pose x, ω into low dimensional appearance space y, fa (x, ω) = Wa Φa (x, ω)
(6)
where the functional mapping fa (x, ω) is approximated by a sparse kernel regressor (RVM) with weight matrix Wa and kernel functions Φa (x). By plugging (6) into (5), we obtain a discrete 2d probability distribution of foreground probabilities Seg(p) over the pixels p in the bounding box. Seg(p) = p(p = f g|fa (x, ω))
(7)
From this pdf, a likelihood measure can then be derived by comparing it to the actually observed segmented image Yobs , also viewed as a discrete pdf Obs(p), using the Bhattacharyya similarity measure [17] which measures the affinity between distributions. Obs(p) =p(p = f g|Yobs ) Seg(p)Obs(p) BC(x, ω, Yobs ) =
(8)
p
We model the likelihood measure as a zero mean Gaussian distribution of the Bhattacharyya distance dBh = −ln(BC(x, ω, Yobs )), and obtain the observation likelihood p(Yobs |x, ω) ∝ exp(−
ln(BC(x, ω, Yobs ))2 ) 2 2σBC
(9)
3 Inference In this section we will show how the 2d image position, body orientation, and body pose of the subject are simultaneously estimated given a video sequence, by using the learned models from the previous section within the framework of particle filtering. The pose estimation as well as the image localisation can benefit from the coupling of pose
Learning Generative Models for Monocular Body Pose Estimation
613
and image location. For example, the known current pose and motion pattern can help to distinguish subjects from each other and track them through occlusions. We therefore believe that tracking should happen jointly in the entire state space Θ, Θt = [ωt , ut , vt , wt , ht , xt ],
(10)
consisting of the orientation ω, the 2d bounding box parameters (position, width and height) u, v, w, h, and the body pose x. Despite the reduced number of pose dimensions, we face an inference problem in 9-dimensional space. Having a good sample proposal mechanism like our dynamical model is crucial for the Bayesian recursive sampling to run efficiently with a moderate number of samples. For the monocular sequences we consider, the posteriors can be highly multimodal. For instance a typical walking sequence, e.g. observed from a side view, has two obvious posterior modes, shifted 180 degrees in phase, corresponding to the left resp. the right leg swinging forward. When taking the orientation of the figure into account, the situation gets even worse, and the modes are no longer well separated in state space, but can be close in both pose and orientation. Our experiments have shown that a strong dynamical model is necessary to avoid confusion between these posterior modes and reduce ambiguities. Some posterior multimodalities do however remain, since they correspond to a small number of different interpretations of the images, which are all valid and feasible motion patterns. The precise inference algorithm is very similar to classical CONDENSATION [18], with normalisation of the weights and resampling at each time step. The prior and likelihood for our inference problem are obtained by extending (4) and (9) to the full state i ) serves as the sample space Θ. In our implementation, the dynamical prior pd (Θti |Θt−1 proposal function. It consists of the learned dynamical prior from eq. (2), and a simple motion model for the remaining state variables θ = [ωt , ut , vt , wt , ht ]. i i pd (Θti |Θt−1 ) = pd (xit |xit−1 )N (θti ; θt−1 , Σθ )
(11)
In practice, one may want to use a standard autoregressive model for propagating θ, omitted here for notational simplicity. The static prior over likely body poses (3) and the likelihood (9) are then used for assigning weights wi to the samples. wti ∝ p(Yti |Θti )ps (Θti ) = p(Yti |xit , ωti )ps (xit )
(12)
Here, i is the sample index, and Yti is the foreground probability map contained in the sampled bounding box (uit , vti , wti , hit ) of the actually observed image. Note that our choice for sample proposal and weighting functions differs from CONDENSATION in that we only use one component (pd ) of the prior (4) as a proposal function, whereas the other component (ps ) is incorporated in the weighting function.
4 Experiments We evaluated our tracking algorithm on a number of different sequences. The main goals were to show its ability to deal with noisy sequences with poor foreground segmentation, very low resolution, and varying viewpoints. Particle filtering was performed using a set of 500 samples, leading to a computation time of approx. 2-3 seconds per image frame in unoptimised Matlab code. The
614
T. Jaeggli, E. Koller-Meier, and L. Van Gool
Fig. 2. Circular walking sequence from [5]. The figure shows full frames (top), and cutouts with bounding box in original or segmented input images and estimated poses. Darker limbs are closer in depth.
Fig. 3. Diagonal walking sequence. Estimated bounding boxes and poses. The intensity of the stick figure limbs encodes depth; lighter limbs are further away.
sample set is initialised in the first frame as follows. Hypotheses for the 2d bounding box locations are either derived from the output of a pedestrian detector that is run on the first image, or from a simple procedure to find connected components in the
Learning Generative Models for Monocular Body Pose Estimation
615
Fig. 4. An extract from a soccer game. The figure shows original and segmented images and with estimated bounding boxes, and estimated 3d poses.
segmented image. Pose hypotheses xi1 are difficult to initialise, even manually, since the LLE parameterisation is not easily interpretable. Therefore, we randomly sample from the entire space of feasible poses in the reduced LLE space to generate the initial hypotheses. Thanks to the low-dimensional representation, this works well, and the sample set converges to a low number of clusters after a few time steps, as desired. The described models were trained on a database of motion sequences from 6 different subjects, walking and running at different speeds. The data was recorded using an optical motion capture system. The resulting sequences of body poses were normalised for limb lengths and used to animate a realistic computer graphics figure in order to create matching silhouettes for all training poses (see Fig. 1c). The figure was rendered from different view points, located every 10 degrees in a circle around the figure. Due to this choice of training data, our system currently assumes that the camera is in an approximately horizontal position. The training set consists of 4000 body poses in total. All the kernel regressors were trained using the Relevance Vector Machine algorithm (RVM) [8], with Gaussian Kernels. Different kernel widths were tested and compared using a crossvalidation set consisting of 50% of the training data, in order to avoid overfitting. 4 LLE dimensions were used, and 15 BPCA components. The first experiment (Fig. 2) shows tracking on a standard test sequence1 from [5], where a person walks in a circle. We segmented the images using background subtraction, yielding noisy foreground probability maps. The main challenge here is the varying viewing angle that is difficult to estimate from the noisy silhouettes. Tracking through another publicly available sequence from the HumanID corpus is shown in Figure 3. The subject walks in an angle of approx. 35 degrees to the camera plane. In addition it is viewed from a slight top-view and shows limb foreshortening due to the perspective projection. These are violations of the assumptions that are inherent in our 1
http://www.nada.kth.se/∼hedvig/data.html
616
T. Jaeggli, E. Koller-Meier, and L. Van Gool
Fig. 5. Traffic scene with low resolution images and noisy segmentation
choice of training data, where we used horizontal views and orthographic projection. Nevertheless the tracker performs well. Figure 4 shows an extract from a real soccer game with a running player. The sequence was obtained from www.youtube.com, therefore the resolution is low and the quality suffers from compression artefacts. We obtained a foreground segmentation by masking the color of the grass. In Figure 5 we show a real traffic scene that was recorded with a webcam of 320 × 240 pixels. The subjects are as small as 40 pixels in height. Noisy foreground segmentation was carried out by subtracting one of the frames at the beginning of the sequence. Our experiments have shown that the dynamical model is crucial for tracking through these sequences with unreliable segmentations and multimodal per-frame likelihoods.
5 Summary and Conclusion We have proposed a learning-based approach to the estimation of 3d body pose and image bounding boxes from monocular video sequences. The relationship between body pose and image appearance is learned in a generative manner. Inference is performed with a particle filter that samples in a low-dimensional body pose representation obtained by LLE. A nonlinear dynamical model is learned from training data as well. Our experiments show that the proposed approach can track walking and running persons through video sequences of low resolution and unfavourable image quality. Future work will include several extensions of the current method. We will explicitly consider multiple activity categories and perform action recognition along with the
Learning Generative Models for Monocular Body Pose Estimation
617
tracking. Also, we will investigate different image descriptors, that do extract the relevant image information more efficiently.
Acknowledgements This work is supported, in parts, by the EU Integrated Project DIRAC (IST-027787), the SNF project PICSEL and the SNF NCCR IM2.
References 1. Rosales, R., Sclaroff, S.: Learning body pose via specialized maps. In: NIPS (2001) 2. Thayananthan, A., Navaratnam, R., Stenger, B., Torr, P., Cipolla, R.: Multivariate relevance vector machines for tracking. In: Ninth European Conference on Computer Vision (2006) 3. Sminchisescu, C., Kanaujia, A., Li, Z., Metaxas, D.: Discriminative density propagation for 3d human motion estimation. In: CVPR (2005) 4. Agarwal, A., Triggs, B.: Monocular human motion capture with a mixture of regressors. In: CVPR. IEEE Workshop on Vision for Human-Computer Interaction, IEEE Computer Society Press, Los Alamitos (2005) 5. Sidenbladh, H., Black, M., Fleet, D.: Stochastic tracking of 3d human figures using 2d image motion. In: Vernon, D. (ed.) ECCV 2000. LNCS, vol. 1843, pp. 702–718. Springer, Heidelberg (2000) 6. Forsyth, D.A., Arikan, O., Ikemoto, L., O’Brien, J.D.R.: Computational studies of human motion: Part 1. Computer Graphics and Vision 1(2/3) (2006) 7. Moeslund, T.B., Hilton, A., Kr¨uger, V.: A survey of advances in vision-based human motion capture and analysis. Comput. Vis. Image Underst. 104(2), 90–126 (2006) 8. Tipping, M.: The relevance vector machine. In: NIPS (2000) 9. Roweis, S., Saul, L.: Nonlinear dimensionality reduction by locally linear embedding. Science 290(5500), 2323–2326 (2000) 10. Agarwal, A., Triggs, B.: A local basis representation for estimating human pose from cluttered images. In: Narayanan, P.J., Nayar, S.K., Shum, H.-Y. (eds.) ACCV 2006. LNCS, vol. 3852, Springer, Heidelberg (2006) 11. Elgammal, A., Lee, C.S.: Inferring 3d body pose from silhouettes using activity manifold learning. In: CVPR (2004) 12. Lim, H., Camps, O.I., Sznaier, M., Morariu, V.I.: Dynamic appearance modeling for human tracking. In: Conference on Computer Vision and Pattern Recognition, pp. 751–757 (2006) 13. Wang, J.M., Fleet, D.J., Hertzmann, A.: Gaussian process dynamical models. Advances in Neural Information Processing Systems 18, 1441–1448 (2006) 14. Sminchisescu, C., Jepson, A.: Generative modeling for continuous non-linearly embedded visual inference. In: ICML. International Conference on Machine Learning (2004) 15. Li, R., Yang, M.H., Sclaroff, S., Tian, T.P.: Monocular tracking of 3d human motion with a coordinated mixture of factor analyzers. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3952, pp. 137–150. Springer, Heidelberg (2006) 16. Zivkovic, Z., Verbeek, J.: Transformation invariant component analysis for binary images. In: CVPR, vol. 1, pp. 254–259 (2006) 17. Bhattacharyya, A.: On a measure of divergence between two statistical populations defined by their probability distributions. Bull. Calcutta Math Soc. (1943) 18. Isard, M., Blake, A.: Condensation - conditional density propagation for visual tracking. Int. J. Computer Vision (1998)
Human Pose Estimation from Volume Data and Topological Graph Database Hidenori Tanaka1 , Atsushi Nakazawa2, and Haruo Takemura2 1
Graduate School of Information Science and Technology, Osaka University 1-32 Machikaneyama, Toyonaka-shi, Osaka, 560-0043 Japan [email protected] 2 Cybermedia Center, Osaka University 1-32 Machikaneyama, Toyonaka-shi, Osaka, 560-0043 Japan {nakazawa,takemura}@cmc.osaka-u.ac.jp
Abstract. This paper proposes a novel volume-based motion capture method using a bottom-up analysis of volume data and an example topology database of the human body. By using a two-step graph matching algorithm with many example topological graphs corresponding to postures that a human body can take, the proposed method does not require any initial parameters or iterative convergence processes, and it can solve the changing topology problem of the human body. First, three-dimensional curved lines (skeleton) are extracted from the captured volume data using the thinning process. The skeleton is then converted into an attributed graph. By using a graph matching algorithm with a large amount of example data, we can identify the body parts from each curved line in the skeleton. The proposed method is evaluated using several video sequences of a single person and multiple people, and we can confirm the validity of our approach.
1
Introduction
Motion capture is widely used in the research fields of computer graphics, humancomputer interaction, medical applications, and robotics. However, in conventional motion capture systems, the actors must attach markers or special devices to their bodies. To solve this problem, markerless motion capture methods have been thoroughly studied by computer vision researchers [1,2]. Many studies have introduced articulated human body models and solve the problem through error minimization frameworks. These studies use different types of three-dimensional primitives to express body parts, for example cylinders and ellipsoids [3], colored blobs [4], and so on. And Kehl et al. [5] have introduced a dense surface model, where joint parameter estimation is carried out through error minimization between model features and input image features, such as silhouettes, optical flows, and contours. These top-down approaches pose some difficulties with regard to the computation time for the convergence process, initial parameter estimation, and recovery from tracking error. In reality, the initial body parameters are provided manually Y. Yagi et al. (Eds.): ACCV 2007, Part I, LNCS 4843, pp. 618–627, 2007. c Springer-Verlag Berlin Heidelberg 2007
Human Pose Estimation from Volume Data
619
in all methods, or users are required to form specific poses at the start frame. Moreover, the estimation is performed by the tracking framework; this implies that the estimation at one frame depends on the results of previous frames. Therefore, the tracking cannot be continued once an error has occurred. To avoid these problems, some methods employ a bottom-up approach. Cheung et al. [6] proposed an algorithm to segment volume data into articulated rigid parts and acquire human kinematics. Other methods directly analyze the input volume data and obtain the medial axes, termed “skeleton”, of the 3D shape. Then, joint positions are estimated from the skeletons. The analysis of each frame is performed by a bottom-up process and is independent of the other frames; this can avoid the problems that occur in the top-down approach. To segment volume data and obtain the skeleton, Chu et al. [7] have used Isomaps. Another approach used by Sundaresan et al. [8] uses the Laplacian eigenspace and fitting of spline curves. During human movement, the topology of the body can change significantly, for example when both hands touch each other or touch the body. This changing topology should be taken into consideration to ensure robust estimation. However, previous studies have only considered a limited number (one or two) of topologies. Our method can be categorized as one of the latter bottom-up methods. Skeletons are extracted from captured volume data and the joint positions are estimated frame-by-frame. In order to consider many topologies in the same manner, example skeletons and a matching framework are used. These steps are explained in the following sections.
2
Proposed Method
Figure 1 shows an overview of our method. First, we capture the time series of the volume data using a visual hull based method (2.1). Then, we apply the volume thinning process and obtain 3D curved lines (2.2),. Next, we identify the body parts formed by the curves using the model graph database (2.3). Finally, the joint positions are estimated using the body part information along with time-series curvature analysis of the skeletons (2.4). In contrast to previous methods, our method has the following advantages: – By adopting a bottom-up approach, we can avoid the difficulty of parameter initialization and having to recover from tracking failure. – The computational cost of our process is less than that of model-based approaches or subspace projections because we do not use any iterative convergence algorithms. – The changing topology problem of the human body is solved by developing many possible examples of the human body with a graph matching and decision tree framework. This approach is very general and there are few heuristic rules. We can easily expand this method for identifying many varieties of topologies, noisy data, multiple person tracking, and other articulated objects.
620
H. Tanaka, A. Nakazawa, and H. Takemura
Volume Reconstruction
Estimation of Joint Positions
Capture Video by 8 cams
3D Volume Thinning
Silhouette-based Visual Hull
Skeleton Conversion
Find Local Maxima from Summed up Curvature
Estimate Joint Positions for All Frames Graph Matching
Model Graph Database Learn
Body Part Identification
Example Graphs
Fig. 1. Algorithm Overview
Fig. 2. Captured volume data and their skeletons
2.1
Volume Reconstruction
We use a voxel-based visual hull algorithm for capturing time-series volume data of the human body[9]. The actor’s image regions are detected by using background subtraction. In this step, the CIELab color space is used to remove the shadows on the floor. Here, the weight used for the L value is smaller than that used for a and b. Finally, a visual hull algorithm is applied to reconstruct the target volume data (figure 2). 2.2
Three Dimensional Volume Thinning
We apply a 3D volume thinning process to the captured data and obtain a skeleton of the human body. Though there are many algorithms to extract skeleton from volume data [10,11], we use Saitoh’s algorithm [10] because it is fast and simple. First, we calculate the depth value (minimum distance between a voxel and the original surface) for each voxel. Then, the surface voxels are sequentially removed beginning from smaller depth voxels. This process is continued until the line width becomes one voxel while preserving the original topology. The resulting skeleton consists of a set of 3D curved lines whose topology is the same as that of the original volume data, passing through the middle of the original shape (figure 2). In this process, some unnecessary short lines are produced. To solve this problem, we first apply a thresholding technique for their removal. In order to remove longer noise lines, we add example graphs into the model graph database as described below.
Human Pose Estimation from Volume Data
a
b
c
d
e
f
621
g
Fig. 3. Variations in the topology of the human body. (a,b): Two arms are connected at the same position on the body, (c,d): Loop structure, (e,f,g): Some body parts merge into each other.
2.3
Identification of Body Parts Using Model Graph Database
In this step, we identify the body parts formed by curves in the skeleton. Here, we must consider the variation in the topologies of the human body. In this section, we first describe the topology of the human body and then describe a method to identify the topology and body parts based on the input skeleton. The topologies of the skeletons may differ from the normal topology of the human (figure 3-a) in the following ways: 1. arms and feet may not be connected at the same position (figure 3-b), 2. multiple body parts contact with each other (figure 3- c,d), 3. multiple body parts merge into each other (figure 3-e,f). In these cases, we apply a split-and-merge algorithm for some of the curves to determine the joint positions. In addition, multiple skeletons, while having the same topology, may differ in the way body parts touch the body (touch pattern) such as figure 3-b and g. So, we must consider not only the topologies, but also other features of each curved line for proper identification. To solve this problem, we introduce an example based approach (figure 4). We develop several example skeletons with different topologies in advance. They are converted into attributed graphs, and body part IDs are manually assigned to their nodes which are then stored in the model graph database (MGDB). The input skeleton is also converted into an attributed graph and then matched with the example graphs. Based on the node-to-node correspondence, we can identify body parts from the input graph. Skeletons are represented as attributed graphs; the line segments become nodes and the intersection points become links (figure 5). As attribute values for the node, we use the length of the curve, volume of the original 3D volume data, and variance of the depth value among the point in the curve. The length attribute value is the length of the line segment normalized over the whole skeleton. The volume is obtained by summing the squares of the depth values (described in Section 2.2) of line voxels. Normalization is also applied to the volume, resulting in the final volume attribute value. Example graphs are obtained from test sequences and stored into the MGDB. IDs and attribute values are assigned to the nodes in the graph. These graph representations of skeletons are invariant with human posture (joint angles). Thus, we only need to develop a limited number of examples, despite the high number of possible human postures.
622
H. Tanaka, A. Nakazawa, and H. Takemura
Input Skeleton
Graph Conversion
Example Graphs
Attributed Graph Model Graph Database Example Graphs
Topology Matching
HEAD
...
ARM_A
ARM_A
BODY
Matching Results Decision Tree Database
LEG_B
ARM_B
LEG_A
LEG_B
HEAD
LEG_A
Example Graphs Example Graphs Topology 2 Topology 1
BODY
Feature Vectors
Feature Vectors
Learning
Learning
...
Learning
Decision Tree Filter
Decision Tree Filter
...
Decision Tree Filter
ARM_B
Decision Tree filter HEAD ARM_A
Node-to-node Correspondence LEG_A
ARM_B
Graphs ... Example Topology n
...
Feature Vectors
BODY
LEG_B
Fig. 4. Left: our body part identification algorithm that uses the MGDB. Right: learning of the decision tree filter.
Two step graph matching algorithm. Graph matching consists of two steps and is performed on the input graph with the example graphs in the MGDB (figure 4-left). First, the topology of the input graph is identified using Messmer’s method [12], which is a combination of graph subdivision and subgraph matching. This algorithm increases the speed of matching by using a network that contains hierarchical relations between subgraphs of the model graphs. However, this matching may produce multiple results and node-to-node correspondences because it compares only the graph topologies. Therefore, we employ decision tree based filters to narrow these down to one correct matching. The Decision Tree. To obtain the correct matching result and node-to-node correspondence, C4.5 decision trees [13] are made for each topology and touch pattern (figure 4-right). During the learning stage of the MGDB, we manually label the nodes in the example graphs and acquire feature vectors using the length and volume of each node. These feature vectors are collected from example graphs with the same touch pattern and used as positive samples (figure 6). We also prepare feature vectors for error patterns, such as the head and a hand being swapped and different touch patterns, and used them as negative samples. Then, C4.5 decision trees are constructed using these samples. During the matching step, several feature vectors are generated from one input graph according to the topology-matching results. After that, they are evaluated by the decision trees of the touch patterns and we can identify the body parts of the input nodes.
Human Pose Estimation from Volume Data
HEAD
HEAD
0
ARM_A
0
3
BODY_U
ARM_B
1 4
ARM_B LEG_A
2 5
6
LEG_B
0
4
8,17,53
ARM_B HEAD
0
3
ARM_B
BODY_L LEG_B
2
6
5
3
2
6
ARM_A
24,24,196
BODY_L LEG_A 6
⎛ HEAD ⎞ ⎜ ⎟ ⎜ BODY_U ⎟ ⎜ BODY_L ⎟ ⎜ ⎟ ⎜ ARM_A ⎟ ⎜ ARM_B ⎟ ⎜ ⎟ ⎜ LEG_B ⎟ ⎜ ⎟ ⎝ LEG_A ⎠
0
BODY_U 4
ARM_B
3
2
BODY_L
5
LEG_B
HEAD
1
LEG_A 6
⎛ ARM_A ⎞ ⎜ ⎟ ⎜ BODY_U ⎟ ⎜ BODY_L ⎟ ⎜ ⎟ ⎜ HEAD ⎟ ⎜ ARM_B ⎟ ⎜ ⎟ ⎜ LEG_B ⎟ ⎜ ⎟ ⎝ LEG_A ⎠
Positive Sample LEG_A 0
BODY_L
BODY_U
2
5
LEG_B
⎛ ARM_A ⎞ ⎜ ⎟ ⎜ BODY_U ⎟ ⎜ BODY_L ⎟ ⎜ ⎟ ⎜ HEAD ⎟ ⎜ ARM_B ⎟ ⎜ ⎟ ⎜ LEG_A ⎟ ⎜ ⎟ ⎝ LEG_B ⎠
LEG_B
1
1
ARM_B
ARM_A
ARM_A 3
BODY_U 4
5
LEG_A
⎛ LEG_A ⎞ ⎟ ⎜ ⎜ BODY_L ⎟ ⎜ BODY_U ⎟ ⎟ ⎜ ⎜ LEG_B ⎟ ⎜ ARM_B ⎟ ⎟ ⎜ ⎜ HEAD ⎟ ⎟ ⎜ ARM_A ⎠ ⎝
25,9,9
Skeleton
6
LEG_A BODY_L
26,26,171
BODY_L LEG_B
HEAD
HEAD
1
Positive Sample
Attribute Swapping
8,6,63
9,18,130
4
⎛ HEAD ⎞ ⎜ ⎟ ⎜ BODY_U ⎟ ⎜ BODY_L ⎟ ⎜ ⎟ ⎜ ARM_A ⎟ ⎜ ARM_B ⎟ ⎜ ⎟ ⎜ LEG_A ⎟ ⎜ ⎟ ⎝ LEG_B ⎠
Sample Graph
volume 8 length 6 variance 63
5
LEG_A
BODY_L
2
0
BODY_U
1
4
ARM_A
ARM_A 3
BODY_U
623
4
ARM_B ARM_A
LEG_B 0
LEG_B
BODY_L
3 1
4
BODY_U
2
5
6
⎛ LEG_A ⎞ ⎟ ⎜ ⎜ BODY_L ⎟ ⎜ BODY_U ⎟ ⎟ ⎜ ⎜ LEG_B ⎟ ⎜ ARM_B ⎟ ⎟ ⎜ ⎜ ARM_A ⎟ ⎟ ⎜ HEAD ⎠ ⎝
Attributed Graph
3
ARM_B
HEAD
HEAD
5
2
LEG_B
LEG_A
BODY_U 6
0
BODY_L
1
ARM_A
⎛ LEG_B ⎞ ⎟ ⎜ ⎜ BODY_L ⎟ ⎜ BODY_U ⎟ ⎟ ⎜ ⎜ LEG_A ⎟ ⎜ ARM_B ⎟ ⎟ ⎜ ⎜ HEAD ⎟ ⎟ ⎜ ARM_A ⎠ ⎝
4
ARM_B ARM_A
5
LEG_A 3
1 2
BODY_U 6
HEAD
⎛ LEG_B ⎞ ⎟ ⎜ ⎜ BODY_L ⎟ ⎜ BODY_U ⎟ ⎟ ⎜ ⎜ LEG_A ⎟ ⎜ ARM_B ⎟ ⎟ ⎜ ⎜ ARM_A ⎟ ⎟ ⎜ HEAD ⎠ ⎝
Feature Vectors
Fig. 5. Graph representation of Fig. 6. Making feature vectors from an example skeleton and attribute values graph. Positive samples are enclosed within blue frames.
ARM_A
ARM_A BODY
ARM_A ARM_B
ARM_B BODY
BODY
BODY
ARM_B
ARM_A
ARM_B
BODY
BODY
Fig. 7. Split and merge algorithm in joint estimation. Left: If there are several line segments identified as the body, they are merged together. Right: The loop structure is also removed by the same rule.
2.4
Estimation of Joint Positions
In this step, we estimate the joint positions from the curved lines that are used to identify the body parts. First, we introduce the split and merge algorithm using body part IDs. If a body part is split by connecting points (figure 7-left), they are removed and the line segments of that body part are merged together. If there is a loop structure (figure 7-right), the same rule is applied and we can obtain a one-to-one relation between the body parts and the curved lines. Next, we perform a time-series curvature analysis and determine the joint positions in each curved line. Our algorithm consists of the following steps: 1. The lengths of the curved lines are normalized, and their curvature at each frame is calculated. 2. The curvatures of each body portion are summed up for all frames. 3. The local maxima in the summed curvature graph are determined and considered as joint positions relative to the body part’s length. Here, we use the same number of maxima as the number of joints given by the body part attribute value.
624
H. Tanaka, A. Nakazawa, and H. Takemura
Fig. 8. Result of body part identification with subject 1. Up: Input image. Bottom: Skeleton color coded by body part.
4. The absolute joint positions are determined for all the frames according to the relative joint positions in the curved line. Through these steps, all joints will be determined if they are “bent” at least once within all the video frames. By using the split and merge algorithm and the timeseries curvature analysis based on the results of body part identification, we can obtain consistent kinematics of the human body in every frame.
3
Experimental Results
We tested our algorithm on 3 subjects (subjects 1, 2, and 3) individually and two subjects (subjects 1 and 4) together. We set up a studio that contained 8 cameras and blue screens. Each camera was connected to a PC and could synchronously capture XGA images at 30 fps. Volume reconstruction was performed at a resolution of 2 cm, and the following processes were done on a single PC. We prepared the MGDB with 190 example graphs (11 topologies) selected from the reconstruction results of several videos and 196 test graphs from these same videos. The body part IDs were manually assigned to the nodes. The results of body part identification with subject 1 are shown in figure 8. Captured images are shown in the top row and the colors of curves in the bottom row show identification of body parts. The results confirm that our method correctly identifys body parts even for loops and mergers. We can also confirm this method works well even when unnecessary curves exist thanks to the prepared example graphs. Figure 9 show the results of subjects 2 and 3 along with that of subjects 1 and 4 together in the scene. The experimental result also shows our method correctly identifys the body parts of two people even when their body parts touch each other. 170 graphs out of 196 test graphs were assigned topology and body part IDs matching the manual assignments. The others were assigned incorrect touch patterns. Figure 10 is a curvature graph of four body parts summed up over 291 frames from one video. Local maxima in the curved lines of arms become wrists (relative
Human Pose Estimation from Volume Data
625
Fig. 9. Result of body part identification with subject 2, 3 and with 2 subjects. Up: Input image. Bottom: Color coded skeleton.
㪘㪩㪤㪶㪘 㪘㪩㪤㪶㪙 㪣㪜㪞㪶㪘 㪣㪜㪞㪶㪙
㪪㫌㫄㩷㫆㪽㩷㫋㪿㪼㩷㪺㫌㫉㫍㪸㫋㫌㫉㪼
㪇㪅㪇㪇㪎 㪇㪅㪇㪇㪍 㪇㪅㪇㪇㪌 㪇㪅㪇㪇㪋 㪇㪅㪇㪇㪊 㪇㪅㪇㪇㪉 㪇㪅㪇㪇㪈 㪇 㪈
㪊㪋
㪍㪎
㪈㪇㪇
㪧㫆㫊㫀㫋㫀㫆㫅㩷㫀㫅㩷㫋㪿㪼㩷㪺㫌㫉㫍㪼㪻㩷㫃㫀㫅㪼㩷㩿㩼㪀
Fig. 10. Sum of the curvature of 291 skeletons
position from 9% to 11%), elbows (40% to 43%), and shoulders (84% to 85%), and those found in one of the legs become ankles (7% to 8%), knees (51% to 52%), and thighs (90% to 94%). The estimated joint positions are shown in figure 11. With regards to processing time, it took 5.75 seconds for volume reconstruction, 480 ms for thinning, 94 ms for graph conversion of the skeleton, and 4.2ms for graph matching and decision tree filtering. Joint estimation needs 40 ms per frame. 3.1
Discussion
The experimental results show that our method can successfully identify body parts even when the topology of the body shape changes in each consecutive frame. Since the algorithm performs the estimation independently for each frame, the estimation in one frame is not affected by errors in previous frames. The system can identify some unnecessary curved lines in the skeletons as noise and ignore them during body part identification. Analyzing the failure cases, we found that sometimes the upper arm is partially merged to the body. Our touch pattern classes did not contain such a
626
H. Tanaka, A. Nakazawa, and H. Takemura
Fig. 11. Result of the pose estimation
pattern; our sample patterns had the upper arm as either completely separate or completely merged with the body. Our example graph-based approach is very easy to expand. As shown in the experiment, we can easily apply our method to multiple people simply by developing new example graphs that express the topologies of multiple human bodies. The time-series curvature analysis effectively obtains consistent kinematics of the human body. Hence, we can obtain joint position data in a manner similar to that of conventional motion capture systems. Our method requires a total processing time of 6.4 s per frame. However, 90% of the total processing time takes place during reconstruction, while the volume thinning process and joint position estimation only require less than 0.7s. Thus, we are attempting to develop faster volume reconstruction and thinning methods to reach real-time estimation. In general, our example based approach is very useful because of its generality and fast processing.
4
Conclusion and Future Study
In this paper, we have proposed a novel markerless motion capture method using volume data. Our method employs the bottom up analysis of the volume data; therefore, we do not require any initial parameters of the human body, and fast estimation can be achieved by using a 3D volume thinning process. Additionally, we have solved the changing topology problem of the body part identification process using an example-based approach. The experimental results indicate that our method can successfully identify different topologies that consist of a single person as well as many persons and can correctly estimate joint positions using time-series curvature analysis. Thus far, it has been difficult to distinguish between left and right body parts because our graph representations do not contain geometric information. To solve this problem, we are currently attempting to incorporate some geometrical attributes into our graph representations.
Human Pose Estimation from Volume Data
627
Acknowledgement This work is supported by the Ministry of Education, Culture, Sports, Science and Technology under the “Development of fundamental software technologies for digital archives” project.
References 1. Gavrila, D.M.: The visual analysis of human movement: A survey. CVIU 73(1), 82–98 (1999) 2. Moeslund, T.B., Granum, E.: A survey of computer vision-based human motion capture. CVIU 81(3), 231–268 (2001) 3. Miki´c, I., Trivedi, M., Hunter, E., Cosman, P.: Human body model acquisition and tracking using voxel data. International Journal of Computer Vision 53(3), 199–223 (2003) 4. Caillette, F., Howard, T.: Real-Time Markerless Human Body Tracking with MultiView 3-D Voxel Reconstruction. In: Proc. of British Machine Vision Conference, vol. 2, pp. 597–606 (2004) 5. Kehl, M.B.R., Gool, L.V.: Full body tracking from multiple views using stochastic sampling. In: Proc. of CVPR 2005, vol. 2, pp. 129–136 (2005) 6. Cheung, G., Baker, S., Kanade, T.: Shape-from-silhouette of articulated objects and its use for human body kinematics estimation and motion capture. In: Proc. of CVPR 2003, vol. 1, pp. 77–84 (2003) 7. Chu, C.W., Jenkins, O.C., Matari´c, M.J.: Markerless kinematic model and motion capture from volume sequences. Proc. of CVPR 2003 2, 475–482 (2003) 8. Sundaresan, A., Chellappa, R.: Segmentation and probabilistic registration of articulated body models. In: Proc. of ICPR 2006, vol. 2, pp. 92–96 (2006) 9. Laurentini, A.: The visual hull concept for silhouette-based image understanding. IEEE Trans. on PAMI 16(2), 150–162 (1994) 10. Saitoh, T., Mori, K., Toriwaki, J.: A sequential thinning algorithm for three dimensional digital pictures using the euclidean distance transformation and its properties. The transactions of the Institute of Electronics, Information and Communication Engineers. D-II J79-D-II (10), 1675–1685 (1996) 11. Brostow, G.J., Essa, I., Steedly, D., Kwatra, V.: Novel skeletal representation for articulated creatures. In: Pajdla, T., Matas, J. (eds.) ECCV 2004. LNCS, vol. 3023, pp. 66–78. Springer, Heidelberg (2004) 12. Messmer, B.T., Bunke, H.: A new algorithm for error-tolerant subgraph isomorphism detection. IEEE Trans. on PAMI 20, 493–504 (1998) 13. Quinlan, R.J.: Programs for Machine Learning. Morgan kaufmann Publishers, San Francisco (1993)
Logical DP Matching for Detecting Similar Subsequence Seiichi Uchida1 , Akihiro Mori2 , Ryo Kurazume1 , Rin-ichiro Taniguchi1 , and Tsutomu Hasegawa1 1
2
Kyushu University, Fukuoka, 819-0395, Japan [email protected] http://human.is.kyushu-u.ac.jp/ Toshiba Corporation, Tokyo, 198-8710, Japan
Abstract. A logical dynamic programming (DP) matching algorithm is proposed for extracting similar subpatterns from two sequential patterns. In the proposed algorithm, local similarity between two patterns is measured by a logical function, called support. The DP matching with the support can extract all similar subpatterns simultaneously while compensating nonlinear fluctuation. The performance of the proposed algorithm was evaluated qualitatively and quantitatively via an experiment of extracting motion primitives, i.e., common subpatterns in gesture patterns of different classes.
1
Introduction
Detection of similar subsequences between two sequential patterns is one of fundamental problems for sequential pattern processing. Especially, this problem is very important when extracting frequent subsequences. For example, frequent subsequences of gesture patterns, called motion primitives, are widely used for analyzing human activities. In addition, frequent subsequences of biological sequences (such as DNA sequence and protein sequence), called motives, also plays quite important role in genome science [1,2]. In this paper, a new algorithm, called logical dynamic programming (DP) matching, is proposed for detecting similar subsequences. The proposed algorithm employs a logical function s(i, j), called support, to evaluate similarity between the ith frame of A = a1 , . . . , ai , . . . , aI and the jth frame of B = b1 , . . . , bj , . . . , bJ . Specifically, if a pair of frames ai and bj are similar, s(i, j) = true. Otherwise, s(i, j) = false. Similar subsequences between A and B can be determined as sets of consecutive frame pairs having true supports. The use of the support allows simultaneous detection of all similar subsequences while compensating nonlinear fluctuations in the subsequences. The remaining part of this paper is organized as follows. After reviewing related work in in Section 2, the logical DP matching algorithm is described in Section 3. In addition, the logical DP matching algorithm is applied to actual gesture patterns for evaluating its performance on the detection of similar subsequences among them. In Section 4, the performance of the proposed algorithm Y. Yagi et al. (Eds.): ACCV 2007, Part I, LNCS 4843, pp. 628–637, 2007. c Springer-Verlag Berlin Heidelberg 2007
Logical DP Matching for Detecting Similar Subsequence
629
is further evaluated quantitatively via an experiment of extracting motion primitives from gesture patterns.
2
Related Work
The algorithms for detecting similar subsequences are classified into two groups. The first group is based on “rigid comparison” between two sequential patterns. In the algorithms of this group, sequential patterns are firstly described by rough representations, such as sequences of a limited number of symbols, by clustering or other quantization technique, and then identical subsequences are detected. The rough representation is necessary for compensating fluctuations between two sequences. Tanaka et al. [3] transformed a gesture sequence into a symbol sequence for extracting motion primitives under an MDL criterion. A similar algorithm can be found in [4]. A problem of this group is that the rough representation will lose exact boundaries of similar subsequences. In fact, a refinement step is introduced in [4] for finding exact boundaries. The second group is based on “elastic comparison”, where two sequential patterns are compared under an optimized nonlinear matching to compensate fluctuations. DP matching algorithm, or dynamic time warping (DTW) algorithm, will be the most popular algorithm in this group. DP matching possesses several useful properties, such as (i) the global optimality of its matching result, (ii) computational feasibility, (iii) high versatility, etc., and thus has been used various sequential pattern processing tasks, such as speech recognition [5] and character recognition [6] and gnome science [1,2]. The difference between the proposed and the conventional DP matching algorithms is the definition of frame (dis)similarity. Specifically, the proposed algorithm employs the logical support function s(i, j), whereas the conventional algorithm has been employed Euclidean distance between feature vectors of the ith and jth frames. This modification brings the following novel abilities to the DP matching: – The proposed algorithm can detect strictly similar subsequences. This means that all the paired frames in the similar subsequences are similar. In contrast, similar subsequences by the conventional DP matching algorithms are averagely similar; that is, several paired frames can be dissimilar if other paired frames are very similar. (This is because of the conventional algorithm is based on accumulated Euclidean distance.) – The proposed algorithm can detect all similar subsequences simultaneously.
3
Detecting Similar Subsequence by Logical DP Matching
This section describes the logical DP matching algorithm and its application to the simultaneous detection of similar subsequences. The performance of the algorithm is also discussed via a detection experiment.
630
S. Uchida et al. detected similar sub-sequence s(i,j) = 1
pattern A
θ1
i pattern B j
feature space
Fig. 1. Similar subsequence detection using support s(i, j)
3.1
Logical DP Matching
Support and Supported Path. Each sequential pattern can be represented as a trajectory in a feature space, as shown in Fig. 1. Similar subsequences between two sequential patterns will be observed as regions where two trajectories are close to each other. This closeness is evaluated by the following logical function, called support: true if d(i, j) < θ1 (1) s(i, j) = false otherwise, where θ1 is a positive constant and d(i, j) is a distance between the ith and the jth frames. In the followings, we will use Euclidean distance d(i, j) = ai − bj , unless otherwise noted. If two trajectories are close at the ith and the jth frames (namely, if ai and bj are similar), the support s(i, j) = true. Otherwise, s(i, j) = false. The dashed lines in Fig. 1 indicates frame pairs where two trajectories are close. Consider an i-j plane of Fig. 2 and assume that the support s(i, j) has been already calculated at each (i, j) node according to (1). Since similar subsequences between A and B can be determined as sets of consecutive frame pairs having true supports, the problem of finding similar subsequences is treated as the problem of finding paths connecting nodes with true supports. Hereafter, this path is called a supported path. A supported path from (is , js ) to (ie , je ) indicates that subsequences ais , . . . , aie and bjs , . . . , bje are similar throughout, that is, strictly similar. DP Recursion. All the supported paths can be found efficiently and simultaneously by a DP-based algorithm. In the algorithm, the following equation, so-called DP recursion, is calculated at each j from i = 1 to I: 3 gk (2) g(i, j) = s(i, j) ∧ k=1
where
⎧ ⎨ g1 = g(i − 1, j − 1) g2 = s(i, j − 1) ∧ g(i − 1, j − 2) ⎩ g3 = s(i − 1, j) ∧ g(i − 2, j − 1)
pattern B
Logical DP Matching for Detecting Similar Subsequence J
j
1
0
0
0
1
0
0
0
0
0
0
0
1
1
0
0
0
1
1
0
0
1
0
0
0
1
0
0
0
0
0
0
1
0
0
1
0
0
0
1
1
0
0
0
1
1
0
1
0
0
0
0
1
1
0
0
0
0
0
0
1
0
0
0
1
i
631
I pattern A
a pair of similar sub-patterns
Fig. 2. Representation of similar subsequences by supported paths. Here, “1” stands for true and “0” for false.
...
...
...
...
...
...
...
...
pattern B ... ... ...
... ...
...
J
...
ending node
... (d)...
starting node
j
g(i,j) (c)
1 1
...
(a)
...
(b)
... ... ... ... I i pattern A
Fig. 3. DP recursion and staring/ending node
and ∨ and ∧ denote logical OR and AND operations, respectively. The function g(i, j) is a logical function taking true or false and indicates whether there is at least one supported path arriving at the node (i, j) or not. The nodes (a), (b), and (c) of Fig. 3 correspond to the nodes with g1 , g2 , and g3 , respectively. The above recursion represents the trial to extend the supported paths to the nodes (a), (b), and (c) by connecting the node (i, j). The starting and the ending nodes of each supported path are also detected while calculating of the DP recursion (2). The node (i, j) is marked as a starting node iff g1 = g2 = g3 = false and s(i, j) = true. Similarly, the node (i, j) is marked as an ending node iff g(i, j) = true and s(i + 1, j + 1) = false1 . Backtracking from every ending node provides all the supported paths simultaneously. The supported paths shorter than θ2 are eliminated as noise. As shown in Fig. 2, supported paths often branch and share the same starting/ending node. In the following experiment, those branched supported paths are to be shrunk into a non-branched one by removing less reliable paths. The reliability of each branch is evaluated by the accumulated distance of d(i, j) along the branch. A branch with a higher accumulated distance is considered as a less reliable branch. 1
Any supported path passing (i, j) will also pass (i + 1, j + 1). Thus, if s(i + 1, j + 1) = false, no path can be extended from (i, j) and therefore (i, j) is an ending node.
hurrah
bye
stop
raise hand point
S. Uchida et al.
circle shrug
632
time
time Fig. 4. Snapshot of assumed gestures
After this removal, only one supported path will start from each starting node and end in each ending node. The computational complexity of the above algorithm is O(IJ). This is the same complexity as the conventional DP matching algorithms. 3.2
Experiment of Detecting Similar Subpatterns Between Gesture Patterns
Experimental Setup. The performance of the proposed algorithm has been evaluated via an experiment of detecting similar subsequences of gesture patterns. Gestures of 18 classes were used in the experiment. Fig. 4 shows snapshots of 7 gestures of them. Each gesture was performed 6 times by one person and thus 108 gesture patterns were prepared. The average frame length was 89. A six-dimensional feature vector was used for representing the performer’s posture at each frame. Specifically, the six elements of the vector were the threedimensional positions of both hands (relative to the head position), which were obtained by stereo measurement with two IEEE1394 cameras. The two parameters θ1 and θ2 were fixed at 200 and 8, respectively. The condition θ1 = 200 implies that the support s(i, j) becomes true if the difference of two postures, i.e., ai − bj , is less than 20cm. The condition θ2 = 8 implies that similar subsequences whose length is less than 500ms are eliminated as noise. Detection Result of Similar Subsequences. Fig. 5 shows five detection results. The images at the leftmost column show d(i, j) on the i–j plane and the images at the middle column show the support s(i, j). The images at the rightmost column show the supported paths detected by the proposed algorithm. Fig. 5 (a) is the result when an identical “(left-hand) bye” pattern was used as both A and B. Since A and B were identical, a long diagonal straight supported path was obtained. The other, more meaningful, two supported paths were obtained because a left-to-right hand motion is repeated twice in “bye” and the first motion and the second motion were correctly detected as similar subsequences. Fig. 5 (b) is the result when two different “bye” patterns were used for A and B. The repeated hand motions were correctly detected like (a).
Logical DP Matching for Detecting Similar Subsequence distance map
90 80 70 60 50 40 30 20 10 0
"left-hand bye" no.1
"left-hand bye" no.1
0
10 20 30 40 50 60 70 80 90
"left-hand bye" no.1 100
"left-hand bye" no.2
(b)
detected similar sub-sequences
support map
"left-hand bye" no.1
(a)
633
80 60 40 20 0
"left-hand bye" no.1
"left-hand bye" no.1
0
10 20 30 40 50 60 70 80 90
(c)
"raise right-hand"
"left-hand bye" no.1
"right-hand bye"
0
"hurrah"
10 20 30 40 50 60 70 80 90
"right-hand bye" 90 80 70 60 50 40 30 20 10 0
"circle"
(d)
"right-hand bye"
45 40 35 30 25 20 15 10 5 0
"hurrah"
0
20
40
60
80
100
80
100
"hurrah" 70 60 50 40
"stop"
(e)
30 20 10 0