Lecture Notes in Computer Science Commenced Publication in 1973 Founding and Former Series Editors: Gerhard Goos, Juris Hartmanis, and Jan van Leeuwen
Editorial Board David Hutchison Lancaster University, UK Takeo Kanade Carnegie Mellon University, Pittsburgh, PA, USA Josef Kittler University of Surrey, Guildford, UK Jon M. Kleinberg Cornell University, Ithaca, NY, USA Alfred Kobsa University of California, Irvine, CA, USA Friedemann Mattern ETH Zurich, Switzerland John C. Mitchell Stanford University, CA, USA Moni Naor Weizmann Institute of Science, Rehovot, Israel Oscar Nierstrasz University of Bern, Switzerland C. Pandu Rangan Indian Institute of Technology, Madras, India Bernhard Steffen TU Dortmund University, Germany Madhu Sudan Microsoft Research, Cambridge, MA, USA Demetri Terzopoulos University of California, Los Angeles, CA, USA Doug Tygar University of California, Berkeley, CA, USA Gerhard Weikum Max Planck Institute for Informatics, Saarbruecken, Germany
6915
Jacques Blanc-Talon Richard Kleihorst Wilfried Philips Dan Popescu Paul Scheunders (Eds.)
Advanced Concepts for Intelligent Vision Systems 13th International Conference, ACIVS 2011 Ghent, Belgium, August 22-25, 2011 Proceedings
13
Volume Editors Jacques Blanc-Talon DGA, 7-9 rue des mathurins, 92 221 Bagneux, France E-mail:
[email protected] Richard Kleihorst VITO-TAP, Boeretang 200, 2400 Mol, Belgium E-mail:
[email protected] Wilfried Philips Ghent University St.-Pietersnieuwstraat 41, 9000 Ghent, Belgium E-mail:
[email protected] Dan Popescu CSIRO ICT Centre P.O. Box 76 Epping, NSW 1710, 1710 Sydney, Australia E-mail:
[email protected] Paul Scheunders University of Antwerp, Universiteitsplein 1 2610 Wilrijk, Belgium E-mail:
[email protected] ISSN 0302-9743 e-ISSN 1611-3349 ISBN 978-3-642-23686-0 e-ISBN 978-3-642-23687-7 DOI 10.1007/978-3-642-23687-7 Springer Heidelberg Dordrecht London New York Library of Congress Control Number: 2011935066 CR Subject Classification (1998): I.4, I.5, C.2, I.2, I.2.10, H.4 LNCS Sublibrary: SL 6 – Image Processing, Computer Vision, Pattern Recognition, and Graphics © Springer-Verlag Berlin Heidelberg 2011 This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, re-use of illustrations, recitation, broadcasting, reproduction on microfilms or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer. Violations are liable to prosecution under the German Copyright Law. The use of general descriptive names, registered names, trademarks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. Typesetting: Camera-ready by author, data conversion by Scientific Publishing Services, Chennai, India Printed on acid-free paper Springer is part of Springer Science+Business Media (www.springer.com)
Preface
This volume collects the papers accepted for presentation at the 13th International Conference on “Advanced Concepts for Intelligent Vision Systems” (ACIVS 2011). Following the first meeting in Baden-Baden (Germany) in 1999, which was part of a large multiconference, the ACIVS conference then developed into an independent scientific event and has ever since maintained the tradition of being a single-track conference. ACIVS 2011 attracted scientists from 28 different countries, mostly from Europe, but also from Australia, Japan, Brazil, Korea, China, Tunisia, Pakistan, Taiwan and the USA. Although ACIVS is a conference on all areas of image and video processing, submissions tend to gather within some major fields of interest. About 40% of the selected papers deal with image and video processing, including filtering and restoration and low-level analysis. This year topics related to vision, with a focus on the analysis of people, were well represented, as were papers on 3D scene estimation, processing and understanding. We would like to thank the invited speakers Marc De Mey (UGent and Koninklijke Vlaamse Academie van Belgi¨e), Peter Meijer (Metamodal BV), Ben Kr¨ose (University of Amsterdam) Lambert Spaanenburg (Lund University) and Rainer Stiefelhagen (Institute for Anthropomatics, Karlsruhe Institute of Technology and Fraunhofer Institute of Optronics, System Technologies and Image Exploitation) for enhancing the technical program with their presentations. A conference like ACIVS would not be feasible without the concerted effort of many people and the support of various institutions. The paper submission and review procedure was carried out electronically and a minimum of three reviewers were assigned to each paper. From 124 submissions, 29 were selected for oral presentation and 37 as posters. A large and energetic Program Committee, helped by additional referees (about 95 people in total) – listed on the following pages – completed the long and demanding reviewing process. We would like to thank all of them for their timely and high-quality reviews. Also, we would like to thank our sponsors, FWO-Vlaanderen, Ghent University, IBBT, Alcatel-Lucent, Philips, Nicta and Object Video, for their valuable support. Last but not least, we would like to thank all the participants who trusted in our ability to organize this conference for the 13th time. We hope they attended a stimulating scientific event and enjoyed the atmosphere of the ACIVS social events in the city of Ghent. July 2011
J. Blanc-Talon D. Popescu W. Philips R. Kleihorst P. Scheunders
Organization
Acivs 2011 was organized by IBBT and Ghent University in Ghent Belgium in conjunction with the ACM/IEEE International Conference on Distributed Smart Cameras.
Steering Committee Jacques Blanc-Talon Wilfried Philips Dan Popescu Paul Scheunders
DGA, France Ghent University - IBBT, Belgium CSIRO, Australia University of Antwerp, Belgium
Organizing Committee Richard Kleihorst Wilfried Philips Paul Scheunders
VITO - Ghent University, Belgium Ghent University - IBBT, Belgium University of Antwerp, Belgium
Sponsors Acivs 2011 was sponsored by the following organizations: – – – – – – –
Flemish Fund for Scientific Research - FWO-Vlaanderen Ghent University - Faculty of Engineering and Architecture Interdisciplinary Institute for Broadband Technology - IBBT Alcatel-Lucent Philips Nicta Object Video
Program Committee Hamid Aghajan Marc Antonini Laure Blanc-Feraud Philippe Bolon Salah Bourennane Dumitru Burdescu Vicent Caselles
Stanford University, USA Universit´e de Nice Sophia Antipolis, France INRIA/I3S, France University of Savoie, France Ecole Centrale de Marseille, France University of Craiova, Romania Universitat Pompeu Fabra, Spain
VIII
Organization
Jocelyn Chanussot Pamela Cosman Yves D’Asseler Jennifer Davidson Arturo de la Escalera Hueso Eric Debreuve Christine Fernandez-Maloigne Don Fraser Jerˆ ome Gilles Georgy Gimel’farb Jean-Yves Guillemaut Markku Hauta-Kasari Dimitris Iakovidis Arto Kaarna Andrzej Kasinski Nikos Komodakis Murat Kunt Kenneth Lam Alessandro Ledda Maylor Leung Yue Li Brian Lovell Anthony Maeder Xavier Maldague Gonzalo Pajares Martinsanz Javier Mateos G´erard Medioni Fabrice M´eriaudeau Alfred Mertins Jean Meunier Amar Mitiche Rafael Molina Adrian Munteanu Michel Paindavoinec Fernando Pereira Aleksandra Pizurica Frederic Precioso William Puech Gianni Ramponi Paolo Remagnino Martin Rumpf Guillermo Sapiro Andrzej Sluzek
INPG, France University of California at San Diego, USA Ghent University, Belgium Iowa State University, USA Universidad Carlos III de Madrid, Spain I3S, France Universit´e de Poitiers, France Australian Defence Force Academy, Australia UCLA, USA The University of Auckland, New Zealand University of Surrey, UK University of Eastern Finland, Finland Technological Educational Institute of Lamia, Greece Lappeenranta University of Technology, Finland Poznan University of Technology, Poland University of Crete, Greece EPFL, Switzerland The Hong Kong Polytechnic University, China Artesis University College of Antwerp, Belgium Nanyang Technological University CSIRO ICT Centre, Australia University of Queensland, Australia University of Western Sydney, Australia Universit´e de Laval, Canada Universidad Complutense, Spain University of Granada, Spain USC/IRIS, USA IUT Le Creusot, France Universit¨ at zu L¨ ubeck, Germany Universit´e de Montr´eal, Canada INRS, Canada Universidad de Granada, Spain Vrije Universiteit Brussel, Belgium Bourgogne University, France Instituto Superior T´ecnico, Portugal Ghent University - IBBT, Belgium Paris 6, France LIRMM, France Trieste University, Italy Kingston University, UK Bonn University, Germany University of Minnesota, USA Nanyang Technological University, Singapore
Organization
Hugues Talbot Jean-Philippe Thiran Matthew Thurley Frederic Truchetet Dimitri Van De Ville Marc Van Droogenbroeck Peter Veelaert Miguel Vega Gerald Zauner Pavel Zemcik Djemel Ziou
IX
ESIEE, France EPFL, Switzerland Lule˚ a University of Technology, Sweden Universit´e de Bourgogne, France EPFL, Switzerland University of Li`ege, Belgium University College Ghent, Belgium University of Granada, Spain Fachhochschule Ober¨osterreich, Austria Brno University of Technology, Czech Republic Sherbrooke University, Canada
Reviewers Hamid Aghajan Marc Antonini Sileye Ba Thierry Baccino Charles Beumier Jacques Blanc-Talon Philippe Bolon Don Bone Alberto Borghese Salah Bourennane Dumitru Burdescu Vicent Caselles Umberto Castellani Jocelyn Chanussot Cornel Cofaru Peter Corke Pamela Cosman Marco Cristani Erik D’Hollander Matthew Dailey Arturo de la Escalera Hueso Jonas De Vylder Francis Deboeverie Eric Debreuve Julie Digne Koen Douterloigne S´everine Dubuisson Lieven Eeckhout Christine Fernandez-Maloigne David Filliat Don Fraser
Stanford University, USA Universit´e de Nice Sophia Antipolis, France Telecom Bretagne, France Lutin Userlab, France Royal Military Academy, Belgium DGA, France University of Savoie, France Cannon Information Systems Research, Australia University of Milan, Italy Ecole Centrale de Marseille, France University of Craiova, Romania Universitat Pompeu Fabra, Spain Universit` a degli Studi di Verona, Italy INPG, France Ghent University, Belgium Queensland University of Technology, Australia University of California at San Diego, USA University of Verona, Italy Ghent University, Belgium Asian Institute of Technology, Thailand Universidad Carlos III de Madrid, Spain Ghent University, Belgium Ghent University College, Belgium I3S, France ENS Cachan, France Ghent University, Belgium Universit´e Paris VI, France Ghent University, Belgium Universit´e de Poitiers, France ENSTA, France Australian Defence Force Academy, Australia
X
Organization
Jerˆ ome Gilles Georgy Gimel’farb Bart Goossens Jean-Yves Guillemaut Markku Hauta-Kasari Monson Hayes Dimitris Iakovidis Arto Kaarna Andrzej Kasinski Richard Kleihorst Nikos Komodakis Murat Kunt Nojun Kwak Olivier Laligant Kenneth Lam Peter Lambert Alessandro Ledda Maylor Leung Yue Li Hiep Luong Xavier Maldague Gonzalo Pajares Martinsanz Javier Mateos G´erard Medioni Fabrice M´eriaudeau Alfred Mertins Jean Meunier Amar Mitiche Rafael Molina Adrian Munteanu Sergio Orjuela Vargas Michel Paindavoine Dijana Petrovska Sylvie Philipp-Foliguet Wilfried Philips Wojciech Pieczynski Aleksandra Pizurica Ljiljana Platisa Dan Popescu Eric Postma Frederic Precioso William Puech Gianni Ramponi Paolo Remagnino
UCLA, USA The University of Auckland, New Zealand Ghent University, Belgium University of Surrey, UK University of Eastern Finland, Finland Georgia Institute of Technology, USA Technological Educational Institute of Lamia, Greece Lappeenranta University of Technology, Finland Poznan University of Technology, Poland VITO - Ghent University, Belgium University of Crete, Greece EPFL, Switzerland Ajou University, Republic of Korea IUT Le Creusot, France The Hong Kong Polytechnic University, China Ghent University, Belgium Artesis University College of Antwerp, Belgium Nanyang Technological University CSIRO ICT Centre, Australia Ghent University, Belgium Universit´e de Laval, Canada Universidad Complutense, Spain University of Granada, Spain USC/IRIS, USA IUT Le Creusot, France Universit¨ at zu L¨ ubeck, Germany Universit´e de Montr´eal, Canada INRS, Canada Universidad de Granada, Spain Vrije Universiteit Brussel, Belgium Ghent University, Belgium Bourgogne University, France SudParis, France ETIS, France Ghent University - IBBT, Belgium TELECOM SudParis, France Ghent University - IBBT, Belgium Ghent University, Belgium CSIRO, Australia University of Tilburg, The Netherlands Paris 6, France LIRMM, France Trieste University, Italy Kingston University, UK
Organization
Martin Rumpf Paul Scheunders V´eronique Serfaty D´esir´e Sidib´e Andrzej Sluzek Dirk Stroobandt Hugues Talbot Jean-Philippe Thiran Matthew Thurley Frederic Truchetet Dimitri Van De Ville Marc Van Droogenbroeck David Van Hamme Peter Van Hese Peter Veelaert Miguel Vega Gerald Zauner Pavel Zemcik Djemel Ziou Witold Zorski
XI
Bonn University, Germany University of Antwerp, Belgium DGA, France University of Bourgogne, France Nanyang Technological University, Singapore Ghent University, Belgium ESIEE, France EPFL, Switzerland Lule˚ a University of Technology, Sweden Universit´e de Bourgogne, France EPFL, Switzerland University of Li`ege, Belgium University College Ghent, Belgium Ghent University, Belgium University College Ghent, Belgium University of Granada, Spain Fachhochschule Ober¨osterreich, Austria Brno University of Technology, Czech Republic Sherbrooke University, Canada Cybernetics Faculty, Military University of Technology, Poland
Table of Contents
Vision Robust Visual Odometry Using Uncertainty Models . . . . . . . . . . . . . . . . . . David Van Hamme, Peter Veelaert, and Wilfried Philips
1
Supervised Visual Vocabulary with Category Information . . . . . . . . . . . . . Yunqiang Liu and Vicent Caselles
13
Nonparametric Estimation of Fisher Vectors to Aggregate Image Descriptors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Herv´e Le Borgne and Pablo Mu˜ noz Fuentes
22
Knowledge-Driven Saliency: Attention to the Unseen . . . . . . . . . . . . . . . . . M. Zaheer Aziz, Michael Knopf, and B¨ arbel Mertsching
34
A Comparative Study of Vision-Based Lane Detection Methods . . . . . . . . Nadra Ben Romdhane, Mohamed Hammami, and Hanene Ben-Abdallah
46
A New Multi-camera Approach for Lane Departure Warning . . . . . . . . . . Amol Borkar, Monson Hayes, and Mark T. Smith
58
Classification, Recognition and Tracking Feature Space Warping Relevance Feedback with Transductive Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Daniele Borghesani, Dalia Coppi, Costantino Grana, Simone Calderara, and Rita Cucchiara Improved Support Vector Machines with Distance Metric Learning . . . . . Yunqiang Liu and Vicent Caselles A Low-Cost System to Detect Bunches of Grapes in Natural Environment from Color Images . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Manuel J.C.S. Reis, Raul Morais, Carlos Pereira, Olga Contente, Miguel Bacelar, Salviano Soares, Ant´ onio Valente, Jos´e Baptista, Paulo J.S.G. Ferreira, and Jos´e Bulas-Cruz
70
82
92
Fuzzy Cognitive Maps Applied to Synthetic Aperture Radar Image Classifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Gonzalo Pajares, Javier S´ anchez-Llad´ o, and Carlos L´ opez-Mart´ınez
103
Swarm Intelligence Based Searching Schemes for Articulated 3D Body Motion Tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Bogdan Kwolek, Tomasz Krzeszowski, and Konrad Wojciechowski
115
XIV
Table of Contents
Combining Linear Dimensionality Reduction and Locality Preserving Projections with Feature Selection for Recognition Tasks . . . . . . . . . . . . . . Fadi Dornaika, Ammar Assoum, and Alireza Bosaghzadeh
127
A New Anticorrelation-Based Spectral Clustering Formulation . . . . . . . . . Julia Dietlmeier, Ovidiu Ghita, and Paul F. Whelan
139
Simultaneous Partitioned Sampling for Articulated Object Tracking . . . . Christophe Gonzales, S´everine Dubuisson, and Xuan Son N’Guyen
150
Segmentation A Geographical Approach to Self-Organizing Maps Algorithm Applied to Image Segmentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Thales Sehn Korting, Leila Maria Garcia Fonseca, and Gilberto Cˆ amara A Multi-Layer ‘Gas of Circles’ Markov Random Field Model for the Extraction of Overlapping Near-Circular Objects . . . . . . . . . . . . . . . . . . . . . Jozsef Nemeth, Zoltan Kato, and Ian Jermyn Evaluation of Image Segmentation Algorithms from the Perspective of Salient Region Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Bogdan Popescu, Andreea Iancu, Dumitru Dan Burdescu, Marius Brezovan, and Eugen Ganea
162
171
183
Robust Active Contour Segmentation with an Efficient Global Optimizer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jonas De Vylder, Jan Aelterman, and Wilfried Philips
195
A Method to Generate Artificial 2D Shape Contour Based in Fourier Transform and Genetic Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Maur´ıcio Falvo, Jo˜ ao Batista Florindo, and Odemir Martinez Bruno
207
Image Segmentation Based on Electrical Proximity in a Resistor-Capacitor Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jan Gaura, Eduard Sojka, and Michal Krumnikl
216
Hierarchical Blurring Mean-Shift . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ˇ Milan Surkala, Karel Mozdˇreˇ n, Radovan Fusek, and Eduard Sojka
228
Image Analysis Curve-Skeletons Based on the Fat Graph Approximation . . . . . . . . . . . . . . Denis Khromov DTW for Matching Radon Features: A Pattern Recognition and Retrieval Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Santosh K.C., Bart Lamiroy, and Laurent Wendling
239
249
Table of Contents
Ridges and Valleys Detection in Images Using Difference of Rotating Half Smoothing Filters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Baptiste Magnier, Philippe Montesinos, and Daniel Diep
XV
261
Analysis of Wear Debris through Classification . . . . . . . . . . . . . . . . . . . . . . . Roman Jur´ anek, Stanislav Machal´ık, and Pavel Zemˇc´ık
273
Fourier Fractal Descriptors for Colored Texture Analysis . . . . . . . . . . . . . . Jo˜ ao B. Florindo and Odemir M. Bruno
284
Efficiency Optimization of Trainable Feature Extractors for a Consumer Platform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Maurice Peemen, Bart Mesman, and Henk Corporaal Salient Region Detection Using Discriminative Feature Selection . . . . . . . HyunCheol Kim and Whoi-Yul Kim Image Analysis Applied to Morphological Assessment in Bovine Livestock . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Horacio M. Gonz´ alez-Velasco, Carlos J. Garc´ıa-Orellana, Miguel Mac´ıas-Mac´ıas, Ram´ on Gallardo-Caballero, and Antonio Garc´ıa-Manso Quantifying Appearance Retention in Carpets Using Geometrical Local Binary Patterns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Rolando Quinones, Sergio A. Orjuela, Benhur Ortiz-Jaramillo, Lieva Van Langenhove, and Wilfried Philips Enhancing the Texture Attribute with Partial Differential Equations: A Case of Study with Gabor Filters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Bruno Brandoli Machado, Wesley Nunes Gon¸calves, and Odemir Martinez Bruno Dynamic Texture Analysis and Classification Using Deterministic Partially Self-avoiding Walks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Wesley Nunes Gon¸calves and Odemir Martinez Bruno
293 305
316
327
337
349
Image Processing Segmentation Based Tone-Mapping for High Dynamic Range Images . . . Qiyuan Tian, Jiang Duan, Min Chen, and Tao Peng Underwater Image Enhancement: Using Wavelength Compensation and Image Dehazing (WCID) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . John Y. Chiang, Ying-Ching Chen, and Yung-Fu Chen Video Stippling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Thomas Houit and Frank Nielsen
360
372 384
XVI
Table of Contents
Contrast Enhanced Ultrasound Images Restoration . . . . . . . . . . . . . . . . . . . Adelaide Albouy-Kissi, Stephane Cormier, Bertrand Zavidovique, and Francois Tranquart
396
Mutual Information Refinement for Flash-no-Flash Image Alignment . . . Sami Varjo, Jari Hannuksela, Olli Silv´en, and Sakari Alenius
405
Virtual Restoration of the Ghent Altarpiece Using Crack Detection and Inpainting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Tijana Ruˇzi´c, Bruno Cornelis, Ljiljana Platiˇsa, Aleksandra Piˇzurica, Ann Dooms, Wilfried Philips, Maximiliaan Martens, Marc De Mey, and Ingrid Daubechies Image Sharpening by DWT-Based Hysteresis . . . . . . . . . . . . . . . . . . . . . . . . Nuhman ul Haq, Khizar Hayat, Neelum Noreen, and William Puech Content Makes the Difference in Compression Standard Quality Assessment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Guido Manfredi, Djemel Ziou, and Marie-Flavie Auclair-Fortier
417
429
437
A Bio-Inspired Image Coder with Temporal Scalability . . . . . . . . . . . . . . . Khaled Masmoudi, Marc Antonini, and Pierre Kornprobst
447
Self-similarity Measure for Assessment of Image Visual Quality . . . . . . . . Nikolay Ponomarenko, Lina Jin, Vladimir Lukin, and Karen Egiazarian
459
Video Surveillance and Biometrics An Intelligent Video Security System Using Object Tracking and Shape Recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Sang Hwa Lee, Siddharth Sharma, Linlin Sang, Jong-Il Park, and Yong Gyu Park 3D Facial Expression Recognition Based on Histograms of Surface Differential Quantities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Huibin Li, Jean-Marie Morvan, and Liming Chen
471
483
Facial Feature Tracking for Emotional Dynamic Analysis . . . . . . . . . . . . . . Thibaud Senechal, Vincent Rapp, and Lionel Prevost
495
Detection of Human Groups in Videos . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Sel¸cuk Sandıkcı, Svitlana Zinger, and Peter H.N. de With
507
Estimation of Human Orientation in Images Captured with a Range Camera . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . S´ebastien Pi´erard, Damien Leroy, Jean-Fr´ed´eric Hansen, and Marc Van Droogenbroeck
519
Table of Contents
Human Identification Based on Gait Paths . . . . . . . . . . . . . . . . . . . . . . . . . . ´ Adam Swito´ nski, Andrzej Pola´ nski, and Konrad Wojciechowski Separating Occluded Humans by Bayesian Pixel Classifier with Re-weighted Posterior Probability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Daehwan Kim, Yeonho Kim, and Daijin Kim
XVII
531
543
An Edge-Based Approach for Robust Foreground Detection . . . . . . . . . . . Sebastian Gruenwedel, Peter Van Hese, and Wilfried Philips
554
Relation Learning - A New Approach to Face Recognition . . . . . . . . . . . . . Len Bui, Dat Tran, Xu Huang, and Girija Chetty
566
Algorithms and Optimizations Temporal Prediction and Spatial Regularization in Differential Optical Flow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Matthias Hoeffken, Daniel Oberhoff, and Marina Kolesnik Parallel Implementation of the Integral Histogram . . . . . . . . . . . . . . . . . . . . Pieter Bellens, Kannappan Palaniappan, Rosa M. Badia, Guna Seetharaman, and Jesus Labarta System on Chip Coprocessors for High Speed Image Feature Detection and Matching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Marek Kraft, Michal Fularz, and Andrzej Kasi´ nski Fast Hough Transform on GPUs: Exploration of Algorithm Trade-Offs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Gert Jan van den Braak, Cedric Nugteren, Bart Mesman, and Henk Corporaal Feasibility Analysis of Ultra High Frame Rate Visual Servoing on FPGA and SIMD Processor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yifan He, Zhenyu Ye, Dongrui She, Bart Mesman, and Henk Corporaal
576 586
599
611
623
3D, Depth and Scene Understanding Calibration and Reconstruction Algorithms for a Handheld 3D Laser Scanner . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Denis Lamovsky and Aless Lasaruk Comparison of Visual Registration Approaches of 3D Models for Orthodontics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Rapha¨el Destrez, Benjamin Albouy-Kissi, Sylvie Treuillet, Yves Lucas, and Arnaud Marchadier
635
647
XVIII
Table of Contents
A Space-Time Depth Super-Resolution Scheme for 3D Face Scanning . . . Karima Ouji, Mohsen Ardabilian, Liming Chen, and Faouzi Ghorbel Real-Time Depth Estimation with Wide Detectable Range Using Horizontal Planes of Sharp Focus Proceedings . . . . . . . . . . . . . . . . . . . . . . . Hiroshi Ikeoka, Masayuki Ohata, and Takayuki Hamamoto Automatic Occlusion Removal from Facades for 3D Urban Reconstruction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Chris Engels, David Tingdahl, Mathias Vercruysse, Tinne Tuytelaars, Hichem Sahli, and Luc Van Gool
658
669
681
hSGM: Hierarchical Pyramid Based Stereo Matching Algorithm . . . . . . . Kwang Hee Won and Soon Ki Jung
693
Surface Reconstruction of Rotating Objects from Monocular Video . . . . . Charlotte Boden and Abhir Bhalerao
702
Precise Registration of 3D Images Acquired from a Hand-Held Visual Sensor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Benjamin Coudrin, Michel Devy, Jean-Jos´e Orteu, and Ludovic Br`ethes A 3-D Tube Scanning Technique Based on Axis and Center Alignment of Multi-laser Triangulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Seung-Hae Baek and Soon-Yong Park Combining Plane Estimation with Shape Detection for Holistic Scene Understanding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Kai Zhou, Andreas Richtsfeld, Karthik Mahesh Varadarajan, Michael Zillich, and Markus Vincze
712
724
736
Simple Single View Scene Calibration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Bronislav Pˇribyl and Pavel Zemˇc´ık
748
Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
761
Robust Visual Odometry Using Uncertainty Models David Van Hamme1,2 , Peter Veelaert1,2 , and Wilfried Philips2 1
University College Ghent (Vision Systems) 2 Ghent University/IBBT (IPI)
Abstract. In dense, urban environments, GPS by itself cannot be relied on to provide accurate positioning information. Signal reception issues (e.g. occlusion, multi-path effects) often prevent the GPS receiver from getting a positional lock, causing holes in the absolute positioning data. In order to keep assisting the driver, other sensors are required to track the vehicle motion during these periods of GPS disturbance. In this paper, we propose a novel method to use a single on-board consumer-grade camera to estimate the relative vehicle motion. The method is based on the tracking of ground plane features, taking into account the uncertainty on their backprojection as well as the uncertainty on the vehicle motion. A Hough-like parameter space vote is employed to extract motion parameters from the uncertainty models. The method is easy to calibrate and designed to be robust to outliers and bad feature quality. Preliminary testing shows good accuracy and reliability, with a positional estimate within 2 metres for a 400 metre elapsed distance. The effects of inaccurate calibration are examined using artificial datasets, suggesting a self-calibrating system may be possible in future work.
1
Introduction
The current generation of GPS navigation systems is insufficiently reliable in urban conditions. Obstacles along and over the road (e.g. tall buildings, overpasses, bridges) often stand in the way of the four-way satellite link that is required for a positional fix. In these situations, the GPS unit temporarily loses track of position. However, it is sufficient to be able to accurately track the relative motion of the vehicle over a short distance to keep a reasonable estimate of the absolute position. In this paper we present a relative motion tracking approach based on a single forward-facing camera. There is a tendency in the automotive industry to equip consumer vehicles with cameras to perform a number of driver assistance tasks. Traffic sign recognition and lane departure warning are two examples that can readily be found on the options list of many new cars. The calculation of the ego-motion of a moving platform from images captured from the platform itself is called visual odometry. A solid mathematical basis for visual odometry was laid in the past 20 years ([1, 10, 11, 13–15]). More recent works focus on how to bring these concepts into J. Blanc-Talon et al. (Eds.): ACIVS 2011, LNCS 6915, pp. 1–12, 2011. c Springer-Verlag Berlin Heidelberg 2011
2
D.V. Hamme, P. Veelaert, and W. Philips
practice. The approaches can be roughly divided into three categories based on the optics used. A first approach uses stereo cameras, with influential works by Obdrˇz´alek et al., 2010 [16], Kitt et al., 2010 [8], Comport et al., 2007 [6], Konolige et al., 2007 [9] and Cheng et al., 2006 [5]. A second approach is the use of an omnidirectonal camera, as in Scaramuzza et al., 2009 [17], Tardif et al., 2008 [18]. The third category consists of methods that use a single, non-panoramic camera. Significant efforts include Campbell et al., 2005 [4] and Mouragnon et al., 2009 [12], Azuma et al., 2010 [2]. Both stereo vision and panoramic cameras boast obvious advantages for solving the visual odometry problem. However, such setups are impractical for use on consumer vehicles, as the stereo methods require precise and repeated calibration and the omnidirectional camera cannot be mounted inconspicuously. We will therefore focus on a solution that involves a single, standard consumer camera. Among the methods cited above, good performance has been demonstrated on indoor test sequences, and in some cases even in controlled outdoor environments. However, the application on consumer vehicles poses new challenges. As road speeds increase, the number of available feature correspondences is reduced, and their displacements in the image can become very large. A third problem is the presence of outliers: other traffic will cause feature correspondences that are of no use to calculate visual odometry. These unsolved problems lead us to propose a novel method for a single, forward-facing camera. Central to the method is the backprojection of image features to the world ground plane, and the uncertainties associated with this backprojection. A Hough-like voting scheme is implemented to track the consistent motion in the uncertainty models of the ground plane features. Careful modelling of the uncertainties both in the backprojection and in the motion parameters, coupled with a robust voting algorithm, allow us to reliably extract the ego-motion from a calibrated system. Experiments on real data show accuracy within 2 metres after an elapsed distance of 400 metres. The details of the method will be further explained in section 3.
2
Camera Model
To accurately model the uncertainties of the backprojection in an elegant way, it is important to choose an appropriate camera model and carefully define the used coordinate systems. In our application, we are working in three different coordinate systems: 3D world coordinates, 3D camera coordinates, and 2D image coordinates. The 3D world coordinate system is chosen as a right-handed system with the origin on the road surface directly below the center of rotation of the vehicle. Note that this means that the world axes remain tied to the vehicle, and vehicle motion manifests itself as moving texture on a fixed plane. The world Z-axis is taken perpendicular to the ground plane, the Y-axis parallel to the straight-ahead driving direction of the vehicle. The camera coordinate system is also a right-handed system, and is similarly aligned. Its origin is in the center of projection of the camera, its Y-axis points in the viewing direction along
Robust Visual Odometry Using Uncertainty Models
3
the principal axis, and the X- and Z-axis are aligned with the horizontal and vertical image sensor directions respectively. While this deviates somewhat from the conventional definitions, it facilitates calibration, as will be explained shortly. Finally, the 2D image coordinate system is defined with the origin in the top left corner of the image, the X-axis pointing right and the Y-axis pointing down. Choosing the axes this way offers some advantages from a programming point of view, as it better reflects image data organization in memory. To describe the perspective projection from 3D world coordinates to 2D image coordinates, we will use an undistorted pinhole camera model. Let x = [x y w]T T denote the 2D image point in homogeneous coordinates, and X = [X Y Z 1] the corresponding point in homogeneous 3D world coordinates. The projection of X onto x is then given by: x = C R|t X. (1) In the above expression, C is the upper triangular intrinsic camera matrix as described in Hartley and Zisserman, 2004 [7], consisting of the horizontal and vertical scaling components αx and αy and the image coordinates of the principal point (x0 , y0 ), multiplied by a substitution matrix arising from our non-standard definition of image axes: ⎤ ⎤⎡ ⎡ 10 0 αx 0 x0 (2) C = ⎣ 0 αy y0 ⎦ ⎣0 0 −1⎦ . 01 0 0 0 1 [R|t] is the rotation matrix R that aligns the world axes with the camera axes, augmented by the 3D translation vector t between their origins. The resulting 4x3 matrix is the projection matrix that maps homogeneous 3D world coordinates onto 2D image coordinates. Because we will only consider points in the world ground plane, Z = 0 for all points and the projection matrix reduces to a 3x3 homography matrix H: ⎡ ⎤ ⎡ ⎤ x X x = ⎣ y ⎦ = H ⎣Y ⎦ . (3) w 1 In our case, we are interested in the inverse transformation: we want to project image points onto the world ground plane. This projection is characterized by the inverse of H: ⎡ ⎤ ⎡ ⎤ X x X = ⎣ Y ⎦ = H−1 ⎣y ⎦ . (4) W 1 If lens distortion has to be taken into account, a distortion function is applied to the left hand side of equations (1) and (3). We will consider the distortion to be rectified in advance, as can easily be done using standard methods [3]. Let us take a closer look at the components of equation (1). The camera matrix C can be obtained using standard calibration methods as described in Bouguet, 1999 [3], and will remain constant for a fixed-zoom camera.
4
D.V. Hamme, P. Veelaert, and W. Philips
The vector t determines the offset between the origin of the world axes and the origin of the camera axes. We can measure t as part of the extrinsic calibration process, as it changes only very slightly with vehicle motion and load. The rotation matrix R can be constructed as a series of three rotations, each along one of the axes. The most common conventions for defining these rotations are through heading, pitch and roll (Z-, X-, and Y-axis rotations) or Euler angles (usually Z-X-Z). We will use the heading, pitch and roll configuration. Due to our non-standard choice of world and camera axes, these three angles can be measured relatively easily in world coordinates. However, only the heading can be assumed to remain constant while the vehicle is moving, as suspension movement does not affect this direction of the vehicle in any significant way. The other rotations however, pitch and roll, have a range of freedom around the calibration values obtained while stationary, as they are affected by the suspension. This range of motion will be the key challenge to overcome, as will be explained in section 3. The ground plane projection of a sample frame while stationary is shown in figure 1.
Fig. 1. Example of a test frame (left), and its ground plane backprojection (right)
3
The Proposed Method
The first step in our proposed method is the backprojection of Harris corners to the world ground plane, using equation (4). The advantage of using backprojected ground features is that the method will still produce useful motion information when there are only few feature correspondences, due to the reduction in degrees of freedom. However, in order to accurately backproject the Harris corners, we need the immediate pitch and roll angles for each frame. These angles are not precisely known, they are only defined within an interval around the calibration values obtained at rest (cfr. section 2). This range of possible suspension angles defines a region of possible backprojected locations in the world plane. Due to the trigonometric functions in the rotation matrix R, this region will not strictly be convex, but as the variation in angles is small, it can be closely approximated by a tetragon. These uncertainty regions on the world plane arising from the range of angles in the camera perspective will be referred to as Perspective Uncertainty Tetragons (PUTs). Note that there is a PUT for every Harris corner detected in the camera image.
Robust Visual Odometry Using Uncertainty Models
5
The PUTs are easily calculated: a rotation matrix R is constructed for each of the four combinations of extremal pitch and roll values. This results in four different backprojections of each Harris corner, yielding the four corners of the tetragon for that feature. It should be noted that any inaccuracies in the feature detector can also be modelled into the PUTs. Suppose for example that the chosen feature detector is known to have poor localization, producing features in positions that can be off by one pixel from the actual point of interest. One way to account for this could be to calculate the PUTs for all possible extremal pixel co¨ordinates of the actual feature, distributed in an area around the feature detector output, and then taking the union of the PUTs to obtain the uncertainty on the world plane position. In our application, the Harris corners are assumed to be accurate within half a pixel in both the horizontal and vertical image dimension.
Fig. 2. Harris corners in the camera view (left), and close-up of some of their associated PUTs (right)
Figure 2 shows the PUTs for an example frame in a test sequence. The left image shows a camera frame in which Harris corners were detected. The background of the right image is the backprojection of the camera frame based on the average angles, with the PUTs of the Harris corners drawn on top of it. Realistic limits on the pitch and roll angles can be established by analysis of test videos in which the vehicle is accelerating, braking or cornering hard. We can see from the shape and size of the PUTs in figure 2 that pitch is the main factor contributing to the uncertainty (vertical elongation), while the effect of roll remains limited. The problem we need to solve, is how to extract accurate translation and rotation parameters from correspondences between the features of consecutive frames when the position of the features is only known up to a region (the PUTs). Additionally, the method must be robust to outliers. Features that do not correspond to object in the ground plane, or that belong to other traffic, will exhibit inconsistent motion and should have as little influence as possible. To solve the above problem we propose a simple, transparent voting mechanism that can extract the incremental trajectory in real-time. In a first step, we must establish possible feature correspondences between consecutive frames. Let
6
D.V. Hamme, P. Veelaert, and W. Philips
us assume that we know the exact ground plane position of the previous frame’s features. As we stated earlier, the origin of the world coordinate system remains affixed to a point below the center of rotation of the vehicle. This means that one frame ahead, the features (which are static in the real world) will have new coordinates in the ground plane. The new coordinates depend on the speed and steering angle between the previous two frames, and on the acceleration and steering input of the driver between the previous and the current frame. The driver inputs are the unknowns that we want to determine. However, the acceleration is bound by the maximum torque of the engine and by the maximum retardation allowed by the braking system, and the steering input is limited by the ratio of the steering rack coupled with the maximum speed at which the driver is physically able to twirl the steering wheel. These bounds can be established from the specification of the vehicle and a simple experiment in which a person tries to turn the steering wheel from lock to lock as fast as possible. The bounds on driver input are also included in our uncertainty model. For each known feature position from the previous frame, the range of possible driver inputs delimits a region in the world plane that represents this feature’s possible world plane positions in the current frame. Again, this region is not strictly convex due to the rotation component, but is closely approximated by a tetragon. We will call this type of tetragon a Motion Uncertainty Tetragon (MUT), as it arises from the uncertainty on the vehicle motion. An example of a set of MUTs is shown in figure 3. Note that features in close proximity to the vehicle have a narrower MUT than distant features. The MUT of a feature position is calculated by displacing the feature along four circle segments, representing the four extremal combinations of possible speed and steering angle. The circle segments are an approximation of the real trajectory, as they assume a constant speed and steering angle over the interframe interval. In reality, the bend radius will change continuously in this interval, but the errors introduced by this approximation are small. The uncertainty of the feature detection and backprojection is now modelled in the PUTs and the uncertainty on the predicted displacement of the vehicle is modelled in the MUTs. The PUTs correspond to features in the current frame’s camera view, while the MUTs correspond to features in the previous frame’s world plane. The problem of finding correspondences between the previous frame and the current frame is thereby reduced to finding overlap between MUTs and PUTs. Once the possible feature correspondences have been established, the second problem is how to extract the correct motion parameters (i.e. speed differential and steering angle differential) from these correspondences. Once we know these differentials, we can reconstruct the trajectory of the vehicle between the two frames. The MUTs are essentially projections of a region of translation-rotation parameter space onto the ground plane. The overlap of the MUTs with the PUTs gives us information about which parameter combinations are plausible according to the observed features. Although the MUTs have slightly different sizes and
Robust Visual Odometry Using Uncertainty Models
7
Fig. 3. Example of MUTs
shapes based on their location, their boundaries correspond to the same extremal values of speed and steering angle, and as such every MUT is a deformation of the same rectangular patch in parameter space. When a PUT overlaps with an MUT, this is evidence that the region of overlap in the MUT contains plausible vehicle motion parameters according to one of the features. We can state that the overlap expresses a vote for this region in parameter space. When we sum the region votes for all areas of PUT-MUT overlap, we obtain a measure of plausibility for every rotation-translation combination in the parameter space patch. Evidence will concentrate on those combinations that agree with the majority of the observations. This is similar in concept to Hough-based shape detection algorithms, where the shapes are found as peaks in a voted parameter space. This type of voting method has the advantage that it provides some robustness against outliers, as the contributions from bad features (e.g. features caused by other traffic or by objects not in the ground plane) will not typically have a common intersection, and as such tend to manifest themselves as noise spread out over the parameter space. Figure 4 shows the typical overlapping of MUTs and PUTs. To perform the parameter space vote in practice, we normalize every MUT to a rectangle of predefined size and represent its overlap with one or more PUTs by a binary image of this predefined size. The horizontal axis of the normalized image is a slightly nonlinearly stretched representation of the rotation differential axis, while the vertical axis is the speed differential axis. Summing the images of all normalized MUTs is essentially the same as summing region votes in the discretized parameter space. An example of a sum image is shown in figure 4. We call this the consensus image, as it is a graphic representation of the parameter consensus between all features. In the example shown, we see that the votes concentrate on an area slightly below center. This translates to a slight deceleration while driving straight ahead. We should note that in order for the evidence to concentrate in the consensus image, we need a horizontal distribution of features over the image as equal as
8
D.V. Hamme, P. Veelaert, and W. Philips
Fig. 4. Overlap between PUTs and MUTs, and corresponding consensus image
possible. If Harris corners are only found on the right side of the driving direction, their PUTs will all be skewed in the same direction, causing their intersections to be large. Contributions of corners from the other side of the driving direction will yield much smaller intersections, and therefore a better concentration of evidence.
4
Results
The proposed method was tested on a trajectory on the parking lot of our campus. The trajectory is a figure of eight with a length of 431 metres. The reconstructed trajectory is shown in figure 5. The backprojections of the entire video frames are used as a background. The comparison with the ground truth of the trajectory is also shown in figure 5, with an aerial photo as background. The positional error at the end of this trajectory is 2.14 metres, while the rotational error is 7.67 degrees. When coupled to an offline map, this is sufficiently accurate to establish the road that the vehicle is travelling on. This proves the validity of the concept for filling in gaps in the GPS reception of a couple of hundred metres. However, more testing is required to evaluate robustness and the effect of other traffic (which was absent during the test run). To evaluate the sensitivity of the method to calibration errors, the method was also tested on artificial data. A camera trajectory of 150 metres consisting of 2 sharp and 2 shallow bends above a checkerboard pattern was rendered and evaluated by the proposed method. The use of a rendered video provides the advantage that the calibration angles and trajectory are exactly known. An example of a frame from the artifical set is shown in figure 6. Figure 7 shows the results of our method compared to the ground truth, shown in black. The red trajectory in figure 7 is the output of our method for correct calibration values. Positional and rotational errors are given in table 1. The green trajectory shows the result when the roll angle would be off by 2 degrees, for
Robust Visual Odometry Using Uncertainty Models
9
Fig. 5. Reconstructed trajectory and surroundings according to our system (left), compared to ground truth (yellow) on aerial photo
Fig. 6. Example frame of artificial test sequence
example due to a calibration error. Two distinct effects are visible. Firstly there is an overall rotational bias that manifests itself in both left and right bends. Secondly, the right bends get truncated while the left bends get elongated. Still, for our trajectory with equal bends left and right, the positional and rotational errors are relatively small. This shows that our method is reasonably robust against roll miscalibration. The blue trajectory in figure 7 results from a 2 degree error in the pitch calibration. The effect is significant: every displacement
10
D.V. Hamme, P. Veelaert, and W. Philips
Fig. 7. Results for artificial test data, showing the effect of various calibration errors Table 1. Positional and rotational errors for different miscalibrations
Exact calibration 2 Degree roll error 2 Degree pitch error 2 Degree heading error
Positional error (m) Rotational error (degrees) 1.51 0.44 2.83 -6.37 16.78 -4.64 18.79 -33.40
gets severely underestimated. Rotational accuracy is still good, but the positional error quickly becomes very large. We can conclude that the method is fairly sensitive to errors in pitch calibration. Finally, the teal trajectory shows the effect of a 2 degree error in heading. Predictably, there is a strong rotational bias, both in the bends and on the straight segments. The end position is off by a very wide margin, and likewise the rotation. Clearly, the method is also sensitive to heading calibration errors. However, it should be noted that when an offline map is present, these systematic errors in position and rotation are easily detected, and the different effects of errors in the different angles should enable us to identify which calibration values are off. This in turn opens the road to a self-correction system, which would be a great asset for consumer applications.
5
Conclusion
We proposed a novel method for calculating visual odometry. The method is shown to work on real data, yielding a positional error of only around 2 metres
Robust Visual Odometry Using Uncertainty Models
11
on trajectory over 400 metres long. However, robustness has to be tested further on real-world data sets with other traffic present, and the algorithm should be connected to an offline map to keep drift under control for longer sequences. Despite the early stage of development though, the technique has already been proven accurate enough to pinpoint the lane in which the vehicle is driving after 300 metres of GPS silence. The effect of calibration errors was examined on artificial data, giving a good understanding of the consequences of inaccuracies in each of the calibration angles. In future work, this will be exploited to make the system self-calibrating to an extent, using the offline map as a reference. Another logical extensions would be implementation of a feedback loop of the trajectory to the vehicle suspension model to reduce perspective uncertainty. Also, the current version makes no use of the established consensus to refine the PUTs and recalculate the consensus. One can reasonably assume that such an iterative method will further improve the results, however at the cost of longer computation. Finally, a detailed analysis of the evolution of the uncertainties over time could be carried out to derive a confidence measure for the estimated position and orientation.
References 1. Amidi, O., Kanade, T., Miller, J.: Vision-based autonomus helicopter research at cmu. In: Proc. of Heli Japan 1998 (1998) 2. Azuma, T., Sugimoto, S., Okutomi, M.: Egomotion estimation using planar and non-planar constraints. In: Intelligent Vehicles Symposium (IV), pp. 855–862. IEEE, Los Alamitos (2010) 3. Bouguet, J.: Visual Methods for Three-Dimensional Modeling. Ph.D. thesis, California Institute of Technology (May 1999) 4. Campbell, J., Sukthankar, R., Nourbakhsh, I., Pahwa, A.: A robust visual odometry and precipice detection system using consumer-grade monocular vision. In: Proc. of IEEE Int. Conf on Robotics and Automation (ICRA) 2005, pp. 3421– 3427 (2005) 5. Cheng, Y., Maimone, M., Matthies, L.: Visual odometry on the mars exploration rovers. IEEE Robotics and Automation Magazine 13(2) (2006) 6. Comport, A., Malis, E., Rives, P.: Accurate quadrifocal tracking for robust 3d visual odometry. In: Proc. of IEEE Int. Conf. on Robotics and Automation (ICRA) 2007, pp. 40–45 (2007) 7. Hartley, R., Zisserman, A.: Multiple View Geometry in Computer Vision. Cambridge University Press, Cambridge (2004) 8. Kitt, B., Geiger, A., Lategahn, H.: Visual odometry based on stereo image sequences with ransac-based outlier rejection scheme. In: Intelligent Vehicles Symposium (IV), pp. 486–492. IEEE, Los Alamitos (2010) 9. Konolige, K., Agrawal, M., Sol, J.: Large-scale visual odometry for rough terrain. In: Int. Symposium on Research in Robotics (2007) 10. Levin, A., Szeliski, R.: Visual odometry and map correlation. In: Proc. of IEEE Int. Conf. on Computer Vision and Pattern Recognition 2004, vol. 1-I, pp. 611–618 (2004)
12
D.V. Hamme, P. Veelaert, and W. Philips
11. Marks, R., Wang, H., Lee, M., Rock, S.: Automatic visual station keeping of an underwater robot. In: Proc. of IEEE Oceans 1994, pp. 137–142 (1994) 12. Mouragnon, E., Lhuillier, M., Dhome, M., Dekeyser, F., Sayd, P.: Generic and real-time structure from motion using local bundle adjustment. Image and Vision Computing 27(8) (2009) 13. Negahdaripour, S., Horn, B.: Direct passive navigation. IEEE Transactions on Pattern Analysis and Machine Intelligence 9(1) (1987) 14. Nist´er, D.: An efficient solution to the five-point relative point problem. IEEE Transactions on Pattern Analysis and Machine Intelligence 26(6) (2004) 15. Nist´er, D., Naroditsky, O., Bergen, J.: Visual odometry for ground vehicle applications. Journal of Field Robotics 23 (2006) 16. Obdrˇza ´lek, S., Matas, J.: A voting strategy for visual ego-motion from stereo. In: Intelligent Vehicles Symposium (IV), pp. 382–387. IEEE, Los Alamitos (2010) 17. Scaramuzza, D., Fraundorfer, F., Siegwart, R.: Real-time monocular visual odometry for on-road vehicles with 1-point ransac. In: Proc. of IEEE Int. Conf on Robotics and Automation (ICRA) 2009, pp. 4293–4299 (2009) 18. Tardif, J.-P., Pavlidis, Y., Daniilidis, K.: Monocular visual odometry in urban environments using an omnidrectional camera. In: Proc. of IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS) 2008, pp. 2531–2538 (2008)
Supervised Visual Vocabulary with Category Information Yunqiang Liu1 and Vicent Caselles2 1
Barcelona Media - Innovation Center, Barcelona, Spain 2 Universitat Pompeu Fabra, Barcelona, Spain
[email protected],
[email protected]
Abstract. The bag-of-words model has been widely employed in image classification and object detection tasks. The performance of bagof-words methods depends fundamentally on the visual vocabulary that is applied to quantize the image features into visual words. Traditional vocabulary construction methods (e.g. k-means) are unable to capture the semantic relationship between image features. In order to increase the discriminative power of the visual vocabulary, this paper proposes a technique to construct a supervised visual vocabulary by jointly considering image features and their class labels. The method uses a novel cost function in which a simple and effective dissimilarity measure is adopted to deal with category information. And, we adopt a prototypebased approach which tries to find prototypes for clusters instead of using the means in k-means algorithm. The proposed method works as the k-means algorithm by efficiently minimizing a clustering cost function. The experiments on different datasets show that the proposed vocabulary construction method is effective for image classification. Keywords: bag of words, visual vocabulary, category information, image classification.
1
Introduction
The bag-of-words model [1] has been widely employed in image classification and object detection tasks due to its simplicity and good performance. It describes images as sets of elementary local features called visual words. Generally, this method consists of several basic steps: (1) keypoints are sampled using various detectors, (2) keypoints are represented using local descriptors such as SIFT (scale-invariant feature transform)[2], (3) descriptors are vector quantized into a fixed-size vocabulary using a clustering algorithm and each resulting cluster corresponds to a visual word, and (4) by mapping keypoints into visual words, images are represented by the histogram of visual words occurrences in the image. This histogram can be used as an input to a classifier in the classification step. The performance of bag-of-words methods depends fundamentally on the visual vocabulary that is applied to quantize the continuous local image features into discrete visual words. Various techniques have been explored to address this quantization process for vocabulary generation e.g. k-means [1] and J. Blanc-Talon et al. (Eds.): ACIVS 2011, LNCS 6915, pp. 13–21, 2011. c Springer-Verlag Berlin Heidelberg 2011
14
Y. Liu and V. Caselles
mean-shift [3]. These approaches are typically designed to build up a vocabulary with the objective of minimizing the reconstruction error. Vocabularies are usually constructed based solely on the statistics of local image patches without taking category information into account, and cannot capture the semantic relationship between image features. This semantic relationship is useful for building up discriminative vocabularies for image classification. Some recent approaches have addressed the problem of building discriminative vocabularies. Winn et al. [4] obtained discriminative vocabulary by reducing a large one using pair-wise word merging based on the mutual information between visual words and categories. Lazebnik et al. [5] proposed to construct supervised vocabularies by minimizing mutual information loss. Moosmann et al. [6] built up discriminative visual vocabularies using extremely randomized clustering forests which consider semantic labels as stopping tests in building supervised indexing trees. Perronnin et al. [7] learned a universal vocabulary and then adapt it to class specific data. Yang et al. [8] proposed a vocabulary learning framework that is integrated with classifier learning. They encode an image by a sequence of visual bits that constitute the vocabulary. Zhang et al. [9] constructed category sensitive vocabulary where the category information is incorporated as an additional term. Lian et al. [10] proposed probabilistic models for supervised vocabulary learning which seek a balance between minimization of distortions of clusters and maximization of discriminative power of histogram representations of images. In order to increase the discriminative power of visual vocabulary, this paper proposes a technique to construct a supervised visual vocabulary by jointly considering image features and their class labels. The method uses a novel cost function in which a simple and effective dissimilarity measure is adopted to deal with category information. And, a prototype-based approach is adopted which tries to find prototypes for clusters instead of the means in k-means algorithm. The proposed method works like the k-means algorithm by efficiently minimizing a clustering cost function.
2
Supervised Vocabulary Construction
Under the bag of words model, images are represented by a histogram of visual word occurrences relative to a given vocabulary. Traditional vocabularies are usually constructed without taking category information into account. Consequently, histogram representations of images may not be optimal for classification tasks. Our aim is to construct a discriminative visual vocabulary by jointly considering image features and their category information. Fig. 1 shows the problem of the traditional visual vocabulary on a toy dataset. The features denoted as ◦ and + belong to classes A and B, respectively. A traditional method, like k-means, focuses on minimizing the overall reconstruction error without considering the category information and would generally produce two words (denoted as × ) as shown in Fig. 1(a). If we use the histogram of visual words as feature vector for classification, the described vocabulary would
Supervised Visual Vocabulary with Category Information
15
not be very attractive since the histograms for classes A and B would be the same in this case (as shown in Fig.2(a) and (b)). A more discriminative vocabulary can be constructed if we jointly consider image features and their category information, trying to find a tradeoff between the overall reconstruction error and class purity. Fig. 1(b) shows the generated words (denoted as × ). Based on the supervised vocabulary, we can obtain histogram representations as shown in Fig. 2(c) and (d), from which we can easily identify classes A and B. Obviously, the supervised vocabulary is more discriminative to classify A and B. Please note that the category information is only used at training stage to obtain the vocabulary, and at testing stage, each feature is assigned to its nearest visual words (as Fig. 1(b) shows) using the traditional distance measurement without using category information. However, we can expect that the visual vocabulary obtained by the supervised method has high class purity for nonannotated testing dataset if it has similar feature distribution with the training dataset. Thus the supervised vocabulary is also discriminative for the testing dataset.
Fig. 1. A toy problem
Fig. 2. Histogram representation for toy problem
16
2.1
Y. Liu and V. Caselles
Supervised Vocabulary Construction
Let F = {f1 , f2 ..., fN } be the set of local image features in the training set, and each feature is represented by a d-dimensional vector, (d=128 for SIFT descriptor). And we assume that the class label of each image is also known so that we have Y = {y1 , y2 ..., yN } where yi is the class label corresponding to feature fi . Our goal is to partition the dataset F into k clusters for generating a vocabulary with k words. The k-means algorithm searches for a partition into k clusters so that the sum of squared errors between the empirical mean of each cluster and its corresponding points is minimized. The process can be formulated as the following constrained optimization problem: min W,V
C(W, V ) =
k n
wi,j d(fi , vj )
j=1 i=1
(1) subject to
k
wi,j = 1, wi,j ∈ {0, 1}
j=1
where W = [wi , j]nk is a partition matrix, V = {v1 , v2 ..., vk } is a set of cluster centers, d(x, y) = ||x−y||2 is the squared Euclidean distance between two vectors x and y, and vj is the mean of cluster j. Solving (1) is known to be an NP-hard problem. K-means solves it using alternating optimization strategy [11]. The main steps of k-means algorithm are as follows: 1. Select an initial V randomly or based on some prior knowledges. 2. Fix V and find the distribution for W minimizing C which is given by: 1, if d(fi , vj ) ≤ d(fi , vt ) for 1 ≤ t ≤ k wi,j = (2) 0, for t = j 3. Fix W and determine V so that C is minimized. Then vj are given by: n wi,j fi vj = 1=1 (3) n 1=1 wi,j 4. Repeat steps 2 and 3 until there is no changes for each cluster. The vocabulary constructed by k-means solely considers the statistics of local image patches without taking category information into account. Then it cannot capture the semantic relation between image features. As discussed above, this vocabulary is not optimal for classification tasks. Inspired by [12], we propose a technique to construct a supervised visual vocabulary by jointly considering image features and their class labels. We use a novel cost function in which a simple and effective dissimilarity measure is adopted to deal with category information. Moreover, we adopt a prototype-based approach which tries to find prototypes for clusters instead of using the means as in the k-means algorithm. The prototype, denoted as P = {(vj , uj ); j = 1, ..., k}, consists of two vector entries, the feature
Supervised Visual Vocabulary with Category Information
17
prototype V and the corresponding label prototype U = {u1 , u2 , ..., uk }. The feature prototype V is used to generate the visual vocabulary. In order to construct a supervised vocabulary, we modify the cost function of (1) as following: min
W,V,U
C(W, V, U ) =
subject to
k
k n
wi,l d (fi , pj )
j=1 i=1
(4)
wi,j = 1, wi,j ∈ {0, 1}
j=1
where d (fi , pj ) = d(fi , vj ) + λδ(yi , uj ), δ(yi , uj ) is the dissimilarity measure between two class labels x and y, defined as: 1, x = y δ(x, y) = (5) 0, x = y and λ is the weight parameter to adjust the influence of label dissimilarities. When λ equals to zero, the method reduces to the standard k-means, and it generates the category specified visual vocabulary when λ is large enough. Similar to k-means, we minimize C by using an alternating optimization strategy. Its main steps are as follows: 1. Select an initial prototype P randomly or based on some prior knowledges. 2. Fix P and find distribution for the partition matrix W minimizing C which is given by: 1, if d (fi , vj ) ≤ d (fi , vt ) for 1 ≤ t ≤ k wi,j = (6) 0, for t = j 3. Fix W and determine P so that C is minimized according to Lemma 1 below. 4. Repeat steps 2 and 3 until there is no changes for each cluster. In the first step, the prototypes (vj , uj ) are initialized within each category. And the number of prototypes is proportional to the number of features in the corresponding category. Specifically, suppose the expected of prototypes (or visual words) is k for the whole dataset, we first assign the number of prototypes for each category according to the number of feature in this catergory, and the total number of prototypes for all the categories equals to k. Once the number of prototypes for each category is determined, we randomly initialize the prototypes within each category. Lemma 1. Suppose that the partition distribution W is fixed, then the function C is minimized if and only if the prototype P is assigned as follows: n wi,j fi vj = 1=1 (7) n 1=1 wi,j uj = arg max(nc,j ) c
(8)
where nc,j is the number of features having the class label c within current cluster j under W .
18
Y. Liu and V. Caselles
Proof. Let Cf (W, V ) =
k
j=1
n
i=1
wi,j d(fi , vj ) Cy (W, U ) =
Then C(W, V, U ) =
k
j=1
n
i=1
k
j=1
n
i=1
wi,j δ(fi , vj )
wi,l d (fi , pj )
can be written as C(W, V, U ) = Cf (W, V ) + λCy (W, U ). Since Cf (W, V ) and Cy (W, U ) are nonnegative and independent of each other, minimizing C(W, V, U ) respect to V , U is equivalent to minimizing Cf (W, V ) and Cy (W, U ) with respect to V and U respectively. It is obvious that Cf (W, V ) is minimized when (7) holds, which is the same as that in the standard k-means. Moreover, Cy (W, U ) can be rewritten as: Cy (W, U ) =
k n j=1 i=1
wi,j δ(fi , vj ) =
k
(nj − nuj ,j )
(9)
j=1
where nj is the number of features in cluster j according W (notice that nj is fixed since W is fixed), and nuj ,j is the number of features with class label uj within the cluster j. Cy (W, U ) is minimized if and only if every nuj ,j is maximal any j = 1, , k. Therefore uj must be the class label with the largest number of features in cluster j.
3
Experiments
We evaluated the proposed supervised vocabulary for image classification task on two public datasets: MSRC-2 [4] and OT [13]. We first describe the implementation setup, and then give the comparison results. 3.1
Experimental Setup
For the two datasets, we use only the grey level information in all the experiments. The keypoints are obtained using dense sampling since the comparative results have shown that using a dense set of keypoints works better than sparsely detected keypoints in many computer vision applications [14]. Specifically, we compute keypoints on a dense grid with spacing d=7 both in horizontal and vertical directions. SIFT descriptors are computed at each patch over a circular support area with radius r=5. Classification is performed using KNN and SVM classifiers on both datasets. The RBF (radial basis function) kernel is used in the SVM classifier. Multiclass classification is obtained from two-class SVM using one-vs-all strategy. The parameters such as the number of neighbors in KNN and the regularization parameter c in SVM are determined using k-fold (k = 5) cross validation.
Supervised Visual Vocabulary with Category Information
3.2
19
Experimental Results
MSRC-2 In the experiments with MSRC-2, there are 20 classes, and 30 images per class in this dataset. We choose six classes out of them: {airplane, cow, face, car, bike, sheep}. The visual vocabulary size is set as 100. Moreover, we divided randomly the images within each class into two groups of the same size to form a training set and a testing set. We repeat each experiment five times over different splits. Fig.3 shows how the classification accuracy varied with the weight parameter λ using a SVM classifier. Observe that the best performance is obtained when λ = 1.0. Therefore, λ is empirically set to 1.0 in the subsequent experiments. Please note that the best value of λ will change when using different feature spaces to represent the image. It is recommend to set the λ experimentally. We report the comparison results in terms of classification accuracy in table 1 for KNN and SVM classifiers. The supervised vocabulary method consistently outperforms the vocabulary obtained using k-means for both classifiers. Moreover, the computation time of the supervised vocabulary generation is very close to that of k-means. On average, supervised vocabulary generation takes 1.79s for each iteration with 114840 features and 100 words, compared with 1.78s for k-means.
Fig. 3. Classification accuracy (%) plotted with λ Table 1. Classification accuracy (%) on MSRC-2
KNN SVM
k-means Ours 81.0 83.5 86.9 89.2
OT Dataset OT dataset contains 8 categories, {coast, forest, highway, insidecity, mountain, opencountry, street, tallbuilding}. We use the first 200 images per class in the experiment. Images within each class are randomly divided into two subsets of the same size to form a training set and a testing set. We set the visual vocabulary size as 200. Table 2 shows the averaged classification results over five experiments with different random splits of dataset. A confusion matrix for the supervised vocabulary when using the SVM classifier is presented in Fig.4
20
Y. Liu and V. Caselles
Fig. 4. Confusion matrix for supervised vocabulary Table 2. Classification accuracy (%) on OT
KNN SVM
k-means Ours 67.2 68.7 72.1 73.4
in order to give more details on the categorization of each category. The first column contains the true labels and the last row lists the referred labels.
4
Conclusions
This paper proposed supervised visual vocabulary construction approach which jointly considers image features and their class labels. A novel cost function is adopted to measure the dissimilarity of category information. The proposed method runs like the k-means algorithm to minimize the clustering cost function. The experiments on different datasets demonstrate that the proposed vocabulary construction method is effective for the application of image classification. Acknowledgements. This work was partially funded by Mediapro through the Spanish project CENIT-2007-1012 i3media and by the Centro para el Desarrollo Tecnol´ogico Industrial (CDTI). The authors acknowlege partial support by the EU project “2020 3D Media: Spatial Sound and Vision” under FP7-ICT. Y. Liu also acknowledges partial support from the Torres Quevedo Program from the Ministry of Science and Innovation in Spain (MICINN), co-funded by the European Social Fund (ESF). V. Caselles also acknowledges partial support by MICINN project, reference MTM2009-08171, by GRC reference 2009 SGR 773 and by “ICREA Acad`emia” prize for excellence in research funded both by the Generalitat de Catalunya.
Supervised Visual Vocabulary with Category Information
21
References 1. Sivic, J., Zisserman, A.: Video google: A text retrieval approach to object matching in videos. In: Proc. ICCV, vol. 2, pp. 1470–1477 (2003) 2. Lowe, G.: Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision 60(2), 91–110 (2004) 3. Jurie, F., Triggs, B.: Creating efficient codebooks for visual recognition. In: Proc. ICCV (2005) 4. Winn, J., Criminisi, A., Minka, T.: Object categorization by learned universal visual dictionary. In: Proc. ICCV, pp. 1800–1807 (2005) 5. Lazebnik, S., Raginsky, M.: Supervised Learning of Quantizer Codebooks by Information Loss Minimization. IEEE Transactions on Pattern Analysis and Machine Intelligence 31(7), 1294–1309 (2009) 6. Moosmann, F., Nowak, E., Jurie, F.: Randomized clustering forests for image classification. IEEE Transactions on Pattern Analysis and Machine Intelligence 30(9), 1632–1646 (2008) 7. Perronnin, F.: Universal and Adapted Vocabularies for Generic Visual Categorization. IEEE Transactions on Pattern Analysis and Machine Intelligence 30(7), 1243–1256 (2008) 8. Yang, L., Jin, R., Sukthankar, R., Jurie, F.: Unifying discriminative visual codebook generation with classifier training for object category recognition. In: Proc. CVPR (2008) 9. Zhang, C., Liu, J., Ouyang, Y., Tian, Q., Lu, H., Ma, S.: Category sensitive codebook construction for object category recognition. In: Proc. ICIP (2009) 10. Lian, X., Li, Z., Wang, C., Lv, B., Zhang, L.: Probabilistic models for supervised dictionary learning. In: Proc. CVPR (2010) 11. Jian, A., Dubes, R.: Algorithms for clustering data. Prentice Hall, Englewood Cliffs (1988) 12. Huang, Z.: Extensions to the k-Means Algorithm for Clustering Large Data Sets with Categorical Values. Data Mining and Knowledge Discovery, 283–304 (1998) 13. Oliva, A., Torralba, A.: Modeling the shape of the scene: a holistic representation of the spatial envelope. International Journal of Computer Vision 42(3), 145–175 (2001) 14. Bosch, A., Zisserman, A., Munoz, X.: Scene classification using a hybrid generative/discriminative approach. IEEE Transactions on Pattern Analysis and Machine Intelligence 30(4), 712–727 (2008)
Nonparametric Estimation of Fisher Vectors to Aggregate Image Descriptors Herv´e Le Borgne and Pablo Mu˜noz Fuentes CEA, LIST, Laboratory of Vision and Content Engineering 18 route du Panorama, BP6, Fontenay-au-Roses, F-92265 France
Abstract. We investigate how to represent a natural image in order to be able to recognize the visual concepts within it. The core of the proposed method consists in a new approach to aggregate local features, based on a non-parametric estimation of the Fisher vector, that result from the derivation of the gradient of the loglikelihood. For this, we need to use low level local descriptors that are learned with independent component analysis and thus provide a statistically independent description of the images. The resulting signature has a very intuitive interpretation and we propose an efficient implementation as well. We show on publicly available datasets that the proposed image signature performs very well.
1 Introduction Contemporary works on image classification (but also for object recognition and image retrieval) showed the efficiency of approaches based on the computation of local features (such as SIFT descriptors [14]).However, since the number of points of interest may vary from one image to another, the dimension of the vector representing an image (i.e number of keypoints × local feature size) is not constant, making it unusable as input of usual machine learning algorithms used in the image classification paradigm. This drawback was circumvented through the bag-of-visterm (BOV) approach [20]. It consists in pre-computing a codebook of visual words then coding each image according to this visual vocabulary. The simplest approach consists here in hard quantization that associate each local feature to one visual word(s) of the dictionary. Significant improvement is obtained by using a soft assignment scheme [5] or sparse coding [21,1]. The BOV signature can be refined by the spatial pyramid matching (SPM) scheme [10] that add spatial information at several scales. It consists of averaging the local features according to hierarchical regular grids over the image (average pooling) although recent work showed that one may benefit to consider their maximum instead (maximum pooling) [21,1]. An alternative to the BOV scheme was recently proposed to aggregate local descriptors. In [16], the visual vocabulary is modeled with a Gaussian mixture model (GMM) then a signature is derived according to the Fisher kernel principle, consisting in computing the gradient of the loglikelihood of the problem with respect to some parameters. It allows to take advantage of a generative model of data and discriminative properties at the same time. The VLAD descriptor [8] can be seen as a derivative of the Fisher kernel too . In practice, it consists of accumulating the difference between (each component of) the local descriptors and the visual words of the codebook. To be applied J. Blanc-Talon et al. (Eds.): ACIVS 2011, LNCS 6915, pp. 22–33, 2011. c Springer-Verlag Berlin Heidelberg 2011
Nonparametric Estimation of Fisher Vectors to Aggregate Image Descriptors
23
to image retrieval, these both signatures were compressed using principal component analysis and product kernel [8] or a simple binarization strategy [17]. The main contribution of this paper is to propose a non-parametric estimation of the Fisher kernel principle then derive an image signature that can be used for image classification.The main difference with the work of Perronnin [16,17] is that he used a parametric method (GMM) to estimate the density of the visual words. Hence, two parameters affect its performance: (i) the number of Gaussian component and (ii) the parameters with respect to which the gradient is computed. By using a non-parametric estimation of the Fisher kernel principle, our approach circumvent these issues, and may as well provide a more accurate model of the loglikelihood. However, the theoretical framework we proposed has to be applied to local features with statistically independent dimensions. Such descriptors were proposed in [11], who directly estimated the density of each dimension of the descriptors with parametric and non parametric methods. Our work differs from them since the non-parametric estimation is only a step in our framework, and the signature we finally obtain from the Fisher kernel derivation is different. A second contribution of this work is to propose an efficient implementation of the image signature obtained according to the proposed theoretical framework. We propose some choices to make the computation of the signature simpler than in theory and show that classification results are maintained or even improved. In section 2 we review the Fisher kernel framework, the non-parametric density estimation used and derive the proposed image signature. We present in section 3 some experimental results on several publicly available benchmarks and compare our approach to recent work in image scene classification.
2 Aggregating the Image Feature 2.1 Fisher Kernel, Score and Vector Let X = {xt , t = 1 . . . T } a set of vectors, for instance used to describe an image (e.g the collection of local features extracted from it). It can be seen as resulting from a generative probability model with density f (X|θ). To derive a kernel function from such a generative model, i.e being able to exhibit discriminative properties as well, Jaakola [7] proposed to use the gradient of the log-likelihood with respect to the parameters, called the Fisher score: UX (θ) = θ log f (X|θ) (1) This transforms the variable length of the sample X into a fixed length vector that can feed a classical learning machine. In the original work of [7] the Fisher information matrix Fλ is suggested to normalize the vector: Fλ = EX [θ log f (X|θ)θ log f (X|θ)T ]
(2)
It then results into the Fisher vector: −1/2
GX (θ) = Fλ
θ log f (X|θ)
(3)
24
H. Le Borgne and P. Mu˜noz Fuentes
2.2 Logspline Density Estimation The traditional way of modeling a distribution density is to assume a classical parametric model such as normal, gamma or Weibull. For instance in [16], the vocabularies of visual words are represented with a Gaussian Mixture Models, for which the parameters (weight, mean and variance of each Gaussian) are estimated by maximum likelihood. Alternatively, we can use a nonparametric estimate of the density, such as a histogram or a kernel-based method.A histogram density estimation can be seen as modeling the unknown log-density function by a piecewise constant function and estimating the unknown coefficients by maximum likelihood. In this vein, Kooperberg [9] proposed to model the log-density function by cubic spline (twice-continuously differentiable piecewise cubic polynomial), resulting into the so-called logspline density estimation. More precisely, given the lower bound of data L and upper one U (L and U can be infinite), and a sequence of K values t1 , . . . , tK such that L < t1 < . . . < tK < U (later referred as knots) and K > 2, we consider the space S consisting of the twicecontinuously differentiable function fs on (L, U ), such that the restriction of fs to each of the intervals [t1 , t2 ] . . . [tK−1 , tK ] is a cubic polynomial and linear on (L, t1 ] and [tK , U ). The functions of the K-dimensional space S are named natural (cubic) splines. Let 1, B1 , . . . , Bp (with p = K−1) a set of basis functions that span the space S, chosen such that B1 is linear with negative slope on (L, t1 ], B2 , . . . , Bp are constant on (L, t1 ], Bp is linear with positive slope on [tK , U ) and B1 , . . . , Bp−1 are constant on [tK , U ). Given θ = (θ1 , . . . , θp ) ∈ Rp such that: U exp (θ1 B1 (y)+, . . . , θp Bp (y)dy) < ∞ (4) L
We can thus consider the exponential family of distribution based on this basis function: f (y, θ) = exp (θ1 B1 (y)+, . . . , θp Bp (y) − C(θ)) Where C(θ)is a normalizing constant such that: f (y, θ)dy = 1
(5)
(6)
R
As shown in [9], it is possible to determine the maximum likelihood estimate of θ with a Newton-Raphson method with step-halving. They also proposed a knot selection methodology based on Akaike Information Criterion (AIC) to select the best model. It finally results into the estimation of the maximum likelihood estimate ˆθ of the coefficients, the knots t1 , . . . , tK and thus the wanted density. Let notice that at convergence (θ = ˆ θ), the loglikelihood is maximal, thus its derivative is null. Let {y1 . . . yn } a random sample from f . We have: n ∂L(Y, θ) ∂C(ˆθ) = 0 = Bj (yt ) − n (7) ∂θj ∂θj θ=ˆ θ t=1 Thus at convergence we have: ∂C(ˆ θ) = Ef [Bj (y)] ∂θj
(8)
Nonparametric Estimation of Fisher Vectors to Aggregate Image Descriptors
25
Where Ef [B(y)] denotes the Monte Carlo estimate of the expectation of B(.) according to f . This will be used later in our development. 2.3 Signature Derivation Let consider that any image is described according to some D-dimensional vectors. Each feature dimension xi can be thought of as arising as a random sample from a distribution having a density hi . We can model the log-density function by a cubic spline, such as explained in section 2.2. Hence, it exists a basis 1, B1i , . . . , Bpi i of S such that: ⎛ i ⎞ p hi (xi , θi ) = exp ⎝ θji Bji (xi ) − C i (θ i ))⎠ (9) j=1
Let {yt , t = 1 . . . T } a set of vectors extracted from a given image, seen as T independent realizations of the D-dimensional random vector Y (for simplicity, Y denotes both the image and the corresponding random vector). The log-likelihood is thus: L(Y, θ) =
T
log (h(yt , θ))
(10)
t=1
Where h(yt , θ) denotes the density of Y . If one assumes the independence of all feature dimensions (this point is discussed in section 2.6), we have: h(yt , θ) =
D
hi (yti , θi )
(11)
i=1
Thus: L(Y, θ) =
T D
log hi (yti , θi )
(12)
t=1 i=1
Each density hi can be estimated on the same basis as the one determined during learning. In other words, each hi is expressed as in (9) with specific value for the coefficients θji . Hence, from (12) it follows: T
∂L(Y, θ) i i ∂C i (θi ) = Bj (yt ) − i ∂θj ∂θji t=1
(13)
The first component of equation (13) is simply the expectation of Bji (y) according to the density of the considered image (i.e estimated from the samples {yt , t = 1 . . . T }). The second component is a function that depend on θji . If one assumes that the considered samples follow a probability law quite close to the one estimated during learning, we can apply equation (8), and finally:
∂L(Y, θ) = Ehi Bji (y) − Ef i Bji (y) (14) i ∂θj i i θj ≈ θj
26
H. Le Borgne and P. Mu˜noz Fuentes
Where hi (.) is the density of the image descriptor and f i (.) the density class descriptor (dimension i), this last being estimated from local descriptor extracted from several learning images. The full gradient vector UY (θ) is a concatenation of these partial D derivatives with respect to all parameters. Its number of components is i=1 pi , where pi is the number of non-constant polynomial of the basis of S for dimension i. The equation (14) leads to a remarkably simple expression, for which the physical interpretation is quite straightforward. It simply reflects the way a specific image (with density hi ) differs from the average world (i.e density f i ), through
a well
chosen polynomial basis, at each dimension (figure 1). The average world Ef i Bji (y) can be
seen as a sort of codebook. If one uses linear polynomials Bji (y) = αj y i , equation (14) relates to the VLAD signature, with an important difference since all vectors are used (i) during learning to estimate the codeword (ii) during test to compute the signature, while (i) K-means uses the closest vectors of a codeword (cluster center) to re-estimate it at each step (ii) VLAD uses as well only nearest neighbors to compute the signature component (see eq. (1) in [8]).
Bedroom
Suburb image
categ
image
categ
image
categ
image
categ
image
categ
Kitchen
Industrial image
categ
image
categ
image
categ
image
categ
image
categ
Livingroom
Forest
image
categ
image
categ
image
categ
image
categ
InsideCity
OpenCountry
Building
categ
Coast
Highway
Mountain
image
Street
Office
Store
Fig. 1. Example images from the scene15 database (1st column), with the response of an ICA filter (dimension i) to a particular image, i.e f i in equation (14) (2nd column) and the average density over the category, i.e hi in equation (14) (3rd column)
2.4 Signature Normalization In his seminal work, Jaakola [7] proposed to normalize the Fisher score by the Fisher information matrix. In [16], it was noted that such an operation improved the efficiency of the method in term of discrimination, by normalizing the dynamic range of the different dimensions of the gradient vector.
Nonparametric Estimation of Fisher Vectors to Aggregate Image Descriptors
27
To normalize the dynamic range of each dimension of the gradient vector UY (θ) (each UY (θji ) is given by the equation (14)), we need to compute the diagonal terms of the Fisher information matrix Fθ . Considering the expression of each UY (θji ) given by he equation (14), the diagonal terms of Fθ are:
2 Fθji = E Ehi Bji (y) − Ef i Bji (y) (15) The dynamic range being computed on the learning database (density f i ), it is thus the variance of Bji (y). From equation (3) the final fisher vector is: UYn (θji )
Ehi Bji (y) − Ef i Bji (y)
= σf i Bji (y)
(16)
Where σf i [.] is the standard deviation computed according to density f i . Hence, our normalized signature UYn (θ) can be regarded as the “standardizing” transformation of the raw description of the image given by the polynomial activity Bji (y). The “normal ity” is here the learning database, from which we compute an average Ef i Bji (y) and
a standard deviation σf i Bji (y) at each dimension. 2.5 Efficient Implementation We discuss in this section the choice of the knots t1 , . . . , tK and the set of basis functions 1, B1 , . . . , Bp (with p = K − 1) that span the space S defined in section 2.2, used to compute our signature according to equation (14). In his article on the logspline density estimation theory [9], Kooperberg proposed an automatic method to place the knots according to an AIC criterion. However, preliminary experiments have convinced us that such a process is not necessary, and that a simpler strategy is more efficient. For this, we fix a given number of shot and place them according to statistic order of the learning data. For instance, if K = 9, knots are places according to the decile of data. Hence, at each dimension, the amount of information is regularly distributed between knots. For low-level features such as those presented in section 2.6, the knots are approximately placed according to a logarithmic distribution. Once the knots (t1 , . . . , tK ) are fixed, a natural choice for the set of basis functions (1, B1 , . . . , Bp ) that span the space S is to consider the polynomials of the form: B0 (y) = 1 B1 (y) = y 3 k Bk (y) = |y−tk |+y−t for k > 1 2
(17)
With such a basis, B0 = 1 has no influence in the computation of the signature (see equation (14)), while B1 (.) leads to compute the difference of the mean between the considered image (density hi ) and the class (density f i ). Further polynomials Bk (.) are the positive part of a cubic function that is null at knot tk . Obviously, such a representation is strongly redundant. Hence, we propose two simplifications to this implementation. First, for polynomials Bk with k > 1, we only
28
H. Le Borgne and P. Mu˜noz Fuentes
consider the values y between knot tk and tk+1 , in order to avoid redundancy. The difference in equation (14) is thus computed on each interval defined by the knots. This can be seen as a sort of filter activity quantization, the limits of the cells being the knots. Secondly, for computational efficiency, we only consider the positive part (absolute value) without computing the power three. The chosen basis thus becomes: B0 (y) = 1 (not used) B1 (y) = y Bk>1 (y) =
|y−tk |+y−tk 2
0
for y < tk+1 for y > tk+1
(18)
Such an implementation is equivalent to compute only (y − tk ) on the interval [tk , tk+1 ] since the polynomial is null elsewhere and y > tk on the interval. The third proposed simplification is to use a binary weighting scheme. It consists in not considering the value of |y − tk | in the computation but only its existence. In other word, one can only count +1 each time a pixel activity y is between tk and tk+1 . Such a binary weighting scheme is commonly used in the design of BOV, in particular when the codebook is large [20]. 2.6 Independent Low Level Features According to equation (11) the signature derivation requires to use independent lowlevel features, such that the image description density could be expressed as a factorial code. Such features can be obtained with Independent Component Analysis (ICA)[2,6] that is a class of methods that aims at revealing statistically independent latent variables of observed data. In comparison, the well-known Principal Component Analysis (PCA) would reveal uncorrelated sources, i.e with null moments up to the order two only. Many algorithms were proposed to achieve such an estimation, that are well reviewed in [6]. These authors proposed the fast-ICA algorithm that searches for sources that have a maximal nongaussianity. When applied to natural image patches of fixed size (e.g Δ = 16 × 16 = 256), ICA results into a generative model composed of localized and oriented basis functions [6]. Its inverse, the separating matrix, is composed of independent filters w1 , . . . , wD (size Δ) that can be used as feature extractors, giving a new representation with mutually independent dimensions. The number of filters (D) extracted by ICA is less or equal to the input data dimension (Δ). This can be reduced using a PCA previously to the ICA. The responses of the D filters to some pixels (p1 , . . . , pT ) of an image I(.) are thus independent realizations of the D-dimensional random vector Y . As a consequence, the density can be factorized as expected: hica (I(pt )) =
D i=1
hiica (I(py )) =
D
wi ∗ I(pt )
(19)
i=1
Where ∗ is the convolution product. These independent low-level features can be further used according to the method presented into section 2.3 since they verify equation (11).
Nonparametric Estimation of Fisher Vectors to Aggregate Image Descriptors
29
3 Experiments 3.1 Datasets and Experimental Setup The first dataset (scene15, see figure 1) [10] was recently used in several works on scene classification [15,19,18,21,1]. It is composed of 4485 images with 200 to 400 images for each category. A given image belong to one category exactly. The original sources of the pictures include the COREL collection, personal photographs and Google image search. We followed the experimental setup of[10] using 100 images per class for training and the rest for testing. As well, only the gray level of the images is considered, making color-based descriptors inoperable on this dataset. The performance is measured using the classification rate to be comparable to previous works. The second dataset (vcdt08) was used as a benchmark in the Visual Concept Detection Task of the campaign ImageCLEF 2008 [3]. It is composed on 1827 images for training and 1000 for testing, all image being annotated according to 17 categories.The categories are not exclusive i.e an image can belong to several of these categories and sometimes to none of them. The performance is measures using the Equal Error Rate i.e the point such that the false acceptance rate is equal to the false rejection rate. Hence, this value is between 0 and 1 such that the smaller the EER the better the system: ideally, false acceptance and false rejection are null. Two machine learning algorithms were used in the following experiments. the first is a SVM with a linear kernel [4] that has a fast convergence. We consider the use of a linear kernel as relevant because our method lead to a large signature (from hundreds to thousands dimensions) and that in such a high dimensional space, data are quite sparse and it is thus easier to find an hyperplane that separate them. The second learning algorithm is the fast shared boosting (FSB) [12] that is specifically designed to tackle the problem of overlapping classes (such as in the dataset vcdt08), by using weak classifiers that shares features among classes. Another benefit of this algorithm is its fast convergence when the umber of classes grows. When images can belong to several classes, it was shown that it has usually better results than a classic one-versus-all strategy. 3.2 Signature Implementation We study the effect of the simplifications proposed in section 2.5 to efficiently implement the proposed signature. In this experiment, only D = 64 filters are extracted with fastICA [6] from 40000 patches of size 16 × 16. Multi-class classification on non-exclusive classes is done with the FSB. All the experiments in this section were conducted with 32 knots. The “normal” implementation uses the polynomial basis given at equation (17). Simplifications (section 2.5) include: (i) computing the polynomial between knots instead of above them (ii) use the absolute value instead of the third power (iii) in the case of the absolute value one can use a binarized weighting scheme instead of the values. Table 1 shows the performances of all these possible implementations on the vcdt08 benchmark. As expected, reducing information redundancy into the signature by computing it between knots improve the performances. Concerning the other simplification, no significant effect is noticed for signatures computed between knots. However,
30
H. Le Borgne and P. Mu˜noz Fuentes
Table 1. Performances on the vcdt08 benchmark for several implementations of our method (see text). The lower the EER, the better the method. Above knots |.|3 0.294 |.| val 0.286 |.| bin 0.256
Between knots 0.254 0.257 0.254
Table 2. Classification accuracy on scene15 for three different implementation (see text). The equation number is given for the centered and the normalized signature. The simple one is only the first part of equation (14). Signatures were computed with 32 knots and no grid nor pyramid. Signature
simple
Ef i Bji (y) 64 filters 74.20 128 filters 75.58 256 filters 78.56
centered normalized see (14) see (16) 74.30 73.92 75.71 76.08 78.86 78.42
when the signature is computed with the “above knots” implementation, the binarized weighting scheme allows to reach similar performances as those obtained with the nonredundant implementation. Other variation in the implementation are possible, since we proposed a centered signature (equation 14) and a normalized one (equation 16). We also consider here a simple signature that only implement the first member of the equation (14), i.e a non-centered version of it. Since of the dimension has few influence on the FSB [12], we tested these three signatures as input of a SVM, on the scene15 dataset (exclusive classes). As well, signatures were computed for three different sets of ICA filters, of size 64, 128 and 256, with 32 knots. Optimal parameters of the SVM were determined on the learning database, with a 5-fold cross validation, and data were previously scaled according to the method proposed by the author [4]. Results presented in table 2 show that results improve when more filters are used (see next section for a discussion on this point) but very few variation from one implementation to another. This is probably due to the data scaling of the SVM we used, that itself standardized the signatures. 3.3 Signature Parameters Above implementation details (previous section), the signature we propose mainly depends on two parameters, namely the number of filters D and the number of knots K. In this section we show the influence of these two parameters on the scene15 dataset. We used a simple signature and computed the polynomial activity between knots with a binarized weighting scheme. The signature size is D × K. Optimal parameters of the SVM were determined on the learning database, with a 5-fold cross validation, and data were previously scaled as proposed in [4].
Nonparametric Estimation of Fisher Vectors to Aggregate Image Descriptors
31
Table 3. Classification accuracy on scene15 for three filter basis of various size D and different number of knots K. Best result for a given D is marked in bold. K = 8 K = 16 D = 64 71.55 75.11 D = 128 71.82 75.88 D = 256 76.78 78.49
K = 32 74.20 75.58 78.56
K = 64 73.47 74.84 77.69
Results are presented in table 3. For a given number of knots K, the best results are always obtained with a maximal number of filters, that is consistent with the previous results. Indeed, with patches of size 16 × 16 the maximal number of filters is 256. To obtain a smaller collection of ICA filter, data is reduced with a PCA during the process (see section 2.6), thereby inducing a loss of information detrimental to the resulting descriptors. For a given number of filter D, the best results are obtained with K = 16 or K = 32. As explained above, knots can be regarded as the limits of quantization cells of the ICA filter activity. Hence, when the number of knots is large, cells are too selective (K = 64) while they generalize too much for small number of knots (K = 8). 3.4 Comparison to Other Works Table 4 shows the classification results for our method in comparison to recent works evaluated on the scene15 dataset. Note that we considered only signatures on the full image for fair comparison. A SPM scheme, dividing the image according to a spatial grid at several scales, usually allows to improve the performances [10,21,1]. We used the same filters as previously with K = 16 knots to build the signature at each dimension. For [10] the results reported here are those with the “strong features” (SIFT). When no spatial pyramid scheme is used, it is equivalent to a BOV approach. With 256 filters, without spatial pyramid scheme, our method achieve an accuracy of 78.5%, that is a gain between 2% and 6% over the best accuracies reported in the literature. Let also notice that with 64 filters our method achieve 75.1%, that is comparable to the state-of-the art. K Table 4. Classification accuracy for scene15. UD (ICA) is our method with D filters and K knots. At the 8 top rows are methods that do not use any spatial pyramid scheme. The bottom rows show the results of methods using such a technique.
Method Accuracy % Lazebnik et al. [10] 74.8 Liu et al. [13] 75.16 Rasiwasia et al. [19] 72.2 Rasiwasia et al. [18] 72.5 Masnadi-Shirazi et al. [15] 76.74 16 U64 (ICA) 75.10 16 U128 (ICA) 75.88 16 U256 (ICA) 78.49
32
H. Le Borgne and P. Mu˜noz Fuentes
Fig. 2. Average EER over the 17 categories of VCDT 2008: ranking of the proposed approach (white) with respect to the other 53 runs (blue)
The result we obtained on vcdt08 (table 1) are competitive as well. On this benchmark, [12] reported an EER of 0.24 while we achieve 0.254 with 64 filters and 32 knots. In figure 2 we compare the average EER of our approach with the one of the runs submitted to the campaign [3]. Although the best run reported performs better, our method is ranked among the best.
4 Conclusion We proposed a non parametric estimation of the Fisher Kernel framework to derive an image signature. This high-dimensional and dense signature can reasonably feed a linear SVM. We showed the effectiveness of the approach in the context of image categorization. Experiments on challenging benchmark showed our approach outperforms recent techniques in scene classification [10,13,15,19,18], even when a spatial pyramid scheme is used [10]. Hence we report the higher accuracy on this state-of-the-art dataset in scene classification. We also reported competitive results on the vcdt08 benchmark of the ImageCLEF campaign [3]. In future work we would be interested in applying the non-parametric Fisher vector estimation presented in this paper on local features that only partially verify the property of statistical independence (see [11]) or not at all [14]. Another direction of research will concern the signature compression in order to apply it to image retrieval. Acknowledgment. This work has been partially funded by I2S in the context of the project Polinum. We acknowledge support from the ANR (project Yoji) and the DGCIS for funding us through the regional business cluster Cap Digital (project Rom´eo).
References 1. Boureau, Y., Bach, F., LeCun, Y., Ponce, J.: Learning mid-level features for recognition. In: CVPR, San Francisco, USA (2010) 2. Comon, P.: Independent component analysis, a new concept? Signal Processing 36(3), 287– 314 (1994) 3. Deselaers, T., Deserno, T.: The visual concept detection task in imageclef 2008. In: ImageCLEF Workshop (2008)
Nonparametric Estimation of Fisher Vectors to Aggregate Image Descriptors
33
4. Fan, R.E., Chang, K.W., Hsieh, C.J., Wang, X.R., Lin, C.J.: Liblinear: a library for large linear classification. Journal of Machine Learning Research 9, 1871–1874 (2008) 5. van Gemert, J.C., Veenman, C.J., Smeulders, A.W.M., Geusebroek, J.M.: Visual word ambiguity. IEEE Transactions on Pattern Analysis and Machine Intelligence 32(7), 1271–1283 (2010) 6. Hyv¨arinen, A., Karhunen, J., Oja, E.: Independent Component Analysis. Wiley-Interscience, Hoboken (2001) 7. Jaakola, T., Haussler, D.: Exploiting generative models in discriminative classifiers. In: NIPS, pp. 1–8 (1999) 8. J´egou, H., Douze, M., Schmid, C., P´erez, P.: Aggregating local descriptors into a compact image representation. In: CVPR, San Francisco, USA (June 2010) 9. Kooperberg, C., Stone, C.J.: Logspline density estimation for censored data. Journal of Computational and Graphical Statistics 1, 301–328 (1997) 10. Lazebnik, S., Schmid, C., Ponce, J.: Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories. In: CVPR, Washington, DC, USA, pp. 2169–2178 (2006) 11. Le Borgne, H., Gu´erin Dugu´e, A., Antoniadis, A.: Representation of images for classification with independent features. Pattern Recognition Letters 25(2), 141–154 (2004) 12. Le Borgne, H., Honnorat, N.: Fast shared boosting for large-scale concept detection. Multimedia Tools and Applications (2010) 13. Liu, J., Shah, M.: Scene modeling using co-clustering. In: ICCV (2007) 14. Lowe, D.G.: Object recognition from local scale-invariant features. In: CVPR 1999, Los Alamitos, CA, USA, vol. 2, pp. 1150–1157 (August 1999) 15. Masnadi-Shirazi, H., Mahadevan, V., Vasconcelos, N.: On the design of robust classifiers for computer vision. In: CVPR, San Francisco, USA, pp. 779–786 (June 2010) 16. Perronnin, F., Dance, C.R.: Fisher kernels on visual vocabularies for image categorization. In: CVPR (2007) 17. Perronnin, F., Dance, C.R.: Large-scale image retrieval with compressed fisher kernels. In: CVPR, San Francisco, USA, pp. 3384–3391 (2010) 18. Rasiwasia, N., Vasconcelos, N.: Holistic context modeling using semantic co-occurrences. In: CVPR, Los Alamitos, CA, USA, pp. 1889–1895 (2009) 19. Rasiwasia, N., Vasconcelos, N.: Scene classification with low-dimensional semantic spaces and weak supervision. In: CVPR, pp. 1–6 (2008) 20. Sivic, J., Zisserman, A.: Video google: A text retrieval approach to object matching in videos. In: ICCV, vol. 2, pp. 1470–1477 (2003) 21. Yang, J., Yu, K., Gong, Y., Huang, T.: Linear spatial pyramid matching using sparse coding for image classification. In: CVPR (2009)
Knowledge-Driven Saliency: Attention to the Unseen M. Zaheer Aziz, Michael Knopf, and B¨arbel Mertsching GET LAB, Universit¨ at Paderborn, 33098 Paderborn, Germany {aziz,mertsching}@upb.de,
[email protected] http://getwww.uni-paderborn.de Abstract. This paper deals with attention in 3D environments based upon knowledge-driven cues. Using learned 3D scenes as top-down influence, the proposed system is able to mark high saliency to locations occupied by objects that are new, changed, or even missing from their location as compared to the already learned situation. The proposal addresses a system level solution covering learning of 3D objects and scenes using visual, range and odometry sensors, storage of spatial knowledge using multiple-view theory from psychology, and validation of scenes using recognized objects as anchors. The proposed system is designed to handle the complex scenarios of recognition with partially visible objects during revisit to the scene from an arbitrary direction. Simulation results have shown success of the designed methodology under ideal sensor readings from range and odometry sensors.
1
Introduction
Living cognitive systems perform various types of activities to perceive and react to their three dimensional environment. Attention towards selected signals coming into the senses help them in quickly responding to the most relevant and important items. Sharp contrast or some element of surprise in the available data lead to bottom-up attention [1][2]. On the hand, a vast amount of literature on psychophysics advocates dominance of top-down in this process of autonomous selection [3][4]. Computational modeling of the attention phenomenon has become a topic of research since more than a decade. Majority of efforts on attention modeling take the bottom-up saliency channels into account whereas higher level processes of top-down attention have been addressed only recently, in which knowledge-driven saliency is a rarely modeled aspect. This paper presents a system able to compute saliency using previously learned knowledge about 3D scenes. It makes theoretical advancements in terms of three contexts. Firstly, it introduces an attention mechanism able to perform attention in 3D space. Secondly, it includes knowledge-driven influence to compute top-down saliency. Thirdly, a multi-sensor mechanism has been devised to learn 3D objects by collecting their several 2D views through a mobile robotic vision system. Hence a complete system has been discussed that is able to learn 3D objects and scenes and, at a later visit to the learned scenes, detect salient locations based upon the changes occurred after the previous visit. J. Blanc-Talon et al. (Eds.): ACIVS 2011, LNCS 6915, pp. 34–45, 2011. c Springer-Verlag Berlin Heidelberg 2011
Knowledge-Driven Saliency: Attention to the Unseen
2
35
Related Literature
The system under discussion has roots in various disciplines. In order to be concise, selected literature is reviewed here that has conceptually close relation to the major contributions of the presented work. The fields worth mentioning here include role of recognition in human attention, efforts on recognition based artificial visual attention, psychological findings on recognition of 3D objects in brain, and work on 3D object analysis in machine vision. Role of cognition or recognition in selection of focus of attention in realworld has been established by research in psychology, e.g., [5] and [6]. The psychophysical experiments reported in [7] have shown memory-driven attentional capture on the basis of content-specific representations. In the context of learning and recognizing objects in 3D environments different theories exist about the brain functions. The theories suggesting incorporation of multiple 2D views into 3D models, e.g. [8], are simpler for implementation in vision systems of mobile robots. The psychological model in [5] suggests learning of canonical views for object representations. The work presented in [9] shows capability of learning three dimensional structure not only by looking at the target object or environment from different directions but by imagining to view it from these orientations as well. The ability of the brain to visualize a scene from an orientation in space that was not actually experienced by it is shown in [10]. A relation between visual attention and spatial memory is established in [11] with a conclusion that a detailed representation of the spatial structure of the environment is typically retained across fixations and it is also used to guide eye movements while looking at already learned objects. Formation of object representations by human vision through snapshots taken from different view angles is discussed in [12] and it is suggested that such procedure is followed only for the objects under visual attention while the unattended scene may be processed as a 2-D representation bound to the background scene as a texture. In machine vision, an analysis on importance of cognitive functions prior to focusing visual attention can be seen in [13]. Some efforts on establishing a relation between visual attention and object recognition already exist, for example [14] localizes and identifies objects using an adaptive color-based visual attention control algorithm and hierarchical neural networks for object recognition. Training of neural network for learning influences of task and context for constructing top-down saliency map was employed by [15] as well. In [16] prior knowledge of the target and distractors is used to compute the top-down gains. The bottom-up saliency maps are multiplicatively weighted by these top-down gains to yield the saliency map. A model incorporating familiarity based saliency can be seen in [17] in which top-down familiarity is computed using SIFT based object recognition. Handling representation and recognition of 3D objects has been a topic of research for some time. One of the recent proposals includes [18] with a learning approach that finds salient points on a 3D object and represents these points in a 2D spatial map based on a longitude-latitude transformation. Results of classification have been demonstrated on isolated objects. Use of canonical views has
36
M. Zaheer Aziz, M. Knopf, and B. Mertsching
been experimented in [19] where isolated objects are converted into representations having predefined size. Recognition of isolated 3D objects using partial views by artificial neural networks and semantic networks was presented in [20]. Merging 3D point cloud views from different observation directions have been used for unsupervised learning of objects in [21]. A similar work on recognition using partial views of 3D point clouds can be seen in [22]. The major aspect lacking in the existing 3D object analysis systems is their requirement of isolated 3D objects as input. The solution proposed makes a step ahead towards simultaneous object segregation and learning. Attention models applying top-down influences have shown success only in 2D scenes. The proposed system extends the scope of top-down attention towards three dimensional space and demonstrates some practical applications for mobile vision systems.
3
Proposed System
Both in learning and learned mode, the first major module of the proposed system include processing on input data from vision, range, and localization sensors to obtain color- and depth-wise segmented regions. This data is then arranged into view descriptors in the sensory memory to formulate a ground for 3D object representations (3DOR). Under learning mode, the 3DORs go to the short term memory (STM) through a data filtering and organization process in the sensory memory. The spatial relations between 3DORs of recognized objects are used to define semantics of a 3D scene. The learned knowledge in the STM is stored in the long term memory for permanent storage. When the system runs in the learned mode, it performs object recognition on the 3DORs fed into the sensory memory and tries to validate the learned scenes. This capability is utilized to detect saliency of spatial locations of some new, altered, or missing objects. The following subsections provides details of these processes. It may be noted that there can be a variety of options for each of the said processes. Since the main focus of this paper is a system level solution, in which each component is itself a research area, we have proposed simple solutions for each component to achieve the overall objectives of the whole system to a level sufficient for the proof of concept. 3.1
Multisensor Input and Preprocessing
The system requires visual input from camera and depth input from a reliable mechanism such as laser range sensor or time of flight camera. In the current status of the solution we have utilized a laser scanner calibrated with a camera by establishing correspondence between points of laser data and image pixels. Depth values are applied on homogeneously colored segments in the camera image to separate depth-wise different regions. Neighboring regions with smooth variation of depths are grouped together. Such groups in the camera image represent 2D views of 3D objects from a particular observing position. A mobile robot can collect many such views after moving around a given object.
Knowledge-Driven Saliency: Attention to the Unseen
3.2
37
View Descriptors
Views represent the basic information unit in the proposed 3D object recognition process. A view includes data about a 3D object’s appearance from a specific angle through the camera’s field of view. It is possible to have more than one views in the sensor input at a time when multiple objects are visible. Scale and, to some extent, rotation invariance is among the desired attributes of view descriptors to tolerate distance of camera from the objects and input taken when the robot stands on an uneven ground. The view descriptors used in the presented status of the system consist of weighted average and standard deviation of hue (h), saturation (s), intensity (i), eccentricity (e), and orientation (o) of the regions formulating the view. Physical width, height, and area (actual number of pixels covered by the regions in the view) are also included as additional features. The physical width and height is computed using the measurements from the camera image and the distance obtained from the range sensor. Hence the view descriptor has the following format: v = {μh , μs , μi , μe , μo , σh , σs , σ i , σ e , σo } For a view consisting of the set of regions R the attribute μh is computed using the hue values of the involved regions as |R| Ai hi μh = i=1 |R| i=1 Ai where Ai is the area and hi is the hue component of color of the considered region. Hence the larger regions in the view will contribute more in the final attribute value. The standard deviation is also weighted by the region region, hence |R| Ai (hi − μh ) h σ = i=1|R| i=1 Ai Magnitudes of other components of the view descriptor are computed in the similar fashion. 3.3
3D Object Construction Using Views
A robot can obtain many view descriptors of an observed 3D object by moving around it. The proposed view-based object representation consists of a set of pairs, each of which consists of an orientation vector pointing to the viewed object from the robot’s location coupled with the view descriptor, i.e., O = {(φ1 , v1 )...(φn , vn )} The first view of an object is taken as reference for that object and considered as the origin for orientations of other views. The odometry sensor is used to know
38
M. Zaheer Aziz, M. Knopf, and B. Mertsching
(a)
(b)
(c)
Fig. 1. (a) Four samples of the orientation vectors from which the example object was viewed to construct its view-based 3D representation. (b) Three vectors with starting from another first view as origin orientation. (c) Aligning and merging of the two object representations.
the robot transformations and to localize it with respect to the origin orientation vector. Figure 1(a) shows four of the orientation vectors from which the system viewed the cubic object, vi being the one taken as reference. Each movement of the robot around the observed object can cause a change in the acquired view descriptors. Storing all these descriptors is not desirable. Using fixed intervals of orientation can lead to redundant storage of similar descriptors at one hand while missing some important one on the other. The proposed solution is to continue the robot’s movement around the object until the current view descriptor vi differs from the previously sensed vi−1 one by a certain amount, i.e. the distance Δ(vi , vi−1 ) < τd . 3.4
Learning of Objects into Knowledge
We propose to accomplish the learning using three levels of memories. The first level consists of sensory memory that remembers the last descriptor vi−1 along with the localization data of vi−1 and the robot itself. The current vi is compared with vi−1 to filter out non-useful descriptors. Useful vi are stored into the shortterm memory (STM). Contents of the STM are stored into long term memory (LTM) for use after restart of the robot. Location information is not stored in STM and LTM because objects are to be recognized without reference to their locations. It is likely that an object may go out of sensor range and come back in view from another perspective due to wandering of the robot. Hence, a mechanism has to be provided for extending learned object representations by merging new ones into them. Consider the situation shown in figures 1(a) and 1(b) where the robot had collected an object representation with four vs into STM and now another one is available with three vs. In order to merge these two sets the first question is to find the transformation to align them together. For this purpose we define a concept of dominator view descriptors. Two consecutive view descriptors {va , vb } in an object Oi will be called dominant on vc ∈ Oj if the distances Δ(va , vc ) and Δ(vb , vc ) both remain below a threshold τd . In the example shown in figure 1 vector vm is dominated by {vk , vl }.
Knowledge-Driven Saliency: Attention to the Unseen
39
As a first step to find the dominating pairs we pick the elements of the lager object representation that have a small vector distance from some element of the smaller one into a set of matches M (Oi , Oj ). Hence, for Oi and Oj such that |Oi | ≤ |Oj | M (Oi , Oj ) = {va |va ∈ Oj ∧ Δ(va , vc ) < τd ∀vc ∈ Oi } vc with minimum distance is picked in case of more than one vc with Δ(va , vc ) < τd . Then a set D(Oi , Oj ) is constructed having pair of dominators for each vc ∈ Oi as D(Oi , Oj ) = {(vcx , vcy )|vcx , vcy ∈ M (Oi , Oj ) ∧ Θ(φ(vcx ), φ(vcy )) ≤ τθ } where Θ(., .) computes the minimal angle between orientation of view descriptors delivered by φ(.). An upper bound τθ has been applied to avoid accepting a pair with very wide angle as dominator pair. For example (vax , vay ) for visually similar faces pointing to opposite directions in space cannot be accepted as dominators for a given vc . The set of dominator pairs is used to find the transformation required by each vc ∈ Oi to align it with a best matching descriptor from its dominator pair. These transformation candidates are collected in a set T (Oi , Oj ) as T (Oi , Oj ) = {(vc , t)|t = Θ(φ(vcz ), φ(vc )) s.t. Δ(vcz , vc ) = min (Δ(vcx , vc ), Δ(vcy , vc ))}
It may be noted that without the above mentioned step-wise refinements a vc ∈ Oi could try to align itself any visually similar va ∈ Oj leading to high probability of merging errors. Ideally all members of T (Oi , Oj ) should have the same value of transformation but practically it may not be the case. T (Oi , Oj ) is filtered to obtain the candidate values for transformation between Oi and Oj by picking those members from T (Oi , Oj ) that occur more than τn times, i.e., CT = {t|(vc , t) ∈ T (Oi , Oj ) ∧ N (t, T (Oi , Oj )) > τn } where N (., .) counts elements in the given set having similar values to the first parameter. Mean of the elements collected in CT is taken as the transformation magnitude to align Oi to Oj . The transformed version OiT = μ(CT )Oj is merged into Oj by inserting new elements of OiT into Oj while maintaining an order of orientations for view descriptors. 3.5
Object Recognition in 3D
For a mobile vision system, most of the parts of viewed objects are hidden, while looking from a particular observation direction, due to occlusions by other objects or hidden back faces. It is not desirable that all sides should be examined before recognizing a 3D object. The object representation mechanism proposed in the previous section retains sufficient information to recognize a learned object when sensed through an arbitrary observation direction or under minor occlusions. The simplest recognition criterion would be to search a single observed view into the learned knowledge using the merging process described above. A more confident recognition would be available when a system could collect two or more views that could fulfill the merging criteria.
40
M. Zaheer Aziz, M. Knopf, and B. Mertsching
3.6
Learning of 3D Scenes
A scene comprises of a specific configuration of objects. A structure to learn scenes requires references to the learned object representations and spatial relations between these objects. A graph based approach in adopted for extracting and storage of spatial relations. A scene representation graph Si = {O, E, β} consists of nodes O referring to the involved objects, edges E ⊆ O × O between the objects and 3D displacement vectors β : E → R3 assigned to each E. The object recognition mechanism mechanism provides input for nodes O. The displacement vectors β depends upon accuracy of odometry and range sensor data that lead to absolute locations in world coordinates. The sensory memory stores this information and establishes an edge E between the involved objects as soon as more than one entries are available to it. Since a mobile robot faces dynamic scenarios in which objects may appear and get occluded with the passage of time, the established graph may may rapidly change especially during scene learning phase. At the time of merging two object representations their respective nodes are merged, the edge between them are deleted, and the edges from other objects to these are updated to point to the merged one. It is notable that scenes can be learned even with partially learned objects. 3.7
Knowledge-Driven Saliency
The proposed mechanisms for object and scene representations provide an adequate base for computation of familiarity-based saliency in 3D environments. Top-down saliency could be determined using learned knowledge of objects and scenes in two scenarios. Firstly, it may be desirable to let know objects popout in an unlearned environment, e.g. known landmarks should attract attention in surrounding distractors. On the other hand in some applications unknown or unrecognizable objects should be treated as more salient. This type of scenarios include attention to objects in a known scene that have gone through alteration, have been replaced by another object, are are missing from their place. In the current status of the system we compute the saliency keeping the above mentioned two scenarios separate from each other. For OS being the object representations available in the sensory memory and OM the set of learned objects in the STM, the familiarity-based saliency of an object OiS would be ψ f = 1 − Δ (OiS , OkM ) where Δ (OiS , OkM ) = min(Δ (OiS , OjM )∀OjM ∈ OM ) Here Δ (., .) computes the normalized vector difference between the given objects (0 ≤ Δ (., .) ≤ 1). The saliency based upon change detection is computed using the learned scenes. For this purpose a scene configuration is selected from the STM using the recognized objects in the current view that serve as anchors to activate the relevant scene configuration S a (O, E, β). Saliency ψ l of a location
Knowledge-Driven Saliency: Attention to the Unseen
41
pointed by displacement vectors βik (S a ) associated with edges Eik (S a ) of the recognized anchor object Ok (S a ) is computed as ψl = Δ (Ol , Oik (S a )) where Ol is the object visible at location pointed by βik (S a ) and Oik (S a ) is the object expected at location l according the stored knowledge. It is obvious that a high saliency will be signaled when Ol is empty. The saliency of objects OiS available in sensory memory with respect to the active scene configuration S a is computed as ψis = 1 − Δ (Ois , L(Oj(S a ))) where L(.) picks the object at the same location as Ois from the active scene configuration. Combining the saliencies ψ l and ψis based upon the objects actually sensed in the current observation provides a saliency map that indicates locations where some got added, changed, or is missing as compared to the known configuration around the anchor object(s).
4
Results
It is apparent that the system discussed in this paper involves integration of many modules that are currently under research by robotics community. The sensor noise and accumulation of related errors are among some of the serious problems to be tackled. Since the proposed system relies upon ideal solutions for robot localization and pose estimation we report here experiments done in a robot simulation framework in order to concentrate on the conceptual testing without being distracted by branch issues. The robot simulation framework SIMORE [23][24] is able to simulate robots will fully functional virtual cameras, laser range sensor, and localization feedback. The robots can interact with their virtual environment with simulated physical forces. Hence experiments performed in this framework are equivalent to doing them in real world with the advantage that the sensor data is noise free. Hence this experimentation platform provides sufficient feedback for the proof of concept of the current state of the proposed system. Secondly, it is possible to show results in scenes with controlled complexity using this framework in order to make a clear demonstration of the contributions of this work. Handling issues related to sensor noise and localization inaccuracies in planned as part of future work. Figure 2(a) demonstrates a snap shot of the experiment in which the virtual robot drives around in a scene with two objects. The beams of laser scanner intercepting with the objects are also visualized by the simulation framework. The view seen by the robot through its camera is shown in figure 2(b). Figure 3(a) shows maneuver of the system in learning mode in a scene with three objects. In figure 3(b) it can seen that the system has marked two already learned objects as known, represented by green rectangles and crosses on the
42
M. Zaheer Aziz, M. Knopf, and B. Mertsching
(a)
(b)
Fig. 2. Experimentation platform used to obtain results of the current state of the proposed system (a) A simulated robot roaming in a virtual reality scene with visualization of its laser range scanner beams. (b) Camera input as seen by the robot.
(a)
(b)
(c)
Fig. 3. Familiarity-based saliency detected by the proposed system (a) The system observes a scene with two known and one unknown objects. (b) Regions involved in the view descriptors of known objects marked by the system using green rectangles and crosses. (c) Saliency map showing high conspicuity of unknown object among known ones.
region involved in the view descriptors. The third object is yet unknown hence its familiarity-based saliency (ψ f ) is high, as seen in figure 3(c). Figures 4, 5, and 6 demonstrate samples from experiments that show the ability of the system to detect changes in a learned scene (the scene shown in figure 3(a)). It can be observed in figure 4 that the left most object (cuboid with blue stripes) is replaced by a yellow object. The system approaches the scene from almost the opposite direction as that of figure 3(a) and it is able to recognize the central object (having red and yellow stripes) as anchor. Using this anchor it can detect the changed object and mark it salient for attention. In figure 5, an additional object has been placed in the scene. The system approaches the scene from an arbitrary direction and recognizes two known objects and marks them as anchors. The newly added object is detected on scene validation and is marked as salient by the system. On the other hand, In figure 6 demonstrates the case where one of the objects (the middle one with red and yellow stripes) is missing from the scene known to the system (figure 3(a)). The system is approaching the scene from yet another direction. After recognizing
Knowledge-Driven Saliency: Attention to the Unseen
(a)
(b)
43
(c)
Fig. 4. (a) The system observes an already known scene (the one shown in figure 3(a)). (b) A known object recognized (striped one at right) using view descriptors of current observation and marked as anchor. (c) Already learned scene recognized using anchor object and the replaced object detected as salient.
(a)
(b)
(c)
Fig. 5. (a) Observation of an already known scene (the one shown in figure 3(a)) from a different view direction. (b) Known objects recognized (left and right ones) and marked as anchors. (c) Already learned scene recognized using anchor object and an added object detected as salient.
(a)
(b)
(c)
Fig. 6. (a) Observation of the scene shown in figure 3(a) from a different view direction. (b) A known object recognized and marked as anchor. (c) Already learned scene recognized using the anchor object and a missing object detected as salient.
the visible object as anchor it expects the missing object to be visible in its sensor range. Failing to do so it detects the missing object and marks is location as salient in the saliency map.
44
5
M. Zaheer Aziz, M. Knopf, and B. Mertsching
Conclusion
The topic of knowledge driven saliency has been addressed in this work with a system level perspective. Methodology is developed to learn 3D objects and then scenes formed by these objects. The knowledge is managed by sensory, short-term, and long-term memories. The designed structure allows recognition at a later visit from an arbitrary direction and detects saliency with respect to the occurred changes. Results with ideal sensor conditions are very promising and show success of the developed solution. The proposed approach is novel in three contexts, namely, visual attention in 3D environments by mobile vision systems, application of knowledge for finding top-down saliencies, and a learning mechanism for 3D allowing recognition after viewing from arbitrary direction at revisit. Future work in this direction include dealing with the complexity of real-life scenarios and noisy sensor data. Acknowledgment. We gratefully acknowledge the funding of this work by the German Research Foundation (DFG) under grant Me 1289/12-1(AVRAM).
References 1. Treisman, A.M., Gelade, G.: A feature-integration theory of attention. Congnitive Psychology 12, 97–136 (1980) 2. Posner, M., Snyder, C., Davidson, B.: Attention and the detection of signals. Journal of Experimental Psychology: General 109(2), 160–174 (1980) 3. Chen, X., Zelinsky, G.: Real-world visual search is dominated by top-down guidance. Vision Research 46(24), 4118–4133 (2006) 4. Chen, X., Zelinsky, G.J.: Is visual search a top-down or bottom-up process? Journal of Vision 6, 447 (2006) 5. Blanz, V., Tarr, M., B¨ ulthoff, H., Vetter, T.: What object attributes determine canonical views? PERCEPTION-LONDON 28, 575–600 (1999) 6. Endres, D., Vintila, F., Bruce, N., Bouecke, J., Kornprobst, P., Neumann, H., Giese, M.: Hooligan detection: the effects of saliency and expert knowledge. Perception 39 ECVP Abstract Supplement 39, 193 (2010) 7. Olivers, C., Meijer, F., Theeuwes, J.: Feature-based memory-driven attentional capture: Visual working memory content affects visual attention. Journal of Experimental Psychology 32(5), 1243–1265 (2006) 8. Palmer, S.: Vision science: Photons to phenomenology, vol. 2. MIT Press, Cambridge (1999) 9. Oman, C.M., Shebilske, W.L., Richards, J.T., Tubr´e, T.C., Bealli, A.C., Natapoffi, A.: Three dimensional spatial memory and learning in real and virtual environments. Spatial Cognition and Computation 2, 355–372 (2000) 10. Shelton, A.L., Mcnamara, T.P.: Spatial memory and perspective taking. Memory & Cognition 32, 416–426 (2004) 11. Aivar, M.P., Hayhoe, M.M., Chizk, C.L., Mruczek, R.E.B.: Spatial memory and saccadic targeting in a natural task. Journal of Vision 5, 177–193 (2005) 12. Hoshino, E., Taya, F., Mogi, K.: Memory formation of object representation: Natural scenes. In: Wang, R., et al. (eds.) Advances in Cognitive Neurodynamics, pp. 457–462 (2008)
Knowledge-Driven Saliency: Attention to the Unseen
45
13. Follet, B., Le Meur, O., Baccino, T.: Modeling visual attention on scenes. Studia Informatica Universalis 8, 150–167 (2010) 14. Fay, R., Kaufmann, U., Markert, H., Palm, G.: Adaptive visual attention based object recognition. In: Proceedings of the IEEE SMC UK-RI Chapter Conference (2005) 15. Rasolzadeh, B., Tavakoli, A.T., Eklundh, J.O.: Attention in cognitive systems. theories and systems from an interdisciplinary viewpoint, pp. 123–140. Springer, Heidelberg (2008) 16. Navalpakkam, V., Itti, L.: An integrated model of top-down and bottom-up attention for optimizing detection speed. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 2, pp. 2049–2056. IEEE, Los Alamitos (2006) 17. Lee, S., Kim, K., Kim, J., Kim, M., Yoo, H.: Familiarity based unified visual attention model for fast and robust object recognition. Pattern Recognition 43, 1116–1128 (2010) 18. Atmosukarto, I., Shapiro, L.: A learning approach to 3D object representation for classification. Structural, Syntactic, and Statistical Pattern Recognition, 267–276 (2010) 19. Denton, T., Demirci, M., Abrahamson, J., Shokoufandeh, A., Dickinson, S.: Selecting canonical views for view-based 3-D object recognition. In: Proceedings of the 17th International Conference on Pattern Recognition, ICPR 2004, vol. 2, pp. 273–276. IEEE, Los Alamitos (2004) 20. B¨ uker, U., Hartmann, G.: Knowledge-based view control of a neural 3-D object recognition system. In: ICPR, vol. 4, pp. 24–29. IEEE, Los Alamitos (2002) 21. Ruhnke, M., Steder, B., Grisetti, G., Burgard, W.: Unsupervised learning of 3d object models from partial views. In: ICRA 2009, pp. 801–806. IEEE, Los Alamitos (2009) 22. Salamanca, S., Cerrada, C., Adan, A., Cerrada, J., Ad´ an, M.: Free-Shaped Object Recognition Method from Partial Views Using Weighted Cone Curvatures. Progress in Pattern Recognition, Image Analysis and Applications, 222–232 (2005) 23. Kutter, O., Hilker, C., Simon, A., Mertsching, B.: Modeling and simulating mobile robots environments. In: 3rd International Conference on Computer Graphics Theory and Applications (GRAPP 2008), Funchal, Portugal (2008) 24. Kotth¨ auser, T., Mertsching, B.: Validating Vision and Robotic Algorithms for Dynamic Real World Environments. In: Second International Conference on Simulation, Modeling and Programming for Autonomous Robot, SIMPAR (2010)
A Comparative Study of Vision-Based Lane Detection Methods Nadra Ben Romdhane1, Mohamed Hammami2, and Hanene Ben-Abdallah1 1
MIRACL-FSEG, Sfax University, Rte Aeroport Km 4, 3018 Sfax, Tunisia
[email protected],
[email protected] 2 MIRACL-FS, Sfax University, Rte Sokra Km 3 BP 802, 3018 Sfax, Tunisia
[email protected]
Abstract. Lane detection consists of detecting the lane limits where the vehicle carrying the camera is moving. The aim of this study is to propose a lane detection method through digital image processing. Morphological filtering, Hough transform and linear parabolic fitting are applied to realize this task. The results of our proposed method are compared with three proposed researches. The method presented here was tested on video sequences filmed by the authors on Tunisian roads, on a video sequence provided by Daimler AG as well as on the PETS2001 dataset provided by the Essex University. Keywords: Driver assistance system, Lane detection, Hough transform, Morphological filtering.
1 Introduction In the majority of the cases, the lane markings exist on both sides of the road, in others, the road is not marked. The lane detection consists of detecting the lane marking limits. This task is relatively easy when the texture of the road is uniform and when the lane presents very clear markings. However, it becomes non trivial particularly with the presence of various markings (cf., Figure 1.a), the presence of obstacles that cover lane markings (cf., Figure 1.b), the effects of weather conditions (cf., Figure 1.c) and the time of acquisition (cf., Figure 1.d). Its difficulty gets further accentuated in cases of camera movements, blur effects, and abrupt or unexpected actions of conductors, which are likely to distort the result of detection. Hence, all these circumstances must be considered to ensure the effectiveness of lane detection; this is the objective of our presented approach.
(a)
(b)
(c)
Fig. 1. Different environment conditions J. Blanc-Talon et al. (Eds.): ACIVS 2011, LNCS 6915, pp. 46–57, 2011. © Springer-Verlag Berlin Heidelberg 2011
(d)
A Comparative Study of Vision-Based Lane Detection Methods
47
In this paper, we are interested in marked roads where markings are presented in the form of two parallel linear or curved ribbons, with continuous or dashed lines. The remainder of this paper is organized as follows: Section 2 describes a state of the art of the lane detection step. Section 3 details our proposed method. Section 4 presents a set of experimental results and performance evaluations. Section 5 concludes by summarizing the major contributions of the present work.
2 State of the Art The aim of lane detection and tracking based on image processing is to locate, in the first sequence frame, the Lane Limits (LL) of the road in which the vehicle is engaged and then to track these limits in the remaining frames. Various lane detection methods have been proposed in the literature. These methods can be broadly grouped in two approaches, namely the model-based approach [5-10] and the feature-based approach [11-15]. 2.1 Model-Based Approach The methods in this category detect lane limits based on the assumption that the form of a given lane can be modeled by a curve that can be linear, parabolic etc. These methods follow a top-down step to detect the lane limits. Thus, the detection consists in searching the adequate curve parameters to be modeled while using parametric or explicit models. Parametric models. The widely used algorithm in lane detection applies the Hough transform on approximate pixels extracted by a gradient operator. Yeh [6] applies the Hough transform on oblique image edges. On the other hand, Collado [12] apply the Hough transform, he fixes the orientation ranges to limit the number of lines to be detected. The two required lines that refer respectively to the right and to the left sides are those having the smallest and the greatest value of the orientations. With the Hough transform model, the LL is detected very quickly. Nevertheless, it is characterized by a weak detection precision since it only represents the linear portions of the limits. To overcome this problem, various algorithms were proposed to detect slightly curved lane based on a local detection of the LL instead of the global detection. In particular, Tian [10] divides the lower half of the image contour into five sections of decreasing width from the bottom to the top, and then applies the Hough transform in each section. As for Jung [9], he divides horizontally the image into two regions corresponding to the near field and the far field. Based on the near and far region delimitation, we presented in our previous work [21], a lane detection algorithm that is based on Hough transform and linear parabolic fitting. Explicit models. In opposition to parametric models, with explicit models, it is possible to detect arbitrary shapes of roads. The majority of these models such as deformable and recursive based models are initialized with points positioned either on road boundaries or on lane markings. Deformable models are complex models, but they have a certain degree of freedom in the evolution compared to parametric models. They require an approximate initialization of the model on the segmented image. Then, the model evolves repeatedly in order to minimize an energy function. The
48
N. Ben Romdhane, M. Hammami, and H. Ben-Abdallah
model stops when a specified criterion is reached. Researches [10, 11] rely on interpolation by B-snake to model linear and curved portions of the road starting from the image edges. They simplify the problem of detecting the two lane boundaries to the detection of the middle of the lane. After the approximate pixels extraction step, they apply the Hough transform to different sections of the image to take into account the curved roads. They use the position of the vanishing point (intersection of the right and left limits) detected in each section to determine the middle pixels of the lane. These pixels will be adjusted by applying the deformable model. A recursive method in detecting lane markings is proposed by Aufrère [28]. He describes the LL using the positions of the left and right boundaries of the road in several horizontal lines of the image. He first of all delimits the region of interest which has a trapezoid form, and then he detects the lane marking segments by searching the pixel having the maximum value of the gradient for each line. He defines thereafter the two new regions of interest. The updated model will be initialized close to the detected segment and the algorithm will be stopped based on a depth criterion. In spite of explicit models effectiveness in modelling various road shapes, the segmentation result depends enormously on the initialization of these models. Moreover, they are characterized by their great sensitivity to noises and they require considerable execution time which makes these methods inadequate for real-time systems. 2.2 Feature-Based Approach The feature-based methods detect lane limits based on characteristics extracted from marking regions or from marking edges. These limits can be detected using different methods, such as geometric filtering methods, supervised training methods, or tracking methods. Rule-based method. The rule-based methods consist in searching the forms which are similar to those of lane marking limits according to user-defined classification rules. D'Cruz [18] uses the position characteristics to extract the required regions. He adjusts the retained pixels by a quadratic linear function. Cheng [19] uses colour for the image segmentation, and then he calculates the form, geometry and motion characteristics to eliminate false detections. As for Yim [29], he extracts the characteristics of the starting position, the direction and the intensity. These methods are conceptually fast and simple but they are not general. They are specific to the conditions defined by the user. Supervised training methods. The supervised training methods produce automatically the classification rules based on a training image dataset containing examples of processed cases. Gonzalez [30] extracts the position, form and geometry characteristics from the segmented image (the size of each segmented object, the gravity centre, the orientation, the maximum width, etc). Then, he selects the marking areas from the segmented regions based on a decision tree classifier. The author detects the LL by adjusting the gravity centres of the retained regions by a quadratic linear function. Kim [15] extracts 81 parameters, from small windows, related to the RVB components. Then he applies a multi-layer perceptron neural network to classify the retained regions into markings and non-markings.
A Comparative Study of Vision-Based Lane Detection Methods
49
Tracking methods. The basic idea of the tracking methods is to consider iteratively, from a starting point, different segments of a defined length and having different directions. In each iteration, only the segment which corresponds best to the required structure based on a given criteria is kept. Authors [15, 27] apply the Markov process to link the segments retained in the approximate pixels extraction step using a morphological gradient operator method. The aim of this process is to predict the position of a following segment knowing only the position of a current one. In each iteration, they consider different segments of a given length and oriented in different directions compared to the initial direction’s segment. The segment which corresponds the best to the LL is kept and added to the list of the pixels of the tracked limit.
3 A New Method for Lane Detection This section describes our proposed lane detection method. Our process is illustrated in Figure 2. Pre-processing • Delimitation of the region of interest • Median filtering • Contrast enhancement
Extraction of approximate pixels • Image segmentation • Edge pixels extraction Lane detection • Detection in near lane region • Detection in far lane region
Fig. 2. Lane detection process
In the pre-processing step, we apply a list of transformations to delimit the region of interest and to improve the contrast of the image. For the extraction of approximate pixels step, we first segment the image to extract the approximate regions of lane limit’s markings, and then we select their edge pixels. In the lane detection step, as the road may be formed by linear and curved portions, we present the techniques we apply to detect the corresponding lane markings. 3.1 Pre-processing In the pre-processing step, we treat the image to restrict the region of interest and improve the contrast of the image. We resize each processed image to half of the initial
50
N. Ben Romdhane, M. Hammami, and H. Ben-Abdallah
size, and then delimit our region of interest to the lower part which occupies third of the image width. We delimit two rectangular regions to detect the linear portions of the road: ROIr to detect the right limit, and ROIl to detect the left limit (cf., Figure 3.a). In some cases, the presence of shadows, lights and object reflections as well as blurred lane markings is noticed in the image. For deal with these cases, and in order to improve the result of approximate pixel extraction, only the pixels belonging to the clear areas are extracted from each region of interest, since lane markings areas are clear. Hence, we proceed according to the following three steps: 1) We apply a median filtering on the gray level image given that it is the most used method to reduce impulsive noise while preserving the sharp edges. 2) We apply the top hat transformation to extract the clear regions regardless of background variations; thus, the extraction of lane markings will not be affected by the presence of strong shadows or illuminations. The result of top hat transformation is illustrated in Figure 3.b. 3) We increase the contrast of the approximate lane markings regions in order to facilitate their extraction. Since in ROIr and ROIl generally the road is free, the mean value ‘M’ of the pixels composing each region in the grayscale image indicates the road condition. Thus, lower values may be caused either by the presence of strong shadows, cloudy day or obscurity. In these cases, lane markings are generally blurred. In opposition, upper values indicate the presence of a sunny day. In these cases, lane markings are generally clear. Thus, we enhance the pixels intensities by determining the square of the intensities of the top hat transformed image if ‘M’ is upper than a threshold, otherwise, we determine their cubic values to more enhance the image. We have determined the value of this threshold after a series of experiments. As a result of this operation, the clearer pixels that refer in majority to the lane markings will have as intensity value: 255 which facilitate their extraction in the following step. This enhancement will not be affected in case of light reflections due to the use of the top hat transformation as we have indicated above. The result of the contrast enhancement is illustrated in Figure 3.c.
ROIr
ROIl (a)
(b)
(c)
Fig. 3. Pre-processing step. (a) Original image, (b) image after top hat transformation, (c) image after contrast enhancement.
Our proposed contrast enhancement step allows us to improve the contrast of the lane markings in different environment conditions such as the presence of strong shadows, illumination variation, objects reflection, obscurity and in the presence of cloudy, heavy rain and intense sunny day.
A Comparative Study of Vision-Based Lane Detection Methods
51
3.2 Extraction of Approximate Pixels In this step, we extract the approximate pixels of lane markings based on the preprocessed image. To reach this objective, we proceed in 2 steps: 1) We retain the pixels having intensity equal to 255 (cf. Figure 4.a). 2) We label the obtained binary masque, eliminate small regions and extract edge pixels of the retained regions to obtain the approximate pixels. This was done by first delimiting each retained region by a rectangle then, by collecting the first white pixel met in each line of a rectangle by scanning the lines from left to right in ROIr and from right to left in ROIl. The result is illustrated in Figure 4.b.
(a)
(b)
Fig. 4. Extraction of approximate pixels step. (a) segmented image, (b) retained pixels.
3.3 Lane Detection In this section, we propose the lane detection step which is based on the model approach. As the road may be formed by linear and curved portions, we present the techniques we apply to detect the lane markings in the near and far lane regions. Detection in the near lane region. The near lane region includes ROIr and ROIl. In this region, we apply the Hough transform to detect the lane limits considering its rapidity and its robustness in the presence of noises and partial coverage of markings. In Hough transform, each line is a vector of parametric coordinates that are the orientation θ and the intercept to the origin P. The first coordinate corresponds to the angle formed by the vertical axis of the image and the segment passing by the origin and which is perpendicular to this line. From the set of obtained segments, we limited the range of search to lines that have an orientation ranging between 25° and 70° in ROIl of the road, and an orientation ranging between -25° and -70° in ROIr. Among these lines, we retained the segment with the maximum length in each region (cf., Figure 5.a). Finally, we project the two retained limits up to the sides of ROIr and ROIl (cf., Figure 5.b).
(a)
(b)
Fig. 5. Lane detection step based on model approach
52
N. Ben Romdhane, M. Hammami, and H. Ben-Abdallah
Using the linear equation of each detected limit, we determine the coordinates u and v of their pixels and we organise them in two different lists: Listr for the right limit, and Listl for the left limit:
{
List r = ( u ir , vir ) i = 1,2 ,...m 1
{
Listl = ( uil , vil ) i = 1,2,...m 2
}
(1)
}
(2)
where m1 and m2 are the numbers of pixels collected respectively in Listr and Listl. Detection in the far lane region. To consider the presence of an eventual curved lane portion, we delimited a third region of interest ROIc by Hz and Y according to the position of the detected limits in the near region (cf., Figure 6.b). Hz is the horizontal line passing through the intersection of the two detected limits, namely the vanishing line, Y is the horizontal line that delimits the near lane region. The detection process is detailed in our previous work [21]. To detect the lane markings in ROIc, we proceed in 2 steps: 1) We apply the same pre-processing step as in ROIl and ROIr (cf., Figure 6.c). 2) We extract the approximate pixels by following a mid to side strategy: we scan upwards the horizontal lines of ROIc, and from the middle to the right and left sides (cf., Figure 6.d). The lane centre position for each row can be determined based on the width of the lane in the corresponding row. For each side, we retain the first white pixel met satisfying certain constraints, and we add it to the appropriate list (Listr or Listl). We begin by collecting some approximate pixels from the two sides of the lane. Then we apply the Hough transform to detect the main line segment on each side and we search the position of the new vanishing point. By comparing this new position to the initial one, we can have an idea on the road trajectory and then we continue the extraction process.
ROIc ROIl
Hz Y
ROI ROIrr
(a) Detection of linear portions in ROIr and ROIl
(b) Delimitation of ROIc
(c) Preprocessing of ROIc
(d) Extraction of approximate pixels
(e) Lane detection
Fig. 6. Lane detection in near and far lane regions
To have a continuous layout of the limits in near and far lane regions, we apply to each obtained list, a linear parabolic model that corresponds to a quadratic polynomial function. The final result is illustrated in Figure 6.e.
4 Experimental Results and Performance Evaluation In order to evaluate the performance of our proposed method, we carried out a series of experiments on various sequences. In order to clarify the experimental conditions, the following sections will start by providing a brief overview of the test dataset and then proceed to present the experimental results and performance evaluation.
A Comparative Study of Vision-Based Lane Detection Methods
53
4.1 The Test Dataset The test dataset is composed by several video sequences divided in three sets. The first set (S1) contains four video sequences that are (Video 1) composed of 1400 frames, (Video 2) composed of 1220 frames, (Video 3) composed of 860 frames and (Video 4) composed of 500 frames, which we had captured in Tunisian roads. They are tacked on highways and on main roads during different moments of the day and in sunny, intense sunny, cloudy and rainy days. The roadway comprises two lanes separated by continuous and/or discontinuous white lines. The images present various road conditions, such as the presence of blurred, contrasted and well delimited markings, as well as various types of markings and partial coverage of markings by obstacles. The second set (S2) contains a night vision stereo sequence, composed of the 600 frames provided by Daimler AG1. The third set (S3) contains the video sequences of the PETS20012 dataset provided by the Essex University (England), a sequence (Video 5) composed of 2866 frames and a sequence (Video 6) composed of 2867 frames. They were tacked during a sunny and intense sunny days, and they illustrate a roadway that comprises three lanes separated by discontinuous lane markings; two continuous white lines delimit the edges of the road. 4.2 Experimental Results This section presents a series of experiments obtained through our proposed lane detection method. In order to evaluate and compare the performance of our lane detection method, we have implemented three proposed algorithms of the most known methods. For the model-based approach, we implement the Canny/Hough Estimation of Vanishing Points ‘CHEVP’ algorithm proposed by wang et al [10] and adopted by Tian et al [11] that is based on the local parametric detection method (Method A) and the global parametric detection method (Method B) proposed by Zhu et al [9]. For the feature-based approach, we implement the tracking method (Method C) proposed by Tsai et al [16]. We also present the detection results based on our previous work [21]. In this comparison, we discarded the explicit parametric methods that is based on the model approach and both the geometric filtering and the supervised training methods that are based on the feature approach, for two main reasons: i) their great sensitivity to noises and partial coverage of lane markings by obstacles; and ii) our quest for a robust and fast algorithm to delimit the lane limits. Table 1 illustrates the detection results of the different methods in different environment conditions. The images of Table 1 illustrate the detection results of the different methods. The first column indicates the different environment conditions. The second column illustrates the original images tacked from S1 (cf. Figures a (Video 1), b (Video 2), e (Video 3), c and h (Video 4)), S2 (cf. Figure d) and S3 (cf. Figures g (Video 5) and f (Video 6)). The following columns illustrate successively the lane detection results obtained based on our proposed method, our previous work, the Method A, the Method B and the Method C. 1 2
http://www.mi.auckland.ac.nz ftp://ftp.cs.rdg.ac.uk
54
N. Ben Romdhane, M. Hammami, and H. Ben-Abdallah Table 1. Comparative evaluation of the lane detection step Environment condition
Original Image
Proposed Method
previous work
Method A
Method B
Method C
Strong bridge a shadow + objects reflection b Cloudy day
c Heavy Raining
d Night time
e Normal condition
f Obstacles
g
Strong trees Shadow
h Intense raining
Most of the proposed methods give good results in normal conditions (cf. Figure e) and in case of night time (cf. Figure d). With Method A, a misalignment was caused by obstacles (cf. Figure f) and some misdetections were caused by strong shadows, objects reflection, intense lighting where the lane markings have a very low contrast (cf. Figures a and c). Such failures are caused by the use of Canny edge detector on the original image to extract the edge pixels without applying a pre-processing step to enhance the lane markings. As for Method B which performs a global detection of the lane markings, some misdetections were caused by strong shadows, objects reflection and intense lighting where the lane markings have a very low contrast (cf. Figures a, c and g). Such failures are caused by the use of the histogram equalization technique in preprocessing step, to enhance the image, which is not efficient in different environments Some misalignments were obtained with Method C. As illustrated in Figures (a, b and f), these misalignments are caused by a false detection of the first or the followings lane segments in the presence of objects reflexion on the windshield of the car, obstacles or lack of markings. We noted that this method gives good results in the presence of a heavy rain, strong shadows and intense lighting due to the use of morphological transformations in the pre-processing step (cf. Figures c and g). With our previous work, we obtained good results in the normal environments conditions. It detects efficiently the lane limits in the presence of curved roads
A Comparative Study of Vision-Based Lane Detection Methods
55
(cf. Figures b and e) and in the presence of obstacles (cf. Figure f). Furthermore, our previous work detects the limits during a cloudy day (cf. Figure b) and at night time (cf. Figure d). The weakness of our method is seen in presence of strong shadows (cf. Figures a and g), objects reflection and intense lighting (cf. Figure a) and in the presence of rain (cf. Figure c). The robustness of our new method in comparison with our previous work and the three others is seen in frames presenting a combination of strong shadow, too spaced blurred lane markings, illumination variation and intense lighting (cf., Figures a, c and g) due to the efficiency of our pre-processing step. Our method gives, as for the Method C, good results in the presence of a heavy rain (cf., Figure c). Furthermore, it detects efficiently the limits in case of curved lanes (cf., Figures b and e) and in the presence of obstacles (cf., Figure f). There remain few cases where the detection fails with our method and with the others such as the presence of sharp curved lane, non flat road, congested road and also in case of intense raining as illustrated in Figure (h). 4.3 Performance Evaluation In order to further evaluate our lane detection method and puts it in its context, we compared its detection performance with the three methods in terms of Recall, Precision and F-measure. We performed this evaluation on the set S3 (the PETS2001 dataset) since it is a known dataset and to provide our lane detection rates for comparison with other researches. We have manually detected the limits of 1000 frames: the first 500 frames of Video 5 and the first 500 frames of Video 6. Considering the reference lane limits, which correspond to the lane limits that we have detected manually, and the extracted lane limits, which correspond to the automatic extracted ones, these measurements are defined by the following equations: R ecall =
Length of matched reference lenght of reference
P recision =
length of matched extraction length of extraction
F - mesure = 2 ×
Recall × Pr ecision Recall + Pr ecision
The performance measure of the different methods is illustrated in Figure 7.
Fig. 7. Comparative performance evaluation of the five methods
(4) (5) (6)
56
N. Ben Romdhane, M. Hammami, and H. Ben-Abdallah
Based on our proposed method, we obtained an average rate of 92.04% for the Recall, 91.53% for the Precision and 91.76% for the F-measure. These results prove the improvement of our method previous work given an average rate of 78.59% for the Recall, 78.16% for the Precision and 72.55% for the F-measure. Our results are almost equal to the Method C. However, this method is characterized by its sensitivity to noises which lead it to present more cases of false detections than our method. On the other hand, our previous work, Method A and Method B give lower rates especially because of the presence of strong shadows and intense lighting in some frames.
5 Conclusion In this study, we presented our lane detection method. Our experimental evaluation showed that, with our method, we can detect lane limits in the majority of the images in the presence of strong shadows, partial coverage of the limits by obstacles, reflection of objects on the windshield of the car, presence of different lane markings that can be blurred, contrasted, and continuous or dashed, and in different weather conditions. The comparison of our results with three known methods proves its robustness in the cases of complex environment conditions. The findings of the present study are promising. For this reason, studies are currently underway in our laboratories to investigate the tracking of the limits to have a continuous detection of the limits while minimizing noises and execution time.
References 1. Dai, X., Kummert, A., Park, S.B., Neisius, D.: A warning algorithm for lane departure warning system. In: IEEE Intelligent Vehicles Symposium (2009) 2. Yu, B., Zhang, W., Cai, Y.: A Lane Departure Warning System based on Machine Vision. In: IEEE Pacific-Asia Workshop on Computational Intelligence and Industrial Application (2008) 3. Guo, L., Wang, J., Li, K.: Lane Keeping System Based on THASV-II Platform. In: IEEE International Conference on Vehicular Electronics and Safety (2006) 4. Dariu, M.G., Uwe, F., Wohler, C., Gorzig, S.: Real-time vision for intelligent vehicles. IEEE Instrumentation and Measurement Magazine (2001) 5. Assidiq, A., Khalifa, O., Islam, R., Khan, S.: Real Time Lane Detection for Autonomous Vehicles. In: Proceedings of the International Conference on Computer and Communication Engineering, Kuala Lumpur, Malaysia (2008) 6. Yeh, C., Chen, Y.: Development of Vision-Based Lane and Vehicle Detecting Systems via the Implementation with a Dual-Core DSP. In: IEEE Int. Transp. Systems Conference, Canada (2006) 7. Lin, H., Ko, S., Hi, W., Kim, Y., Kim, H.: Lane departure identification on Highway with searching the region of interest on Hough space. In: International Conference on Control, Automation and Systems, Seoul, Korea (2008) 8. Jung, C.R., Christian, R.K.: A Robust Linear-Parabolic Model for Lane Following. In: Proceedings of the XVII Brazilian Symposium on Computer Graphics and Image Processing (2004)
A Comparative Study of Vision-Based Lane Detection Methods
57
9. Zhu, W., Chen, Q., Wang, H.: Lane Detection in Some Complex Conditions. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, Beijing, China (2006) 10. Wang, Y., Shen, D., Teoh, E.K.: Lane Detection Using Spline Model. Pattern Recognition Letter 21, 677–689 (2000) 11. Tian, M., Fuqiang, L., Wenhong, Z., Chao, X.: Vision Based Lane detection for Active Security in Intelligent Vehicle. In: IEEE Conference on Vehicular Electronics and Safety, pp. 507–511 (2006) 12. D’Cruz, C., Zou, J.J.: Lane detection for driver assistance and intelligent vehicule applications. In: International Symposium on Communications and Information Technologies (2007) 13. Cheng, H., Jeng, B., Tseng, P., Fan, K.: Lane detection with moving vehicle in the traffic scenes. IEEE Trans. on Intelligent Transportation Systems 7, 571–582 (2006) 14. Gonzalez, J.P., Omit, O.: Lane Detection Using Histogram-Based Segmentation and Decision Trees. In: Int. Transp. Systems, Conference Proceedings Dearborn (MI), USA, (2000) 15. Zu, K.: Realtime Lane Tracking of Curved Local Road. In: Proceedings of the IEEE Intelligent Transportation Systems Conference Toronto, Canada (2006) 16. Tsai, M., Hsu-Yung, C., Yu, C., Tseng, C., Fan, K., Hwang, J., Jeng, B.: Lane Detection Using Directional Random Walks. In: International Conference on Acoustics, Speech, and Signal Processing (2008) 17. Asif, M., Arshad, M.R., Yousuf, M., Zia, I., Yahya, A.: An Implementation of Active Contour and Kalman Filter for Road Tracking. International Journal of Applied Mathematics (2006) 18. Lim, K.H., Seng, K.P., Ngo, A.C., Ang, L.: Real-time Implementation of Vision-based Lane Detection and Tracking. In: Int. Conference on Intelligent Human-Machine Systems and Cybernetics (2009) 19. Wanga, Y., Teoha, E.K., Shenb, D.: Lane detection and tracking using B-Snake. Image and Vision Computing 22, 269–280 (2004) 20. Boumediene, M., Ouamri, A., Dahnoun, N.: Lane Boundary Detection and Tracking using NNF and HMM Approaches. In: IEEE Intelligent Vehicles Symposium Istanbul, Turkey (2007) 21. Ben Romdhane, N., Hammami, M., Ben-Abdallah, H.: An Artificial Vision-based Lane Detection Algorithm. In: International Conference on Computer Science and its Applications, Korea (2009)
A New Multi-camera Approach for Lane Departure Warning Amol Borkar1, Monson Hayes2 , and Mark T. Smith3 1
Georgia Institute of Technology, Atlanta, GA, USA
[email protected] 2 Chung-Ang University, Seoul, Korea
[email protected] 3 Kungliga Tekniska H¨ ogskolan, Stockholm, Sweden
[email protected]
Abstract. In this paper, we present a new multi camera approach to Lane Departure Warning (LDW). Upon acquisition, the captured images are transformed to a bird’s-eye view using a modified perspective removal transformation. Then, camera calibration is used to accurately determine the position of the two cameras relative to a reference point. Lane detection is performed on the front and rear camera images which are combined using data fusion. Finally, the distance between the vehicle and adjacent lane boundaries is determined allowing to perform LDW. The proposed system was tested on real world driving videos and shows good results when compared to ground truth. Keywords: Lane departure warning, multi camera calibration, data fusion, non-overlapping camera networks.
1
Introduction
Driver Assistance (DA) systems are common accessory in today’s passenger and commercial vehicles. DA systems as the name suggests are systems that provide aid or feedback to the driver of a vehicle. One example of such a system is Lane Departure Warning (LDW). LDW is a safety feature that informs the driver if the vehicle appears to change lanes unless certain conditions are met e.g. turn indicator is on. The hardware often consists of a single camera which acquires images out of the front windshield that are analyzed to determine if the vehicle is within a certain distance of a lane boundary. Although LDW has been in research for many years, it has only been recently that LDW systems have started appearing in luxury cars and as after market products. Variations of the Time to Lane Crossing (TLC) are some of the most popular techniques used for LDW [1–4]. Using a “look-ahead” approach, the future position of the vehicle is estimated based on the steering angle and by extrapolating detected lane positions, then the driver is warned if a lane crossing is predicted. The lane detection component of LDW often produces noisy results; consequently, the extrapolated estimates may not accurately represent J. Blanc-Talon et al. (Eds.): ACIVS 2011, LNCS 6915, pp. 58–69, 2011. c Springer-Verlag Berlin Heidelberg 2011
A New Multi-camera Approach for Lane Departure Warning
59
future lane positions leading to false signaling. Other LDW techniques involve determining the angles or orientations of detected lane boundaries to hypothesize a distance metric [5–7]. Actual distance to the adjacent boundaries is not computed. In the proposed work, we compute the immediate distance between the vehicle and adjacent boundaries using known camera geometries. In addition, we add a second camera that looks out of the rear windshield. This provides redundant lane boundary position information which is used in data fusion. Since we are combining data from multiple cameras, calibration is important. Calibration for stereo and n-camera setups is often performed using variations of standard test patterns like checkerboard to establish a baseline [8]. However, these techniques require each camera to be able to see the same checkerboard. Since our cameras are facing in opposite directions, the checkerboard methods are ineffective. Work on calibration for non-overlapping camera networks has been conducted in [9–11]. The method shown in [9, 10] uses a planar mirror while [11] uses a moving rig to calibrate each camera. However, the rig movement appears arbitrary which can severely affect the estimates of camera positions. A similar experiment with a multi camera setup for vehicular applications has been shown in [12]; however, the method used to merge information from the two cameras is only stated briefly and lacks details. Additionally, calibration of opposite facing cameras is not discussed. In our work, we have devised a new yet simple calibration technique that allows to accurately determine the position of the two cameras in the vehicle. This paper is organized as follows: subsequent to the introduction, preprocessing in the form of a geometric transformation applied to acquired images is explained. Then, the details of camera calibration and data fusion are described. Finally, LDW is completed by explaining the technique used to measure the distance to lane boundaries. The proposed approach was compared to a single camera LDW system as well as ground truth and showed good results when tested on real world driving videos.
2
Pre-processing
In a camera image, lane markings appear as lines that decrease in thickness and converge near the horizon. This can be seen on the left in Fig. 1. Consequently, feature extraction can become difficult since the line features change as a function of distance from the camera due to the effects of perspective. As a result, the camera acquired images first undergo a geometric transformation known as Inverse Perspective Mapping (IPM) to remove the effects of perspective from the image [13]. With IPM, the captured images are transformed to appear as a birds-eye view with lane markers now appearing as nearly parallel lines. In addition, the thickness of these lines remain constant in the IPM image as shown on the right in Fig. 1. The camera configuration is shown in Fig. 2. The mapping from world co-ordinates (x, y, z = 0) to image co-ordinates (r, c) is given as M −1 h − x · tan θo r(x) = 1+ · cot αv + 1 (1) 2 h · tan θo + x
60
A. Borkar, M. Hayes, and M.T. Smith
Fig. 1. Pre-processing stage Z
ew ield of Vie
0
T0 Ima g Pla e ne
0
Optica
Image Plane
l Axis
c
M
h
iew fV ld o Fie
Horizontal Axis
Dv
r
o^ Came ra
Fie ld
of
^o Vie
w
Y
Du
X
Z
J
Camera
N
Y Road Surface
Optical Axis
Field
of Vie
w
X
(a) Side view.
(b) Top view.
Fig. 2. Camera parameters
N −1 y c(x, y) = 1− · cot αu + 1 2 h · sin θo + x · cos θo
(2)
where h is the height of the camera, θo is the pitch, αu is the horizontal field of view, αv is the vertical field of view, γ is the yaw, and (M × N ) is the size of the camera image. Furthermore x cos γ − sin γ x = (3) y sin γ cos γ y provides the yaw correction by rotating the world co-ordinates about the optical center of the camera. This mapping differs slightly from the technique described in [13] where the horizontal and vertical fields of view of the camera are assumed to be equal. However, in most commercial camera hardware, these values are different.
3
Camera Calibration
In camera calibration, an origin needs to be specified. This point is chosen as T the center of the vehicle as shown in Fig. 3 and denoted as Po = 0 0 0 . The positions of the front and rear cameras relative to Po are then determined. These position are denoted as Pf for the front camera and Pr for the rear camera. Pr is also oriented looking out of the back of the vehicle which results in its IPM image
A New Multi-camera Approach for Lane Departure Warning
61
Y
Pr
Pf
Z
Po
X
Fig. 3. Position of the origin and two cameras
(a) Front camera image.
(b) Rear camera image.
Fig. 4. Sample camera images
being rotated 180◦ . Since small measurement errors in Pf and Pr could lead to critical errors during data fusion used later, the accuracy of these measurements is verified by comparing to reference data subject to certain constraints. The reference data is made up of images captured while driving at a constant speed near the middle of a straight road where the distance between adjacent lane boundaries is constant and uniform. Both cameras undergo basic calibration to determine yaw (γ) and pitch (θo ) which prevents the lane boundaries from appearing skewed or tilted in the IPM image. Therefore, the cameras are now 180◦ opposed and facing in opposite directions along the X-axis of the reference co-ordinate system shown in Fig. 3. Sample front and rear camera images are shown in Fig. 4. Correctness is assumed in the measured X and Y co-ordinates of Pf . The techniques described below accurately determine the height of Pf as well as the translation of Pr relative to Po . 3.1
Height Measurement
We combine Eq. (1) and (2) to determine the mapping of a pixel from the camera to world co-ordinates as [1 − 2( Nc−1 −1 )] tan αu Y (r, c) = h (4) r−1 sin θo − [1 − 2( M −1 )] tan αv cos θo We choose two points (r, c1 ) and (r, c2 ) on adjacent lane boundaries in the camera image and find their corresponding mappings to the IPM domain as Y (r, c1 ) and Y (r, c2 ) respectively as shown in Fig. 5. Since it not possible to park on the
62
A. Borkar, M. Hayes, and M.T. Smith
Y(r,c1)
(r,c1)
Y(r,c2)
(r,c2)
(a) Two points in the camera image.
(b) Two points in the IPM image.
Fig. 5. Mapping points from the camera image to IPM image
highway and physically measure the distance between adjacent lane boundaries, we refer to Federal Highway Administrations (FHA) handbook regarding specification for painted markings [14] where it is stated that the distance between lane boundaries on highways is on average 12ft. Using this information, we can set Y (r, c1 ) − Y (r, c2 ) = ΔY = 12 feet (5) Hence, we substitute Eq. (4) in Eq. (5) and solve for h as r−1 sin θo − [1 − 2( M−1 )] tan αv cos θo h = ΔY 1 2[ cN2 −c ] tan αu −1
(6)
Similarly, we can repeat this procedure to determine the height of the rear camera. 3.2
Offset between Cameras
If the two cameras are not aligned properly, there will be a visible offset between the lane markings when observed in each IPM image. This is illustrated in Fig. 6 by placing the two IPM images side by side. Since the Pf is inline with the origin, the offset is adjusted by translating the rear camera IPM image along the Y-axis till the lane markings line up correctly. The translation in the IPM image is scaled to world units to determine the actual offset between the two cameras. 3.3
Distance between Cameras
To determine the distance between the cameras, we need to find objects with corners in each IPM image. In our case, we use the diamond from the high occupancy vehicle (HOV) lane. This is shown in Fig. 7. Since the vehicle is commuting at a constant speed and knowing the time delay since the diamond from the front camera IPM image reappears in the rear camera IPM image, the total covered distance can be computed as w = Speed × Time Delay
(7)
A New Multi-camera Approach for Lane Departure Warning
63
Y Z
X
Fig. 6. Error in camera camera alignment. This results in lane markings being offset from one view to the other.
j
k
w Y Z
X
Fig. 7. Determining the distance between the cameras
By setting (x = 0, y = 0) in Eq. (3), the optical center of each camera relative to its IPM image can be determined and is illustrated by the red dots in Fig. 7. Since the lens used in this implementation has a focal length of 5mm, it is safe to assume that optical center is analogous to the camera position. As a result, it is possible to determine the distance j or k between a specific point on the diamond and the camera itself. Consequently, the distance between the cameras can estimated as Distance between cameras = w − (j + k)
(8)
After completing the three verification steps described above, the position of T the front camera is determined as Pf = 2.0 0.0 4.79 and rear camera as T Pr = −7.51 −0.48 4.94 relative to Po . The units are in feet.
4
Lane Detection
Lane detection is performed using the technique presented in [15] on both forward and rear camera views. Following IPM, the transformed image is converted
64
A. Borkar, M. Hayes, and M.T. Smith
from RGB to YCbCr to aid in color segmentation. Then, the images are crosscorrelated with a collection of predefined templates to find candidate lane regions. These regions then undergo connected components analysis, morphological operations, and elliptical projections to approximate positions of the lane markers. The implementation of the Kalman filter enables tracking lane markers on curved roads while RANSAC helps improve estimates by eliminating outliers [15].
5
Data Fusion
In a standalone IPM image, the optical center e.g. Pf is at the origin. Each pixel in the IPM image has a certain (x, y) co-ordinate relative its optical center. This approach is acceptable when dealing with a single camera; however, when dealing with multiple cameras, a common co-ordinate system needs to be established. In our implementation, Po is chosen as the origin of this space and shown as a red dot in Fig. 8. The two green dots in Fig. 8 represent Pf and Pr . Next, a transformation Tf that maps only the (x, y) co-ordinates of Po to Pf is determined. This transformation when applied to the IPM image maps the (x, y) co-ordinates of each pixel to a location in the common co-ordinate system that is now relative to Po . Consequently, the lane boundaries detected in the front camera IPM image are also mapped to the common space and described by (x, y) co-ordinates relative to Po . A transformation mapping Po to Pr is also computed and used to relocate the rear camera IPM pixels to the common co-ordinate space.
Pr
Pf Po
Y Z
X
Fig. 8. Position of the origin and two cameras
Upon completing the front and rear lane detections, data fusion in the form of a parametric linear spline connects the detected lane boundaries. Using an end point from each boundary as shown in Fig. 9, the locations of lane boundaries outside the cameras view can be estimated using Eq. (9). Q1 (9) Q(t) = (1-t) t Q2
A New Multi-camera Approach for Lane Departure Warning
Q2
Q1 Y Z
65
Q3
X
Fig. 9. Spline estimated boundary shown as dotted green line between the two end points Q1 and Q2
6
Lane Departure Warning
Upon completing camera calibration and data fusion, the following is known: 1. Center of the vehicle. 2. Dimensions of a reference vehicle. 3. Location of lane boundaries. Using the available information, the distance to the lane boundary (in feet) on either side of the vehicle can be calculated as: Distance to marking = |LBx=0 − sign[LBx=0 ](
Car Width )| feet 2
(10)
where LBx=0 is simply the y-value of the estimated lane boundary from Eq. (9) evaluated at x=0. The orange bounding box in Fig. 10 depicts the dimensions of the reference vehicle in the common co-ordinate space with Po as the red dot. Using Eq. (10), LDW can be systematically performed by warning the driver if the vehicle is within a minimum distance from a lane boundary.
3ft
2.5ft Y Z
X
Fig. 10. Distance to lane boundaries on either side
7
Results and Analysis
Two time-synchronized NTSC cameras were used in acquiring video streams for testing. The streams were captured while driving on roadways in and around
66
A. Borkar, M. Hayes, and M.T. Smith
Atlanta, GA, USA. Several minutes worth of videos were used in testing. The rear camera was inoperable at night without artificial illumination; hence, testing was restricted to day time only. Additionally, testing was conducted only on highway footage as most commercial LDW systems operate above 45mph [16, 17].
(a)
(b)
(c)
Fig. 11. Front camera image and estimated position of the vehicle within the lane
In Fig. 11, a few examples of the images captured by the front camera with an overhead visualization estimating the position of the vehicle within a lane are shown. Fig. 12 shows an example of the vehicle approaching very close to a lane boundary over the duration of a clip. Fig. 12a - 12c shows the camera captured images with corresponding visualization. When the vehicle is within 0.75ft of a lane boundary, a warning is signaled. Hysteresis is used whereby the vehicle needs to travel more than 1ft away from the lane boundary for the warning to be disabled. This reduces flicker around the minimum distance threshold. Fig. 12d shows a plot of the estimated distance between the right lane boundary and vehicle over the duration of the video. The plot also shows a comparison between the multi camera approach, single camera approach and ground truth. To estimate distance using a single camera, we refer back to Fig. 9 and evaluate the line formed by Q1 and Q3 at x=0. A straight line model is less affected by noise in lane detection results; hence, it is used for extrapolation. Ground truth estimates are produced using cubic interpolation between the ground truth lane boundary locations in the front and rear camera images. In Fig. 12d, it can be observed that the estimates of the multi camera approach are often very close to the ground truth. The single camera approach fares well on straight roads but face difficult when encountered with a curve. Fig. 12e provides a visual comparison between the ground truth, single camera and multi camera approach when the vehicle encounters a turn in the test clip. The advantage of using the described multi camera setup is that estimating distances to the lane boundaries becomes an interpolation problem. With a single camera setup, it is an extrapolation problem which appears to be less accurate.
A New Multi-camera Approach for Lane Departure Warning
(a) Frame 73.
(b) Frame 289.
67
(c) Frame 429.
4
Distance (ft)
3 2 1
Ground Truth Single Camera Multi Camera
0 −1
0
100
200
300 Frames
400
500
600
(d) Distance between vehicle and right lane marker.
(e) Comparison between ground truth (red), single camera (green) and multi camera (blue) approach. Detected lane markings are shown in orange. Fig. 12. An example of Lane Departure Warning (LDW)
Since the lane boundary estimates are calculated based on the findings of the front and rear lane detectors, errors in lane detection reflect directly on the lane boundary estimates. A few instances of incorrect lane detections shown in Fig. 13 resulted in incorrect estimates of the distances to the lane boundaries. In addition, when an incomplete lane detection occurs i.e. only one boundary is detected in either front or rear camera images, the estimates around the vehicle cannot be determined. This is shown in Fig. 13c.
68
A. Borkar, M. Hayes, and M.T. Smith
(a)
(b)
(c)
Fig. 13. A few examples where the multi-camera approach faces difficulty
8
Conclusion
Presented in this paper is a novel approach to Lane Departure Warning (LDW) using multiple cameras. First, the captured images undergo an improved Inverse Perspective Mapping (IPM) transform to produce a bird’s-eye view image. Then, camera calibration is used to accurately determine the position of two cameras. Following lane detection, data fusion is used to connect the detected lane boundaries from one camera view to the other. By determining the distance between the vehicle and lane boundaries, an LDW system is realized. The proposed setup showed good results when tested with real world driving videos. The multi camera approach also produced estimates close to the ground truth on both straight and curved roads in the test data while the single camera approach faced difficulty on the curved roads.
9
Future Work
The presented work is part of on going research. The use of smooth curves instead of straight line estimates for lane boundaries will be explored. Handling situations when an incomplete lane detection occur needs to be investigated. Kalman or particle filters will be used to track LBx=0 on either side of the vehicle. Additional tests with complex scenarios will be conducted. Illumination hardware to enable night time multi camera usage will be purchased.
References 1. LeBlanc, D., Johnson, G., Venhovens, P., Gerber, G., DeSonia, R., Ervin, R., Lin, C., Ulsoy, A., Pilutti, T.: CAPC: A road-departure prevention system. IEEE Control Systems Magazine 16(6), 61–71 (1996) 2. Kwon, W., Lee, J., Shin, D., Roh, K., Kim, D., Lee, S.: Experiments on decision making strategies for a lane departure warning system. In: Proceedings of IEEE International Conference on Robotics and Automation, vol. 4, pp. 2596–2601. IEEE, Los Alamitos (1999)
A New Multi-camera Approach for Lane Departure Warning
69
3. Kim, S., Oh, S.: A driver adaptive lane departure warning system based on image processing and a fuzzy evolutionary technique. In: Proceedings of IEEE Intelligent Vehicles Symposium, pp. 361–365. IEEE, Los Alamitos (2003) 4. Dai, X., Kummert, A., Park, S., Neisius, D.: A warning algorithm for Lane Departure Warning system. In: IEEE Intelligent Vehicles Symposium, pp. 431–435. IEEE, Los Alamitos (2009) 5. Hsiao, P., Yeh, C., Huang, S., Fu, L.: A portable vision-based real-time lane departure warning system: day and night. IEEE Transactions on Vehicular Technology 58(4), 2089–2094 (2009) 6. Lin, Q., Han, Y., Hahn, H.: Real-time Lane Detection Based on Extended Edgelinking Algorithm. In: Second International Conference on Computer Research and Development, pp. 725–730 (2010) 7. Leng, Y.-C., Chen, C.-L.: Vision-based lane departure detection system in urban traffic scenes. In: 11th International Conference on Control Automation Robotics Vision, ICARCV 2010, pp. 1875–1880 (2010) 8. Hartley, R., Zisserman, A.: Multiple View Geometry in Computer Vision, 2nd edn. Cambridge University Press, Cambridge (2004) 9. L´ebraly, P., Deymier, C., Ait-Aider, O., Royer, E., Dhome, M.: Flexible extrinsic calibration of non-overlapping cameras using a planar mirror: Application to visionbased robotics. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2010, pp. 5640–5647. IEEE, Los Alamitos (2010) 10. Kumar, R., Ilie, A., Frahm, J., Pollefeys, M.: Simple calibration of non-overlapping cameras with a mirror. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2008. IEEE, Los Alamitos (2008) 11. Pagel, F.: Calibration of non-overlapping cameras in vehicles. In: IEEE Intelligent Vehicles Symposium (IV), pp. 1178–1183. IEEE, Los Alamitos (2010) 12. Ieng, S., Vrignon, J., Gruyer, D., Aubert, D.: A new multi-lanes detection using multi-camera for robust vehicle location. In: Proceedings of IEEE Intelligent Vehicles Symposium, pp. 700–705. IEEE, Los Alamitos (2005) 13. Bertozzi, M., Broggi, A.: GOLD: a parallel real-time stereo vision system for generic obstacle and lane detection. IEEE Transactions on Image Processing 7(1), 62–81 (1998) 14. Federal Highway Administration. Manual Uniform Traffic Control Devices (November 2009), http://mutcd.fhwa.dot.gov/ 15. Borkar, A., Hayes, M., Smith, M.T.: A template matching and ellipse modeling approach to detecting lane markers. In: Blanc-Talon, J., Bone, D., Philips, W., Popescu, D., Scheunders, P. (eds.) ACIVS 2010, Part II. LNCS, vol. 6475, pp. 179–190. Springer, Heidelberg (2010) 16. Infiniti USA, 2012 Infiniti M Specs & Options (2011), http://www.infinitiusa. com/m/specs-options 17. Mercedes-Benz USA, 2011 Mercedes-Benz S550 Specs (2011), http://www.mbusa. com/mercedes/vehicles/explore/specs/class-S/model-S550V
Feature Space Warping Relevance Feedback with Transductive Learning Daniele Borghesani, Dalia Coppi, Costantino Grana, Simone Calderara, and Rita Cucchiara University of Modena and Reggio Emilia, Modena, Italy http://imagelab.ing.unimore.it
Abstract. Relevance feedback is a widely adopted approach to improve content-based information retrieval systems by keeping the user in the retrieval loop. Among all, the feature space warping has been proposed as an effective approach for bridging the gap between high-level semantics and the low-level features. Recently, combination of feature space warping and query point movement techniques has been proposed in contrast to learning based approaches, showing good performance under different data distributions. In this paper we propose to merge feature space warping and transductive learning, in order to benefit from both the ability of adapting data to the user hints and the information coming from unlabeled samples. Experimental results on an image retrieval task reveal significant performance improvements from the proposed method. Keywords: Relevance feedback, covariance matrices, transductive learning, feature space warping.
1
Introduction
The use of relevance feedback strategies in information retrieval and in particular in content-based image retrieval systems is widely considered a very precious (sometimes necessary) addition to the system itself. At the present time, it is the most effective way to capture user’s search intentions. The reason is pretty straightforward: the automatic association of low-level features to high-level semantics is still a very open problem, and the only practical way to identify what the user is looking for is by including him in the retrieval loop, letting him provide feedbacks (positive, negative or both) about what is going on. The common scenario in which relevance feedback is used within content-based image retrieval systems is the following: 1. An initial query-by-keyword or query-by-example is performed, in the form of a list of results ranked with increasing distances from the query in the feature space; 2. The user provide some good (and bad, implicitly or explicitly) feedbacks given the displayed images, choosing in other words what is relevant and what is irrelevant; J. Blanc-Talon et al. (Eds.): ACIVS 2011, LNCS 6915, pp. 70–81, 2011. c Springer-Verlag Berlin Heidelberg 2011
Feature Space Warping Relevance Feedback with Transductive Learning
71
3. An algorithm uses these information to change the displayed results in a “refinement” step to accommodate user’s judgments; 4. Back to step 2 and loop until a certain condition (or satisfaction) is reached. In this paper, we focus on the third step proposing a new and effective strategy for relevance feedback based on Transductive Learning. The main contribution of this work is the joint use of Transductive Learning (successfully applied to a wide variety of pattern recognition and classification tasks) with the Feature Space Warping, a widely used technique for relevance feedback. Feature Space Warping in particular allows to modify accordingly the relations between feature points: objects similar to positive feedbacks go closer to the query, while object dissimilar are pushed away. We show that the union of these two techniques overcome the respective limitations providing a significant boost in performance over the two techniques used alone.
2
Related Work
The literature on this topic is countless [22, 5], since this problem can be faced from several point of view (computer vision, database management, humancomputer interaction, artificial intelligence, even psychology). Moreover, aside the research on the algorithms for relevance feedback, there is a wide literature about the way in which the performance of a system with relevance feedback can be safely evaluated in order to provide fair comparison with different techniques. Regarding the algorithms, we can identify very generally three classes. In the first one (called Query Point Movement, QPM in short), we try to move the query point in order to create a more complete query (a fast technique to overcome to slow convergence is proposed by [10]). In the second one (called Feature Space Warping, FSW in short), we try instead to manipulating the feature space or the metric space, in order to shape it in the direction of the users’ feedbacks [1, 12]. The third one applies some machine learning procedures (like SVM or Adaboost) to learn how to separate relevant samples from irrelevant ones [16,17]. Among the usual techniques, based on SVM or boosting, we preferred testing a transduction-based learning. Some author followed the same path (see [20,15,14, 13]). The idea is to take advantage both of the unlabeled and labeled samples in a transductive inference manner, learning from an incremental amount of training samples (feedbacks, in this case). Given the algorithm, the problem of evaluation is controversy. Even back in the Seventies, Williamson [19] proposed an evaluation methodology to tackle the so called “ranking effect” in the “fluid” relevance feedback evaluation, i.e. the overestimated performance improvement (in terms of recall and precision) due to the reposition of positive feedback in the top of the rank, aside the underestimated performance improvement of the “frozen” relevance feedback evaluation, which maintain the original ranks of documents along the sessions. In his “reranked original” ranking proposal, the best ranks are assigned to thy relevant documents and the worst ranks to the non-relevant documents; those documents not yet judged would remain in their original order, but with a rank decreased
72
D. Borghesani et al.
by the number of non-relevant documents identified. In [9] a comprehensive analysis tries to find out the reasons of relevance feedback evaluation problems, in particular problems with the dataset (characteristics and relative ground truth), problem with the comparison (different measures, different ranking approaches that make a comparison unfair, the need of rank normalization), and finally the problem of the parameter settings which can be impractical in real context. In our opinion, a quite fair and complete set of measures has been proposed in [11], where authors proposed: – actual recall and precision (computed at each iteration and relative to the current set of retrieved images solely) – new recall and precision (computed at each iteration and relative to the previous set of retrieved images solely) – cumulative recall and precision (computed at each iteration and relative to the whole set iterations so far) In this way, we can describe the behavior of the retrieval system both in terms of speed (how fast valuable images are retrieved over time) and in terms of completeness (how many good images the retrieval system finds out globally). Finally, as suggested in [8], we tried to concentrate the analysis on feasible search tasks, i.e. visual topics with a good number of representatives with a low degree of uncertainty in the evaluation, in order to assure a valuable reference ground truth.
3
Visual Similarity Using Covariance Matrices
In order to accomplish an effective similarity retrieval upon these images, we relied on a simple yet effective feature which allows to consider both color and edge based information, that is covariance matrices. Computing the covariance region descriptor from multiple information sources yields a straightforward technique for a low-dimensional feature representation [18]. In particular, we use normalized pixel locations (x/W, y/H), RGB values in the range [0, 1] and the norm of the first derivatives of the intensity with respect to x and y, calculated through the filter [−1 0 1]T . The covariance of a region is thus a 7 × 7 matrix. Notice that the covariance matrices do not form a vector space, but if we concentrate on nonsingular covariance matrices, we can observe that they are symmetric positive definite, and as such they lay on a Riemannian manifold. In order to rank images by visual similarity to a given query, we need to measure the distance between covariance matrices. In [6] the following distance measure for positive definite symmetric matrices is proposed: d ρ(X, Y) = ln2 λi (X, Y) (1) i=1
where {λi (X, Y)}i=1..d are the generalized eigenvalues of two non singular covariance matrices X and Y.
Feature Space Warping Relevance Feedback with Transductive Learning
73
Unfortunately distance alone is not enough for our purposes. In fact, in order to enable the user to provide relevance feedbacks, we need to move the query and the other points with linear combinations. To this aim two steps are required: the projection on an Euclidean tangent space, and the extraction of the orthonormal coordinates of the tangent vector. By combining these two steps, the projection y of a covariance matrix Y on the hyperplane tangent to covariance matrix X can be written as 1 1 y = vec log X− 2 YX− 2 (2) where the vector operator is defined as √ √ √ vec(y) = y1,1 2y1,2 2y1,3 . . . y2,2 2y2,3 . . . yd,d
(3)
In this way, after the selection of an appropriate projection origin, every covariance matrix gets projected to a 28-dimensional feature vector laying on an Euclidean space. This process is easily invertible. We can compute the relative covariance matrix in the Riemannian Manifold starting from the 28-dimensional feature vector using the following formulation:
1 1 Y = X 2 exp vec−1 (y) X 2
4
(4)
Mean-Shift Feature Space Warping with Remapping
In this work, we started from the relevance feedback technique proposed by Chang et al. [4] called Mean Shift Feature Space Warping (MSFSW). Given a query point q in the feature vector space, k samples are retrieved by nearest neighbor search. By examining the results, the user provides his feedback by specifying the relevance of M of these samples, forming two sets: {fp } and {fn }, the relevant and irrelevant sets respectively. These are employed to move all data samples {p} toward or away from the warping center w. In particular, for each p, its warped point p is given by p = p + λ
M
uj exp (−c|p − fj |) (w − p)
(5)
j=1
where the scalar value uj is set to +1 if fj ∈ {fp }, and −1 if fj ∈ {fn }. Two global coefficients c and λ are required to control the influence of each feedback to each sample and the maximum moving factor of any point p toward or away from the warping center w. The original FSW algorithm fixes the warping center w to q. Thus, the query point will always stay in its original position. Other points will move toward or far away from q based on its proximity to relevant and irrelevant sets. But, according to the analysis proposed in [4], FSW algorithm tends to perform poorly under Gaussian distributions when the query point is far away from the cluster center.
74
D. Borghesani et al.
For this reason, in the MSFSW, authors proposed to move the warping center instead of staying at q. They suggest to adopt the Rocchio’s query movement formula: w = αw + βfp − γfn (6) where w is the warping center (initially set to q), fp and fn are the mean of the set {fp } and {fn }. Another set of parameters α,β and γ is required, and must be tuned to optimize the performance. The MSFSW algorithm provides a flexible parameterization for switching between the two extreme algorithms: QPM by setting α = γ = λ = 0 and β = 1, and FSW by setting α = 1 and β = γ = 0. Given the final user target of our application, exposing the parameters configuration to the user was out of question. Thus, we determined the parameters configuration which provided best results on a small initial training set, using an automatic exhaustive search procedure. From the above equations, it is clear that we need a way to compute a linear combination of the feature vectors. For this reason, we employed the projection of the covariance matrices on the tangent space previously described (Eq. 2). As mentioned before, the projection requires a point from which determine the orthonormal coordinates of the tangent vector (i.e. the vector in the Euclidean space). Our experiments confirm that the choice of this point is fundamental to guarantee an optimal correspondence between the distances computed on the Riemannian manifold and those computed on the tangent space. Thus, when the user requires a refinement of a similarity search of a previously selected image, we project the whole feature space on the chosen query point (i.e. the covariance matrix of the selected image), then we rank the results and show them to the user in order to perform further refinements. Since this mapping is a homeomorphism around the neighborhood of the point, the structure of the manifold is locally preserved. The problem here is that the first step of MSFSW moves the warping center away from the current one, and this may impact on the quality of the projected vectors. For this reason we propose to employ an intermediate step of reprojection around the new warping center. The proposed relevance feedback approach works iteratively according with the following sequence of steps: 1. Given the previous warping center wi−1 and feedbacks {f}i−1 , the new warping center wi is computed by means of Eq. 6; 2. All points {p}i−1 and wi are reprojected on the manifold, defining the set of remapped points {R}i , exploiting Eq. 4: 1
12 2 Ri = Wi−1 exp vec−1 pi−1 Wi−1 I
(7)
3. Now the tangent space at the new warping center wi is taken into consideration: all remapped points on the manifold {R}i are mapped into the new Euclidean space, exploiting Eq. 2: −1 −1 ri = vecI log Wi 2 Ri Wi 2 (8)
Feature Space Warping Relevance Feedback with Transductive Learning
75
4. At this point, we can apply the FSW on the set {r}i , as in Eq. 5, finally obtaining the new set of points {p}i . Notice that on the first iteration only the feedbacks {f}0 must be initially mapped to the Euclidean space tangent at the query point (for step 1). Thus in step 2, only the new warping center w1 must be remapped, since the set of remapped points {R}1 would be equal to the original dataset itself.
5
Transductive Relevance Feedback
As mentioned in the Section 2, the relevance feedback problem can be analyzed as a semisupervised learning problem, in which the positive and the negative feedbacks given by the users constitute iteratively (and incrementally) the training set of the algorithm. In this paper, we propose a graph-based transductive learning method to tackle this purpose, defining a graph where the vertices represent the labeled and unlabeled images of the dataset, while the edges incorporate the similarity between them, in our case obtained from the distance between covariance matrices. Graph-based methods are nonparametric, discriminative, and transductive by definition [21], and labels can be assumed to be smooth over the graph. Starting from the whole dataset with n images, let’s define a set L of labeled images (x1 , y1 ), . . . , (xl , yl ) in which C classes are defined, so yi ∈ 1 . . . , C. The other images belongs to a set U of u unlabeled images, with n = l + u. Now let’s define a function f : Rn → [0, 1] which denotes the confidence of each image to one class. Formally, we can define a cost function J on f as: J(f ) =
n
2
f (xi ) − f (xj ) wij + λ
l
2
f (xi ) − yi
(9)
i=1
(i,j)=1
with λ as a regularization parameter (in our case, λ = 1). This equation, in a minimization process, tries to match the confidence wij between samples with the the true confidence yi with respect of the current confidence. Once converted in matrix notation, Eq. 9 becomes: J(f ) = (f (X) − Y )T (f (X) − Y ) + λf (X)T Lf (X)
(10)
where L = D − W is the graph Laplacian. W is the weight matrix, while D is the matrix which represents the degree of vertices: dii =
n
wij
(11)
j=1
The cost function minimization has a closed solution, that is: f = (I − λL)− 1Y
(12)
At the end of the computation, f contains the class confidence of each new sample.
76
D. Borghesani et al.
In the transductive process, we want to transfer labels from labeled samples to unlabeled ones: in other words, we want the samples which are close in the feature space to share the same label. To satisfy this local constraint, we construct accordingly the weight matrix W , that is the matrix in which each element wij contains the relation between two vertices (thus two images) xi, xj. To move from distances to affinities, we use the following formulation:
2 xi − xj wi,j = exp − (13) 2σ 2 where σ is a bandwidth parameter to tune the relations between vertices, and the distance is computed as the L2 norm after the conversion of covariance matrices in the Riemannian Manifold to vectors in the Euclidean Space. W can be subdivided into four submatrices: ll W W lu W = (14) W ul W uu where W ll (a full connected graph) denotes relations between labeled data, W uu (a k-nearest neighbor graph) denotes relations between the candidate images yet to label, and the symmetric subgraphs W ul and W lu (still k-nearest neighbor) denote the relations between positive and candidate images. The relations used to compute values in these graphs are the following: ll Wi,j =
1 n
xi −xj 2 exp − 2σ if xi ∈ knn(xj ); 2 uu = 0 otherwise; xi −xj 2 exp − 2σ if xi ∈ knn(xj ); 2 ul uu = Wi,j = 0 otherwise;
(15)
uu Wi,j
lu Wi,j
(16)
(17)
where k = 10 and σ = 1. After the first ranking by similarity, the user selects the positive feedbacks while the unselected samples are considered as negative. Then the process described in this section is iterated following the user’s need or until no more changes in the rank occurs.
6
Transductive Relevance Feedback with Feature Space Warping
The main benefit regarding the use of FSW is the possibility to move potentially irrelevant samples away from the query center, while attracting far away relevant samples toward the query center. This characteristic turns out to be very important in the the transductive learning context, especially when the samples
Feature Space Warping Relevance Feedback with Transductive Learning
77
retrieved by the system using similarity only are poor. Recalling that covariance matrices express points lying on a Riemannian Manifold and their distances, according to (1), are the geodesic distances on the manifold, we can motivate the choice of the graph Laplacian transductive approach observing that during feedback iterations we try to learn the underlying geometry of the manifold composed by positive query results points as realizations. Since the Laplacian is the discretized approximation of the Laplace-Beltrami operator on the manifold [3], we can learn the manifold structure directly from the analysis of the graph Laplacian itself. The basic idea is to strengthen, in the Laplacian, the contribution of positive query points while weakening the contribution of negatives one. This is equivalent, under a continuous relaxation of the graph affinities, to adding and removing links and path in the graph. With this premises, the user intervention allows to enhance the geodesic distances to better represent the geometry of the manifold where the query results are lying on. Additionally, it is important to remark that for transductive methods, the ratio between the number of labeled and unlabeled samples is fundamental: if the number of training samples is low as well as the number of unlabeled data goes to infinity, the learning procedure leads to uninformative membership functions [2]. To overcome this problem, we introduced a further step in the transductive learning procedure, modifying the value of W lu and W ul elements in the affinity matrix in analogy with Eq. 5. In particular, given P the set of positive samples indexes and N the set of negative ones, for each element of the affinity matrix Wijlu , its warped version Wijlu is corrected by factors δjp and δjn : δjp = − exp (−cWij ) (18) i∈P
δjn =
exp (−cWij )
(19)
i∈N
The warped elements Wijlu are finally described by the following equation:
(20) Wijlu = Wijlu + λ δjp + δjn where the global coefficients c and λ assume the same tuning functionalities as described in Section 4.
7
Experimental Results
In order to verify the effectiveness of the proposed approach, we report the results on an historical images dataset, created using the procedure described in [7], and composed of 2282 pictures. We performed an automatic simulation of relevance feedback interaction, in order to avoid human errors. We evaluate the techniques using the 171 visual queries provided with the dataset annotation, in which several prototypes of 6 object classes were retrieved (Fig. 1). We compared the following algorithms:
78
D. Borghesani et al.
Fig. 1. Samples taken from the 6 query classes
– Naive relevance feedback (actually no relevance feedback at all): the system discards the current set of n results and proposes to the user the next n, following the original rank given by the visual similarity; – MSFSW: original Mean Shift Feature Space Warping proposal by [10], with an empirically optimized set of parameter α = 0.2, β = 0.5 and γ = 0.3 for the means-shift part and λ = 0.7 and c = 0.8; – MSFSW with remapping: our modification of the original MSFSW algorithm which performs a remapping of the entire feature set using the mean of positive feedbacks as tangent point for the conversion from Riemannian Manifold to Euclidean Space; – TL with NN: the transductive learning approach which uses positive feedbacks as samples and defines the relevance feedback as a process to assign a label to unlabeled samples. The affinity matrix is filled only for the k = 20 nearest neighbors; – TL with FSW: the same transductive learning approach jointly used with Feature Space Warping on the affinity matrix Performance has been evaluated with a user-centric perspective: we choose to use metrics clearly easy to comprehend by a user in front of the application, and we included a fixed number T = 10 of iterations, to convey that the user will get bored and stop pursuing in the search at most after 10 refinements. We chose cumulative recall as representative metric, namely the recall provided by the system at each step i = 1 . . . T . While the first steps give an idea of the convergence capabilities of the algorithm, the last step give an overall evaluation of the algorithm itself. The results are presented in Table 1 and Fig. 2. The first step in the chart is the retrieval by similarity, sorted by increasing distances. The naive procedure is the baseline for the comparison. The original MSFSW proposal has a good behavior in the first step, but the deformation after the first projection limits its effectiveness in the following steps, probably due to the side effects highlighted in Section 4. Much better performance are obtained with remapping on the mean of positive feedbacks, which continually improve in substantial way up to the forth step. In the next steps, the improvement remains marginal, mimicking the behavior of the original version. The Transductive Learning approach has a milder gradient: the performance is comparable in the first steps, but the number of steps required to gather the
Feature Space Warping Relevance Feedback with Transductive Learning
79
100% 90% 80%
averagerecall
70% 60% 50% 40% 30% 20% 10% 00% 0
1
2
Naive
3
MSFSW
4
5 iterations MSFSWw/R
6
TLNN
7
8
9
10
TLFSW
Fig. 2. Comparison of the proposed techniques in terms of recall at each iteration step
Table 1. Recall values at different iteration steps method 1 2 Naive 26,7 31,1 MSFSW 34,5 40,3 MSFSW w R 36,6 50,8 TLNN 30,5 39,3 TLFSW 37,0 52,3
3 34,1 41,9 59,9 46,8 65,7
4 37,1 43,7 63,5 52,8 75,3
5 39,5 45,4 65,3 57,8 80,7
10 47,5 54,1 68,6 68,0 87,0
same performance of the other technique is too high. The proposal of this paper, instead, shows the steepest gradient in the first steps and, from the fifth steps on, it maintains an considerable increasing improvement over the best so far (MSFSW with remapping) up to 19%.
8
Conclusions
In this paper we presented a relevance feedback strategy designed to merge a transduction-based relevance feedback with the advantages of the feature space warping. In our tests, this procedure appears to be extremely promising. In the future, we plan to improve this technique in the scalability side, in other words to test it on large-scale image collections.
80
D. Borghesani et al.
References 1. Bang, H.Y., Chen, T.H.: Feature space warping: an approach to relevance feedback. In: IEEE International Conference on Image Processing, pp. 968–971 (2002) 2. Belkin, B., Srebro, N., Zhou, X.: Statistical analysis of semi-supervised learning: The limit of infinite unlabelled data. Neural Information Processing Systems (2009) 3. Belkin, M., Niyogi, P.: Semi-supervised learning on riemannian manifolds. Machine Learning 56(1-3), 209–239 (2004) 4. Chang, Y.J., Kamataki, K., Chen, T.H.: Mean shift feature space warping for relevance feedback. In: IEEE International Conference on Image Processing, pp. 1849–1852 (2009) 5. Crucianu, M., Ferecatu, M., Boujemaa, N.: Relevance feedback for image retrieval: a short survey. In: State of the Art in Audiovisual Content-Based Retrieval, Information Universal Access and Interaction including Datamodels and Languages DELOS2 Report (2004) 6. F¨ orstner, W., Moonen, B.: A metric for covariance matrices. Technical report, Stuttgart University (1999) 7. Grana, C., Borghesani, D., Cucchiara, R.: Picture Extraction from Digitized Historical Manuscripts. In: ACM International Conference on Image and Video Retrieval (July 2009) 8. Huiskes, M.J., Lew, M.S.: Performance evaluation of relevance feedback methods. In: ACM International Conference on Image and Video Retrieval, pp. 239–248 (2008) 9. Jin, X., French, J., Michel, J.: Toward consistent evaluation of relevance feedback approaches in multimedia retrieval. In: International Workshop on Adaptive Multimedia Retrival (July 2005) 10. Liu, D., Hua, K.A., Vu, K., Yu, N.: Fast query point movement techniques for large cbir systems. IEEE Transactions on Knowledge and Data Engineering 21(5), 729–743 (2009) 11. Luo, J., Nascimento, M.A.: Content-based sub-image retrieval using relevance feedback. In: ACM International Workshop on Multimedia Databases, pp. 2–9 (2004) 12. Nguyen, G.P., Worring, M., Smeulders, A.W.M.: Interactive search by direct manipulation of dissimilarity space. IEEE Transactions on Multimedia 9(7), 1404– 1415 (2007) 13. Radosavljevic, V., Kojic, N., Zajic, G., Reljin, B.: The use of unlabeled data in image retrieval with relevance feedback. In: Symposium on Neural Network Applications in Electrical Engineering, pp. 21–26 (September 2008) 14. Sahbi, H., Audibert, J.-Y., Keriven, R.: Graph-cut transducers for relevance feedback in content based image retrieval. In: IEEE International Conference on Computer Vision, pp. 1–8 (October 2007) 15. Sahbi, H., Etyngier, P., Audibert, J.-Y., Keriven, R.: Manifold learning using robust graph laplacian for interactive image search. In: IEEE International Conference on Computer Vision and Pattern Recognition, pp. 1–8 (June 2008) 16. Tao, D., Tang, X., Li, X., Wu, X.: Asymmetric bagging and random subspace for support vector machines-based relevance feedback in image retrieval. IEEE Transactions on Pattern Analysis and Machine Intelligence 28(7), 1088–1099 (2006) 17. Tieu, K., Viola, P.: Boosting image retrieval. In: IEEE International Conference on Computer Vision and Pattern Recognition, vol. 1, pp. 228–235 (2000) 18. Tuzel, O., Porikli, F., Meer, P.: Pedestrian Detection via Classification on Riemannian Manifolds. IEEE Transactions on Pattern Analysis and Machine Intelligence 30(10), 1713–1727 (2008)
Feature Space Warping Relevance Feedback with Transductive Learning
81
19. Williamson, R.E.: Does relevance feedback improve document retrieval performance? SIGIR Forum 13, 151–170 (1978) 20. Wu, Y., Tian, Q., Huang, T.S.: Integrating unlabeled images for image retrieval based on relevance feedback. In: International Conference on Pattern Recognition, vol. 1, pp. 21–24 (2000) 21. Zha, Y., Yang, Y., Bi, D.: Graph-based transductive learning for robust visual tracking. Pattern Recognition 43, 187–196 (2010) 22. Zhou, X.S., Huang, T.S.: Relevance feedback in image retrieval: A comprehensive review. Multimedia Syst. 8(6), 536–544 (2003)
Improved Support Vector Machines with Distance Metric Learning Yunqiang Liu1 and Vicent Caselles2 1
Barcelona Media - Innovation Center, Barcelona, Spain 2 Universitat Pompeu Fabra, Barcelona, Spain
[email protected],
[email protected]
Abstract. This paper introduces a novel classification approach which improves the performance of support vector machines (SVMs) by learning a distance metric. The metric learned is a Mahalanobis metric previously trained so that examples from different classes are separated with a large margin. The learned metric is used to define a kernel function for SVM classification. In this context, the metric can be seen as a linear transformation of the original inputs before applying an SVM classifier that uses Euclidean distances. This transformation increases the separability of classes in the transformed space where the classification is applied. Experiments demonstrate significant improvements in classification tasks on various data sets. Keywords: Classification, support vector machine, metric learning, kernel function.
1
Introduction
Support Vector Machine (SVM) algorithm [10] has been widely adopted in classification and pattern recognition tasks because of its very good performance. SVM is a learning technique based on statistical learning theory and the principle of structural risk minimization. By means of a kernel function, SVMs map the input data into a higher dimensional feature space making its separation easier. The kernel operation permits to avoid the explicit mapping and unnecessary computations. The kernel function plays a very important role in the performance of the SVM classifier. Between the most popular kernels let us mention the Gaussian, the polynomial and the exponential radial basis function kernels. Most of the existing kernels employed in SVM measure the similarity between pairs of (input) features using their Euclidean inner product or Euclidean distance, assuming that all features are equally scaled and equally relevant, having, thus, equal contribution to construct the separating hyperplane. However, some features can be more informative than others under the specific data distribution. Most of SVM classifiers ignore the statistical regularities which can be estimated from a training set with labeled examples. J. Blanc-Talon et al. (Eds.): ACIVS 2011, LNCS 6915, pp. 82–91, 2011. c Springer-Verlag Berlin Heidelberg 2011
Improved Support Vector Machines with Distance Metric Learning
83
Similar considerations have been partially explored in the context of feature selection and feature weighting. For example, [8] directly exploits the informativeness of features in order to determine which features to remove and which to retain. SVM-RFE (Recursive Feature Elimination) [4] uses a backwards feature elimination procedure to remove features based on the values of their weights which are obtained by a linear kernel SVM. The Entropy-based Recursive Feature Elimination(E-RFE) method [3] eliminates uninteresting features according to the entropy of the weights distribution of a SVM classifier. Wang et al. [11] incorporate data distribution information of the input space into the kernel function by constructing a weighted Mahalanobis distance kernel. The data structure for each class in the input space is found via agglomerative hierarchical clustering. This paper proposes to exploit the discriminatory information of features using a learned distance metric. Recently, distance metric learning was designed for clustering and kNN classification [6],[9]. Xing et al. [14] learn a distance metric for clustering by minimizing the distances between similarly labeled data while maximizing the distances between differently labeled data. Domeniconi et al. [1] use the decision boundaries of SVMs to induce a locally adaptive distance metric for kNN classification. Weinberger et al. [12] proposed a large margin nearest neighbor (LMNN) classification approach by formulating the metric learning problem in a large margin setting for kNN classification. In this paper, we introduce a novel classification approach which improves the performance of support vector machines (SVMs) by learning a distance metric. The metric is a Mahalanobis metric which has been learned using a training set so that examples from different classes are separated with a large margin. Then we use the learned metric to compute a kernel function for SVM classification. In this context, the metric can be seen as a linear transformation of the original inputs before applying an SVM classifier that uses Euclidean distances. Thanks to this transformation we increase the separability of classes in the transformed space where the classification is applied. Let us describe the plan of the paper. In Section 2 we briefly review the theory of SVMs for binary classification and we describe the proposed improvement using a learned Mahalanobis metric. In Section 3 we describe the method to learn the metric. We display some experiments in Section 4 showing the performance of the proposed method. We summarize our conclusions in Section 5.
2
Support Vector Machine with Metric
In this section, we propose an improvement of SVMs classification based on the use of a Mahalanobis metric. For that, let us briefly review the theory of SVMs for binary classification. Assume that we are given a set of training data {(xi , yi )}ni=1 , where xi are in the vector space Rd , and yi ∈ {−1, 1} is the class label of vector xi , i = 1, 2, ..., n. Recall that, to obtain the optimal separating
84
Y. Liu and V. Caselles
hyperplane which minimizes the training error and maximizes the margin, one solves the following constrained optimization problem: 1 T w w+C ξi 2 i=1 n
min
w,b,ξi
subject to ξi ≥ 0
yi (wT φ(xi ) + b) ≥ 1 − ξi ,
(1)
i = 1, . . . , n,
where φ : Rd → Rm is a kernel feature mapping, w ∈ Rm , b ∈ R determine the hyperplane in the feature space, ξi are slack variables, and C is the regularization parameter which determines the trade-off between the maximization of margin and the amount of misclassification errors. The above optimization problem is solved using its dual: max ai
n
ai −
i=1
subject to
n 1 ai aj yi yj k(xi , xj ) 2 i,j=1 n ai yi = 0, 0 ≤ ai ≤ C,
(2) i = 1, . . . , n,
i=1
which in turn is solved using quadratic programming. The kernel function k(xi , xj ) = φ(xi ), φ(xj ), where , denotes an inner product between two vectors, is introduced to handle nonlinearly separable cases without an explicit construction of the feature mapping φ. The input vector x is classified according to the output value n f (x) = sign ai yi k(xi , x) + b . (3) i=1
In SVM classification, a proper choice of the kernel function is necessary to obtain good results. The kernel function determines the degree of similarity between two data vectors. One of the most common being the exponential radial basis function (RBF) kernel krbf (xi , xj ) = exp(−γd(xi , xj )), γ > 0,
(4)
where γ > 0 is the width of the Gaussian, and d(xi , xj ) is the squared distance between xi and xj . In many cases, the distance is just the Euclidean distance. This is reasonable when all features are equally scaled and equally relevant. Unfortunately, some features can be more informative than others under the specific data distribution. Euclidean distances ignore the statistical regularities which can be estimated from a training set with labeled examples. Ideally, we would like to find an adaptive distance metric that can be adapted to the particular data set being resolved.
Improved Support Vector Machines with Distance Metric Learning
85
Based on this consideration, we propose an improved support vector machine, named SVM-M, by learning a Mahalanobis metric. Specifically, we wish to learn a squared distance of the form: dM (xi , xj ) = (xi − xj )T M (xi − xj ),
(5)
where M = P T P is a positive semi-definite matrix (by AT denotes the transpose of the matrix A). As we will describe in next section, the distance is trained so that examples from different classes are separated with a large margin. Using the matrix M , we can transform the examples by x ˆi = P T xi , i = 1, . . . , n. The purpose of the linear transformation is to separate the differently labeled examples by a large margin in the transformed space, so that examples from the same class move closer and those from different classes move away, increasing the separability of classes. Once the distance metric is obtained, we could construct the SVM-M by ˆi . In practice, substituting the input feature vector xi by the transformed one x it is not necessary to know exactly the transformation matrix P , it suffices to define the kernel function using the metric obtained. Taking the exponential RBF kernel, we define: krbf,M (xi , xj ) = exp(−γdM (xi , xj )), γ > 0,
(6)
where dM (xi , xj ) is the squared distanced given in (5). Then SVM-M is built by substituting the original kernel function in (2) and (3) by the kernel (6).
3
Distance Metric Learning
We describe in this section the proposed method to learn a squared distance of the form (5). The desired metric is trained with the goal that examples from different classes are separated with a large margin, while the distance between examples in the same class is minimized. Ideally, in order to separate the examples from different classes, the squared distance dM should satisfy the constraints dM (xi , xk ) > dM (xi , xj ) ∀(i, j, k) ∈ D,
(7)
where D is the set of triple indexes: D := {(i, j, k) : yi = yj , yi = yk }.
(8)
Since it may be not possible to satisfy all these constraints simultaneously, as usual, we introduce the slack variables ξijk , so that: dM (xi , xk ) − dM (xi , xj ) ≥ 1 − ξijk
∀(i, j, k) ∈ D.
(9)
86
Y. Liu and V. Caselles
On the other hand, the distance between examples of the same class should be small. Therefore, we formulate the following optimization problem: min
M,ξijk
dM (xi , xj ) + C
ξijk ≥ 0
ξijk
i=1
(i,j)∈S
subject to
n
dM (xi , xk ) − dM (xi , xj ) ≥ 1 − ξijk ,
(10)
∀(i, j, k) ∈ D,
M ≥ 0, where S is the set of example pairs which belong to the same class, and C is a positive constant. The constraint M ≥ 0 indicates that the matrix M has to be positive semi-definite. As usual, the slack variables ξijk allow a controlled violation of the constraints. A non-zero value of ξijk allows a triple (i, j, k) ∈ D not to meet the margin requirement at a cost proportional to ξijk . The optimization problem (10) is an SDP problem and can be solved using standard optimization software. However, the large amount of constraints in (9) makes the problem computationally intensive for large data sets. As in [12], in order to overcome this difficulty, we select a subset of these constraints and redefine S and D as: S := {(i, j) : yi = yj , ηij = 1},
(11)
D := {(i, j, k) : yi = yj , yi = yk , ηij = 1, ηik = 1},
(12)
where ηij indicates whether example j is a neighbor of example i, the neighboring relations being defined by the Euclidean distance. Notice that (11) and (12) restrict the pairs of neighbors. This is reasonable, since only the examples which are neighbor but do not share the same class label should be separated using the learned metric, while we do not care about pairs which belong to different classes and have originally a large distance. We solved the problem (10) using the algorithm proposed in [12]. In our experiments we constructed the sets (11) and (12) using the five nearest neighbors (with respect to the Euclidean distance) of each input data.
4
Experiments
We evaluated the SVM-M approach for classification tasks on several public data sets: Iris and Balance data sets from the UCI Machine Learning Repository [2], US Postal Service (USPS) data set [5], Graz-02 [7], and MSRC-2 [13]. Iris and Balance are low dimensional data sets, both with 4 dimensions. USPS is a data set of gray level images of the 10 digits from ’0’ to ’9’ and we represent each of them using a vector obtained by applying principal component analysis (PCA). Graz-02 and MSRC-2 are also image data sets, and we use the
Improved Support Vector Machines with Distance Metric Learning
87
bag-of-words model to represent those images by a high dimensional descriptor. We compare the proposed approach with standard KNN, LMNN (large margin nearest neighbor) and standard SVM classification methods. In the SVM and SVM-M methods, we use the RBF kernels (4) and (6), respectively. Parameters such as the number of neighbors in KNN and the regularization parameter C are determined using k-fold (k = 5) cross validation. 4.1
Low Dimension Data Sets
Iris and Balance are low dimensional data sets, both with 4 dimensions. The Iris data set includes 150 examples in 3 classes: Setosa, Versicolour, Virginica. The Balance data set includes 625 examples in 3 classes: {L, B, R}. Examples are randomly divided into two subsets of the same size to form a training set and a testing set. Multiclass classification is obtained from two-class SVM using the one-vs-all strategy. We repeat each experiment five times with different splits, and report the average results. Table 1 shows the classification ratio on the two data sets. Our results show a significant improvement on both data sets. The confusion matrix for the Balance data set using the SVM-M classifier is presented in Table 2 in order to give more details on the categorization of each class (for one implementation). The first column contains the true labels and the first row lists the referred labels. The computation time required to compute the matrix M by solving (10) is around 1.2s and 4.7s for Iris and Balance data sets, respectively (using a CPU of 2.67GHz). Table 1. Classification results (%) on low dimension data sets KNN LMNN SVM SVM-M Balance 87.5 89.9 95.8 97.2 Iris 94.7 96.3 96.0 98.7 Table 2. Confusion matrix for Balance using SVM-M Class L B R L 143 0 0 B 4 17 3 R 0 0 145
4.2
Image Data Sets
The USPS handwritten digits data set was created for testing classification methods for handwritten digits on envelopes. It contains 11,000 normalized gray-scale digit images, with 1,100 examples for each of the 10 digits (from ’0’ to ’9’). Images have a normalized size of 16 × 16 pixels. We apply principal component analysis (PCA) to reduce its dimensionality before classification and each image is projected onto their leading principal components. In our experiments we use
88
Y. Liu and V. Caselles
Fig. 1. Classification ratio (%) on USPS
Table 3. Confusion matrix for the USPS data ser using SVM-M 1 2 3 4 5 6 7 8 9 0 1 548 0 2 0 0 0 0 0 0 0 2 2 538 2 1 1 1 2 0 1 2 3 1 5 525 0 8 0 5 5 1 0 4 1 1 0 540 0 4 0 1 3 0 5 0 1 6 0 531 1 0 8 1 2 6 0 7 0 6 2 533 0 1 0 1 7 0 0 0 1 0 0 546 0 3 0 8 1 0 2 2 8 0 2 533 2 0 9 0 0 3 2 1 0 5 1 538 0 0 1 0 0 0 0 2 2 0 0 545
the first 30 leading components since they permit to obtain a good classification ratio. The training and testing sets are created by randomly dividing the data set into two parts (with specified percentages). We repeat each experiment five times with different splits for each fixed percentage of the training set, and we report the average classification ratio. Fig. 1 shows the classification results obtained varying with the percentage of training set. Table 3 shows the confusion matrix for the USPS data set (for one execution) corresponding to the SVM-M classifier when the percentage of the training set is 50%. The first column lists the true labels and the first row are the obtained labels. The numbers of correctly classified images for each category are shown in the diagonal. In this case, the computation time for obtaining the metric M takes on average 41s. For the image data sets Graz-02 and MSRC-2, we use a bag-of-words model to represent the images. In our experiments, we use only the gray level information, although there may be room for further improvement by including color information.
Improved Support Vector Machines with Distance Metric Learning
a) bike
b) car
c) person
d) background
89
Fig. 2. Example images from Graz-02, one image per class
Our image representation follows the standard bag of words model. Each image is divided into equivalent blocks on a regular grid with spacing d = 7 both in horizontal and vertical directions. We take the set of grid points as keypoints, each with a circular support area of radius r = 5 (in our experiments). Each support area can be taken as a local patch. Notice that the patches overlap. Each patch is described by a SIFT (scale-invariant feature transform) descriptor. Then a visual vocabulary is built-up by vector quantizing the descriptors using a clustering algorithm such as K-means. Each resulting cluster corresponds to a visual word. With this vocabulary, each descriptor is assigned to its nearest visual word in it. The vector containing the number of each visual word in the image is used as a feature vector. We carried out binary classification experiments with Graz-02. Graz-02 contains 3 object classes {bike, car, person} and a background class. It is a challenging data set and, as example, some images are shown in Fig. 2. We used the first 200 images of each class. Moreover, images within each class are randomly divided into two subsets of the same size to form a training and a testing sets. We set the visual vocabulary size as 200. The binary classification is performed as object class against background. We report in Fig. 3 the averaged results over five experiments with different random splits of the data set. The SVM-M method consistently outperforms the standard SVM and LMNN classifiers. In the data set MSRC-2, there are 20 classes and 30 images per class. In our experiment, we choose six classes out of them: {tree, cow, face, car, bike, and book}. The visual vocabulary size is set as 100. Moreover, we randomly divided
90
Y. Liu and V. Caselles
Fig. 3. Classification ratio (%) on image data set Graz-02 Table 4. Classification results (%) on MSRC-2 KNN LMNN SVM SVM-M 82.2 83.3 86.7 88.9
the images within each class into two groups of the same size to form a training and a testing sets. We repeated each experiment five times using different splits, and we report the averaged results in Table 4.
5
Conclusions
In this paper, we have proposed to improve the performance of support vector machines classification by learning a distance metric which captures the discriminatory information of features. The learned metric is used to define the kernel function for SVM classification. In this context, using the new metric can be interpreted as a linear transformation of the original inputs before applying the SVM classifier with Euclidean distances. Because the distance metric is trained so that the examples from different classes are separated with a large margin, we increase their separability in the transformed space where the classification is applied. Experiments on various data sets show that the proposed method permits to obtain significant improvements in classification tasks. Acknowledgements. This work was partially funded by Mediapro through the Spanish project CENIT-2007-1012 i3media and by the Centro para el Desarrollo Tecnol´ogico Industrial (CDTI). The authors acknowlege partial support by the EU project “2020 3D Media: Spatial Sound and Vision” under FP7-ICT. Y. Liu also acknowledges partial support from the Torres Quevedo Program from the Ministry of Science and Innovation in Spain (MICINN), co-funded by the European Social Fund (ESF). V. Caselles also acknowledges partial support by MICINN project, reference MTM2009-08171, by GRC reference 2009 SGR 773 and by “ICREA Acad`emia” prize for excellence in research funded both by the Generalitat de Catalunya.
Improved Support Vector Machines with Distance Metric Learning
91
References 1. Domeniconi, C., Gunopulos, D., Peng, J.: Large margin nearest neighbor classifiers. IEEE Transactions on Neural Networks 16(4), 899–909 (2005) 2. Frank, A., Asuncion, A.: UCI machine learning repository, university of california, irvine, school of information and computer sciences (2010), http://archive.ics. uci.edu/ml 3. Furlanello, C., Serafini, M., Merler, S., Jurman, G.: An accelerated procedure for recursive feature ranking on microarray data. Neural Networks 16(5-6), 641–648 (2003) 4. Guyon, I., Weston, J., Barnhill, S., Vapnik, V.: Gene selection for cancer classification using support vector machines. Machine learning 46(1), 389–422 (2002) 5. LeCun, A., Boser, B., Denker, J.S., Henderson, D., Howard, R.E., Hubbard, W., Jackel, L.D.: Backpropagation applied to handwritten zip code recognition. Neural Computation, 541–551 (1989) 6. Nguyen, N., Guo, Y.: Metric Learning: A Support Vector Approach. Machine Learning and Knowledge Discovery in Databases, 125–136 (2008) 7. Opelt, A., Pinz, A., Fussenegger, M., Auer, P.: Generic object recognition with boosting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 416– 431 (2006) 8. Rakotomamonjy, A.: Variable selection using svm based criteria. The Journal of Machine Learning Research 3, 1357–1370 (2003) 9. Ramanan, D., Baker, S.: Local distance functions: A taxonomy, new algorithms, and an evaluation. In: International Conference on Computer Vision, pp. 301–308. IEEE, Los Alamitos (2009) 10. Vapnik, V.N.: The nature of statistical learning theory. Springer, Heidelberg (2000) 11. Wang, D., Yeung, D.S., Tsang, E.C.: Weighted mahalanobis distance kernels for support vector machines. IEEE Transactions on Neural Networks 18(5), 1453–1462 (2007) 12. Weinberger, K.Q., Saul, L.K.: Distance metric learning for large margin nearest neighbor classification. The Journal of Machine Learning Research 10, 207–244 (2009) 13. Winn, J., Criminisi, A., Minka, T.: Object categorization by learned universal visual dictionary. In: IEEE International Conference on Computer Vision, vol. 2, pp. 1800–1807. IEEE, Los Alamitos (2005) 14. Xing, E.P., Ng, A.Y., Jordan, M.I., Russell, S.: Distance metric learning with application to clustering with side-information. Advances in Neural Information Processing Systems, 521–528 (2003)
A Low-Cost System to Detect Bunches of Grapes in Natural Environment from Color Images Manuel J.C.S. Reis1, , Raul Morais2, Carlos Pereira3, Olga Contente4 , Miguel Bacelar3 , Salviano Soares1 , Ant´onio Valente1 , Jos´e Baptista3 , Paulo J.S.G. Ferreira5, and Jos´e Bulas-Cruz3 1
4
IEETA/UTAD, ECT, Dept. Engenharias, 5001-801 Vila Real, Portugal {mcabral,avalente,salblues}@utad.pt 2 CITAB/UTAD, ECT, Dept. Engenharias
[email protected] 3 UTAD, ECT, Dept. Engenharias
[email protected], {mbacelar,baptista,jcruz}@utad.pt Escola Superior de Tecnologia de Viseu, Dep.Eng.Mec.Ges.Ind., 3504-510 Viseu
[email protected] 5 IEETA/Universidade de Aveiro, Campus Univ. Santiago, 3810-193 Aveiro
[email protected]
Abstract. Despite the benefits of precision agriculture and precision viticulture production systems, its adoption rate in the Portuguese Douro Demarcated Region remains low. One of the most demanding tasks in wine making is harvesting. Even for humans, the environment makes grape detection difficult, especially when the grapes and leaves have a similar color, which is generally the case for white grapes. In this paper, we propose a system for the detection and location, in the natural environment, of bunches of grapes in color images. The system is also able to distinguish between white and red grapes, at the same time, it calculates the location of the bunch stem. The proposed system achieved 97% and 91% correct classifications for red and white grapes, respectively. Keywords: precision viticulture, visual inspection, grape detection, image processing.
1
Introduction
Precision agriculture (PA) and precision viticulture (PV) are production systems that promote variable management practices within a field according to site-specific conditions. The concept is based on new tools and information sources provided by modern technologies, such as yield monitoring devices, soil, plant and pest sensors and remote sensing [1, 2]. PA and PV generally have two main objectives: to render the production more cost-effective and to reduce its
Corresponding author.
J. Blanc-Talon et al. (Eds.): ACIVS 2011, LNCS 6915, pp. 92–102, 2011. c Springer-Verlag Berlin Heidelberg 2011
A Low-Cost System to Detect Bunches of Grapes
93
environmental impact. The first objective can be achieved by reducing production costs and improving productivity. The second objective relates to the accuracy and the ability to control the application of production factors, such as chemicals, which should be done within a fair measure of the real needs of crops. Despite the benefits, such diversity restrains the rate of adoption of these technological tools, which varies considerably from country to country, and from region to region [3, 4]. In addition to environmental benefits, such as those related to better water and nutrient management, PA and PV systems can increase worker productivity, augment product throughput and improve selection reliability and uniformity. Additionally, the use of robotic systems in agriculture has seen a sharp increment in recent years. Many systems have been described for the automation of various harvesting processes, such as for fruit location, detachment and transfer, among other, [6]. They have been used in the harvesting of, for example, melons [5], cucumbers [7], tomatoes [8] and oranges [9]. For a survey of several aspects of these systems see, for example, [10] and [11]. The wine industry (see the report of the US Department of Energy [12]) is interested in using autonomous robotic systems for grape harvesting, phytosanitary treatments, among other very time and human resources consuming tasks, for multiple reasons; this is particularly true for the Douro Demarcated Region (DDR) of Portugal as explained below. Harvesting conditions, in particular, affect wine quality, and several techniques need to be used in order to produce quality wines, and including scheduling details [13], and ending up with production details [14, 15, 16]. The DDR (a UNESCO World Heritage Site and the oldest Wine Demarcated Region of the World, and where the Port wine is actually produced and then shipped at the city of OPorto), due to its unique characteristics, poses very specific challenges, mainly due to the topographic profile, pronounced climatic variations and complex soil characteristics. It is located in northeast Portugal and consists mostly of steep hills (slopes reaching 15%) and narrow valleys that flatten out into plateaux above 400m. The Douro river dug deeply into the mountains to form its bed, and the dominant element of the landscape are the vineyards, planted in terraces fashioned from the steep rocky slopes and supported by hundreds of kilometres of drystone walls. Grape harvest and disease predictions, as well as the assessment of the grape value, are currently left to the grape-growers, without the help of decision-support mechanisms, in an environment where no significant irrigation systems exists. Traditionally, vineyards have been harvested by humans. However, harvesting is difficult, particularly in a region with the topographic and climatic characteristics of the DDR. The manpower needs are large and it is getting more and more difficult to find qualified workers. Autonomous systems could not only reduce the harvesting cost and manpower needs, they could also work during the night. There is however one important constraint: although it is not essential that the work rate of a harvesting robot surpasses that of a human, it is crucial that they satisfy quality control levels at least similar to those achieved
94
M.J.C.S. Reis et al.
by humans. Also, the existing machines harvest grapes by striking the vine, a process that is not recommended for some wines, such as champagne, for chemical reasons (e.g., oxidation), but also because of some deposits being collected with the grapes. These machines need, at least, one operator to harvest. They also require previous preparation of the vineyard, such as cutting the tips. Within the harvesting process the first step to take is to locate the bunches of grapes. This can be done using visual inspection. Unfortunately, this location process is much easier for fruits than for vine grapes; see, for example, [17, 18]. Even for humans, the environment makes grape detection difficult, especially when the grapes and leaves have a similar color. To an automatic recognition application, the bunches of grapes can appear at different scales, they can be partially occluded by leaves, the luminance of images can vary widely due to the sun, clouds, leaves, and so on, changing environmental conditions. Many works have been devoted to the definition of object invariant descriptors for simple geometric transformations; see, for example, [19, 20]. The Zernike moments [21, 22, 23] have been developed to overcome the major drawbacks of regular geometrical moments regarding noise effects and image quantization error. Zernike moments were successfully used in the detection of red grapes [24], but nothing is known about their performance in connection with white grapes. Moreover, besides the difficulties associated with the calculation of the Zernike moments, the method proposed by Chamelat et al. [24] implies two distinct phases, training and recognition, the training phase being crucial to the results of the whole system. The computation time for the learning step, for 17 images, is reported to take 5 minutes on a 2.8 GHz Pentium 4. The computation time for the recognition step (identification of each block of size 16 × 16 pixels) takes less than one second. We have tried to use Zernike moments to detect white grapes but with questionable success (with less than 50% of correct classifications). In this paper we propose a much simpler method for the detection and location, in natural environment, of bunches of grapes in color images, using only very simple and basic image processing techniques. The system is also able to distinguish between white and red grapes. Additionally, it can also calculate the location of the bunch stem, and can be used to help guiding a harvesting robot. The system presented here represents part of an effort that is being made by our team to help with the introduction of PA and PV in the farmers’ everyday practices in the DDR [25, 26], and is intended to be used in an autonomous harvesting robotic system. This paper is organized as follows. In the next section, we present the proposed system. Its performance and efficiency is discussed in section 3, where experimental results obtained with real images are given. Section 4 is devoted to the conclusions and future work.
2
The Grape Recognition System
Very soon in our work we have realized that if we can solve the problem of detecting bunches of white grapes we also solve the problem of detecting red grapes,
A Low-Cost System to Detect Bunches of Grapes
95
as it will become clear in the results section. Consequently, we have focused our efforts on the detection of bunches of white grapes. For the reasons presented in the previous section, we want that the whole system to be able to work during night conditions (i.e., darkness, with very little, or none, brightness variations), and also have the ability to distinguish between red and white grapes. To this end, the system simply makes a first pass through the original (night captured) image, counting the number of pixels that are “inside” limits of the Red, Green and Blue (RGB) components of (044, 051, 064), (033, 041, 054), (055, 062, 075), and (018, 024, 036), for red grapes, and (102, 108, 089), (095, 104, 085), (076, 090, 078), and (083, 089, 038) for white grapes. These four central values (colors) were experimentally (trial and error) determined during the development phase. The system seeks for pixels within the “limits” of these central values: the default limit contains all the values within 8% of these central values for red grapes and 15% for white grapes (these were the values experimentally determined from the night captured images). The biggest counting indicates the type of grape, i.e., red or white.
(a)
(b)
Fig. 1. Processing steps: (a) original image containing bunches of red grapes; (b) color mapping
If initial conditions are known, i.e., if we know that a parcel consists of a single type of grape (say white), then the system can be switched to that type of grape mode (say white mode), skipping the grape identification phase. Most of the vineyards in the DDR are characterized by their small area size, and by having more than one type of grape (red or white and even different types of caste) per parcel and even for bard, particularly in the case of old vines (the more recent vines were constructed and organized having in mind better control of the number and type of castes—and color, i.e., red and white—, but still, more than 120 different castes are used in the region). Once the type of grape (red or white) is established, the system follows three additional steps: color mapping, morphological dilation, black areas and stem
96
M.J.C.S. Reis et al.
detection. The color mapping step is done based on the conditions established during the grape identification step. At the end of this step we will have a binary image (black for the pixels in the range and white for the other pixels). Figure 1(a) shows the original image, containing bunches of red grapes, and figure 1(b) presents the resulting image of the application of color mapping step.
0 1 (a)
(b)
Fig. 2. Processing steps: (a) morphological dilation; (b) black areas (bunches) and stem detection step
As we can see in figure 1(b) the resulting image does not generally have a uniform (continuous) black region, but several regions where the concentration of black pixels is greater. The morphological dilation operation is meant to fill in the gaps between the pixels, yielding uniform black regions. We have used a square of 3 × 3 pixels as the structuring element, typically one of the most simple and used structuring elements. We should recall that the shape of this structuring element is not very far from the shape of a grape berry. Because the regions resulting from the color mapping step when applied to red grapes and white grapes are very different, with a greater sparsity when dealing with white grapes, we have typically used 60 iterations of the morphological dilation for red grapes, and 100 iterations for white grapes. Figure 2(a) shows the resulting image after the dilation operation, applied to the image presented in figure 1(b). Obviously, we have tried different sizes for the structuring element, but when its size was increased, even with a lower number of morphological dilation iterations, the number of overlapping uniform black regions tends to increase; these overlappings will correspond to incorrect overlapping bunches, leading to incorrect identifications. The final step is concerned with black regions and stem detection. First, the number of contiguous regions are counted and labeled (numbered) using 8-connectivity. Then, for each region, its width, height and area are calculated
A Low-Cost System to Detect Bunches of Grapes
(a)
97
(b)
Fig. 3. Example of identification final result: (a) original image; (b) identification result
(in pixels) so that we can discard false detections of very small bunches or very large regions to contain only one bunch of grapes. Note that in this last case the region will be composed of two or more bunches and the system will need to capture more images (from a different angles) to correctly handle these situations. Now, these parameters, and also the total number of admissible areas, are all adjustable. Next, we count the number of regions, center, area, width, height, perimeter and boundaries, for each region. For each region, and based on the pixel distribution and density around its center, it is determined the “horizontal” (width) and “vertical” (hight) axes of the bunch, i.e., the bunch orientation. Then, with these axes (and orientation), and with the region’s limits, we locate the bunch stem. Figure 2(b) shows the resulting image after the final step and figure 3 shows another example consisting of the original image and the resulting identification. Table 1. Summary of grape detection results. Correct means that all bunches present in each image were correctly identified. Incorrect means that the system had classified some areas of the image has if they were bunches although no bunches at all were present, or missed the identification of an existing bunch. White Correct 172 (91%) Incorrect 18 (9%) Totals 190 (100%) images
3
Red 34 (97%) 1 (3%) 35 (100%) images
Results and Discussion
The images presented here and used to test the system were captured during the night, with a Panasonic FZ28 camera (http://www.panasonic.co.uk/ html/en_GB/1258590/index.html), simply using its internal flash, i.e., no other lightning system was used. By capturing the imagens during night we avoid any spurious reflections or bad illumination during sunny day time, like momentary presence of clouds, but also, as explained above, some importante wine chemical
98
M.J.C.S. Reis et al.
properties (e.g., oxidation). In total, there were 190 images containing bunches of white grapes, and 35 images of red grapes (the number of images containing bunches white grapes is much higher than that of red grapes because the identification of bunches of red grapes is much simpler, as explained above). A summary of the results can be seen in table 1. As we can see, there were 172 correct results for bunches of white grapes; this means that all bunches present in each of these 172 images were correctly identified. We emphasize this fact, because we want that in a near future this system can be part of a harvesting robot, and we know that most of the infield images captured by this robot could contain more than one bunch per image, which clearly is not an optimal situation, but rather real. Recall that in a practical infield situation the robot can take as many images as needed to ensure that no more bunches are present for harvesting. However, we also have 18 images with incorrect or false detections. This means that the system classified some areas of the image as if they were bunches although no bunches at all were present, or missed the identification of an existing bunch. Also, if two regions were merged during the morphological dilation process they were counted as a “incorrect” result. As we have foreseen, the system’s performance, in percentage, is better for bunches of red grapes than for white grapes. Recall that the color of red grapes is very different (contrasting) from their surroundings (e.g., color of the leaves); however one must note the low number of red grape sample images.
(a)
(b)
Fig. 4. Example of a possible robot’s positioning correction by noticing “how far” it is from the bunches: (a) original image; (b) identification result
Additionally, we also argue that the system presented here can help guiding the robot. As it can be seen in figure 4, although the picture was captured very far way form the grape bunches the system manages to identify the presence of bunches of grapes. So, the system can help to tell the robot to move along that direction, adjusting (fine tuning) its position or trajectory. Usually, the presence of many bunches of grapes in one image indicates that the robot is at a considerably far distance from those bunches. Obviously, this can only be a contribution to the robot’s trajectories or positioning. In the examples of figures 5 and 6, for example, we can see that the bunch in the center of the figure is correctly identified. However, we can see that there are more bunches to be harvested.
A Low-Cost System to Detect Bunches of Grapes
99
Fig. 5. Example of overlapping bunches and identification result
Fig. 6. Example of a possible robot’s positioning correction by noticing the presence of more bunches; (a) original image, (b) in this case, only the central bunch was correctly identified, so more pictures are needed (see text).
By capturing one more image after the harvesting of the central bunch the system will detect the presence of more bunches and then correct the robots trajectory, as in the previous situation. As noted before, the presence of leaves may (partially) occlude a bunch; in these cases a mechanic system like the one presented in [27], basically a blower/fan to remove leaves, can be used to help solve this problem. We tested the system on a computer running Microsoft Windows XP Home Edition, with an Intel Core Duo Processor T2300 at 1.66 GHz, Mobile Intel 945PM Express Chipset, 3GB DDR2 667MHz SDRAM, and NVDIA GeForce Go 7300 External 128MB VRAM video card. In order to reduce the identification time, we tested several image resolutions, as presented in table 2. As we can see, even with a resolution of 1.3 Mega-pixel (MP) the system is able to produce accurate results. Note that the identification of bunches of red grapes takes less time, because it uses a lower number of iterations during the dilation operation step.
100
M.J.C.S. Reis et al.
Table 2. Summary of grape detection time versus image resolution (seconds versus Mega Pixels) 10MP 3MP 1.3MP White 1.5 s 0.22 s 0.16 s Red 1.0 s 0.15 s 0.08 s
As for the system cost, nowadays the price of the Panasonic FZ28 camera is about 300 e. The processing unit may be one of the ones used to control the robot, but if we choose to use another separate computer board we may purchase it bellow 200 e. This gives us a total bellow 500 e.
4
Conclusions and Ongoing Work
Within the context of Precision Agriculture/Viticulture, and because the Douro Demarcated Region has its own very particular characteristics, a vision inspection system was developed in order to identify bunches of grapes, for later inclusion in a robotic system for night conditions harvesting. As explained above, this system is also able to automatically distinguish between white and red grapes. Additionally, it can also calculate the location of the bunch stem, and can be used to help the robot’s location and guiding system. The system was targeted to identify bunches of grapes, during night conditions, and we have achieved 97% and 91% of correct classifications, for red and white grapes, respectively. Concerning the identification algorithm we are trying to make it more flexible. For example, during the morphological dilation step, the number of iterations should depend on the merging of the different regions, i.e., the iteration process should stop if the regions are already contiguous or if two distinct regions are merged. Also, the tilling and distribution of these contiguous regions should be analyzed in order to prevent incorrect identifications, like two or more staked regions, regions with much greater width than wight, among other. Obviously, a dedicated (specific) lighting system may help in producing better identification results (starting with a simple diffuser), but it would also increase the system cost and needed resources (e.g., power supply). We are also currently testing multi-spectral cameras. This solution is far more expensive (when compared with the about 300 e of the Panasonic FZ8 camera), but it could bring information about grapes maturation and alcoholic level. A cheaper alternative may include infra-red cameras, but red grapes yield more thermal information than white grapes, and so, once again, the identification of red grapes seems simpler than white grapes. In addition to the tasks of detection and harvesting, the robotic system may also contribute to mitigate the environmental impact of chemical plant protection products, as its application by robotic systems would be made only at points of interest identified by the vision system, among other possible applications (pruning, trimming the vines, disease detection, etc.).
A Low-Cost System to Detect Bunches of Grapes
101
References [1] Bramley, R.G.V.: Precision viticulture — research supporting the development of optimal resource management for grape and wine production (2001), http://www. crcv.com.au/research/programs/one/workshop14.pdf [2] Bramley, R.G.V.: Progress in the development of precision viticulture — variation in yield, quality and soil properties in contrasting australian vineyards (2001), http://www.crcv.com.au/research/programs/one/bramley1.pdf [3] Seelan, S., Laguette, S., Casady, G., Seielstad, G.: Remote sensing applications for precision agriculture: a learning community approach. Remote Sensing of Environment 88(1-2), 157–169 (2003) [4] Tongrod, N., Tuantranont, A., Kerdcharoen, T.: Adoption of precision agriculture in vineyard. In: ECTI-CON: 2009 6th International Conference on Electrical Engineering/Electronics, Computer, Telecommunications and Information Technology, vol. 1 and 2, pp. 695–698 (2009) [5] Edan, Y., Rogozin, D., Flash, T., Miles, G.: Robotic melon harvesting. IEEE Transactions on Robotics and Automation 16, 831–835 (2000) [6] Sistler, F.: Robotics and intelligent machines in agriculture. IEEE Journal of Robotics and Automation 3, 3–6 (1987) [7] Van Henten, E., Hemming, J., Van Tuijl, B., Kornet, J., Meuleman, J., Bontsema, J., Van Os, E.: An autonomous robot for harvesting cucumbers in greenhouses. Autonomous Robots 13, 241–258 (2002) [8] Monta, M., Kondo, N., Ting, K.: End-effectors for tomato harvesting robot. Artifficial Intelligence Review 12, 11–25 (1998) [9] Recce, M., Taylor, J., Plebe, A., Tropiano, G.: Vision and neural control for an orange harvesting robot. In: Proceedings of the 1996 International Workshop on Neural Networks for Identification, Control, Robotics, and Signal/Image Processing (NICROSP 1996), p. 467 (1996) [10] Jimenez, A., Jain, A., Ceres, R., Pons, J.: Automatic fruit recognition: A survey and new results using range/attenuation images. Pattern Recognition 32, 1719– 1736 (1999) [11] Sarig, Y.: Robotics of fruit harvesting: A state-of-the-art review. Journal of Agricultural Engineering Research 54, 265–280 (1993) [12] U.S. Department of Energy. Assessment study on sensors and automation in the industries of the future:reports on industrial controls, information processing, automation, and robotics. Technical report, U.S. Department of Energy, Energy Efficiency and Renewable Energy, Industrial Technologies Program (2004) [13] Bohle, C., Maturana, S., Vera, J.: A robust optimization approach to wine grape harvesting scheduling. European Journal of Operational Research 200(1), 245–252 (2010) [14] Mateo, E.M., Medina, A., Mateo, F., Valle-Algarra, F.M., Pardo, I., Jimenez, M.: Ochratoxin A removal in synthetic media by living and heat-inactivated cells of Oenococcus oeni isolated from wines. Food Control 21(1), 23–28 (2010) [15] Lopes, M.S., Mendonca, D., Rodrigues dos Santos, M., Eiras-Dias, J.E., da Camara Machado, A.: New insights on the genetic basis of Portuguese grapevine and on grapevine domestication. Genome 52(9), 790–800 (2009) [16] Marmol, Z., Cardozo, J., Carrasquero, S., Paez, G., Chandler, C., Araujo, K., Rincon, M.: Evaluation of total polyphenols in white wine dealt with chitin. Revista De La Facultad De Agronomia De La Universidad Del Zulia 26(3), 423–442 (2009)
102
M.J.C.S. Reis et al.
[17] Rosenberger, C., Emile, B., Laurent, H.: Calibration and quality control of cherries by artificial vision. International Journal of Electronic Imaging, Special Issue on Quality Control by Artificial Vision 13(3), 539–546 (2004) [18] Lu, J., Gouton, P., Guillemin, J., My, C., Shell, J.: Utilization of segmentation of color pictures to distinguish onions and weeds in field. In: Proceeding of International Conference on Quality Control by Artificial Vision (QCAV), vol. 2, pp. 557–562 (2001) [19] Petrou, M., Kadyrov, A.: Affine invariant features from the trace transform. IEEE Transactions on Pattern Analysis and Machine Intelligence 26(1), 30–44 (2004) [20] Jain, A., Duin, R., Mao, J.: Statistical pattern recognition: A review. IEEE Transactions on Pattern Analysis and Machine Intelligence 22(1), 4–37 (2000) [21] Choksuriwong, A., Laurent, H., Emile, B.: Comparison of invariant descriptors for object recognition. In: IEEE International Conference on Image Processing, ICIP 2005, vol. 1, pp. I-377–I-380 (September 2005) [22] Chong, C., Raveendran, P., Mukundan, R.: Mean shift: A comparative analysis of algorithms for fast computation of Zernike moment. Pattern Recognition 36, 731–742 (2003) [23] Khotanzad, A., Hong, Y.: Invariant image recognition by Zernike moments. IEEE Transactions on Pattern Analysis and Machine Intelligence 12(5), 489–497 (1990) [24] Chamelat, R., Rosso, E., Choksuriwong, A., Rosenberger, C., Laurent, H., Bro, P.: Grape detection by image processing. In: IECON 2006 - 32nd Annual Conference on IEEE Industrial Electronics, vol. 1-11, pp. 3521–3526 (2006) [25] Morais, R., Fernandes, M., Matos, S., Serˆ odio, C., Ferreira, P., Reis, M.: A ZigBee multi-powered wireless acquisition device for remote sensing applications in precision viticulture. Computers and Electronics in Agriculture 62(2), 94–106 (2008) [26] Morais, R., Matos, S., Fernandes, M., Valente, A., Soares, S., Ferreira, P., Reis, M.: Sun, wind and water flow as energy supply for small stationary data acquisition platforms. Computers and Electronics in Agriculture 64(2), 120–132 (2008) [27] Edan, Y., Miles, G.E.: Systems engineering of agricultural robot design. IEEE Transactions on Systems, Man and Cybernetics 24(8), 1259–1264 (1994)
Fuzzy Cognitive Maps Applied to Synthetic Aperture Radar Image Classifications Gonzalo Pajares1, Javier Sánchez-Lladó2, and Carlos López-Martínez2 1
Dpto. Ingeniería del Software e Inteligencia Artificial, Facultad de Informática, University Complutense of Madrid, 28040 Madrid, Spain 2 Remote Sensing Laboratory (RSLab), Signal Theory and Communications Department Universitat Politècnica de Catalunya (UPC), 08034 Barcelona, Spain
[email protected],
[email protected],
[email protected]
Abstract. This paper proposes a method based on Fuzzy Cognitive Maps (FCM) for improving the classification provided by the Wishart maximumlikelihood based approach in Synthetic Aperture Radar (SAR) images. FCM receives the classification results provided by the Wishart approach and creates a network of nodes associating a pixel to a node. The activation levels of these nodes define the degree of membeship of each pixel to each class. These activations levels are iteratively reinforced or punished based on the existing relations among each node and its neighbours and also taking into account the own node under consideration. Through a quality coefficient we measure the performance of the proposed approach with respect to the Wishart classifier. Keywords: Fuzzy Cognitive Maps, Wishart classifier, Synthetic Aperture Radar (SAR), Polarimetric SAR (POLSAR), classification.
1 Introduction Nowadays, the increasing technology in Polarimetric Synthetic Aperture Radar (PolSAR) remote sensors is demanding solutions for different applications based on the data they provide. One of such applications is data classification to identify the nature of the different structures in the imaged surfaces and volumes based on the scattering of microwaves. Terrain and land-use classification are probably the most important applications of POLSAR, where many supervised and unsupervised classification methods have been proposed [1,2,3,4,5]. In [6], the fuzzy c-means clustering algorithm is applied for unsupervised segmentation of multi-look PolSAR images. A statistical distance measure is derived from the complex Wishart distribution of the complex covariance matrix. Methods based on the complex Wishart distribution have gained interest because of their performance. The scattering of microwaves coming from the surfaces is mapped as a coherence matrix for each pixel. Cloude and Pottier [7,8], based on the eigenvalues and eigenvectors of the coherence matrix, obtained measures of the average scattering mechanism (alpha- α ) and randomness of the scattering (entropyH). For classification, the H- α plane is divided in nine zones. Lee et al. [9] derive a J. Blanc-Talon et al. (Eds.): ACIVS 2011, LNCS 6915, pp. 103–114, 2011. © Springer-Verlag Berlin Heidelberg 2011
104
G. Pajares, J. Sánchez-Lladó, and C. López-Martínez
probability density function for the coherence matrix assuming that this matrix has the complex Wishart distribution. Based on the H- α classification, each class or cluster is characterized by its centre. Ersahin et al. [10] applied graph partitioning and human perceptual skills based on the Wishart distribution. In this paper, class and cluster are terms used without distinction. Each pixel is classified as belonging to a cluster based on the minimum distance between its coherence matrix and the cluster centres. The Wishart classification approach is an iterative process where each pixel in the whole image is re-classified until no more pixel assignments occur or a termination criterion is met. We have verified that the first rarely occurs and we apply as criterion the one giving the best partition, where a quality measurement is obtained through a specific coefficient. Once we obtain the best partition with the Wishart classifier, each pixel has been classified as belonging to a cluster. We have designed a new iterative process based on the Fuzzy Cognitive Maps (FCM) paradigm, assuming that the Wishart classifications can be still improved. FCM is a well-developed modelling methodology for complex systems that allow to describe the behaviour of a system in terms of concepts. Under its most general approach, each concept represents an entity where the concepts are joined by causal edges and have assigned an activation level [11,12]. With the goal of improving the partitions, we build a set of networks, as many as clusters provided by Wishart, i.e. nine in our approach. Each concept at each network represents a pixel in the original image and its activation level the degree of membership to the class that originated the network. Initially, the activation levels are computed from the distances between the coherency matrix associated to each pixel and the cluster centre provided by Wishart classifier. The values of the causal edges, linking a concept with other concepts, are computed considering the activation levels of all involved concepts and also under the goal of achieving the best partition as possible, based on the measure provided by the quality coefficient. The design of the FCM, as a whole, makes the main finding of this paper. Several experiments allow us to verify the improvement in the classification results achieved by Wishart. The paper is organized as follows. In Section 2 the FCM scheme is proposed for SAR data classification, giving details about the complex Wishart classifier and the theory about the cluster separability measures, as they are required by the FCM process. The performance of the method is illustrated in Section 3, where a comparative study against the original Wishart-based approach is carried out. Finally, Section 4 provides a discussion of some topics and conclusions.
2 Fuzzy Cognitive Maps Framework in SAR Classification 2.1 The Combined H − α Decomposition and Maximum Likelihood-Based Wishart Before the FCM paradigm is applied, the Wishart-based classification process described in [9] is carried out, synthesized as follows: 1) The polarimetric scattering information can be represented by a target vector,
kl = ⎡ Shh ⎣
T
2 Shv
Svv ⎤ , or another way to work is to use the Pauli scattering vector ⎦
Fuzzy Cognitive Maps Applied to Synthetic Aperture Radar Image Classifications
for
a
pixel,
k p = 2 −1 2 [ Shh + Svv
where
in
this
Shh − Svv
2Shv ]
T
case
we
and
Ti = ki ki
let
a
T
new
feature
105
vector,
is the hermitian product of
target vectors of the one-look ith pixel. POLSAR data are generally multilook processed for speckle noise reduction by averaging n neighbouring pixels. The coherency matrix is given in (1) where superscripts, * and T denote complex conjugate and matrix transposition, respectively. 1 n *T (1) T = ∑ ki k i n i =1 2) From the coherency matrices, we apply the H/α decomposition process as a refined scheme for parameterizing polarimetric scattering problems. The scattering entropy, H, is a key parameter in determining the randomness about the model. The α angle characterizes the scattering mechanism proposed in [7]. 3) The next step is to classify the image into nine classes in the H − α plane. 4) We have available nine zones (classes) where each class is identified as wj; i.e. in our approach j varies from 1 to 9. Compute the initial cluster centre from the coherency matrices for all pixels belonging to each class wj according to the number of pixels nj in that class wj, t
Vj =
n
1 j ∑ T n j i =1
(2)
i
t
where t denotes the iteration, i.e. V j is the mean for the class wj at the iteration t. 5) Compute the distance measure for each pixel i characterized by its coherence matrix T i to the cluster centre as follows,
d
(T
t
i
)
t
,V j = ln V j + Tr
((
)
t −1
Vj
T
i
)
(3)
6) Assign the pixel to the class with the minimum distance measure
i ∈ w j iff
d
(T
t
i
,V j
)
(T
t
i
,Vm
)
∀ w j ≠ wm
(4)
7) Verify if a small number of pixels change their assignment to the clusters, otherwise set t = t + 1 and return to step 1. Always stop if a prefixed number of iterations tmax is reached, in which case we choose the partition with the greatest quality, according to a coefficient measuring the cluster separation (next section). 2.2 Cluster Separation Measures
Three useful measures can be used for quantitative statistical analysis [13], namely: (a) the dispersion within a class; (b) the distance between classes and (c) a combination of the above that gives the class separability. 1) The dispersion within clusters (Dii): is defined as the averaged distance from the pixels in cluster wi to the cluster centre Vi for all pixels. It measures the compactness of cluster wi and is given by 1 ni Dii = ∑ d T k , Vi = ln ( Vi ) + Tr Vi −1Vi (5) ni k =1 large Dii indicates the dispersion of the pixels into the cluster.
(
)
(
)
106
G. Pajares, J. Sánchez-Lladó, and C. López-Martínez
2) The distance between two clusters (Dij) defined as, 1 Dij = ln ( Vi ) + ln ( Vi ) + Tr Vi −1V j + Vj−1Vi 2 large Dij values indicate the high separation of these two clusters. 3) The cluster separability (Rij): is defined by the clusters wi and wj as follows,
{
)}
(
(6)
Rij = ( Dii + D jj ) Dij
(7)
a small Rij value indicates that these two clusters are well separated. Based on the above measures, the goal is to achieve small dispersion of the clusters and large distances between two clusters, which lead to small Rij values. To verify quantitatively the performance and also in order to choose the best partition, we compute the global averaged cluster separability through the following equation, 1 R= (8) ∑∑ Rij , i ≠ j nw i j where nw is the number of Rij combinations with i ≠ j , because in our experiments we have nine zones (clusters) and we combine in two by two regions, nw = 36 . 2.3 The Fuzzy Cognitive Maps Process 2.3.1 Problem Formulation: Previous Considerations According to equation (4), a pixel i belongs to the cluster wj if the distance to the corresponding cluster centre is minimum, among all distances to the remainder cluster centres. Based on these distances, we define the support received by the pixel i for belonging to the cluster wj as follows,
{ (
μi (t ) = 2 exp − d T i ,V tj j
)}
{ (
m t ∑ exp − d T i ,Vh h =1
)} − 1
(9)
where t denotes, as indicated before, the iteration number which controls the FCM iterative process, as we will see later; the subindex h varies from 1 to m, i.e. it represents the nine zones from h = w1 to h = m = w9. As we can see, the support μij (t ) varies in the range (–1,+1] which is considered as the fuzzy causal interval for our approach. Indeed, if d
(T
)
,V j = 0 then μi (t ) = +1 and if d t
i
j
(T
t
i
)
,V j → ∞
then μi (t ) → −1 . With this transformation, the decision rule in (4) can be expressed j
as a function of the support at the iteration t, according to the equation (10), which expresses that the pixel i belongs to the cluster wj because the support received by the pixel for this cluster is the greatest of all supports received for the remainder clusters. j m (10) i ∈ w j iff μi (t ) > μi (t ) ∀ w j ≠ wm 2.3.2 Problem Formulation: Architecture For each cluster wj, we build a network of nodes, netj, where the topology of this network is established by the spatial distribution of the pixels in the image to be classified with size M × N. Each node i in the netj, is associated to the pixel location
Fuzzy Cognitive Maps Applied to Synthetic Aperture Radar Image Classifications
107
(x,y) in the image, i.e. i ≡ ( x, y) . So, the node i in the netj is initialized with the support provided by the Wishart classifier through equation (9) at the last iteration executed. These initial support values are also the initial activation levels associated to the concepts in the networks under the FCM paradigm, as described in the next section. Through the FCM the activation level of each concept is reinforced or punished iteratively based on the influences exerted by their neighbours. Figure 1 displays the architecture and the set of networks. As we can see, from the original image we build the j nets (j = 1 to 9). Every node with its activation level or support μij (t ) on each netj is associated to a pixel i on the original image, both with identical locations (x,y). The activation levels at each network are updated according to the number of iterations t. x y
μ
μ k1
net1
1 i
t+1
x k y
i
j = 1,…,9 nets
x original image pixels i and k
y
μ
μk9
net9
9 i
t+1
Fig. 1. Network architecture from the original image
2.3.3 FCM Process and Network Topology As mentioned before, FCMs allow describing the behaviour of a system in terms of concepts. More specifically, they are fuzzy signed directed graphs with feedback [11]. The directed edge eik from causal concept Ci to concept Ck measures how much Ci causes Ck. Edges eik take values in the fuzzy causal interval [ −1, +1] , eik = 0 indicates
no causality; eik > 0 indicates causal increase, this means that Ck increases as Ci increases and vice versa, Ck decreases as Ci decreases; eik < 0 indicates causal decrease or negative causality, Ck decreases as Ci increases and Ck increases as Ci decreases. Given a FCM with a number n of concepts Ci, i.e. i = 1,…,n, the value assigned to each concept, called activation level, can be updated iteratively, until convergence, based on the external influences exerted by the other nodes Ck on Ci and its selfinfluence through Ci. Several approaches have been proposed to map these influences, such as the one proposed by Tsardias and Margaritis [10], which introduces a mechanism for achieving high network stability, as we will see below.
Ai (t + 1) = f ( Ai (t ), ∑ nk =1 eki (t ) Ak (t ) ) − d i Ai (t )
(11)
108
G. Pajares, J. Sánchez-Lladó, and C. López-Martínez
The explanation of the terms in the equation (11) is as follows: 1) Ai (t ) and Ak (t ) are respectively the activation levels of the concepts Ci and Ck at the iteration, t. The sum is extended to all n concepts available. Nevertheless, only the concepts with edge values different from zero exert influences over the concept Ci trying to modify its current activation level Ai (t ) towards Ai (t + 1) . 2) eki(t+1) are the fuzzy causalities between concepts, defined as above but considering that they could vary dynamically with the iterations. 3) di ∈ [0,1] is the decay factor of certainty concept Ci. It determines the fraction of the current activation level that will be subtracted from the new one as a result of the concept’s natural intention to get stable activation levels. The bigger the decay factor, the stronger the decay mechanism. This factor was introduced in [11] as a mechanism for introducing a degree of instability, so that those concepts destabilised intentionally but with a high degree of real stability tend towards its stabilized activation level. On the contrary, if the activation level is unstable, the decay mechanism induces a continuous variability on the activation level. 4) f is a non-linear function that determines the activation level of each concept, the sigmoid function is commonly used, i.e. f ( x ) = tanh( x ) . The variable x in f(x) represents a combination of the following two terms Ai (t ) and
∑
n
e (t ) Ak (t ) , in this
k =1 ki
paper we have chosen the arithmetic mean for combination. Based on the network architecture defined above, it is easy to associate each concept Ci in the netj to the pixel location (x,y). We can define a concept for each node at each netj, Ci j , where its activation level results Ai j (t ) . We define the activation level as the support received by the pixel i for belonging to the cluster wj at the iteration t, defined in equation (9), as Ai j (t ) ≡ μi j (t ) , these supports are reinforced or punished through the FCM mechanism, equation (11); the decision is made according to equation (10) based on the supports updated. Now we concentrate the effort on defining the fuzzy causalities between concepts eki(t) and the decay factor di, both involved in equation (11), which are required for updating μij (t ) . Both terms must be expressed considering the netj in which they are involved, i.e. ekij (t ) and di j . The term ekij (t ) is a combination of two coefficients representing the mutual influence exerted by the k neighbours over i, namely: a) a regularization coefficient which computes the consistency between the activation levels or supports of the nodes in a given neighbourhood for each netj; b) a separation coefficient which computes the consistency between the clusters in terms of separability, where high separability values are suitable. The neighborhood N in is defined as the n-connected spatial region in the network around the node i, taking into account the mapping between the pixels in the images and the nodes in the networks. The regularization coefficient is computed at the iteration t as follows, n ⎧ 1 − μ j (t ) − μ j (t ) k ∈ Ni , i ≠ k i k ⎪ j (12) rik (t ) = ⎨ n ⎪⎩0 k ∉ N i or i = k
Fuzzy Cognitive Maps Applied to Synthetic Aperture Radar Image Classifications
109
From (12) we can see that rikj (t ) ranges in (-1, +1] where +1 is obtained with
μij (t ) = μ kj (t ). This means that both supports have identical values, i.e. maximum consistency between nodes. On the contrary, if μij (t ) and μkj (t ) take most extreme opposite values, such as μij (t ) = +1 and μkj (t ) = −1 or vice versa, then rikj (t ) = −1 , which is its lower limit expressing minimum consistency between nodes i and k. The separation coefficient at the iteration t is computed taking into account the labels assigned to the pixels associated to the nodes according to the classification decision rule given in the equation (10). Assume that pixels i and k are classified as belonging to clusters wr and ws respectively, i.e. labelled as r and s. Because we are trying to achieve maximum separability between clusters, we compute the averaged cluster separability according to the equation (7). We compute the separabilities between i and its k neighbors in N in . A low Rrs value, equivalently a high Rrs−1 , expresses that clusters wr and ws are well separated. Based on this reasoning, the separation coefficient is defined as follows, −1
cik (t ) = 2
Rrs (t ) −1
∑n Rru (t )
−1
(13)
Ni
Through the fraction in equation (13) we normalize the values to range in the interval (0, +1], because we compute the cluster separabilities between the clusters wr where the pixel i belongs and the clusters where the neighbours of i in N in belongs to, i.e. wu. One of such clusters is ws, the cluster where k belongs to. The coefficients 2 and 1 in the equation (13) are introduced so that cik (t ) ranges in (-1, +1]. This mapping is made to achieve the same range than rik (t ) . Note that the separation coefficient is independent of j, i.e. of the netj, because the labelling for its computation involves the activation levels of all networks. This implies that it is identical for all networks. Both coefficients, regularization and separation, are combined as the averaged sum, taking into account the signs as follows,
Wikj (t ) = γ rikj (t ) + (1 − γ ) cik (t ) ; eikj (t ) = Wikj (t )
(14)
γ ∈ [0,1] is a constant value representing the trade-off between both coefficients; j ik
e (t ) represents the degree of consistency between nodes i and k in the netj at the iteration t. We define the decay factor based on the assumption that high stability in the network activation levels implies that the activation level for the concept Ci in the network netj would be to lose some of its activation with such purpose. We build an accumulator of cells of size q = L × M , where each cell i is associated to the concept Ci. Each cell i contains the number of times hi j , that the concept Ci has changed significantly its activation level in the netj. Initially, all hi j values are set to zero and then hi j = hi j + 1 if μi j (t + 1) − μi j (t ) > ε . The stability of the node i is measured as
110
G. Pajares, J. Sánchez-Lladó, and C. López-Martínez
the fraction of changes accumulated by the cell i compared with the changes in its neighbourhood k ∈ Nim and the number of iterations t. The decay factor is computed as follows, ⎧⎪ 0 hi j = 0 and hkj = 0 j (15) di = ⎨ j j j ⎪⎩hi ⎡⎣ hk + hi t ⎤⎦ otherwise
(
)
where hij is defined above and hk j is the average value accumulated by the concepts k ∈ Nim . As one can see, from equation (15), if hi j = 0 and hkj = 0 , the decay factor takes the null value, this means that no changes occur in the activation levels of the concepts, i.e., high stability is achieved; if the fraction of changes is small, the stability of the node i is also high and the decay term tends towards zero. Even if the fraction is constant the decay term also tends to zero as t increases, this means that perhaps initially some changes can occur and then no more changes are detected, this is another sign of stability. The decay factor subtracts from the new activation level a fraction; this implies that the activation level could take values less/greater than -1 or +1. In these cases, the activation level is set to -1 or +1, respectively. Once the FCM process ends, each concept Ci has achieved an activation j level μi (t ) that determines de degree of belonging of the pixel i, represented by Ci, to the cluster j. This final decision for classifying the pixel i is made according to (10).
3 Results 3.1 Design of a Test Strategy
In order to asses the validity and performance of our proposed classification approach we use the well-tested NASA/JPL AIRSAR image of the San Francisco Bay. The images are 900 x 1024 pixels in size. Because our proposed FCM approach starts after the iterative complex Wishart process has finished, the first task consists of the determination of the best number of iterations suitable for the Wishart process. This is carried out by executing this process from 1 to tmax, fixed to 7 in our experiments. For every iteration, we compute the averaged separability value according to equation (8) for the classification obtained at this iteration and select the number of iteration tw with the minimum averaged separability coefficient. The classification results obtained for tw are the inputs for the FCM. Because, the FCM is depending on several parameters, we have carried out several experiments for adjusting them. The strategy is based on the following steps: 1) Fix the γ coefficient involved in equation (14), at this initial step. It was fixed to γ = 0.6 after experimentation. 2) Fix the maximum number of iterations tmax. We have set tmax to 4 because after experimentation we have verified that more iterations are not suitable due to an over smoothing of the textured regions.
Fuzzy Cognitive Maps Applied to Synthetic Aperture Radar Image Classifications
111
3) Fixed the above parameters, we test the FCM for the following three neighbourhood regions: N i8 , N i24 and N i48 . As before, bigger neighbourhoods produce smoothed excessively. The best neighbourhood is that with the minimum averaged separability value R . 4) Once, we have obtained the best neighbourhood and tmax as described before, we test for several γ values ranging from 0.2 till 1.0 in steps of 0.2 and for each iteration from t = 1 to tmax = 4. For each γ at each iteration, we compute the R values, eq. (8). 3.2 Results
According to the above strategy, we show the results obtained at the different steps of the process. The averaged separability coefficient values for the complex Wishart classifier, equation (8), are displayed in table 1 for iterations 1 to tmax = 7. Table 1. Averaged separability values for Wishart against the number of iterations
# of iterations R
1 91.78
2 78.29
3 116.12
Wishart 4 145.22
5 112.78
6 92.98
7 106.62
As one may observe, the best cluster separability is achieved for two iterations, because the minimum averaged separability coefficient value is obtained. This is the number of iterations employed for the complex Wishart approach in our experiments, i.e. kw = 2. Table 2 displays the values for a number of iterations, obtained by the FCM, varying from 1 to tmax and for the neighbourhoods specified with γ = 0.6. Table 2. Averaged separability values for the FCM process against the number of iterations and different neighborhoods with γ = 0.6 R Neighbourhood 8 Ni
1 79.5
FCM: # of iterations 2 3 79.0 71.9
4 85.9
N i24
189.4
177.8
198.1
197.7
48 Ni
361.3
337.5
449.5
455.3
As one can see from results in table 3, the best performance is achieved for two iterations with a neighbourhood of N i8 . This is finally the neighbourhood used for all experiments. The results obtained with N i24 and N i48 are obviously worse; one explanation is because with these values the number of neighbours forcing the change
112
G. Pajares, J. Sánchez-Lladó, and C. López-Martínez
Table 3. Averaged separability values for the FCM process against the number of iterations with different γ values
γ R Iteration
1 2 3 4
0.0
0.2
0.4
0.6
0.8
1.0
82.9 72.5 75.7 73.8
78.7 73.5 72.5 71.6
81.7 79.6 82.7 78.3
78.5 66.8 71.1 70.3
77.7 69.8 71.5 73.8
77.9 72.8 72.8 74.1
of the activation levels of the central pixel is big, according to the equations (12) and (13), this implies that there are pixels belonging to different clusters trying to modify the value of the central pixel. Table 3 displays the averaged separability values for different γ values ranging from 0 to 1 and for four iterations. From table 2, the best performance during the four iterations is achieved for γ = 0.6 . These values are obtained for two iterations of Wishart where R was 78.29 and this value is improved with FCM for iterations 2, 3 and 4. This fact verifies the performance of the proposed FCM approach against Wishart. Figure 2(a) displays the classification results obtained by the complex Wishart approach after the two programmed iterations based on the reasoning expressed above. Figure 2(b) displays the classification results obtained by the proposed FCM approach after two iterations according to the discussion above, based on the results shown in table 3, i.e. N i8 and γ = 0.6 at iteration 2. The colour bar, at the right, indicates the colour assigned to each one of the nine clusters, the number i identifies the cluster wi. The data and classification results in figure 2(a) are the inputs for the FCM approach. Figures 2(c)-(f) display two expanded areas extracted from the images in (a) and (b) obtained with Wishart (c,d) and FCM (e,f).
( a)
(c)
( d)
( b)
(e)
(f)
Fig. 2. (a) Classification by Wishart after two iterations; (b) Classification obtained by FCM after two iterations with a neighbourhood of 3 × 3 and γ = 0.6 ;(c)-(d) expanded areas obtained with Wishart; (e)-(f) the same areas processed with FCM
Fuzzy Cognitive Maps Applied to Synthetic Aperture Radar Image Classifications
113
3.3 Discussion
Two points of view may be considered for the analysis: quantitative and qualitative. From the first one, our FCM approach was focused on trying to achieve maximum cluster separability. According to the results in tables 2 and 3, the best performance in terms of separability is achieved with two iterations under a neighbourhood of N i8 , where the average cluster separability is less than the one obtained with Wishart approach. More than two iterations produce an over-homogenization effect and all pixels in a given region are re-classified as belonging to the same cluster. The results obtained by FCM represent a quality improvement with respect the result obtained by complex Wishart approach, both displayed in figure 2. As in [9] we have verified that during the FCM process several cluster centres are shifted between the clusters changing their positions. In the referenced work some qualitative improvements are justified based on this fact, therefore the same is applicable in our approach, as described below. So, under the qualitative point of view, the observation of the images in figure 2 allows to make the following considerations: a) The high entropy vegetation consisting of grass and bushes belonging to the cluster w2 have been clearly homogenized, this is because there are many pixels belonging to w4 in these areas re-classified as belonging to w2. b) The three distinct surface scattering mechanisms from the ocean surface identified in [9] are clearly displayed in figure 2(b), i.e. they appear under the cluster labelled as w6 (area of high entropy), w8 (ripples, near the coast) and w9 (smooth ocean surface). c) Also, in accordance with [9] the areas with abundant city blocks display medium entropy scattering. We have homogenized the city block areas removing pixels in such areas that belong to clusters w1 and w2, so that they are re-classified as belonging to w4 and w5 as expected. d) Some structures inside of other broader regions are well isolated. This occurs in the rectangular area corresponding to a park, where the internal structures with high entropy are clearly visible [14]. e) Additionally, the homogenization effect can be interpreted as a mechanism for speckle noise reduction during the classification phase, avoiding the early filtering for classification tasks.
4 Conclusions We have proposed an effective iterative approach based on the FCM paradigm, which outperforms the complex Wishart approach. Advantages of this mechanism are that it achieves an important homogenization effect in the classified areas preserving the important structures inside of broader areas. Also it has the ability for discovering different scattering mechanisms. The main drawback of the proposed approach is its high computational cost, about 12 minutes, and also that it requires the setting of some parameters. Nevertheless, Wishart spends itself about 7 minutes, both executed under Matlab R2009 with CPU 2.4GHz. The effectiveness of this classification approach has been illustrated by the well tested NASA/JPL AIRSAR L-band data of San Francisco bay.
114
G. Pajares, J. Sánchez-Lladó, and C. López-Martínez
In the future, for reducing the computational cost could be implemented under parallel realizations. Also, because the iterative process has become effective, we hope that other methods of the same nature will be intended in the nearest future. Acknowledgements. This work has been partially funded by the Ramon y Cajal program and the National I+D project TEC2007-65690/TCM.
References 1. Lee, J.S., Grunes, M.R., Pottier, E., Ferro-Famil, L.: Unsupervised Terrain Classification Preserving scattering characteristics. IEEE Trans. Geoscience Remote Sens. 42(4), 722– 731 (2004) 2. Lee, J.S., Pottier, E.: Polarimetric Radar Imaging: From Basics to Applications. CRC Press, Boca Raton (2009) 3. Sharma, J.J., Hajnsek, I., Papathanassiou, K.P., Moreira, A.: Polarimetric decomposition over Glacier Ice using long-wavelength Airborne PolSAR. IEEE Trans. Geoscience Remote Sens. 49(1), 519–535 (2011) 4. Ferro-Famil, L., Pottier, E., Lee, J.S.: Unsupervised Classification of Multifrequency and Fully Polarimetric SAR Images Based on the H/A/Alpha-Whisart Classifier. IEEE Trans. Goscience Remote Sensing 39(11), 2332–2342 (2001) 5. Wang, Y., Han, C., Tupin, F.: PolSAR Data Segmentation by Combining Tensor Space Cluster Analysis and Markovian Framework. IEEE Geoscience and Remote Sensing Letters 7(1), 210–214 (2010) 6. Du, L., Lee, J.S.: Fuzzy classification of earth terrain covers using complex polarimetric SAR data. Int. J. Remote Sensenig 17(4), 809–826 (1996) 7. Cloude, S.R., Pottier, E.: Application of the H/A/α polarimetric decomposition theorem for land classification. In: Proc. SPIE, vol. 3120, pp. 132–143 (1997) 8. Cloude, S.R., Pottier, E.: An entropy based classification scheme for land applications of POLSAR. IEEE Trans. Geosci. Remote Sensing 35(1), 68–78 (1997) 9. Lee, J.S., Grunes, M.R., Ainsworth, T., Du, L.J., Schuler, D., Cloude, S.: Unsupervised classification using polarimetric decomposition and the complex Wishart classifier. IEEE Trans. Geosci. Remote Sensing 37(5), 2249–2258 (1999) 10. Ersahin, K., Cumming, I.G., Ward, R.K.: Segmentation and Classification of Polarimetric SAR Data Using Spectral Graph Partitioning. IEEE Trans. Geoscience Remote Sens. 48(1), 164–174 (2010) 11. Tsardias, A.K., Margaritis, K.G.: An experimental study of the dynamics of the certainty neuron fuzzy cognitive maps. Neurocomputing 24, 95–116 (1999) 12. Kosko, B.: Neural Networks and Fuzzy Systems: a Dynamical Systems Approach to Machine Intelligence. Prentice-Hall, NJ (1992) 13. Davies, D.L., Bouldin, D.W.: A cluster separation measure IEEE Trans. Pattern Anal. Machine Intell. 1, 224–227 (1979) 14. Google Earth, The Google Earth product family (2011), http://www.google.es/intl/es_es/earth/download/ge/agree.html
Swarm Intelligence Based Searching Schemes for Articulated 3D Body Motion Tracking Bogdan Kwolek, Tomasz Krzeszowski, and Konrad Wojciechowski Polish-Japanese Institute of Information Technology Koszykowa 86, 02-008 Warszawa http://www.pjwstk.edu.pl Abstract. We investigate swarm intelligence based searching schemes for effective articulated human body tracking. The fitness function is smoothed in an annealing scheme and then quantized. This allows us to extract a pool of candidate best particles. The algorithm selects a global best from such a pool. We propose a global-local annealed particle swarm optimization to alleviate the inconsistencies between the observed human pose and the estimated configuration of the 3D model. At the beginning of each optimization cycle, estimation of the pose of the whole body takes place and then the limb poses are refined locally using smaller number of particles. The investigated searching schemes were compared by analyses carried out both through qualitative visual evaluations as well as quantitatively through the use of the motion capture data as ground truth. The experimental results show that our algorithm outperforms the other swarm intelligence searching schemes. The images were captured using multi-camera system consisting of calibrated and synchronized cameras.
1
Introduction
Vision-based tracking of human bodies is a significant problem due to various potential applications like user friendly interfaces, virtual reality, surveillance, clinical analysis and sport. The aim of articulated body tracking is to estimate the joint angles of the human body at any time. It is one of the most challenging problems in computer vision due to body self-occlusions, high dimensional and nonlinear state space and large variability in human appearance. To alleviate some of the difficulties, much previous work has investigated the use of 3D human body models of various complexity to recover the position, orientation and joint angles from 2D image sequences [3][4][11][13][15]. An articulated human body can be considered as a kinematic chain consisting of at least eleven parts, corresponding naturally to body parts. This means that around twenty six parameters might be needed to describe the full body pose. The state vectors describing the human pose are computed by fitting the articulated body model to the observed person’s silhouette. Thus, the 3D model-based approaches rely on seeking the pose space to find the geometrical configuration of 3D model that matches best the current image observations [12]. In order to cope with practical difficulties arising due to occlusions and depth ambiguities, multiple cameras and simplified background are typically used by the research community [13][15]. J. Blanc-Talon et al. (Eds.): ACIVS 2011, LNCS 6915, pp. 115–126, 2011. c Springer-Verlag Berlin Heidelberg 2011
116
2
B. Kwolek, T. Krzeszowski, and K. Wojciechowski
Searching Schemes for Articulated Human Body Tracking
In articulated 3D human body tracking the techniques based in particle filtering are widely used. Particle filters [5] are recursive Bayesian filters that are based on Monte Carlo simulations. They approximate a posterior distribution for the configuration of a human body given a series of observations. The high dimensionality of articulated body motion requires huge number of particles to represent well the posterior probability of the states. In such spaces, sample impoverishment may prevent particle filters from maintaining multimodal distribution for long periods of time. Therefore, many efforts have been spent in developing methods for confining the search space to promising regions with true body pose. Deutscher and Reid [3] proposed an annealed particle filter, which adopts an annealing scheme with the stochastic sampling to concentrate the particle spread near the global maximum. In the discussed approach the fitness function is smoothed using annealing factors 0 = α1 < α2 , . . . , < αn = 1, and the particles migrate towards the global maximum without getting stuck in local minima. Additionally, a crossover operation is utilized in order to maintain the diversity of the particles. The configuration space can also be constrained using a hierarchical search. In such an approach, a part of the articulated model is localized independently in advance, and then its location is used to constrain the search for the remaining limbs. In [6], an approach called search space decomposition is proposed, where on the basis of color cues the torso is localized first and then it is used to confine the search for the limbs. However, in realistic scenarios, among others due to occlusions, it is not easy to localize the torso and to extract reliably such a good starting guess for the search. Compared with the ordinary particle filter, the annealed particle filter greatly improves the tracking performance. However, it still requires a considerable number of particles. Since the particles do not exchange information and do not communicate with each other, they have reduced capability of focusing the search on some regions of interest in dependency on the previous visited values. In contrast, the particle swarm optimization (PSO) [7], which is population-based searching technique, has high search efficiency by combining local search (by self experience) and global one (by neighboring experience). In particular, a few simple rules result in high effectiveness of exploration of the search space. The PSO is initialized with a group of random particles (hypothetical solutions) and then it searches hyperspace (i.e. Rn ) of a problem for optima. Particles move through the solution space, and undergo evaluation according to some fitness function f (). Much of the success of PSO algorithms comes from the fact that individual particles have tendency to diverge from the best known position in any given iteration, enabling them to ignore local optima, while the swarm as a whole gravitates towards the global extremum. If the optimization problem is dynamic, the aim is no more to seek the extrema, but to follow their progression through the space as closely as possible. Since the object tracking process is a dynamic optimization problem, the tracking can be achieved through
Swarm Intelligence Based Searching Schemes
117
incorporating the temporal continuity information into the traditional PSO algorithm. This means, that the tracking can be accomplished by a sequence of static PSO-based optimizations to determine the best person’s pose, followed by re-diversification of the particles to cover the possible states in the next time step. In the simplest case, the re-diversification of the particle i can be realized as follows: (i) xt ← N (ˆ xt−1 , Σ) (1) where x ˆt−1 is the estimate of the state in time t − 1. In order to improve the convergence speed, Clerc and Kennedy [2] proposed to use the constriction factor ω in the following form of the equation for the calculation of the i-th particle’s velocity: vi,k+1 = ω[vi,k + c1 r1 (pi − xi,k ) + c2 r2 (g − xi,k )]
(2)
where constants c1 and c2 are used to balance the influence of the individual’s knowledge and that of the group, respectively, r1 and r2 are uniformly distributed random numbers, xi is position of the i-th particle, pi is the local best position of particle, whereas g stands for the global best position. In our approach the value of ω depends on annealing factor α in the following manner: ω = −0.8α + 1.4 (3) k where α = 0.1 + K+1 , k = 0, 1, . . . , K, and K is the number of iterations. The annealing factor is also used to smooth the objective function. The larger the iteration number is, the smaller is the smoothing. In consequence, in the last iteration the algorithm utilizes the non-smoothed function. The algorithm, which we term as annealed particle swarm optimization (APSO) can be expressed in the following pseudo-code:
1. For each particle i 2. initialize vti,0 3. xi,0 t ∼ N (gt−1 , Σ0 ) i,0 i 4. pit = xi,0 t , ft = f (xt ) 5. uit = fti , u˜it = (uit )α0 ∗ ∗ 6. i∗ = arg mini u ˜it , gt = pit , wt = uit 7. For k = 0, 1, . . . , K 8. update ωα on the basis of (3) 9. G = arg mini round(num bins · u ˜it ) 10. For each particle i 11. Select a particle from {G ∪ gt } and assign it to gti i,k i 12. vti,k+1 = ωα [vti,k + c1 r1 (pit − xi,k t ) + c2 r2 (gt − xt )] i,k+1 i,k i,k+1 13. xt = xt + vt 14. fti = f (xi,k+1 ) t 15. if fti < uit then pit = xit , uti = fti , u˜it = (uit )αk 16. if fti < wt then gt = xit , wt = fti
118
B. Kwolek, T. Krzeszowski, and K. Wojciechowski
The smoothed objective functions undergo quantization, which constrains the real numbers to relatively small discrete set of bin values (integers), see 9th line in the pseudo-code. Thanks to such an operation the similar function values are clustered into the same bins. In each iteration the algorithm determines the set G of the particles, which after the quantization of the smoothed fitness function from the previous iteration, assumed the smallest values (the best fitness scores), see 9th line in the pseudo-code. For each particle i the algorithm selects the global best particle gti from {G ∪ gt }, where gt determines the current global best particle of the swarm. That means that the whole swarm selects the global best location from a set of candidate best locations. Figure 1 depicts the number of particles in each bin, which has been determined in one of the experiments, where 200 particles, 10 iterations and a quantization into 30 bins were employed in estimating the human pose.
number of part. 150 100 50 0 0
5
10
15 bin number
20
25
6
30
5
4
3
2
1
7 8 9 10 it. number
Fig. 1. Number of the particles in each bin in iterations 1,..,10
In [15], an annealed PSO based particle filter has been proposed and evaluated in tracking articulated 3D human body. Our approach is different from the discussed algorithm, in particular, it relies on annealing scheme that is based on the smoothing and the quantization of the fitness function. Additionally, the constriction factor that controls the convergence rate depends on the annealing factor. In the global-local particle swarm optimization (GLPSO) algorithm [9], at the beginning of each frame we estimate the pose of the whole body using PSO. Given the pose of the whole body, we construct state vectors consisting of the estimated state variables for pelvis and torso/head, the arms and the legs. At this stage the state variables describing the pose of the legs are perturbed by normally distributed motion. Afterwards, we execute particle swarm optimization in order to calculate the refined estimate of the legs pose. Such refined state variables are then placed in the state vector of the whole body. The state variables describing the hands are refined analogously. Our global-local annealed particle swarm optimization algorithm (GLAPSO) operates analogously, but instead of the ordinary PSO algorithm we employ APSO optimizations.
Swarm Intelligence Based Searching Schemes
3
119
Tracking Framework
The skeleton of the human body is modeled as a kinematic tree. The articulated 3D model consists of eleven segments with limbs represented by the truncated cones, which model the pelvis, torso/head, upper and lower arm and legs. The configuration of the model is defined by 26 DOF. It is parameterized by the position and the orientation of the pelvis in the global coordinate system and the relative angles between the connected limbs. In order to obtain the 3D human pose each truncated cone is projected into 2D image plane via perspective projection. In such a way we obtain an image with the rendered model in a given configuration. Such image features are then matched to the person extracted by image analysis. In most of the approaches to articulated 3D human body tracking, the cameras are static and background subtraction algorithms are utilized to extract the object of interest [14]. In addition, image cues like edges, ridges, color are used frequently to get better delineation of the person [11]. In [10], face detection, head-shoulders contour matching and elliptical skin-blob detection techniques were used in estimating the 3D human poses in static images. In our approach, the background subtraction algorithm [1] is used to extract the binary image of the person, see Fig. 2b). It is then used to calculate a silhouette-overlap term. a)
b)
c)
d)
e)
f)
Fig. 2. Person segmentation. Input image a), foreground b), gradient magnitude c), edge distance map d), skin color patches e), extracted body segments f).
Image edges complement silhouette features and contribute toward precise aligning the body limbs. The most common type of edge detection process uses a gradient operator. Gradient features share many properties with optical flow. In particular, they do not depend on background subtraction. Gradient angle is invariant to global changes of image intensities. In contrast to optical flow, gradients features are discriminative for both moving and non-moving body parts. In our approach, the gradient magnitude, see Fig. 2c), is masked with the closed image of the foreground and then used to generate the edge distance map, see also Fig. 2d). It assigns each pixel a value that is the distance between that pixel and the nearest nonzero edge pixel. In our implementation we employ chessboard distance and limit the number of iterations on the chain propagation to three. Additionally, in the GLPSO and GLAPSO algorithms we perform the segmentation of the person’s shape into individual body parts. To accomplish this we model the distribution of skin color using 16 × 16 histogram in rg color space. The histogram back-projection is employed to identify the skin patches and to
120
B. Kwolek, T. Krzeszowski, and K. Wojciechowski
extract the skin binary masks, see Fig. 2e. Such masks are then used to delineate the skin segments in the person binary images. Taking into account the height of the extracted person we perform rough segmentation of the legs and feet, see Fig. 2f). The fitness score is calculated on the basis of following expression: f (x) = 1 − (f1 (x)w1 · f2 (x)w2 ), where w denotes weighting coefficients that were determined experimentally. The function f1 (x) reflects the degree of overlap between the segmented body parts and the projected model’s parts into 2D image. The overlap degree is calculated through checking the overlap from the binary image to the considered rasterized image of the model and vice versa. The larger the degree of overlap is, the larger is the fitness value. In GLPSO and GLAPSO algorithms the silhouette-overlap term is calculated with consideration of the distinguished body parts. The second function reflects the degree of overlap between model edges and image edges. At this stage the above mentioned edge-proximity term is utilized.
4
Experimental Results
The algorithms were compared by analyses carried out both through qualitative visual evaluations as well as quantitatively through the use of the motion capture data as ground truth. The images were captured using multi-camera system consisting of four calibrated and synchronized cameras. The system acquires color images of size 1920 × 1080 with rate 24 fps. Each pair of the cameras is approximately perpendicular to the other two. The placement of the video cameras in our laboratory is shown in Fig. 3. A commercial motion capture (MoCap) system from Vicon Nexus provides ground truth motion of the body at rate of 100 Hz. It utilizes reflective markers and sixteen cameras to recover the 3D location of such markers. The cameras are all digital and are capable to
Fig. 3. Layout of the laboratory and camera setup. The images illustrate human motion tracking in frame #20 seen in view 1 and 2 (upper row), and in view 3 and 4 (bottom row).
Swarm Intelligence Based Searching Schemes
121
differentiate overlapping markers from each camera’s view. The synchronization between the MoCap and multi-camera system is done through hardware from Vicon Giganet Lab. Figure 4 demonstrates some tracking results that were obtained using particle swarm optimization (PSO), see images in the first row, global-local PSO, see images in the second row, annealed PSO, see images in the third row, and globallocal annealed PSO, see images in the last row. Each image consists of two subimages, where the left sub-image contains the model overlaid on the images from the view 1, whereas the second one illustrates the model overlaid on the images from the view 4. In the experiments presented below we focused on analyses of motion of walking people with bared and freely swinging arms. The analysis of the human way of walking, termed gait, can be utilized in several applications ranging from medical applications to surveillance. This topic is now a very active research area in the vision community.
Fig. 4. Articulated 3D human body tracking. Shown are results in frames #20, 40, 60, 80, 100, 120, 140, obtained by PSO (1st row), GLPSO (2nd row), APSO (3rd row), GLAPSO (4th row), respectively. The left sub-images are seen from view 1, whereas the right ones are seen from view 4.
For fairness, in all experiments we use the identical particle configurations. For the global-local PSO and global-local annealed PSO the sum of particles responsible for tracking the whole body, arms and legs corresponds to the number of the particles in the PSO and APSO. For instance, the use of 300 particles in PSO or APSO is equivalent to the use of 200 particles for tracking the full body, 50 particles for tracking the arms and 50 particles for tracking both legs
122
B. Kwolek, T. Krzeszowski, and K. Wojciechowski
in GLPSO or GLAPSO. The use of 200 in PSO and APSO corresponds to the exploitation of 150, 25 and 25 particles, respectively, whereas the use of 100 particles equals to utilization 80 particles for tracking the global configuration of the body, along with 10 and 10 particles for tracking hands and legs, respectively. In Tab. 1 we can see some quantitative results that were obtained using two image sequences. For each sequence the results were averaged over ten runs with unlike initializations. They were achieved using image sequences consisting of 150 and 180 frames. In the quantization the number of bins was set to 30. Figure 4 demonstrates the images from the sequence two. Table 1. Average errors for M = 39 markers in two image sequences
#particles 100 100 PSO 300 300 100 100 GLPSO 300 300 100 100 APSO 300 300 100 100 GLAPSO 300 300
Seq. 1 Seq. 2 it. error [mm] std. dev. [mm] error [mm] std. dev. [mm] 10 86.17 50.58 73.89 35.98 20 77.71 39.36 67.58 32.15 10 75.31 41.50 65.56 30.26 20 75.11 38.42 63.43 28.63 10 80.95 42.69 68.50 32.00 20 67.66 27.15 67.17 30.08 10 68.58 30.98 64.40 28.01 20 67.96 30.03 62.87 26.00 10 71.56 36.26 65.04 29.74 20 68.81 31.87 61.29 26.86 10 66.51 29.63 61.78 26.69 20 64.63 28.91 59.70 24.98 10 69.44 31.21 63.37 30.74 20 63.71 28.79 60.42 26.72 10 60.07 21.07 60.71 24.41 20 58.96 19.43 57.62 22.49
The pose error in each frame was calculated using M = 39 markers mi (x), i = 1, . . . , M , where mi ∈ R3 represents the location of the i-th marker in the world coordinates. It was expressed as the average Euclidean distance: E(x, x ˆ) =
M 1 ||mi (x) − mi (ˆ x)|| M i=1
(4)
where mi (x) stands for marker’s position that was calculated using the estimated pose, whereas mi (ˆ x) denotes the position, which has been determined using data from our motion capture system. From the above set of markers, four markers were placed on the head, seven markers on each arm, 6 on the legs, 5 on the torso and 4 markers were attached to the pelvis. For the discussed placement of
Swarm Intelligence Based Searching Schemes
123
the markers on the human body the corresponding marker’s assignment on the 3D model was established. Given the estimated human pose the corresponding 3D positions of virtual markers were determined. On the basis of data stored in c3d files the ground truth was extracted and then utilized in calculating the average Euclidean distance given by (4). The errors that are shown in Tab. 1 were calculated on the basis of the following equation: Err(x, xˆ) =
L M 1 ||mi (x) − mi (ˆ x)|| LM i=1
(5)
k=1
where L denotes the number of frames in the utilized test sequences. As Tab. 1 shows, the particle swarm optimization algorithm allows us to obtain quite good results. The GLPSO outperforms the PSO algorithm. However, such considerably better results can only be obtained if skin and legs segmentation is used in GLPSO. Due to global-local searching strategy the GLPSO algorithm is superior in utilizing the information about the location of the hands and the legs. The APSO algorithm tracks better in comparison to GLPSO and PSO. Moreover, it requires no segmentation of skin patches or body parts. The GLAPSO takes advantages of both algorithms and achieves the best results. The plots shown in Fig. 5 illustrate the pose error versus frame number that was obtained by each algorithm. We can see that the PSO-based algorithm is able to provide estimates with errors that are sporadically larger than 100 mm. The average error of the GLAPSO is far below 60 mm. In the GLPSO algorithm, at the beginning of each PSO cycle, the estimation of the pose of the whole body takes place and then the poses of the limbs are refined locally using smaller number of particles. In the algorithms based on the annealing the particles are beforehand weighted by smoothed versions of the weighting function, where the
Fig. 5. Pose error [mm] versus frame number
124
B. Kwolek, T. Krzeszowski, and K. Wojciechowski
influence of the local minima is weakened first but increases gradually. This leads to consistent tracking of the 3D articulated body motion. In consequence, the errors in our algorithm, which takes the advantages the two different strategies for exploration of the search space are far smaller. The discussed algorithm is capable of achieving better results because of its ability to thinking globally and acting locally. As a result, the GLAPSO algorithm outperforms GLPSO and APSO algorithms both in terms of the tracking accuracy as well as consistency in tracking of the human motion. In particular, the standard deviation is far smaller in comparison to the standard deviation of the other investigated algorithms. Figure 6 presents the pose estimation errors for particular body parts. The results were obtained in 20 iterations using the GLAPSO algorithm with 200 particles for the whole body tracking and 2 × 50 particles for tracking the arms and legs. As we can observe, our algorithm can track body limbs with lower errors and it is robust to ambiguous configurations such as self occlusion. In more detail, from the discussed plot, we can see that for all body parts except the left forearm the maximal error does not exceed 100 mm. The complete human motion capture system was written in C/C++. The system runs on Windows and Linux in both 32 bit and 64 bit modes. It operates on color images with spatial resolution of 960 × 540. The entire tracking process takes approximately 2.1 sec. per frame on Intel Core i5 2.8 GHz using a configuration with 100 particles 10 iterations. If Open Multi-Processing (OpenMP) is employed the tracking is completed in 1.12 sec. The image processing and analysis takes about 0.45 sec. One of the future research directions of the presented approach is to explore the CUDA/GPU in order to speed-up the computations [8].
Fig. 6. Tracking errors [mm] versus frame number
Swarm Intelligence Based Searching Schemes
5
125
Conclusions
We have presented a vision system that effectively utilizes swarm intelligence searching schemes to achieve better articulated 3D human body tracking. By combining two searching strategies, namely, annealed and global-local, the proposed method can tackle the inconsistency between the observed body pose and the estimated model configurations. Due to better capability of exploring the search space, the combination of above-mentioned searching strategies leads to superior tracking the articulated 3D human motion. Our global-local annealed (GLAPSO) algorithm is able to track the articulated 3D human motion reliably in multi-camera image sequences. In particular, the resulting algorithm is robust to ambiguous body configurations such as self occlusion. Moreover, it performs satisfactory even when small number of particles is employed, say 100 particles and 10 iterations. The fitness function is smoothed in an annealing scheme and then quantized. This allows us to maintain a pool of candidate best particles. Furthermore, the constriction factor that controls the convergence rate depends on the annealing factor. To show the advantages of our algorithm, we have conducted several experiments on walking sequences and investigated global-local and annealed searching strategies. The algorithms were compared by analyses carried out both through qualitative visual evaluations as well as quantitatively through the use of the motion capture data as ground truth. Acknowledgment. This paper has been supported by the project “System with a library of modules for advanced analysis and an interactive synthesis of human motion” co-financed by the European Regional Development Fund under the Innovative Economy Operational Programme - Priority Axis 1.
References 1. Arsic, D., Lyutskanov, A., Rigoll, G., Kwolek, B.: Multi camera person tracking applying a graph-cuts based foreground segmentation in a homography framework. In: IEEE Int. Workshop on Performance Evaluation of Tracking and Surveillance, pp. 30–37. IEEE Press, Piscataway (2009) 2. Clerc, M., Kennedy, J.: The particle swarm - explosion, stability, and convergence in a multidimensional complex space. IEEE Trans. on Evolutionary Computation 6(1), 58–73 (2002) 3. Deutscher, J., Blake, A., Reid, I.: Articulated body motion capture by annealed particle filtering. In: IEEE Int. Conf. on Pattern Recognition, pp. 126–133 (2000) 4. Deutscher, J., Reid, I.: Articulated body motion capture by stochastic search. Int. J. Comput. Vision 61(2), 185–205 (2005) 5. Doucet, A., Godsill, S., Andrieu, C.: On sequential Monte Carlo sampling methods for bayesian filtering. Statistics and Computing 10(1), 197–208 (2000) 6. Gavrila, D.M., Davis, L.S.: 3-D model-based tracking of humans in action: a multiview approach. In: Proc. of the Conf. on Computer Vision and Pattern Recognition (CVPR 1996), pp. 73–80. IEEE Computer Society, Washington, DC (1996) 7. Kennedy, J., Eberhart, R.: Particle swarm optimization. In: Proc. of IEEE Int. Conf. on Neural Networks, pp. 1942–1948. IEEE Press, Piscataway (1995)
126
B. Kwolek, T. Krzeszowski, and K. Wojciechowski
8. Krzeszowski, T., Kwolek, B., Wojciechowski, K.: GPU-accelerated tracking of the motion of 3D articulated figure. In: Bolc, L., Tadeusiewicz, R., Chmielewski, L.J., Wojciechowski, K. (eds.) ICCVG 2010. LNCS, vol. 6374, pp. 155–162. Springer, Heidelberg (2010) 9. Krzeszowski, T., Kwolek, B., Wojciechowski, K.: Model-based 3D human motion capture using global-local particle swarm optimizations. In: Burduk, R., Kurzy´ nski, ˙ lnierek, A. (eds.) Computer Recognition Systems 4. Advances M., Wo´zniak, M., Zo in Intelligent and Soft Computing, vol. 95, pp. 297–306. Springer, Heidelberg (2011) 10. Lee, M.W., Cohen, I.: A model-based approach for estimating human 3D poses in static images. IEEE Trans. Pattern Anal. Mach. Intell. 28, 905–916 (2006) 11. Schmidt, J., Fritsch, J., Kwolek, B.: Kernel particle filter for real-time 3D body tracking in monocular color images. In: IEEE Int. Conf. on Face and Gesture Rec., pp. 567–572. IEEE Computer Society Press, Southampton (2006) 12. Sidenbladh, H., Black, M., Fleet, D.: Stochastic tracking of 3D human figures using 2D image motion. In: Vernon, D. (ed.) ECCV 2000. LNCS, vol. 1843, pp. 702–718. Springer, Heidelberg (2000) 13. Sigal, L., Balan, A., Black, M.: HumanEva: Synchronized video and motion capture dataset and baseline algorithm for evaluation of articulated human motion. Int. Journal of Computer Vision 87, 4–27 (2010) 14. Sminchisescu, C., Kanaujia, A., Li, Z., Metaxas, D.: Discriminative density propagation for 3D human motion estimation. In: IEEE Int. Conf. on Computer Vision and Pattern Recognition, pp. I:390–I:397 (2005) 15. Zhang, X., Hu, W., Wang, X., Kong, Y., Xie, N., Wang, H., Ling, H., Maybank, S.: A swarm intelligence based searching strategy for articulated 3D human body tracking. In: IEEE Workshop on 3D Information Extraction for Video Analysis and Mining in Conjuction with CVPR, pp. 45–50. IEEE, Los Alamitos (2010)
Combining Linear Dimensionality Reduction and Locality Preserving Projections with Feature Selection for Recognition Tasks Fadi Dornaika1,2 , Ammar Assoum3 , and Alireza Bosaghzadeh1 2
1 University of the Basque Country, San Sebastian, Spain IKERBASQUE, Basque Foundation for Science, Bilbao, Spain 3 LaMA Laboratory, Lebanese University, Tripoli, Lebanon
Abstract. Recently, a graph-based method was proposed for Linear Dimensionality Reduction (LDR). It is based on Locality Preserving Projections (LPP). It has been successfully applied in many practical problems such as face recognition. In order to solve the Small Size Problem that usually affects face recognition, LPP is preceded by a Principal Component Analysis (PCA). This paper has two main contributions. First, we propose a recognition scheme based on the concatenation of the features provided by PCA and LPP. We show that this concatenation can improve the recognition performance. Second, we propose a feasible approach to the problem of selecting the best features in this mapped space. We have tested our proposed framework on several public benchmark data sets. Experiments on ORL, UMIST, PF01, and YALE Face Databases and MNIST Handwritten Digit Database show significant performance improvements in recognition. Keywords: Linear Dimensionality Reduction, Locality Preserving Projections, feature selection, nearest neighbor classifier, object recognition.
1
Introduction
In most computer vision and pattern recognition problems, the large number of sensory inputs, such as images and videos, are computationally challenging to analyze. In such cases it is desirable to reduce the dimensionality of the data while preserving the original information in the data distribution, allowing for more efficient learning and inference. During the last few years, a large number of approaches have been proposed for constructing and computing the embedding. We categorize these methods by their linearity. The linear methods, such as Principal Component Analysis (PCA) [1], Multi-Dimensional Scaling (MDS) [2] are evidently effective in observing the Euclidean structure. The nonlinear methods such as Locally Linear Embedding (LLE) [3], Laplacian eigenmaps [4], Isomap [5] focus on preserving the geodesic distances. Most of the previous researches, e.g. LDA and its variants, aimed to find a good subspace in which the different class mass can be separated in a global way. In general, it might be hard to find a subspace which has a good separability for the whole data J. Blanc-Talon et al. (Eds.): ACIVS 2011, LNCS 6915, pp. 127–138, 2011. c Springer-Verlag Berlin Heidelberg 2011
128
F. Dornaika, A. Assoum, and A. Bosaghzadeh
set, which motivates us to consider local methods, since empirically the local methods may have stronger discriminative power than global methods. Linear Dimensionality Reduction (LDR) techniques have been increasingly important in pattern recognition [6] since they permit a relatively simple mapping of data onto a lower-dimensional subspace, leading thus to simple and computationally efficient classification strategies. The issue of selecting the new components in the new-subspace has not received much attention. In most practical cases, relevant features in the new embedded space are not known a priori. Finding out what features to use in a classification task is referred to as feature selection. Although there has been a great deal of work in machine learning and related areas to address this issue, these results have not been fully explored or exploited in emerging computer vision applications. Only recently there has been an increased interest in deploying feature selection in applications such as face detection, vehicle detection, and pedestrian detection. Most efforts in the literature have largely ignored the feature selection problem and have focused mainly on developing effective feature extraction methods and applying powerful classifiers. The main trend in feature extraction has been to represent the data in a lower dimensional space computed through a linear or non-linear transformation satisfying certain properties. The goal is to find a new set of features that represents the target concept in a more compact and robust way but also to provide more discriminative information. Recently, a graphbased method was proposed for Linear Dimensionality Reduction (LDR). It is based on Locality Preserving Projections (LPP). LPP is a typical linear graphbased dimensionality reduction (DR) method that has been successfully applied in many practical problems such as face recognition. LPP is essentially a linearized version of Laplacian Eigenmaps. When dealing with face recognition problems, LPP is preceded by a Principal Component Analysis (PCA) step in order to avoid possible singularities. All face recognition works using LPP compute the recognition rates for a wide range of dimensions/eigenvectors. However, no work has dealt with the estimation of the optimal dimension. Furthermore, no work has addressed the selection of the relevant eigenvectors. The main contributions of the paper are as follows. First, we propose a recognition scheme based on concatenating the features provided by PCA and LPP. We show that this concatenation can improve the recognition performance of LPP. Second, we propose a feasible approach to the problem of selecting the best features in this mapped space: how many and which eigenvectors to retain for both PCA and LPP. Our proposed method is inspired by [7]. In [7], the authors use feature selection paradigm with PCA for the task of object detection. They demonstrate the importance of feature selection (eigenvector selection) in the context of two object detection problems: vehicle detection and face detection. However, our proposed work differs from [7] in three aspects. First, we address Linear Dimensionality Reduction and feature selection for recognition tasks–a more generic problem than object detection. Second, our study is not limited to the use of the classical PCA. Third, we propose a novel scheme for selecting eigenvectors associated with a concatenation of two linear embedding techniques: PCA+LPP.
Combining LDR and LPP with Feature Selection for Recognition Tasks
129
The remainder of the paper is organized as follows. Section 2 describes the classical Linear Dimensionality Reduction schemes. Section 3 presents the proposed approach for the estimation of embedding and feature selection. Section 4 presents some experimental results obtained with five benchmark data sets.
2
Linear Dimensionality Reduction
In this section, we provide a brief description about PCA and LPP. 2.1
Principal Component Analysis (PCA)
Principal components analysis (PCA) is a popular linear feature extractor to unsupervised dimensionality reduction [8]. We assume that we have a set of N D samples {xi }N i=1 ⊂ R . The mean sample, x, is subtracted from every sample xi . The set of centered samples is represented by a D × N matrix X. Then the covariance matrix is computed as S=
1 X XT N
(1)
The new embedded space will have its origin at x and its principal directions given by the eigenvectors of the covariance matrix S associated with the largest eigenvalues. A new sample x can be approximated by the M principal modes: x∼ =x+
M
yl wl
(2)
l=1
= x + Wy
(3)
where the D×M matrix W represents the first M eigenvectors. The M-dimensional vector y is the low dimensional representation of the original sample x. It is computed by a simple projection onto the principal modes: y = WT (x − x)
(4)
The number of principal modes, M , is chosen such that the variability of the retained modes corresponds to a high percentage of the total variability. In brief, the PCA embedding takes as input a set of sample data and produces an orthonormal set of vectors representing the eigenvectors of the sample covariance matrix associated with the M < D largest eigenvalues, i.e., w1 , w2 , . . . , wM . Turk and Pentland [1] proposed the first application of PCA to face recognition. Since the basis vectors constructed by PCA had the same dimension as the input face images, they were named “Eigenfaces”. Principal Component Analysis (PCA) has been successfully applied to construct linear models of shape, graylevel, and motion. A full review of PCA applications in computer vision is beyond the scope of this paper.
130
F. Dornaika, A. Assoum, and A. Bosaghzadeh
2.2
Locality Preserving Projections (LPP)
Many dimensionality reduction techniques can be derived from a graph whose nodes represent the data samples and whose edges quantifies both the similarity and closeness among pairs of samples [9]. Locality Preserving Projection (LPP) [10,11,12] can be seen as a linearized version of Laplacian Eigenmaps (LE) [4]. In [13], the authors have extended the LPP to the supervised case. D We assume that we have a set of N samples {xi }N i=1 ⊂ R . Define a neighborhood graph on these data, such as a K-nearest-neighbor or -ball graph, or a full mesh, and weigh each edge xi ∼ xj by a symmetric affinity function Aij = K(xi ; xj ), typically Gaussian: xi − xj 2 Aij = exp − (5) β where β is usually set to the average of squared distances between all pairs. Let A denote the symmetric affinity matrix whose elements are defined by (5). 1 L 2 We seek latent points {yi }N ⊂ R that minimizes i=1 i,j yi − yj Aij , 2 which discourages placing far apart latent points that correspond to similar observed points. For the purpose of presentation simplicity, we present the one dimensional mapping case in which the original data set {xi }N i=1 is mapped to a line. Let z = (y1 , y2 , . . . , yN )T be such a map (a column vector). Note that here every data sample is mapped to a real value. A reasonable criterion for choosing a good map is to optimize the following objective function under some constraints: min
1 (yi − yj )2 Aij 2 i,j
(6)
Minimizing function (6) imposes a heavy penalty if neighboring points xi and xj are mapped far apart. By simple algebra formulation, function (6) can be written as 1 (yi − yj )2 Aij 2 i,j
1 2 (y Aij + yj2 Aij − 2yi Aij yj ) 2 i,j i = yi2 Dii − yi Aij yj =
i T
i,j T
= z Dz − z Az = zT L z
(7)
where D is the diagonal weight matrix, whose entries are column (or row, since A is symmetric) sums of A, and L = D − A is the Laplacian matrix. In the LPP formulation, the latent data are simply given by a linear mapping of the original data. This means that the projection of xi is given by yi = wT xi .
Combining LDR and LPP with Feature Selection for Recognition Tasks
131
Thus, the one dimensional map z = (y1 , y2 , . . . , yN )T (column vector) is giving by
z = XT w
(8)
where X = (x1 , x2 , . . . , xN ) is the data matrix. Finally, by combining (8) and (7) and by imposing the constraint zT Dz = 1 the minimization problem reduces to finding: (9) min wT XLXT w s.t. wT XDXT w = 1 w The constraint is used in order to remove the arbitrary scale associated with the mapping. The transformation vector w that minimizes the objective function is given by the minimum eigenvalue solution to the generalized eigenvalue problem: XLXT w = λ XDXT w
(10)
For a multi dimensional mapping, each data sample xi is mapped into a vector yi . The aim is to compute the projection directions (WLP P = (w1 , w2 , . . . , wL )). These vectors are given by the generalized eigenvectors of (10), ordered according to their eigenvalues, 0 ≤ λ1 ≤ λ2 ≤ . . . ≤ λL . Then, the mapping of an unseen sample x is given by y = WTLP P x (11) 2.3
PCA+LPP
In many real world problems such as face recognition, the matrix XDXT is sometimes singular. This stems from the fact that sometimes the number of images in the training set N is much smaller than then number of pixels in each image. To overcome the complication of a singular XDXT , original data are first projected to a PCA subspace so that the resulting matrix XDXT is non-singular. This method when applied on faces is called Laplacianfaces. The global transform is given by: W = WP CA WLP P
3 3.1
Feature Selection Feature Selection
In a great variety of fields, including pattern recognition and machine learning, the input data are represented by a very large number of features, but only few of them are relevant for predicting the label. Many algorithms become computationally intractable when the dimension is high. On the other hand, once a good small set of features has been chosen, even the most basic classifiers (e.g., 1-nearest neighbor) can achieve desirable performance. Therefore, feature selection, i.e. the task of choosing a small subset of features which is sufficient to predict the target labels well, is critical to minimize the classification error. At the same time, feature selection also reduces training and inference time and
132
F. Dornaika, A. Assoum, and A. Bosaghzadeh
leads to a better data visualization as well as to a reduction of measurement and storage requirements. Roughly speaking, feature selection algorithms have two key problems: search strategy and evaluation criterion. According to the criterion, feature selection algorithms can be categorized into filter model and wrapper model. In the wrapper model, the feature selection method tries to directly optimize the performance of a specific predictor (classification or clustering algorithm). This is usually achieved through an evaluation function and a given search strategy. The main drawback of this method is its computational deficiency. In the filter model, the feature selection is done as a preprocessing, without trying to optimize the performance of any specific predictor directly [14,15]. A comprehensive discussion of feature selection methodologies can be found in [16,17]. 3.2
Eigenvector Selection for Object Recognition
In the sequel, we will adopt a wrapper technique for eigenvector selection. The evaluation strategy will be directly encoded by the recognition accuracy over validation sets. Without any loss of generality, the classifier used after Linear Dimension Reduction and eigenvector selection will be the KNN classifier. This classifier is one of the oldest and simplest methods for pattern classification and it is one of the top 10 algorithms in data mining [18]. The adopted search strategy will be carried out by a Genetic Algorithm (GA). We use a simple encoding scheme in the form of a bit string whose length is determined by the number of eigenvectors. Each eigenvector is associated with one bit in the string. If the ith bit is 1, then the ith eigenvector is selected, otherwise, that component is ignored. Each string thus represents a different subset of eigenvectors. Evaluation criterion. The goal of feature selection is to use less features to achieve the same or better performance. Therefore, the fitness evaluation contains two terms: (1) accuracy and (2) the number of selected features. The performance of the KNN classifier is estimated using a validation data set which guides the GA search. Each feature subset contains a certain number of eigenvectors. If two subsets achieve the same performance, while containing different number of eigenvectors, the subset with fewer eigenvectors is preferred. Between accuracy and feature subset size, accuracy is our major concern. We used the fitness function shown below to combine the two terms: F itness = c1 Accuracy + c2 Zeros
(12)
where Accuracy corresponds to the classification accuracy on a validation set for a particular subset of eigenvectors, and Zeros corresponds to the number of eigenvectors not selected (i.e., zeros in the individual). c1 and c2 are two positive coefficients with c2 << c1 . In our implementation, c1 was set to 104 and c2 to 1. Search strategy: a genetic algorithm. Genetic Algorithms (GAs) are biologically motivated adaptive systems based on natural selection and genetic
Combining LDR and LPP with Feature Selection for Recognition Tasks
133
recombination [19]. In the standard GA, candidate solutions are encoded as fixed length vectors–strings. The initial population of solutions is chosen randomly. These candidate solutions are allowed to evolve over a certain number of generations. At each generation, the fitness of each string is calculated; this is a measure of how well the string optimizes the objective function. Subsequent generations are created through a process of selection, recombination, and mutation. An individual string fitness is used to probabilistically select which individuals will recombine. Crossover operators merge the information contained within pairs of selected parents by placing random subsets of the information from both parents into their respective positions in a member of the subsequent generation. Due to the random factors involved in producing children strings, the children may, or may not, have higher fitness values than their parents. Nevertheless, because of the selective pressure applied through a number of generations, the overall trend is towards higher fitness strings. Mutations are used to help preserve diversity in the population. Mutations introduce random changes into the chromosomes. The genetic search process is iterative: evaluating, selecting, and recombining strings in the population during each iteration (generation) until reaching some termination condition. The basic algorithm, where P (t) is the population of strings at generation t, is given below: Algorithm GA() 1. t = 0 2. initialize P(t) 3. evaluate P(t) 4. while (termination condition is not satisfied) 5. do 6. begin 7. select P(t+1) from P(t) 8. recombine P(t+1) 9. evaluate P(t+1) 10. t = t +1 11. end The initial population is generated randomly. This, however, would produce a population where each individual contains approximately the same number of 1s and 0s on the average. To explore subsets of different numbers of features, the number of 1s for each individual is generated randomly. Then, the 1s are randomly scattered in the string. In all of our experiments, we used a population size of 100 and 50 generations. In our experiments, we have observed that the GA converged in less than 50 generations. Selection Our selection strategy was cross generational. Assuming a population of size I, the offspring double the size of the population and we select the best I individuals from the combined parentoffspring population. Crossing We use uniform crossing in which each bit of the offspring is selected randomly from the corresponding bits of the parents. The crossover probability used in all of our experiments was 0.6. Mutation We use
134
F. Dornaika, A. Assoum, and A. Bosaghzadeh
the traditional mutation operator which just flips a specific bit with a very low probability. The mutation probability used in all of our experiments was 0.05. Figure 1 illustrates how the fitness of every individual string is evaluated using training and validation sets. The matrix P and L denote the whole set of eigenvectors associated with PCA and LPP, respectively. The matrices P and L denote the selected subset of eigenvectors. We point out that in this proposed scheme both the PCA and LPP output are used to form the new feature vectors. The corresponding recognition rate will be used as the first term in Eq. (12) in order to compute the fitness of the string.
Fig. 1. Evaluating the fitness of a given individual string by the Genetic Algorithm. The matrix P denotes the whole set of eigenvectors associated with PCA. The matrix P denotes a putative subset of eigenvectors. The corresponding recognition rate that combine selected PCA eigenvevtors and selected LPP eigenvectors will be used for computing the fitness of the string (Eq. (12)).
4
Experimental Results
4.1
Benchmark Data Sets
The data sets used are ORL face data set, UMIST face data set, YALE face data set, PF01 face data set and MNIST handwritten digit data set1 . The details of these data sets are described in Table 1. Some samples of these data sets are shown in Figure 2. 1
ORL, UMIST, YALE, PF01, and MNIST data sets can be retrieved respectively from http://www.cl.cam.ac.uk/research/dtg/attarchive/facedatabase.html http://www.shef.ac.uk/eee/research/vie/research/face.html http://see.xidian.edu.cn/vipsl/database_Face.html http://nova.postech.ac.kr/special/imdb/imdb.html http://yann.lecun.com/exdb/mnist/
Combining LDR and LPP with Feature Selection for Recognition Tasks
ORL
UMIST
YALE
PF01
MNIST Fig. 2. Some samples of ORL, UMIST, YALE, PF01 and MNIST data sets
135
136
F. Dornaika, A. Assoum, and A. Bosaghzadeh Table 1. Details of benchmark data sets Data set Data set size (N ) Sample dimension (D) Nb. classes (C) ORL 400 10304 40 UMIST 575 10304 20 YALE 165 77760 15 PF01 1819 2304 103 MNIST 7000 784 10
Table 2. Average recognition rates obtained with five data sets using four LPP embedding schemes. The figures in parentheses denote the percentage of the training set.
ORL (50%) UMIST (50%) YALE (70%) PF01 (70%) MNIST (70%)
4.2
LPP LPP with selection PCA:LPP PCA:LPP with selection 85.0% 92.2% 92.2% 95.4% 93.2% 97.9% 96.7% 99.2 % 78.5% 94.0% 81.0% 96.5% 43.0% 70.1% 45.4% 71.6% 72.9% 88.1% 84.2% 93.%
Method Comparison
We empirically evaluate the improvement obtained by our proposed methods on the above data sets. We have performed a number of experiments and comparisons to demonstrate the importance of feature selection for face recognition based on linear embedding. First, recognition experiments are conducted on the data sets using the classical Linear Dimensionality Reduction schemes, i.e. using the top eigenvectors, followed by the KNN classifier (K=1). Second, recognition experiments are conducted on the same data sets using the Linear Reduction schemes with feature selection. In this scheme, we use ten fold cross-validation scheme. For each fold (unless stated otherwise), 50 % of the samples are used for training and the remaining samples are used for testing. Table 2 summarizes the average recognition rate over the 10 partitions for the five data sets. The first column illustrates the average recognition rates obtained by the LPP embedding. The second column corresponds to LPP with feature selection (i.e., the eigenvectors of LPP were selected by a GA). The third column illustrates the average recognition rates obtained by concatenating the PCA output and the LPP output. The fourth column corresponds to the feature selection scheme on the concatenated features. As can be seen, the feature selection framework has provided high recognition rates for the data sets. We can observe that by concatenating the PCA output and the LPP output the recognition rate gets improved for all data sets. We can also observe that the recognition rate is improved by adopting the feature selection paradigm for the concatenated features. Figure 3 illustrates the recognition rates obtained for YALE data set for all 10 partitions. In this experiment, the percentage of training samples was set to 70%. The blue bars correspond to the classical LPP. The red bars correspond to
Combining LDR and LPP with Feature Selection for Recognition Tasks
137
100 90
Recognition rate (%)
80 70 60 50 LPP Selective LPP
40 30 20 10 0
Set 1
Set 2
Set 3
Set 4
Set 5
Set 6
Set 7
Set 8
Set 9
Set 10
Fig. 3. Recognition rates using LPP and selective LPP applied on YALE data set. The blue bars correspond to the classical LPP. The red bars correspond to the selective LPP.
the mapping obtained by the selective LPP. As can been seen, the use of feature selection paradigm has improved the recognition rate. Using Matlab on an Intel Core I3 530 CPU (2.93GHz), the CPU times needed for selecting the LPP features with ORL, UMIST, YALE, PF01 and MNIST data sets are respectively 68.8, 133.8, 8.5, 3983.6 and 110.3 seconds. The number of individuals was set to 100 and the number of generations to 10.
5
Conclusion
We presented a new method for identifying critical eigenvectors for multi-class recognition problems. The eigenvectors correspond to the linear embedding given by a concatenation of Principle Component Analysis (PCA) and Locality Preserving Projections (LPP). The procedure of eigenvectors identification is based on maximizing the recognition rate with a genetic algorithm. Experiments are conducted on five benchmark data sets. These experiments have shown that the recognition rate has increased and that the compression of data is improved. Acknowledgment. This work was supported by the Spanish Government under the project TIN2010-18856.
References 1. Turk, M., Pentland, A.: Eigenfaces for recognition. Journal of Cognitive Neuroscience 3, 71–86 (1991) 2. Borg, I., Groenen, P.: Modern Multidimensional Scaling: theory and applications. Springer, New York (2005) 3. Roweis, S.T., Saul, L.K.: Nonlinear dimensionality reduction by locally linear embedding. Science 290, 2319–2327 (2000) 4. Belkin, M., Niyogi, P.: Laplacian eigenmaps for dimensionality reduction and data representation. Neural Computation 15(6), 1373–1396 (2003) 5. Tenenbaum, J.B., de Silva, V., Langford, J.C.: A global geometric framework for nonlinear dimensionality reduction. Science 290, 2319–2323 (2000)
138
F. Dornaika, A. Assoum, and A. Bosaghzadeh
6. Martinez, A.M., Zhu, M.: Where are linear feature extraction methods applicable? IEEE Trans. Pattern Analysis and Machine Intelligence 27(12), 1934–1944 (2005) 7. Suna, Z., Bebisa, G., Miller, R.: Object detection using feature subset selection. Pattern Recognition 37, 2165–2176 (2004) 8. Jolliffe, I.T.: Principal Component Analysis. Springer, New York (2002) 9. Yan, S., Xu, D., Zhang, B., Zhang, H., Yang, Q., Lin, S.: Graph embedding and extension: a general framework for dimensionality reduction. IEEE Trans. on Pattern Analysis and Machine Intelligence 29(1), 40–51 (2007) 10. He, X., Niyogi, P.: Locality preserving projections. In: Conference on Advances in Neural Information Processing Systems (2003) 11. He, X., Yan, S., Hu, Y., Niyogi, P., Zhang, H.J.: Face recognition using laplacianfaces. IEEE Trans. Pattern Anal. Mach. Intelligence 27(3), 328–340 (2005) 12. Zhang, L., Qiao, L., Chen, S.: Graph-optimized locality preserving projections. Pattern Recognition 43, 1993–2002 (2010) 13. Yu, W., Teng, X., Liu, C.: Face recognition using discriminant locality preserving projections. Image and Vision Computing 24, 239–248 (2006) 14. Mitra, P., Murthy, C., Pal, S.: Unsupervised feature selection using feature similarity. IEEE Trans. Pattern Analysis and Machine Intelligence 24, 301–312 (2002) 15. Liua, H., Suna, J., Liua, L., Zhang, H.: Feature selectionwith dynamic mutual information. Pattern Recognition 43, 1330–1339 (2009) 16. Guyon, I., Elisseeff, A.: An introduction to variable and feature selection. Journal of Machine Learning Research 3, 1157–1182 (2003) 17. Liu, H., Yu, L.: Toward integrating feature selection algorithms for classification and clustering. IEEE Trans. Knowledge Data Engineering 17, 494–502 (2005) 18. Wu, X., Kumar, V., et al.: Top 10 algorithms in data mining. Knowledge Information Systems 14(1), 1–37 (2008) 19. Srinivas, M., Patnaik, L.: Genetic algorithms: a survey. IEEE Computer 27(6), 17–26 (1994)
A New Anticorrelation-Based Spectral Clustering Formulation Julia Dietlmeier, Ovidiu Ghita, and Paul F. Whelan Centre for Image Processing and Analysis, Dublin City University, Dublin, Ireland
[email protected],
[email protected],
[email protected]
Abstract. This paper introduces the Spectral Clustering Equivalence (SCE) algorithm which is intended to be an alternative to spectral clustering (SC) with the objective to improve both speed and quality of segmentation. Instead of solving for the spectral decomposition of a similarity matrix as in SC, SCE converts the similarity matrix to a column-centered dissimilarity matrix and searches for a pair of the most anticorrelated columns. The orthogonal complement to these columns is then used to create an output feature vector (analogous to eigenvectors obtained via SC), which is used to partition the data into discrete clusters. We demonstrate the performance of SCE on a number of artificial and real datasets by comparing its classification and image segmentation results with those returned by kernel-PCA and Normalized Cuts algorithm. The column-wise processing allows the applicability of SCE to Very Large Scale problems and asymmetric datasets. Keywords: Spectral clustering, Image segmentation, Dimensionality reduction, Latent variables.
1
Introduction
Recent years have witnessed an enormous increase in research and applications devoted to spectral clustering (SC) where the problem of grouping is reformulated in an induced feature space. This attention comes not undeserved for several reasons. Firstly, SC can be referred to as a fully unsupervised classification method [13]. Secondly, SC excels in discovering hidden and secondary relationships [20], managing non-convex cluster shapes, non-metric data [14] and noise reduction [11] in a well defined and theoretically sound framework. Finally, the segmentation and grouping based on eigenvectors is able to return the perceptual organization features present in an image [16,5,6]. Conceptually, SC belongs to the domain of manifold learning methods aimed at the unsupervised extraction of a low-dimensional representation [19]. The term spectral therein refers to a broad family of clustering methods that make use of the eigenvectors of some normalized similarity matrix [15]. Different SC algorithms formalize the grouping problem in different ways and differ widely in the retained number and ranking of eigenvectors and matrix normalization steps [8,7]. By far, the most popular application to image segmentation is the normalized cuts (Ncut) J. Blanc-Talon et al. (Eds.): ACIVS 2011, LNCS 6915, pp. 139–149, 2011. c Springer-Verlag Berlin Heidelberg 2011
140
J. Dietlmeier, O. Ghita, and P.F. Whelan
algorithm presented by Shi and Malik in 1997 as the first application of SC to computer vision and image analysis domain [4]. Despite its merits, SC has also its limitations associated with the computational complexity of spectral decomposition [19,6] and the problem of discretization of continuous eigenvectors [16]. Particularly, pixel classification tasks for large indefinite (possibly asymmetric) and fully dense similarity matrices form a considerable computational bottleneck for SC. Given an image with N pixels, the size of the similarity matrix increases to N 2 × N 2 and the decompositionbased implementations of SC become quickly infeasible. This is a well-known fact which has been continually emphasized over the past decade [19,6]. Approaches dealing with this paradigm range from exploiting the sparsity, subsampling of an image or similarity matrix to the low-rank approximation methods such as Nystr¨ om algorithm [21,23]. We infer that this is still an open problem from the most recent work by Chen et al. in [25] where the team of researchers presents a parallel HPC implementation of SC. The other recent work by Tung et al. in [24] approaches the scalability problem of SC by using a combination of blockwise processing and stochastic ensemble consensus. In this contribution we do not seek numeric or platform-related solutions to speed-up SC but rather a method which questions the optimality of eigenvectors. We therefore raise a question if the important aspects of the data can be represented through a less expensive alternative. This consideration opens the door to a much broader range of techniques in statistical machine learning and dimensionality reduction which forms the basis for any SC implementation. Therein, the family of learning methods encompasses but is not limited to principal component analysis (PCA), kernel-PCA, Linear Discriminant Analysis (LDA), other generative, discriminative, latent variable [18] and Independent Component Analysis (ICA) methods [11]. In this paper, we assume a low-dimensional manifold and search for a computationally less expensive alternative to eigenvector-based analysis. We next highlight the idea behind our algorithm that given a separable dataset [18], the core information related to the leading eigenvectors is contained in the columns of a kernel matrix. We call our algorithm Spectral Clustering Equivalence (SCE) and in the next sections we show its connection with kernel-PCA (Sections 2,3), Ncut (Section 4) and the Ising model [3] (Section 5). The major contribution of our paper is the reformulation of the standard spectral clustering through the construction of uncorrelated, orthogonal and centered components without the use of eigenvectors.
2
Development of Our Method
In multidimensional scaling (MDS), the spectral decomposition is carried out on the inner product matrix (Gram) G in the feature space with the main emphasis to preserve the inner-point distances [9]. Let us denote a similarity matrix by S and a dissimilarity matrix by A (see Table 1). Either A or S can be viewed as a dot product matrix G in some feature space according to Sch¨ olkopf and Smola
A New Anticorrelation-Based Spectral Clustering Formulation
141
Table 1. SCE notations
S A C R ρmin c1,2 t1,2 ⇒ M z Φ X ˜ G λmin
similarity matrix kernel matrix (dissimilarity matrix) column-centered kernel matrix correlation matrix computed on the columns of C minimal signed Pearson correlation coefficient, ρ ≡ cos ∠c1 , c2 a pair of most anticorrelated columns of C, ∠c1 , c2 ≡ θ1 → π orthogonal complement to c1,2 , θ2 → π/2 next following operation mixing matrix M = Cz equivalence coefficient centered and uncorrelated SCE feature vectors eigenvector-based feature vectors embedded (psd) Gram matrix [9] minimal eigenvalue (additive shift constant) [9]
[27] or transformed to G with kernelization and normalization [9] according to a particular SC formulation and application [6]. We assume that A is a generally indefinite (possibly asymmetric) matrix and interpret it as a multidimensional space spanned by its columns. Further, we center the columns of A, call the new matrix C and consider a 2-class data partitioning problem. In order to answer the question about which columns in C carry more information about the binary class labels, we proceed with the analysis of linear dependencies present in C. From the related works on linear dependency analysis, Srebro and Jaakkola in [17], for example, also seek to identify a low-dimensional subspace that captures the dependent and the ”important” aspects of the data, and separate them from independent variations. Thus, a natural way to conduct dependency analysis is to analyze correlations between different variables and the first step, prior to applying correlation analysis, is the centering of variables. Contrary to the formation of G in kernel-PCA which involves double-centering [9], our normalization of A in order to obtain C does not involve row centering. Next, we define the correlation between two centered columns, c1 and c2 , according to the formula of Pearson product-moment correlation coefficient [1]: N N N ρ1,2 = c1,i c2,i / c21,i c22,i . (1) i=1
i=1
i=1
A strong negative correlation provides a suitable measure of discrimination according, for example, to [2] and also indicates that the decrease in one variable
142
J. Dietlmeier, O. Ghita, and P.F. Whelan
is controlled by the increase in the second variable. In regards to natural images, it is reasonable to view foreground and background as the two most distinct and thus most anticorrelated image structures. We therefore are interested in the lower bound of ρ ∈ [−1, 1] and define a pair of observations (columns of C) with a strong negative correlation ρij → −1 dissociation patterns. In order to construct the orthogonal and uncorrelated kernel-PCA estimates we first draw on the idea of canonical spaces [28]. Let X and Y be the two unknown subspaces spanned by the columns of C. The largest canonical angle between X and Y is defined as θ(X , Y) = maxx∈X miny∈Y ∠(x, y) [28]. We know that the correlation between centered variables is equivalent to the cosine of the angle between these variables [1]. The cosine of the largest (canonical) angle θ → π can therefore be interpreted as the minimal signed Pearson correlation coefficient between the columns of C, ρmin → −1 [1]. According to the cosine input image ⇒ features ⇒ S ⇒ A ⇒
output Dimensionality ⇒ X1,2 (SC) Reduction Φ1,2 (SCE)
˜ =U ˜ Λ˜U ˜T ⇒ X = U ˜ A ⇒ G = UΛUT ⇒ λmin ⇒ G
Λ˜
set of discrete clusters
⇒ X1,2 . (2)
A ⇒ C ⇒ R ⇒ ρmin ⇒ c1,2 ⇒⊥ ⇒ t1,2 ⇒ Mt1,2 ⇒ decorrelate ⇒ Φ1,2 . (3) with 2 × 2 PCA and center Fig. 1. Algorithm description and the comparison of SC (2) and SCE (3). Both methods take a similarity matrix S as an input and produce a pair of orthogonal and uncorrelated feature vectors. SC is based on the eigen-decomposition of the pseudo-Gram matrix G. Conversely, SCE is a decomposition-free method which is based on the dependency analysis of the column-centered kernel matrix C. Because SCE works on the columns of C, it can generally be applied on asymmetric, non-PSD and rectangular datasets. The convention to write a pipeline of equations with ”⇒” has been adopted from [9].
definition, π is the maximum possible angle corresponding to cos(θ) = ρmin = −1 [1]. Thus, ρmin not only defines a pair of mostly anticorrelated columns c1 and c2 but also provides the link with the first canonical angle θ1 . According to Stewart in [28], the number of canonical angles in the case of dim(X ) < dim(Y) is equal to dim(X ), which in our case dim(X ) = 2. This fact allows the construction of the second SCE-based feature component by employing the orthogonality constraint to obtain θ2 . For this purpose for ∀k ∈ [1, N 2 ] we seek an orthogonal
A New Anticorrelation-Based Spectral Clustering Formulation
143
complement vector ck to c1,2 , and in this process we discard the least orthogonal pair of vectors: c1,k , if |∠c1 , ck − π/2| < |∠c2 , ck − π/2| t1,2 = (4) c2,k , otherwise . Further, we multiply t1,2 by Cz (”C to the power of z”) to maximize linear dependency and subsequently decorrelate (and center) with PCA on the computed 2 × 2 covariance matrix (refer to Table 1 for notations) in order to obtain the pair Φ1,2 of orthogonal, uncorrelated and centered SCE feature vectors. In this formulation, the SCE approximation of kernel-PCA feature vectors X1,2 is controlled by the coefficient z. After decorrelation and centering and similarly to the ranking of the PCA components [11], our selection of the leading feature vector Φ1 is based on the maximum variance principle such as σ 2 (Φ1 ) > σ 2 (Φ2 ). The product Cz t can also be viewed as a multivariate polynomial regression model with t being a vector of N 2 × 2 regression coefficients. The algorithm description and the progression from c to Φ is outlined in Fig. 1.
3
Results of Comparison to Kernel-PCA
In this section we compare SCE-based classification to the kernel-PCA-based result and consider an experiment with an asymmetric dataset. We created an interlocked spirals dataset, shown in Fig. 2, which is considered to be a challenging benchmark for spectral clustering [22]. To alleviate this challenging clustering SC on symmetrized Ssym
SC on symmetrized Ssym
( -1)
SCE on asymmetric S; ρ
= - 0.4
SCE on asymmetric S
140
2
120 100 80
1
1
40
t
X
2
60
0
20 0
-1
20 40 60 5
-2 0
5
10
X
1
(a)
15
20 x 10
-5000
23
0
t
(b)
5000
2
(c)
(d)
Fig. 2. Segmentation of asymmetric data with binary k-means in SC (a) and SCE (c) constructed feature spaces. An interlocked two spirals dataset is considered as a challenging benchmark for SC [22]. As can be seen in (c), SCE results in a better intercluster separation in the feature space. As the symmetry condition constitutes one of the four metric axioms [12], this example also tests the non-metric invariance of SCE.
problem, Chang and Yeung present in [22] a robust path-based spectral clustering algorithm with the use of a Gaussian kernel. The main objective of our experiment is to demonstrate the asymmetric and non-metric invariance of SCE and we designed the following asymmetric similarity measure:
sij = b1 · dxi − dxj , (5) sji = b2 · dyi − dyj .
144
J. Dietlmeier, O. Ghita, and P.F. Whelan
In the experiment outlined in Fig. 2 each spiral consists of 151 data points and is generated according to the equation of Archimedean spiral. Two separately computed coordinate vectors have been further concatenated to form x and y vectors. In (5) we are using the first derivatives dx and dy of the raw coordinates x and y. The model parameters b1 and b2 control the degree of asymmetry and with b1 = 20 and b2 = 2 we obtain a highly asymmetric S. Due to the symmetric formulation of SC we decompose S into its symmetric and skew-symmetric parts S = Ssym + Sskew according to [10]. Because of its symmetric formulation, SC disregards Sskew and diagonalizes only Ssym . Given a high degree of asymmetry, SC fails to correctly identify the two separate spirals as illustrated in Fig. 2(b). Conversely, SCE fully utilizes the information in the asymmetric component of S to achieve the correct separation (Fig. 2(d)) and results in a better projection and thus higher inter-cluster separability than kernel-PCA. The application of SCE to image segmentation and its relation to Ncut will be investigated in the next section.
4
Connection of SCE with Normalized Cuts
Ncut is the graph-theoretic formulation of SC with the objective to minimize a normalized measure of disassociation [7]. Ncut operates on the 2nd generalized eigenvector of a normalized weight matrix W where the normalization procedure has the purpose to penalize large image segments. Ncut then computes the diagonal matrix D containing the sum of all edges solves for the eigenvectors of and 1 1 N = D− 2 WD− 2 with N(i, j) = W(i, j)/ D(i, i) D(j, j). The second smallest generalized eigenvector λ2 of W is a componentwise ratio of the second and first largest eigenvectors of N [6]. We are interested if our Φ-based approximation can provide computational savings over the Ncut algorithm while maintaining the same image partition. For comparative purposes we have acquired the Ncut demo software from [26] and used the supplied parameters for the calculation of the adjacency matrix based on intervening contour similarities. In order to compare the segmentation results, we take the returned N matrix and compute Φ as outlined in Fig. 1. There are two aspects which are non-trivial in connection with Ncut: definition of a feature similarity and selection of a partitioning threshold which can take the values of 0, median or a point that minimizes N cut(A, B) [7]. In our SCE formulation we center the columns of N before mixing with t for a number of z iterations. We can also implement Nz t mixing iteratively by centering only the mixed N 2 × 2 components after each iteration. The successive centering results in the ideal non-parametric case, where we partition the graph according only to the signs (A = {Φ1 > 0}, B = {Φ1 ≤ 0}). For our experiments we used a Dell Precision M6300 dual-core notebook with 2GB RAM and Matlab R2009a environment. The results of an experiment on the full resolution 321×481 images from the Berkeley segmentation database are shown in Fig. 3. The returned binary Ncut partition (Fig. 3, third column) is given by the second computed eigenvector (fifth column). We observe the qualitative equivalence
A New Anticorrelation-Based Spectral Clustering Formulation SCE, thresholding Φ1
145
Ncut returned result seg=2 -0.2
0.5
-0.4 0
-0.6
-0.5
-0.8 -1
SCE, thresholding Φ1
Ncut returned result seg=2 -0.3 -0.4
0.2
-0.5
0
-0.6
-0.2
-0.7
SCE, thresholding Φ1
0.4
-0.4
-0.8
-0.6
-0.9
-0.8
Ncut returned result seg=2 -0.2
0.5 -0.4 -0.6 -0.8
0 -0.5
Fig. 3. Original 321 × 481 images from the Berkeley database are shown in the first column and SCE-based results in the second column. The last three columns show the Ncut-based result and the first and the second eigenvector returned by Ncut. This diagram is best viewed in color.
between SCE segmentation based on Φ1 and the first eigenvector (fourth column) returned by Ncut algorithm which also outputs a very narrow-banded sparse matrix N with ≈ 0.1% non-zero elements. Due to the inherent sparsity advantage which has its roots in the definition of similarities [7] Ncut does not rely on the direct eigen-decomposition of N. Instead, it uses the iterative Lanczos eigensolver [28] which, similarly to SCE, is also based on sparse matrix-vector multiplications. In our experiments with the returned sparse N, the ρmin is marginally low ρmin = −8.6025e − 004 which explains the high number of SCE iterations (z ≈ 1e4) needed to approximate the first Ncut eigenvector.
5
SCE Extension with Latent Variables
It is known that although the eigenvectors are efficient in capturing the perceptual organization features [16,5,6], binary Ncut solution does not guarantee the correct discrete image partitions [7]. Conversely, in our SCE formulation, the approximation to the first Ncut eigenvector is connected with the rotation and scaling of two hyperplanes, implemented through iterative N by t multiplications. Therefore, it is reasonable to assume that the foreground innovations can be extracted ahead of the eigenvector equivalence condition and in a much shorter time. To approach these problems, we initially followed the idea of a greedy search and designed an optimization procedure detailed in Table 2. Therein, we view the matrix N as the matrix of features and consider that N is sparse. Further, we view the signs of a pair of columns as two latent (hidden)
146
J. Dietlmeier, O. Ghita, and P.F. Whelan Table 2. Ising-based SCE extension
function [Ω, A, B] = scecut(N, t, itmax) k = cov(t); [u, v] = eigs(k); t = t ∗ u; t = t − ¯t; it = 0; while it
std(b2 ) t1 = −sign(b1 ) ∗ b2 ; t2 = b2 ; else t1 = −sign(b2 ) ∗ b1 ; t2 = b1 ; end it = it + 1; end Ω(:, 1) = t1 ; Ω(:, 2) = t2 A : Ω(:, 1) > 0; B : Ω(:, 1) ≤ 0;
binary support variables s1 (+) and s2 (−) and thus establish the connection with the Ising model which is a special Markov random field (MRF) [3]. In Table 2 we define s1 = sign(b1 ) if std(b1 ) > std(b2 ) and s2 = sign(b2 ) if std(b1 ) ≤ std(b2 ), where ”std” denotes the standard deviation. Instead of using the Ising model to represent pixels, we work with similarities contained in the normalized sparse matrix N. We denote the optimized feature vectors by Ω. The initial condition is given by the pair of columns t1,2 which, with the change in notation such that S ≡ N, can be obtained according to: N ⇒ P ∈ RN
2
×H
⇒ pi = pi − p¯i ⇒ R ⇒ c1,2 ⇒ t1,2 .
(6)
Due to memory limitations we did not search for the global minimum on correlation in N but instead operated on a subset matrix P of H randomly selected columns. Thus, in the experiment shown in Fig. 4 we selected H = 100 in order to process a 321 × 481 image from the Berkeley database. Although the automatic selection of the stopping criterion is still an ongoing work, we note that one possibility to obtain the optimal partition is to analyze the dynamic oscillatory behavior of the correlation coefficient ρ(t1 , t2 ) (first column in Fig. 4) and we observed that the optimal figure ground cut occurs at the change in phase of ρ. The binary A (figure) and B (ground) partitions have to be computed twice for the two successive iterations corresponding to the ρ transition. The final segmentation result F combines the intermediate results at different iterations such that F = F1 ∩ F2 , where F1 = A1 ∪ B1 and F2 = A2 ∪ B2 (see Fig. 4 last row). Random subsampling of N (6) explains somewhat different, but consistent with perceptual meaning, segmentation results in the second and the third row of Fig. 4, where we used different random subsets of N. The results in Fig. 4 show that not only the Ising-based SCE detects the foreground innovations in the analyzed image but also has a factor 2 speed-up compared to Ncut.
A New Anticorrelation-Based Spectral Clustering Formulation
SCE, itmax=298, 41.24 sec
147
SCE, itmax=438, 55.13 sec
Ncut output seg=2, 126.3 sec
SCE, itmax=439, 54.26 sec
SCE combined output
SCE, itmax=299, 37.3 sec
SCE combined result
Fig. 4. Concept of the Ising-based SCE. We analyze the dynamic oscillatory behavior of the correlation coefficient to find the optimal transition. For the two iterations near the transition point we compute the binary classification and combine the results to yield the optimal figure ground segmentation. The transition is given by the point where the hyperplanes are flipped around the Ω1 (strongest) axis as can be seen in the second column. On our computer SCE runs faster (see the computational time above the diagrams) than Ncut and returns more perceptually meaningful binary segmentation.
6
Conclusions and Future Work
In this paper we developed an efficient alternative to eigenvector-based feature classification. We started by examining the conditions of the feature space equivalence between the proposed SCE and the kernel-PCA outputs. We further have shown that the proposed algorithm reduces the dimension of the feature space while improving classification performance and thus results in a better projection and higher inter-cluster separability than kernel-PCA. In regard to image segmentation, we demonstrated that the proposed method has potential to replace eigenvector-based computation at least for applications considering the detection of foreground innovations. Our future work will concentrate on generalizing SCE to multiclass problems as well as investigating the regularization and stopping criteria of the proposed Ising-based SCE extension.
148
J. Dietlmeier, O. Ghita, and P.F. Whelan
Although the Ising model takes SCE beyond the equivalence pursuit, it shows that segmentation without eigenvectors is a more flexible framework than that offered by the standard spectral clustering. Acknowledgments. This research was supported by the National Biophotonics and Imaging Platform Ireland funded under the HEA PRTLI Cycle 4, co-funded by the Irish Government and the European Union - Investing in your future. Our special thanks go to the anonymous referees whose invaluable suggestions and comments have helped us to improve this paper.
References 1. Rodgers, J.L., Nicewander, W.A.: Thirteen Ways to Look at the Correlation Coefficient. The American Statistician 42, 59–66 (1988) 2. Tran, H.T., Romanov, D.A., Levis, R.J.: Control Goal Selection Through Anticorrelation Analysis in the Detection Space. Journal of Physical Chemistry A 110, 10558–10563 (2006) 3. Cevher, V., Duarte, M.F., Hegde, C., Baraniuk, R.G.: Sparse Signal Recovery Using Markov Random Fields. Neural Information Processing Systems (NIPS), 257–264 (2008) 4. Shi, J., Malik, J.: Normalized Cuts and Image Segmentation. In: Computer Vision and Pattern Recognition (CVPR), pp. 731–737 (1997) 5. Perona, P., Freeman, W.T.: A factorization approach to grouping. In: Burkhardt, H.-J., Neumann, B. (eds.) ECCV 1998. LNCS, vol. 1406, pp. 655–670. Springer, Heidelberg (1998) 6. Weiss, Y.: Segmentation Using Eigenvectors: A Unifying View. In: International Conference on Computer Vision, ICCV (1999) 7. Shi, J., Malik, J.: Normalized Cuts and Image Segmentation. IEEE Trans. on Pattern Analysis and Machine Intelligence (TPAMI) 22, 888–905 (2000) 8. Ng, A.Y., Jordan, M.I., Weiss, Y.: On Spectral Clustering: Analysis and an Algorithm. Neural Information Processing Systems (NIPS) 14, 849–856 (2001) 9. Roth, V., Laub, J., Kawanabe, M., Buhmann, J.M.: Optimal Cluster Preserving Embedding of Nonmetric Proximity Data. IEEE Trans. on Pattern Analysis and Machine Intelligence (TPAMI) 25, 1540–1551 (2003) 10. Constantine, A.G., Gower, J.C.: Graphical Representation of Asymmetric Matrices. Journal of the Royal Statistical Society. Series C (Applied Statistics) 27, 297– 304 (1978) 11. Jolliffe, I.T.: Principal Component Analysis. Springer, Heidelberg (2002) 12. Anderberg, M.R.: Cluster Analysis for Applications. Academic Press Inc., London (1973) 13. Zheng, N., Xue, J.: Statistical Learning and Pattern Analysis for Image and Video Processing. Springer-Verlag London Limited (2009) 14. Laub, J., Roth, V., Buhmann, J.M., M¨ uller, K.-R.: On the Information and Representation of Non-Euclidean Pairwise Data. Pattern Recognition 39, 1815–1826 (2006) 15. Alzate, C., Suykens, J.A.K.: Multiway Spectral Clustering with Out-of- Sample Extensions Through Weighted Kernel PCA. IEEE Trans. on Pattern Analysis and Machine Intelligence (TPAMI) 32, 335–347 (2010)
A New Anticorrelation-Based Spectral Clustering Formulation
149
16. Monteiro, F.C., Campilho, A.C.: Spectral Methods in Image Segmentation: A Combined Approach. In: Marques, J.S., P´erez de la Blanca, N., Pina, P. (eds.) IbPRIA 2005. LNCS, vol. 3523, pp. 191–198. Springer, Heidelberg (2005) 17. Srebro, N., Jaakkola, T.: Linear Dependent Dimensionality Reduction. Advances in Neural Information Processing Systems (NIPS) 16, 145–152 (2003) 18. Sanguinetti, G.: Dimensionality Reduction in Clustered Data Sets. IEEE Trans. on Pattern Analysis and Machine Intelligence (TPAMI) 30, 535–540 (2008) 19. Talwalkar, A., Kumar, S., Rowley, H.: Large-Scale Manifold Learning. In: Computer Vision and Pattern Recognition, CVPR (2008) 20. P¸ekalska, E., Harol, A., Duin, R.P.W., Spillmann, B., Bunke, H.: Non-Euclidean or Non-metric Measures Can Be Informative. In: Yeung, D.-Y., Kwok, J.T., Fred, A., Roli, F., de Ridder, D. (eds.) SSPR 2006 and SPR 2006. LNCS, vol. 4109, pp. 871–880. Springer, Heidelberg (2006) 21. Belabbas, M.-A., Wolfe, P.: Spectral Methods in Machine Learning and New Strategies for Very Large Datasets. Proceedings of National Academy of Sciences (PNAS) of the USA 106, 369–374 (2009) 22. Chang, H., Yeung, D.-Y.: Robust Path-based Spectral Clustering. Pattern Recognition 41, 191–203 (2008) 23. Fowlkes, C., Belongie, S., Chung, F., Malik, J.: Spectral Grouping Using the Nystr¨ om Method. IEEE Trans. on Pattern Analysis and Machine Intelligence (TPAMI) 26, 214–225 (2004) 24. Tung, F., Wong, A., Clausi, D.A.: Enabling Scalable Spectral Clustering for Image Segmentation. Pattern Recognition 43, 4069–4076 (2010) 25. Chen, W.-Y., Song, Y., Bai, H., Lin, C.-J., Chang, E.Y.: Parallel Spectral Clustering in Distributed Systems. IEEE Trans. on Pattern Analysis and Machine Intelligence (TPAMI) 33, 568–586 (2010) 26. Cour, T., Yu, S. and Shi, J.: Ncut demo software, http://www.cis.upenn.edu/ ~jshi/software 27. Sch¨ olkopf, B., Smola, A.J.: Learning with Kernels. MIT Press, Cambridge (2002) 28. Stewart, G.W.: Matrix Algorithms, Eigensystems, vol. II. SIAM, Philadelphia (2001)
Simultaneous Partitioned Sampling for Articulated Object Tracking Christophe Gonzales, S´everine Dubuisson, and Xuan Son N’Guyen Laboratoire d’Informatique de Paris 6 (LIP6/UPMC) 4 place Jussieu, 75005 Paris, France [email protected]
Abstract. In this paper, we improve the Partitioned Sampling (PS) scheme to better handle high-dimensional state spaces. PS can be explained in terms of conditional independences between random variables of states and observations. These can be modeled by Dynamic Bayesian Networks. We propose to exploit these networks to determine conditionally independent subspaces of the state space. This allows us to simultaneously perform propagations and corrections over smaller spaces. This results in reducing the number of necessary resampling steps and, in addition, in focusing particles into high-likelihood areas. This new methodology, called Simultaneous Partitioned Sampling, is successfully tested and validated for articulated object tracking.
1
Introduction
Articulated object tracking is an important computer vision task for a wide variety of applications including gesture recognition, human tracking and event detection. However, tracking articulated structures with accuracy and within a reasonable time is challenging due to the high dimensionality of the state and observation spaces. In the optimal filtering context, the goal of tracking is to estimate a state sequence {xt }t=1,...,T whose evolution is specified by a dynamic equation xt = ft (xt−1 , nxt ) given a set of observations. These observations {yt }t=1,...,T , are related to the states by yt = ht (xt , nyt ). Usually, ft and ht are vector-valued and time-varying transition functions, and nxt and nyt are Gaussian noise sequences, independent and identically distributed. All these equations are usually considered in a probabilistic way and their computation is decomposed in two main steps. First the prediction of the density function p(xt |y1:t−1 ) = xt−1 p(xt |xt−1 )p(xt−1 |y1:t−1 )dxt−1 with p(xt |xt−1 ) the prior density related to transition function ft , and then a filtering step p(xt |y1:t ) ∝ p(yt |xt )p(xt |y1:t−1 ) with p(yt |xt ) the likelihood density related to the measurement function ht . When functions ft and ht are linear, or linearizable, and when distributions are Gaussian or mixtures of Gaussians, the sequence {xt }t=1,...,T can be computed analytically by Kalman, Extended Kalman or Unscented Kalman Filters [4]. Unfortunately, most vision tracking problems involve nonlinear functions and non-Gaussian distributions. In such cases, tracking methods based on particle J. Blanc-Talon et al. (Eds.): ACIVS 2011, LNCS 6915, pp. 150–161, 2011. c Springer-Verlag Berlin Heidelberg 2011
Simultaneous Partitioned Sampling for Articulated Object Tracking
151
filters [4,6], also called Sequential Monte Carlo Methods (SMC), can be applied under very weak hypotheses: their principle is not to compute the parameters of the distributions, but to approximate these distributions by a set of N weighted (i) (i) samples {xt , wt }, also called particles, corresponding to hypothetical state realizations. As optimal filtering approaches do, they consist of two main steps: (i) a prediction of the object state in the scene (using previous observations), (i) (i) that consists in propagating the set of particles {xt , wt } according to a pro(i) posal function q(xt |x0:t−1 , yt ), followed by (ii) a correction of this prediction (using a new available observation), that consists in weighting the particles ac(i)
(i)
(i)
p(x
(i)
|x
(i)
)
t t−1 cording to a likelihood function, so that wt ∝ wt−1 p(yt |xt ) , with (i) q(xt |x0:t−1 ,yt ) N (i) = 1. Particles can then be resampled, so that those with highest i=1 wt weights are duplicated, and those with lowest weights are suppressed. There exist many models of particle filters, each one having its own advantages. For example, the Condensation algorithm [9] has proved to be robust to clutter and occlusion due to its multiple hypotheses. Unfortunately, the computational cost of particle filters highly depends on the number of dimensions of the state space and, for large state spaces, costs may be unrealistically high due to the large number of particles needed to approximate the distributions and to the costs (i) of computing weights wt . In this paper, we propose a new way to reduce the number of particles necessary to treat high-dimensional state spaces, by reducing the number of resampling steps and the variance of the particle set. This paper is organized as follows. Section 2 gives a short overview of the existing approaches that try to solve this high-dimensionality problem. Section 3 recalls the Partitioned Sampling approach and its limitations, then details our approach. Section 4 gives comparative tracking results on challenging synthetic video sequences. Finally, concluding remarks and perspectives are given in Section 5.
2
Reducing Particle Filter’s Complexity: The Problem of High-Dimentional State Spaces
Dealing with high-dimensional state and observation spaces is a major concern of the vision tracking research community, especially when using the particle filter framework for articulated object tracking. There exist essentially two ways to tackle high-dimensional problems: either reduce the dimension of the state space/search space or exploit conditional independences naturally arising in the state space to partition the latter into low-dimensional spaces where few particles are needed. Among those algorithms that follow the first way, some exploit tailored proposal functions to better guide particles during the prediction step. For instance, in [3], attractors corresponding to specific known state vectors are used to better diffuse particles and then to explore more efficiently the state space. For articulated body tracking, many model-driven approaches using prior knowledge [7] on the movement of articulated parts derived from physical models, have been successfully applied. In [8], the environment is assumed to influence the movement
152
C. Gonzales, S. Dubuisson, and X.S. N’Guyen
of body parts and environmental constraints are thus integrated into the tracking scheme. Search into the state space may also be improved using optimization techniques. In [18], a population-based metaheuristic is for instance exploited: a path relinking scheme is used to resample particles so as to avoid missing modes of the probability distribution to estimate. Deutscher et al. [5] also proposed the Annealed Particle Filter that consists of adding to the resampling step simulated annealing iterations to diffuse particles into high-likelihood areas. In [2] a new optimization technique is also considered, which is more efficient than previous classical gradient methods because it incorporates constraints. The second family of approaches consists of reducing the number of necessary particles by exploiting conditional independences in the state space to divide it into small parts. For instance, in [16], graphical models are used to derive conditional density propagation rules and to model interpart interactions between articulated parts of the object. A belief inference is also used in [19], where the articulated body is modeled by a Dynamical Bayesian Network in which inference is computed using both Belief Propagation and Mean Field algorithms. In [1] a body is modeled by a factor graph, and the marginal of each part of the body is computed using Belief Propagations and a specific particle filter. Then, the global estimation consists of recomputing all the weights by taking into account the links between parts of the body. In [17] Bayesian Networks are exploited to factor the representation of the state space, hence reducing the complexity of the particle filter framework. Partitioned Sampling (PS) [11,12] was proposed by MacCormick and Isard in 2000 and is one of the most popular frameworks. The key idea, that will be described in Section 3.1, is to divide the joint state space xt into a partition of P elements, i.e. one element per object part, and for each one, to apply the transition function and to perform a weighted resampling operation. PS was first applied in multiple object tracking. The order of treatment of the objects was fixed over time, which made it fail when there were occlusions. In Dynamic Partition Sampling [20], the posterior distribution is represented by a mixture model, whose mixture components represent a specific order of treatment of the objects. However, when the set of configurations is large, this approach becomes intractable. The Ranked Partition Sampling [21] proposes to simultaneously estimate the order of treatment of the objects and their distributions. For the articulated object tracking purpose, PS suffers from numerous resampling steps that increase noise as well as decrease the tracking accuracy over time. We propose in this paper an adaptation of PS to efficiently deal with an articulated object by exploiting independence between its parts to reduce the number of resampling steps. We call this new methodology Simultaneous Partitioned Sampling (SPS) and derive its modeling in the next section.
3 3.1
Proposed Approach Partitioned Sampling (PS)
Partitioned Sampling (PS) is a very effective Particle Filter that exploits some decomposition of the system dynamics in order to reduce the number of particles
Simultaneous Partitioned Sampling for Articulated Object Tracking
153
needed to track objects when the state space dimensions are large. The basic idea is to divide the state space into an appropriate set of partitions and to apply sequentially a Particle Filter on each partition, followed by a specific resampling ensuring that the sets of particles computed actually represent the joint distributions of the whole state space. This specific resampling is called a “Weighted Resampling”. Let g : X → R be a strictly positive continuous function on X called a weighting function. Given a (i) (i) (i) set of particles Pt = {xt , wt }N i=1 with weights wt , weighted resampling pro(i) (i) duces a new set of particles Pt = {x t , w t }N i=1 representing the same distribution as Pt while located at the peaks of function g. To achieve this, let an “imporN (i) (j) tance distribution” ρt be defined on {1, . . . , N } by ρt (i) = g(xt )/ j=1 g(xt ) for i = 1, . . . , N . Select independently indices k1 , . . . , kN according to proba(i) (i) bility ρt . Finally, construct a set of particles Pt = {x t , w t }N i=1 defined by (i) (i) (k ) (k ) i i x t = xt and w t = wt /ρt(ki ). MacCormick [10] shows that Pt represents the same probability distribution as Pt while focusing on the peaks of g. The basic idea underlying Partitioned Sampling is to exploit a “natural” decomposition of the system dynamics w.r.t. subspaces of the state space in order to apply Particle Filtering only on those subspaces. This allows for a significant reduction in the number of particles needed to track complex objects. More precisely, assume that state space X can be partitioned as X = X 1 × · · · × X P , i.e., the system is viewed as being composed of P parts. For instance, a system representing a hand could be decomposed as X hand = X palm × X thumb × X index × X middle × X ring × X little . In addition, assume that the dynamics of the whole system follows this decomposition, i.e., that there exist functions fti : X → X satisfying that the projection of x over X 1 ×· · ·×X i−1 equals that of x whenever x = fti (x) and such that: ft (xt−1 , nxt ) = ftP ◦ ftP −1 ◦ · · · ◦ ft2 ◦ ft1 (xt−1 ),
(1)
where ◦ is the usual function composition operator. By definition, each function fti can propagate the particles over subspace X i × · · · × X P , i.e., it can only modify the substates of the particles defined on X i × · · · × X P . However, in practice, function fti usually just modifies the substate defined on X i . One step of a “standard” Particle Filter would resample particles, propagate them using proposal function ft and finally update the particle weights using the observations at hand. Here, exploiting the features of weighted resampling, Partitioned Sampling achieves the same result by substituting the ft propagation by a sequence of applications of the fti followed by weighted resamplings, as shown in Fig. 1. In this figure, operations “∗fti ” refer to propagations of particles using proposition function fti as defined above and operations “∼ gti ” refer to weighted resamplings w.r.t. importance function gti . Of course, to be effective, PS needs gti to be peaked with the same region as the posterior distribution restricted to X i . As an example, assume that X = X 1 × X 2 and consider that the large square on Fig. 2.a represents the whole of X . Then, the effect of propagating particles according to ft1 and resampling w.r.t. gt1 corresponds to direct the set of particles into the vertical shaded rectangle (where the peaks of gt1 are
154
C. Gonzales, S. Dubuisson, and X.S. N’Guyen p(xt |xt−1 , y1:t−1 )
∼
∗ft1
∼ gt1
∗ft2
∼ gt2
∗ftP
×p(yt|xt )
p(xt |y1:t )
Fig. 1. Partitioned Sampling condensation diagram X2
X3
X1
X2
a) Partition Sampling
b) Simultaneous Partition Sampling
Fig. 2. Interpretation of Partitioned Sampling ans Simultaneous Partition Sampling
located). Further propagating these particles using ft2 and resampling w.r.t. gt2 head them toward the small shaded rectangle that precisely corresponds to the peaks of p(xt |y1:t ). This scheme can be significantly improved when the likelihood function decomposes on subsets X i , i.e., when: P
p(yt |xt ) =
pi (yti |xit ),
(2)
i=1
where yti and xit are the projections of yt and xt on X i respectively. Such a decomposition naturally arises when tracking articulated objects. In these cases, PS condensation diagram can be substituted by that of Fig. 3. MacCormick and Isard show that this new diagram produces mathematically correct results [12]. 3.2
Our Contribution: Simultaneous Partitioned Sampling (SPS)
In a sense, the hypotheses used by Partition Sampling can best be explained on a dynamic Bayesian Network (DBN) representing the conditional independences
p(xt |xt−1 , y1:t−1 )
∗ft1
×p1t
∼
∗ft2
×p2t
∼
∗ftP
×pP t
p(xt|y1:t )
Fig. 3. Improved Partitioned Sampling condensation diagram
Simultaneous Partitioned Sampling for Articulated Object Tracking
x2t−1
x2t 2 yt−1
x2t+1 yt2
x1t−1
2 yt+1
x1t 1 yt−1
x1t+1 yt1
x3t−1
1 yt+1
x3t 3 yt−1
x3t+1 yt3
time slice t − 1
155
time slice t
3 yt+1
time slice t + 1
Fig. 4. A Dynamic Bayesian network
between random variables of states and observations [13]. Assume for instance that an object to be tracked is composed of 3 parts: a torso, a left arm and a right arm. Let x1t , x2t , x3t represent these parts respectively. Then, the probabilistic dependences between these variables and their observations yt1 , yt2 , yt3 , can be represented by the DBN of Fig. 4. In this figure, Eq. (2) implicitly holds because, conditionally to states xit , observations yti are independent of the other random variables. In addition, the probabilistic dependences between substates x1t , x2t , x3t suggest that the dynamics of the system is decomposable on X 1 × X 2 × X 3 . As a consequence, the condensation diagram of Fig. 3 can be exploited to track the object. Through the d-separation criterion [14], DBNs offer a strong framework for analyzing probabilistic dependences among sets of random variables. By this criterion, it can be remarked that, on Fig. 4, x3t is independent of x2t conditionally to x1t and x3t−1 . Similarly, x2t can be shown to be independent of x3t conditionally to x1t and x2t−1 . As a consequence, propagations/corrections over subspaces X 2 and X 3 can be performed simultaneously (since they are independent). This suggests the condensation diagram of Fig. 5, which we call a Simultaneous Partitioned Sampling (SPS). It is easily seen that, as for PS, the set of particles resulting from SPS represents probability distribution p(xt |y1:t ). The major difference with PS is that, by resampling only after both x2t and 3 xt have been processed, we can gain in accuracy. Actually, consider Fig. 2.b in which the shaded rectangles explain how PS achieves concentrating iteratively on the peaks of p(xt |y1:t ). After processing subspace X 2 , PS focuses on the light
p(xt |xt−1 , y1:t−1 )
∗ft1
×p1t
∗ft2
×p2t
∗ft3
×p3t
∼ ∼
p(xt |y1:t )
Fig. 5. Basic Simultaneous Partitioned Sampling condensation diagram
156
C. Gonzales, S. Dubuisson, and X.S. N’Guyen
gray rectangle. Therefore, this rectangle is determined only using observation yt2 . If, for a given particle, substate x1t was near the edge of a rectangle (i.e., not too close to a peak), then one observation yt2 may be insufficient to discard this value of x1t whereas two observations yt2 and yt3 may well be sufficient. In other words, taking into account multiple independent observations can focus the particles on smaller peaked regions of the state space. For instance, on Fig. 2.b, it may well be the case that, instead of ending up with particles located in the dark shaded area, SPS focus them on the smaller dashed rectangle. Of course, to be effective, SPS needs that propagations/corrections on all the subspaces processed simultaneously be “good” in the sense that they end up with high weights. A naive approach to SPS would not guarantee this property. For instance, on the example of Fig. 4, a particle may be close to the true state of the left arm and far from that of the right arm. In such a case, the overall weight of the particle would be low. If numerous particles had this feature, SPS would perform poorly. Fortunately, the conditional independences exploited by SPS also enable a substate swapping operation that improves significantly the concentration of the particles in the high likelihood areas. The idea is that if 1,(i) 2,(i) 3,(i) 1,(j) 2,(j) 3,(j) two particles, say (xt , xt , xt ) and (xt , xt , xt ), have the same 1,(i) 1,(j) 1 substate on X , i.e., xt = xt , then we can swap their substates on X 2 1,(i) 2,(j) 3,(i) 1,(j) 2,(i) 3,(j) or X 3 (thus creating new particles (xt , xt , xt ) and (xt , xt , xt )) without altering the probability distribution represented by the set of particles. This feature is actually guaranteed by the d-separation criterion. Consequently, if one particle is close to the true state of the left arm and far from that of the right arm while another is close to the true state of the right arm and far from that of the left arm, provided they have the same value on X 1 , we can substitute them by a new particle close to the true states of both arms and a new particle far from those true states. Of course, after swapping, resampling will essentially take the best particle into account. By having the best values on both X 2 and X 3 , this particle allows SPS to concentrate on smaller high-peaked regions than PS. This leads to the new condensation diagram of Fig. 6, where 2,3 represents substate swapping on X 2 and X 3 . i Of course, this scheme can be easily generalized. Assume that X = P i=1 X . i P j i Partition set {X }i=1 into subsets Y = {X }i∈Ij such that ∪Ij = {1, . . . , P } and such that each pair X i1 , X i2 belonging to the same set Y j are independent conditionally to subspaces X i ∈ Y j with j < j. Then, for each j, the simultaneous propagation/correction/swapping/resampling scheme described above can
p(xt |xt−1 , y1:t−1 )
∗ft1
×p1t
∗ft2
×p2t
∗ft3
×p3t
∼ 2,3
∼
p(xt |y1:t )
Fig. 6. Complete Simultaneous Partitioned Sampling condensation diagram
Simultaneous Partitioned Sampling for Articulated Object Tracking
157
be applied on the sets of Y j (note however that, to guarantee that probability distributions remain unchanged, swapping some substate xit ∈ Y j actually requires swapping accordingly all the substates xst that are not d-separated from xit conditionally to {xrt ∈ Y k : k < j}). In the next section, we highlight the advantages of SPS by comparing it to PS on challenging synthetic video sequences.
4
Experimental Results
We have chosen to test our method and to compare it with PS on synthetical video sequences because we wanted to highlight its interest in terms of dimensionality reduction and tracking accuracy without having to take into account specific properties of images (noise, etc.). Moreover, it is possible to simulate specific motions and then to test and compare with accuracy our method with PS. We have generated our own synthetic video sequence, each one containing 300 frames, showing a P -part articulated object (P = {3, 5, 7, 9, 11}) translating and distorting over time, see examples of frames in Fig. 7. The goal, here, is then to observe the capacity of PS and SPS to deal with articulated objects composed of a varying number of parts and subject to weak or strong motions.
Fig. 7. Some frames of synthetic sequences: an articulated object with 7 and 9 parts
The tracked articulated object is modeled by a set of P rectangles. The state space contains parameters describing each rectangle, and is defined by xt = p p p p p p {x1t , x2t , . . . , xP t }, with xt = {xt , yt , θt }, where (xt , yt ) denotes the center of the (i) 1,(i) 2,(i) P,(i) pth rectangle, and θtp is its orientation. A particle xt = {xt , xt , . . . , xt } is then a possible configuration of an articulated object. In the first frame, particles are uniformly generated around the object. During the prediction step, particles are propagated following a random walk whose variance has been manually chosen. The weights of the particles are then computed using the current observation (i.e. the current frame). A classical approach consists in integrating the color distributions given by histograms into particle filtering [15], by measuring the similarity between the distribution of pixels in the region of the estimated parts of the articulated object and of the corresponding reference region. This similarity is determined by computing the Bhattacharyya distance d between the histograms of the target and the reference regions. Finally, the particle’s weights 2 (i) (i) (i) (i) are given by wt = wt−1 p(yt |xt ) ∝ wt−1 e−λd , with λ = 50 in our tests. For both approaches, the articulated object global joint distribution is estimated by starting from its center part. PS then propagates and corrects particles part after part to derive a global estimation of the object. SPS considers the left and right
C. Gonzales, S. Dubuisson, and X.S. N’Guyen
140000
PS SPS
40000 Tracking error
Tracking error
45000
PS SPS
120000 100000 80000 60000 40000 20000
Tracking error
158
35000 30000 25000 20000 15000 10000 5000
0
0 0
50
100 150 200 250 300
0
50
200
250
300
0
1800
PS SPS
100 150 200 Frame number
250
1200 1000 800 600 400
300
150
200
250
300
250
300
PS SPS
1200
1400
1000 800 600 400 200
0 50
100
1400
PS SPS
1600
200 0
50
Frame number
Tracking error
2200 2000 1800 1600 1400 1200 1000 800 600 400 200 0
100 150
PS SPS
Frame number
Tracking error
Tracking error
Frame number
50000 45000 40000 35000 30000 25000 20000 15000 10000 5000 0
0 0
50
100 150 200 Frame number
250
300
0
50
100 150 200 Frame number
(a)
200000 150000 100000 50000 0 0
50
100000 90000 80000 70000 60000 50000 40000 30000 20000 10000 0
100 150 200 250 300
PS SPS Tracking error
PS SPS
250000
Tracking error
Tracking error
300000
0
50
Frame number
2500
2000 1500 1000 500 0 50
100
150
200
Frame number
50
250
300
PS SPS
2000 1500 1000 500
0
50
100
150
200
Frame number
100
150
200
250
300
250
300
Frame number
0 0
0
Tracking error
PS SPS
2500
100 150 200 250 300
PS SPS
Frame number
Tracking error
Tracking error
3000
50000 45000 40000 35000 30000 25000 20000 15000 10000 5000 0
250
300
2000 1800 1600 1400 1200 1000 800 600 400 200 0
PS SPS
0
50
100
150
200
Frame number
(b) Fig. 8. Convergence study (50 runs): comparison between PS and SPS, from left to right, top to bottom: N = 2, N = 5, N = 10, N = 20, N = 30 and N = 40 for an articulated composed of (a) 7 parts, (b) 9 parts
parts as totally independent, and thus propagates and corrects simultaneously in these parts. PS and SPS are compared in terms of tracking accuracy. For that, we measure the tracking error as the distance between the ground truth and the estimated articulated object at each instant. This distance is given by the sum of the Euclidean distances between the corners of the estimated rectangles and their corresponding corners of the ground truth shape. We also measure the variance of the particle set. In the first test, we compare the convergence of PS and SPS. For that, we use synthetic sequences showing the same articulated object. We compute the tracking errors by averaging over 50 different runs, and repeat these 50 runs for different numbers N of particles. Tracking errors of articulated objects composed of 7 and 9 parts are shown in Figure 8.(a-b). As we can see on this figure, SPS always outperforms PS, even when N becomes very high. One can also notice
Simultaneous Partitioned Sampling for Articulated Object Tracking 450
350
PS SPS
400
250
Variance
Variance
PS SPS
300
350 300 250 200
159
200 150 100
150 100
50 0
50
100
150
200
250
0
300
50
100
Frame number
150
200
250
300
Frame number
Fig. 9. Comparison of variances (50 runs) obtained for PS and SPS, for a set of N = 10 particles for an articulated composed of, from left to right, 7 and 9 parts
0 0
50
100
150
200
250
300
1000 800 600 400
1400 1200 1000 800 600 400
200
200
0
0 0
Frame number
50
100
150
200
250
300
2000 1500 1000 500 0
0
Frame number
25000
PS SPS
2500
Tracking error
50
1200
3000
PS SPS
1600
1400
Tracking error
100
1800
PS SPS
1600
Tracking error
1800
PS SPS Tracking error
Tracking error
300 250 200 150
50
100
150
200
250
300
Frame number
PS SPS
20000 15000 10000 5000 0
0
50
100
150
200
250
300
Frame number
0
50
100 150
200 250
300
Frame number
Fig. 10. PS vs. SPS: case of strong motions at the beginning of the sequence, N = 20 (50 runs). Tracking errors for an articulated object of, from left to right: 3, 5, 7, 9 and 11 parts. Table 1. Execution times in seconds of PS and SPS, for N = 10 to 200 particles (the times reported are averages over 20 runs) # parts 3 5 7 9 11
tracker PS SP S PS SP S PS SP S PS SP S PS SP S
10
20
40
60
80
100
200
0.54 0.56 0.71 0.74 0.95 1.00 1.21 1.25 1.41 1.46
0.88 0.91 1.26 1.32 1.72 1.76 2.18 2.23 2.61 2.77
1.58 1.61 2.33 2.42 3.21 3.28 4.12 4.25 4.99 5.11
2.99 2.33 3.34 3.43 4.7 4.84 6.16 6.32 7.38 7.61
2.82 2.84 4.43 4.51 6.22 6.46 8.12 8.40 10.02 10.33
3.45 3.50 5.43 5.60 7.86 7.99 10.35 10.59 12.69 13.01
6.80 6.92 11.62 12.05 17.10 17.73 22.06 22.77 26.37 27.18
that with a very small number of particles (N = 2), SPS shows more robustness. In fact SPS is more stable during periods in the sequence where the motion is stronger, when PS totally looses the object to track. Figure 9 confirms the stability of the proposed approach: the variance of the particle set is lower with SPS. This proves that the particles are more concentrated around high likelihood values than with PS. This is mainly due to the fact that PS does perform twice more resampling steps than SPS, introducing more noise. To test the stability of our approach, we have generated video sequences in which the motion at the beginning of the sequence is strong. Comparative results of tracking errors of PS and SPS are reported in Figure 10, for different articulated object (i.e. containing 3, 5, 7, 9 and 11 parts). Here again we see the
160
C. Gonzales, S. Dubuisson, and X.S. N’Guyen
3500
2500
Variance
Tracking error
1800 1600
PS SPS
3000
2000 1500 1000
PS SPS
1400 1200 1000 800 600 400
500
200 0
0 0
50
100 150 200 Frame number
250
300
0
50
100 150 200 Frame number
250
300
Fig. 11. PS vs. SPS: case of strong motions for a 5-part object, N = 10, from left to right: two frames, tracking errors and variance of the particle set (50 runs). SPS outperforms PS because of its capacity to more concentrate particles around highlikelihood areas.
SPS is less disturbed by this strong motion that PS. Moreover, Table 1 shows that the efficiency of SPS over PS is not achieved at the expense of response times (SPS is usually not more than 4% slower than PS). We have also compared the efficiency of PS and SPS in cases of very strong and erratic movements of parts of the articulated object throughout the sequence. Comparative tracking results for a 5-part articulated object are given in Figure 11. We can see two examples of frames of this sequence and the deformation that undergoes the articulated object. Tracking errors are considerably decreased with SPS, as is the variance.
5
Conclusion
We have presented a new methodology, the Simultaneous Partitioned Sampling, that uses independence properties to simultaneous propagate and correct particles in conditionally independent subspaces. As a result, the particle set is more concentrated into high-likelihood areas. Thus, the estimation of the probability density of the tracked object is more accurate. Empirical tests have shown that SPS outperforms PS, especially in cases where the object motion is strong and when the dimension of the state space increases (i.e., the number of parts is large). There still remains to validate this approach on real video sequences. There is still room for improving SPS, especially its swapping method. Currently, we are working on linear programming-based techniques to determine the optimal swappings.
References 1. Bernier, O., Cheungmonchan, P., Bouguet, A.: Fast nonparametric belief propagation for real-time stereo articulated body tracking. Computer Vision and Image Understanding 113(1), 29–47 (2009) 2. Bray, M., Koller-Meier, E., M¨ uller, P., Schraudolph, N.N., Van Gool, L.: Stochastic optimization for high-dimensional tracking in dense range maps. IEE Proceedings Vision, Image and Signal Processing 152(4), 501–512 (2005)
Simultaneous Partitioned Sampling for Articulated Object Tracking
161
3. Chang, W.Y., Chen, C.S., Jian, Y.D.: Visual tracking in high-dimensional state space by appearance-guided particle filtering. IEEE Transactions on Image Processing 17(7), 1154–1167 (2008) 4. Chen, Z.: Bayesian filtering: from kalman filters to particle filters, and beyond (2003) 5. Deutscher, J., Davison, A., Reid, I.: Automatic partitioning of high dimensional search spaces associated with articulated body motion capture. In: CVPR, vol. 2, pp. 669–676 (2005) 6. Gordon, N.J., Salmond, D.J., Smith, A.F.M.: Novel approach to nonlinear/nongaussian bayesian state estimation. IEE Proceedings of Radar and Signal Processing 140(2), 107–113 (1993) 7. Hauberg, S., Sommer, S., Pedersen, K.: Gaussian-like spatial priors for articulated tracking. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010. LNCS, vol. 6311, pp. 425–437. Springer, Heidelberg (2010) 8. Hauberg, S., Pedersen, K.S.: Stick it! articulated tracking using spatial rigid object priors. In: Kimmel, R., Klette, R., Sugimoto, A. (eds.) ACCV 2010, Part III. LNCS, vol. 6494, pp. 758–769. Springer, Heidelberg (2011) 9. Isard, M., Blake, A.: Condensation - conditional density propagation for visual tracking. International Journal of Computer Vision 29, 5–28 (1998) 10. MacCormick, J.: Probabilistic modelling and stochastic algorithms for visual localisation and tracking. Ph.D. thesis, Oxford University (2000) 11. MacCormick, J., Blake, A.: A probabilistic exclusion principle for tracking multiple objects. In: ICCV, pp. 572–587 (1999) 12. MacCormick, J., Isard, M.: Partitioned sampling, articulated objects, and interfacequality hand tracking. In: Vernon, D. (ed.) ECCV 2000. LNCS, vol. 1843, pp. 3–19. Springer, Heidelberg (2000) 13. Murphy, K.: Dynamic Bayesian Networks: Representation, Inference and Learning. Ph.D. thesis, UC Berkeley, Computer Science Division (2002) 14. Pearl, J.: Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. Morgan Kaufman Publishers, Inc., San Francisco (1988) 15. P´erez, P., Hue, C., Vermaak, J., Gangnet, M.: Color-based probabilistic tracking. In: Heyden, A., Sparr, G., Nielsen, M., Johansen, P. (eds.) ECCV 2002. LNCS, vol. 2350, pp. 661–675. Springer, Heidelberg (2002) 16. Qu, W., Schonfeld, D.: Real-time decentralized articulated motion analysis and object tracking from videos. IEEE Transactions on Image Processing 16(8), 2129– 2138 (2007) 17. Rose, C., Saboune, J., Charpillet, F.: Reducing particle filtering complexity for 3D motion capture using dynamic bayesian networks. In: AAAI, pp. 1396–1401 (2008) 18. S´ anchez, A., Pantrigo, J., Gianikellis, K.: Combining Particle Filter and Population-based Metaheuristics for Visual Articulated Motion Tracking. Electronic Letters on Computer Vision and Image Analysis 5(3), 68–83 (2005) 19. Shen, C., van den Hengel, A., Dick, A., Brooks, M.: 2D articulated tracking with dynamic Bayesian networks. In: Das, G., Gulati, V.P. (eds.) CIT 2004. LNCS, vol. 3356, pp. 130–136. Springer, Heidelberg (2004) 20. Smith, K., Gatica-perez, D.: Order matters: a distributed sampling method for multi-object tracking. In: BMVC, pp. 25–32 (2004) 21. Widynski, N., Dubuisson, S., Bloch, I.: Introducing fuzzy spatial constraints in a ranked partitioned sampling for multi-object tracking. In: Bebis, G., Boyle, R., Parvin, B., Koracin, D., Chung, R., Hammoud, R., Hussain, M., Kar-Han, T., Crawfis, R., Thalmann, D., Kao, D., Avila, L. (eds.) ISVC 2010. LNCS, vol. 6453, pp. 393–404. Springer, Heidelberg (2010)
A Geographical Approach to Self-Organizing Maps Algorithm Applied to Image Segmentation Thales Sehn Korting, Leila Maria Garcia Fonseca, and Gilberto Cˆamara Image Processing Division National Institute for Space Research – INPE S˜ ao Jos´e dos Campos – SP, Brazil {tkorting,leila,gilberto}@dpi.inpe.br
Abstract. Image segmentation is one of the most challenging steps in image processing. Its results are used by many other tasks regarding information extraction from images. In remote sensing, segmentation generates regions according to found targets in a satellite image, like roofs, streets, trees, vegetation, agricultural crops, or deforested areas. Such regions differentiate land uses by classification algorithms. In this paper we investigate a way to perform segmentation using a strategy to classify and merge spectrally and spatially similar pixels. For this purpose we use a geographical extension of the Self-Organizing Maps (SOM) algorithm, which exploits the spatial correlation among near pixels. The neurons in the SOM will cluster the objects found in the image, and such objects will define the image segments.
1
Introduction
Image segmentation is currently one of the most challenging tasks in digital image processing. The results of segmentation are used by many other tasks regarding information extraction from images. One simple definition states that a good segmentation should separate the image into simple regions with homogeneous behavior [7]. Algorithms for segmentation covers splitting one image into its components, concerning a specific context. The context includes scale, since regions from segmentation can represent different levels of perception. In an urban remote sensing image, found objects describe buildings in a finer scale, or city blocks in a coarser scale. Depending on the application, both scales provide enough information for object detection. Context also includes neighborhood, in accord to Tobler’s law, which states that near things are more related than distant things [18]. In the image processing area, this law may be interpreted as near pixels are more similar than distant pixels. Extending this interpretation to image segmentation, we can state that near and similar pixels should describe regions in the segmentation results. There are several applications in remote sensing which are depending on the results of image segmentation. The found objects must be in accord to the targets of a satellite image, such as roofs [5], streets [8] and trees in urban areas. In J. Blanc-Talon et al. (Eds.): ACIVS 2011, LNCS 6915, pp. 162–170, 2011. c Springer-Verlag Berlin Heidelberg 2011
A Geographical Approach to SOM Algorithm
163
agricultural imagery, the algorithm should extract different crops [16], or deforested areas in forests [17]. Such different land covers differentiate land uses by classification algorithms. Several initiatives of segmentation algorithms applied to satellite imagery have already been proposed in the literature. However, none of them exploits deeply the spatial correlation among points as suggested by Tobler’s Law. In this scenario we present a novel technique for segmentation which aims to fulfill this gap. The paper is organized as follows. Section 2 presents the literature review, Section 3 presents the methodology, Section 4 shows the results and the Section 5 concludes the paper.
2
Literature Review
Traditional segmentation algorithms are often based on the pixel similarity, by comparing one pixel to its neighbors. Several strategies are based on the technique named “region growing”, which merges neighboring pixels with a certain degree of similarity. Some examples of algorithms are defined in [4] and [1]. The similarity are often based only on spectral properties, and seldom refers to the location of the pixels. Many other strategies for image segmentation have already been proposed in the literature. For further reading on traditional segmentation techniques, please refer to [19]. The present work integrates one classification approach to perform segmentation. The employed classification algorithm is a modification of the well known Self-Organizing Maps – SOM [10]. This work investigates an approach to merge spectrally and spatially similar pixels by using a geographical extension of SOM algorithm, named GeoSOM [2]. 2.1
SOM Applied to Remote Sensing
SOM is a powerful tool for exploring huge amounts of high-dimensional data. It defines an elastic, topology-preserving grid of points that fits to the input space [12]. Initially proposed as a visualization tool for n-dimensional data, SOM have been applied in a variety of applications, such as clustering, dimensionality reduction, classification, sampling, vector quantization, and data mining [10]. Basically, according to [21], the SOM training algorithm is similar to a Kmeans clustering, with an added spatial smoothness constraint. Neurons are initialized randomly, and at each iteration the neuron with most similar response to the input data is called winner. The winner and its neighbors are updated to become more similar to the corresponding input value. The size of the considered neighborhood is decreased along the epochs, so that on latter phases only the winner neurons are updated, with very small refinements. In remote sensing several works employ SOM for classification issues. The work of [21] tested the algorithm to classify different vegetation types using radar images of agricultural areas. The idea of semi-supervised SOM was applied, where the closest neuron to a certain pattern is associated with one class
164
T.S. Korting, L.M.G. Fonseca, and G. Cˆ amara
of the training objects. [11] performed a comparison between the ExpectationMaximization and SOM algorithms applied to intra-urban classification using high spatial resolution images to derive the results. According to the authors, both algorithms achieved good results, and presented their advantages. However, SOM classified the data in less time, which for the current increasingly amount of data, is an important advantage. [9] have concluded that SOM is a good choice of an algorithm for monitoring land cover change. They investigated the optimal classification algorithm applied into multi-temporal data based on monthly phenological characteristics. [13] employed subpixel analysis using Landsat ETM+ data to estimate impervious surface coverage, lawn, and woody tree cover in typical urban landscapes. The authors combined SOM, Learning Vector Quantization (LVQ), and Gaussian Mixture Model (GMM) methods to estimate the posterior probability of the land cover components. [20] have applied SOM in the classification of multi and hyperspectral satellite imagery. Since SOM classification is considered to be a topology preserving projection of high-dimensional data onto a low-dimensional lattice, results emphasized the necessity of a faithful topological mapping for correct interpretation. 2.2
Segmentation Using SOM
Classifying image pixels to perform segmentation is not a new topic. [15] have presented one approach using hue and saturation image components to segment personal photographs. SOM was employed to estimate the main found chromaticities. Then, each pixel was classified according to the identified classes. Neighboring pixels relying in the same class were merged in the resultant segmentation. Also applied to personal photographs, [3] proposed one extension to the traditional SOM algorithm, which tried to map classes of pixel intensities in a hierarchical structure of neurons. More recently, [12] have applied SOM in the segmentation of images for content-based image retrieval (CBIR). CBIR has been targeted at interactive use where the task is to return interesting or relevant images from an unannotated database. In their method, the color and texture values are used to train the map and to classify the segments. SOM have also been applied into magnetic resonance brain image segmentation [14], including spatial constraints using a Markov Random Field (MRF) model to improve the results.
3
Methodology
The main idea is to employ classification to perform segmentation, i.e. neighboring pixels with the same class are merged into segments. The novelty is to take advantage of the spatial correlation among the pixels, since we adpoted a geographical extention of SOM algorithm, which considers the spatial proximity of the pixel elements when merging them into a single object. The SOM extention was proposed by [2] and is named GeoSOM. According to the authors, GeoSOM has the potential to organize the SOM output space according to the geographic proximities of the input patterns.
A Geographical Approach to SOM Algorithm
165
As GeoSOM finds patterns in the image pixels, it must fit homogeneous regions in these patterns, and considers these regions as the resultant segments. A simple way of inserting spatial constraints in classification is to include spatialrelevant variables which are computed as any other variable [2]. In this case remains the choice of what spatial variables to use. The possibilities are endless, and depend on the objectives pursued. Also one has to find the weights to be attributed to the geographic variables, thus giving more (or less) importance to geographic information. The weights of the neurons have their own geographical properties. Since neurons have spatial positions, they are expected to converge to patterns found nearby in the image. In the resultant segmentation clusters reflect the spectral distribution of the data, and are positioned according the spatial occurence of the patterns. Figure 1 shows this approach. In the following we describe the GeoSOM algorithm. Let: – X be the set of N training patterns x1 , x2 , . . . xN , each of these having a set of components geo, i and another set notgeo, i; – W be a p × q grid of units wij where i and j are their coordinates on that grid, each of these units having a set of components wgeo,ij and another set wnotgeo,ij ; – α be the learning rate, assuming values in the interval (0, 1); – h(wij , wmn , r) be a neighborhood function, used to update the neurons wij and wmn with radius r, generally expressed by the gaussian equation 2 2 1 (i−m) +(j−n)
r2 e− 2 – k be the radius of the geographical Best Matching Unit (BMU) to be searched; – f be true if the units are at fixed geographical locations.
1 repeat 2 for m = 1 → N 3 ∀wij ∈ W , calculate dij = ||geom − wgeo,ij || 4 wwinnergeo is the unit that minimizes dij 5 select set Wwinners of wij that ||wwinnergeo − wij || ≤ k 6 ∀wij ∈ Wwinner , calculate dij = ||xm − wij || 7 wwinner is the unit that minimizes dij 8 if f = true 9 update units wij ∈ W : wnotgeo,ij αh(wnotgeo,winner , wnotgeo,ij , r)||xm − wij || 10 else 11 update units wij ∈ W : wij = wij + αh(wwinner , wij , r)||xm − wij || end if 12 decrease α and r 13 until α = 0
=
wnotgeo,ij +
During the training stage the algorithm searches for the nearest neuron to the input value, the so called wwinnergeo . In this case, “nearest” considers the
166
T.S. Korting, L.M.G. Fonseca, and G. Cˆ amara
Fig. 1. Inserting spatial variables as input data – the GeoSOM approach
locations of the input and the neurons. After, it searches for the most similar neuron, in terms of spectral properties, within the neighborhood of wwinnergeo , finding the neuron wwinner . At this point the parameter f plays an important role. If f is true, the neurons are updated only in their spectral properties. If f is false, besides updating the spectral properties of the neurons, their location is changed as well. The learning rate α and the radius r decrease along the epochs, allowing the algorithm to converge smoothly to the discovered patterns. The algorithm stops when one of these two parameters reaches 0, or when a pre-defined number of epochs is performed. Found patterns are then converted into segmentation. When the algorithm stops, neurons have their spectral properties similar to the
A Geographical Approach to SOM Algorithm
(a)
167
(b)
Fig. 2. Segmentation result: a) Input image, and b) segmentation with GeoSOM
main elements found in the image. Summarizinh, the neurons in the SOM will cluster the objects found in the image, and such objects will define the image segments.
4
Results
The following results are composed by input images and their segmentation using GeoSOM. To produce such results we tuned the algorithm with a set of parameters that reached the best results, considering visual inspection. Nevertheless, we selected small image croppings and segmented them with different parameters. According to [6], “a strong and experienced evaluator of segmentation techniques is the human eye/brain combination”, and according [1], “no segmentation result – even if quantitatively proofed – will convince, if it does not satisfy the human eye”. The first image is a crop of a region in the state of Minas Gerais, Brazil, using satellite CBERS-2B, instrument CCD1 . The spatial resolution of the image is 20m. Figure 2 shows the segmentation. In the first result we used a map with 9 neurons (a 3 × 3 grid), with a learning rate α = 0.2 for 50 epochs. 410 regions were generated. Segmented regions describe properly the main objects found in the image. In the central region, which is spectrally heterogeneous, the algorithm also obtained proper segments. As a drawback, some mistakes were noticed in the borders, as they were segmented as new objects. However, the training step in this example used merely 10% of the pixel information present in the image. This fact provides to our approach a significant reduction in the volume of data to be processed, reducing the processing time as well. 1
Free remote sensing imagery at http://www.dgi.inpe.br/CDSR/
168
T.S. Korting, L.M.G. Fonseca, and G. Cˆ amara
(a)
(b)
Fig. 3. Segmentation result: a) Input image, and b) segmentation with GeoSOM
We performed a second test using a high spatial resolution input image, segmenting a small crop of a Quickbird scene from an urban region in the state of S˜ ao Paulo, Brazil. The spatial resolution of the image is 0.5m. Target objects in this image are, differently from the first example, intra-urban objects, such as roofs, trees, shadows, streets, and so on. Figure 3 shows the segmentation. In the second result we used a map with 25 neurons (a 5×5 grid), with learning rate α = 0.2 for 200 epochs. 911 regions were generated. Since we used a higher amount of neurons, the algorithm generated more regions. Even the roofs of the houses were split, in some cases, in more than one region. By inspecting the values of the neurons responsible for the roofs regions, they presented similar spectral values, however different spatial locations. The result shown in Figure 3 are considered over-segmentation, because in some cases the same object reflects in more than one segmented region. However it remains an useful result, since no region contains more than one object inside. In this case the main objects of the image can still be distinguished correctly, by applying proper inferences into the data.
5
Conclusion
This article presented the idea to apply an extension of the SOM algorithm, called GeoSOM, to remote sensing image segmentation. We described the main steps of the algorithm, and how to perform the segmentation. To evaluate the technique, we tested two images with different spatial resolutions and contexts. By tuning the parameters properly, we were able to generate visually appealing results, which makes this algorithm for segmentation encouraging approach. One important aspect of this approach is the reduced amount of data required to perform the segmentation. With 10% of the total pixels in the images, the neurons were trained and the results were able to split properly the images
A Geographical Approach to SOM Algorithm
169
into their component regions. One drawback was detected in the segmentation of some borders, since they do not present spectral properties similar to the bordering objects. However, the smoothness of the resultant objects can be improved by adding a parameter to define a minimum area, or some other threshold applied to spatial properties of the generated segments. Future research include to extend this approach by inspecting the dynamic of the neurons location along training. We believe that it is possible to describe the position of the most relevant objects in the image. If the image is classified after the segmentation, it is possible to infer the spatial distribution of the patterns. An interesting visualization tool can be derived from this approach, by retrieving the position and the amount of the neurons at each place of the image.
References 1. Baatz, M., Schape, A.: Multiresolution Segmentation: an optimization approach for high quality multi-scale image segmentation. In: Wichmann-Verlag (ed.) XII Angewandte Geographische Informationsverarbeitung, Herbert Wichmann Verlag, Heidelberg (2000) 2. Ba¸ca ˜o, F., Lobo, V., Painho, M.: Applications of different self-organising map variants to geographical information science problems. Self-organising Maps: Applications in Geographic Information Science (2008) 3. Bhandarkar, S., Koh, J., Suk, M.: Multiscale image segmentation using a hierarchical self-organizing map. Neurocomputing 14(3), 241–272 (1997) 4. Bins, L., Fonseca, L., Erthal, G., Li, F.: Satellite imagery segmentation: a region growing approach. In: Brazilian Remote Sensing Symposium, vol. 8 (1996) 5. Chesnel, A.L., Binet, R., Wald, L.: Object oriented assessment of damage due to natural disaster using very high resolution images. In: IEEE International Geoscience and Remote Sensing Symposium, IGARSS 2007, pp. 3736–3739 (2007) 6. Gamanya, R., Demaeyer, P., Dedapper, M.: An automated satellite image classification design using object-oriented segmentation algorithms: A move towards standardization. Expert Systems with Applications 32(2), 616–624 (2007) 7. Haralick, R., Shapiro, L.: Image segmentation techniques. Applications of Artificial Intelligence II 548, 2–9 (1985) 8. He, Y., Wang, H., Zhang, B.: Color based road detection in urban traffic scenes. In: Proceedings of IEEE Intelligent Transportation Systems, vol. 1 (2003) 9. Kim, D., Jeong, S., Park, C.: Comparison of Three Land Cover Classification Algorithms-ISODATA, SMA, and SOM-for the Monitoring of North Korea with MODIS Multi-temporal Data. Korean Journal of Remote Sensing 23(3), 181–188 (2007) 10. Kohonen, T.: Self-Organizing Maps, 3rd edn. Springer, Berlin (2001) 11. Korting, T.S., Fonseca, L.M.G., Ba¸ca ˜o, F.: Expectation-Maximization x SelfOrganizing Maps for Image classification. In: IEEE International Conference on Signal Image Technology and Internet Based Systems, SITIS 2008, pp. 359–365 (2008) 12. Laaksonen, J., Viitaniemi, V., Koskela, M.: Application of Self-Organizing Maps and automatic image segmentation to 101 object categories database. In: Proc. Fourth International Workshop on Content-Based Multimedia Indexing (CBMI 2005), Riga, Latvia,Citeseer (2005)
170
T.S. Korting, L.M.G. Fonseca, and G. Cˆ amara
13. Lee, S., Lathrop, R.: Subpixel analysis of Landsat ETM+ Using Self-Organizing Map (SOM) Neural Networks for Urban Land Cover Characterization. IEEE Transactions on Geoscience and Remote Sensing 44(6), 1642–1654 (2006) 14. Li, Y., Chi, Z.: MR Brain image segmentation based on self-organizing map network. International Journal of Information Technology 11(8), 45–53 (2005) 15. Moreira, J., Costa, L.F.: Neural-based color image segmentation and classification using self-organizing maps. Proceedings of the IX SIBGRAPI 12(6), 47–54 (1996) 16. Perez, A., Benlloch, J., Lopez, F., Christensen, S.: Colour and shape analysis techniques for weed detection in cereal fields. Computers and Electronics in Agriculture 25, 197–212 (2000) 17. Silva, M., Cˆ amara, G., Souza, R., Valeriano, D., Escada, M.: Mining patterns of change in remote sensing image databases. In: The Fifth IEEE International Conference on Data Mining, New Orleans, Louisiana, USA. Citeseer (2005) 18. Tobler, W.: A Computer Movie Simulating Urban Growth in the Detroit Region. Economic Geography 46, 234–240 (1970) 19. Verg´es-Llah´ı, J.: Color Constancy and Image Segmentation Techniques for Applications to Mobile Robotics. Ph.D. thesis, UPC (2005) 20. Villmann, T., Mer´enyi, E.: Extensions and modifications of the Kohonen-SOM and applications in remote sensing image analysis. Studies in Fuzziness and Soft Computing 78, 121–144 (2002) 21. Wehrens, R.: Self-organising Maps for Image Segmentation. In: Advances in Data Analysis, Data Handling and Business Intelligence: Proceedings of the 32nd Annual Conference of the Gesellschaft F¨ ur Klassifikation EV, Joint Conference with the British Classification Society (BCS) and the Dutch/Flemish Classification, p. 373. Springer, Heidelberg (2009)
A Multi-Layer ‘Gas of Circles’ Markov Random Field Model for the Extraction of Overlapping Near-Circular Objects Jozsef Nemeth1 , Zoltan Kato1 , and Ian Jermyn2 1 Image Processing and Computer Graphics Department University of Szeged, P.O. Box 652, 6701 Szeged, Hungary 2 Department of Mathematical Sciences, Durham University South Road, Durham DH1 3LE, United Kingdom
Abstract. We propose a multi-layer binary Markov random field (MRF) model that assigns high probability to object configurations in the image domain consisting of an unknown number of possibly touching or overlapping near-circular objects of approximately a given size. Each layer has an associated binary field that specifies a region corresponding to objects. Overlapping objects are represented by regions in different layers. Within each layer, long-range interactions favor connected components of approximately circular shape, while regions in different layers that overlap are penalized. Used as a prior coupled with a suitable data likelihood, the model can be used for object extraction from images, e.g. cells in biological images or densely-packed tree crowns in remote sensing images. We present a theoretical and experimental analysis of the model, and demonstrate its performance on various synthetic and biomedical images.
1 Introduction Object extraction remains one of the key problems of computer vision and image processing. The problem is easily stated: find the regions in the image domain occupied by a specified object or objects. The solution of this problem often requires high-level knowledge about the shape of the objects sought in order to deal with high noise, cluttered backgrounds, or occlusions [4,11,8,1]. As a result, most approaches to extraction have, to differing degrees and in different ways, incorporated prior knowledge about the shape of the objects sought. Early approaches were quite generic, essentially encouraging smoothness of object boundaries [6,9,3,2,10]. For example, [10] uses a Markovian smoothness prior (basically a Potts model, i.e. boundary length is penalized); [6] uses a line process to control the formation of region boundaries and control curvature; while classical active contour models [9] use boundary length and curvature, and region area in order to favor smooth closed curves [3,2].
This research was partially supported by the grant CNK80370 of the National Office for Research and Technology (NKTH) & Hungarian Scientific Research Fund (OTKA); by the European Union and co-financed by the European Regional Development Fund within the project TAMOP-4.2.1/B-09/1/KONV-2010-0005.
J. Blanc-Talon et al. (Eds.): ACIVS 2011, LNCS 6915, pp. 171–182, 2011. c Springer-Verlag Berlin Heidelberg 2011
172
J. Nemeth, Z. Kato, and I. Jermyn
Subsequently there has been a great deal of work on the inclusion of more specific prior shape knowledge in a variational [4,13] or probabilistic [5,15] framework. Many of these methods rely on a kind of template matching: shape variability is modeled as deformations of a reference shape or shapes. Although these methods are useful for many applications, the major drawback of using a reference shape (or shapes) is that handling an unknown number of instances of an object in the same image is difficult. An alternative approach, known as ‘higher-order active contours’ (HOACs), was presented and developed in [11,7,8]. HOAC models integrate shape knowledge without using reference shapes via the inclusion of explicit long-range dependencies between region boundary points. The lack of reference shapes means that they can be used to extract multiple instances of the same object. In [8], Horvath et al. showed how to set the parameters of the model introduced in [11] to favor regions consisting of any number of approximately circular connected components, each component having approximately the same, specified radius. This ‘gas of circles’ (GOC) model was successfully used for the extraction of tree crowns from aerial images. A subsequent reformulation of HOAC models (and active contour models in general) as equivalent phase field models [12,7] brings a number of theoretical and algorithmic advantages. One of the most important of these is that phase field models can be interpreted as real-valued Markov random fields (MRFs), thereby allowing the theoretical and algorithmic toolbox of random field theory to be brought to bear. In [1], this was carried out, and an MRF GOC model equivalent to the phase field GOC model was developed. For many important applications, for example the extraction of cells from light microscope images in biology, or the extraction of densely packed tree crowns in remote sensing images, these methods have limitations. The first is due to the representation: distinct overlapping objects cannot be represented. This is because the representation used is of a region, i.e. a subset of the image domain, and not of objects as such. Thus if the regions corresponding to two objects overlap, they form the single region that is their union. This cannot be distinguished from a single object occupying the same region. The second is due to the model: the same long-range interactions that favor nearcircular shapes also introduce a repulsive energy between nearby objects that means that configurations containing nearby objects have low probability, even if they do not overlap. In this paper, we propose a generalization of the MRF GOC model that overcomes these limitations: the multi-layer MRF GOC model. This consists of multiple copies of the MRF GOC model in [1], each copy being known as a layer. Now overlapping objects can be represented, as subsets of two different layers. The layers interact via a penalty for the overlap of regions in different layers, and this inter-layer interaction is crucial, particularly when a likelihood term is added. In its absence, the maximum probability configuration would simply be the same in all layers and equal to that found using the model in [1]. The result is that rather than the regions corresponding to two overlapping objects necessarily merging into a single region, it may be energetically favourable for the two regions corresponding to the two separate objects to appear in different layers. We begin by recalling the single-layer ‘gas of circles’ model.
A Multi-Layer ‘Gas of Circles’ Markov Random Field Model
173
2 The Single-Layer ‘Gas of Circles’ Model The ‘gas of circles’ model assigns high probability to regions in the image domain consisting of some number of approximately circular connected components, each of which has approximately the same, specified radius, and that are more than a certain distance apart. There are three equivalent formulations of the model: higher-order active contours (HOACs) [8], phase fields [7], and Markov random fields [1]. In the next three subsections, we explain the three formulations, since each provides some insight into the model, and the equivalences between them. 2.1 Contour Representation In the contour formulation, a region R is represented by its boundary ∂R, which is an equivalence class (under diffeomorphisms of their domain) of zero or more closed parameterized curves. The HOAC energy for the GOC model is [8]: βc E(γ) = λc L(γ) + αc A(γ) − n · n G(γ(t) − γ(t )) dt dt , (1) 2 where the contour γ of length L(γ) represents the boundary ∂R of extracted foreground regions with a total area A(γ). The last term of Eq. (1) is responsible for the geometry of extracted regions, where n, n corresponds to the normal vectors at t and t respectively, while G is the so called interaction function z π(z−d) 1 1 2 − − sin if z < 2d, d π d G(z) = 2 (2) 1 − H(z − d) otherwise. where d controls the range of interaction and H is the Heaviside step function. Horvath et al. showed in [8], that parameter triples (λc , αc , βc ) satisfying certain stability conditions will produce circular regions of a given radius r, yielding the first definition of the ’gas of circles’ HOAC model.
0.2 X 0
1
2
3
4
5
0
-0.2
-0.4
-0.6
G(z)
G(z) in 2D
G(x − x ) in 3D
Fig. 1. The interaction function G(z) for d = 2 and corresponding geometric kernel G
174
J. Nemeth, Z. Kato, and I. Jermyn
2.2 Phase Field Representation The phase field framework represents a region R by a function Φ φ : D → defined on the image domain D ⊂ 2 , and a threshold t: R = ζt (φ) = {x ∈ D : φ(x) ≥ t}. The phase field formulation E(φ) of the contour energy Eq. (1) was described in [12]: E(φ) = D
φ4 Df φ2 φ3 |∇φ|2 + λf − + αf φ − 2 4 2 3 βf − ∇φ · ∇ φ G(x − x ) . (3) 2 D×D
It is convenient to integrate the non-local term by parts: βf βf − ∇φ · ∇ φ G(x − x ) = φ φ ∇2 G(x − x )) . 2 D×D 2 D×D G(x−x )
The value φR that minimizes E(φ) for a fixed region R takes the values +1 inside R and −1 outside, away from the boundary ∂R, while changing smoothly from −1 to +1 in a narrow interface region around ∂R. Basically, the linear operator G directly acts on the phase field φ as a geometric kernel (see Fig. 1). In the ‘gas of circles’ model, the parameters of E(φ) are adjusted using the contour stability analysis and the equivalence between the formulations so that a circle of the desired radius is stable [7,8].
s
singleton
r
doubleton & long range
r
long range
Intra-layer interactions
Inter-layer interaction
Fig. 2. MRF neighbourhoods
2.3 Binary MRF Representation Discretizing the field energy Eq. (3) leads to a Markovian interpretation of the phase field model, where φ becomes a random field ω taking the discrete values of ±1 [1]. The resulting energy of the prior distribution P (ω) is given by U (ω) = α
s
ωs +
D
β
(ωs − ωs )2 + Fss ωs ωs , 2 s 2 s ∼s
(4)
s,s
where s denotes lattice sites (or pixels) of the discrete image domain S and ∼ is the nearest neighbour relation. The model parameters are related to the phase field model by 2α 0.82Df α = 3 f ; β = βf ; while D = incorporates the integral over pairs of boundary 4
A Multi-Layer ‘Gas of Circles’ Markov Random Field Model
175
lattice cells. Fss is a discrete approximation of G [1], which also determines the size of the neighborhood: {s ∈ S : |s − s | < 2d} as shown in Fig. 2. The singleton potential αωs of the prior energy corresponds to an area term: a lower α favors more foreground pixels and vice versa, while the doubleton potential D(ωs − ωs )2 acting over a nearest neighborhood of s ensures smoothness by penalizing boundary formation. Finally, the long-range potentials enforce the geometric constraints, thereby forming circles: −βFss if ωs = ωs , βFss ωs ωs = (5) +βFss otherwise. From Fig. 2, it is clear that long-range potentials favour the same label when |s − s | < d (attractive case) and different labels when d < |s − s | < 2d (repulsive case), where d d is the zero of G.
3 The Multi-Layer MRF ‘Gas of Circles’ Model We are now in a position to describe the multi-layer generalization of the MRF GOC model just described. The MRF GOC model has two limitations that render it inappropriate for many applications. First, touching or overlapping objects cannot be represented as separate entities in this model. This is because the representation used is of a region, not of objects as such. If the regions R1 and R2 corresponding to two objects overlap, the result is a single region R = R1 ∪ R2 that cannot be distinguished from the representation of a single object occupying the whole of R. Second, the model energy has a sometimes undesirable effect: it discourages connected components from being too close to one another. This is because the same interactions that favor stable circles also produce a repulsive interaction that raises the energy when two circles are closer than 2d. Thus while this model is able to separate, for example, tree crowns in regular plantations, it cannot represent, nor does it model well, configurations in which objects are touching or overlapping (cf. Fig. 8). The multi-layer MRF GOC model removes both these limitations by using multiple copies of the MRF GOC model, as follows. The domain of the binary random field becomes S˜ = × S, or alternatively, the field is a map from S to , where denotes either ∈ + or the set {1, . . . , }. Hence ω = {ω (i)} for i ∈ , where ω (i) : S → . In principle, we would like = + ,i.e. an infinite number of layers, as this would place no restrictions on the possible configurations. In practice, there is always a maximum number of mutual overlaps, and need be no larger than this. Sites that only differ in the value of i correspond to the same spatial point. Thus S˜ can be thought of as a series of layers, each of which is isomorphic to S, hence the name ‘multi-layer’. It is clear that the multi-layer field can represent overlapping objects, simply by placing the regions corresponding to them on different layers. ˜ of the multi-layer model is the sum of the MRF GOC energies The Gibbs energy U of each layer, plus an inter-layer interaction term that penalizes overlaps (see Fig. 2): ˜ (ω) = U
i=1
U (ω (i) ) +
κ
(1 + ωs(i) )(1 + ωs(j) ) , 4 s i=j
(6)
176
J. Nemeth, Z. Kato, and I. Jermyn
w>0 r
θ1
θ2
r
Fig. 3. Configurations of two overlapping circles and corresponding plots of E(M ) (r, w) and E(S) (r, w) vs. w for two circles of radius r = 10
where κ is a new parameter controlling the strength of the overlap penalty.1 Note that the inter-layer energy is ultralocal: only corresponding sites on different layers interact. Thus two regions in different layers experience no interaction at all unless they overlap. This eliminates the repulsive energy that exists in the single-layer model, because nearby but non-overlapping regions in different layers always have lower energy than the same regions in the same layer, assuming the intra-layer interactions are repulsive. 3.1 Energy of Two Interacting Circles In order better to understand the behaviour of the model, in this section we analyze the energy of two circles, on the same layer and on different layers. We consider the configurations shown in Fig. 3, where w stands for the size of the intersection: w < 0 means the circles do not intersect, while w > 0 represents a non-empty intersection of width w. We want to express the energy of these configurations as a function of w. We take advantage of the equivalence of the ‘gas of circles’ MRF and HOAC models to use the higher-order active contour energy Eq. (1) to compute the energy of the two circles. The parameters of this energy come from the equivalences between the three 4D formulations: βc = 4β; the unit weight of a boundary point is 0.82 ; while the difference in energy between an interior and exterior point is 2α. Thus the MRF energy of a single circle with radius r can be written as 2π 4D E(r) = 2πr + 2απr2 − 2β dθ dθ r2 cos(θ − θ ) G(γ(θ) − γ(θ )) , (7) 0.82 0 where γ is an embedding corresponding to the circle, parameterized, as shown in Fig. 3, by polar angle θ. Different layers: When the two circles are in different layers, the only interaction energy is the inter-layer overlap penalty. Thus the energy is constant until the circles start to overlap. It then starts to increase: 1
˜ is invariant to permutations of the layers. This will remain true even after we Notice that U add a likelihood energy. Thus all configurations, and in particular minimum energy configurations, are ! times degenerate. In practice, this degeneracy will be spontaneously broken by the optimization algorithm.
A Multi-Layer ‘Gas of Circles’ Markov Random Field Model
E(M ) (r, w) = 2E(r) + κA(r, w) , where A(r, w) is the area of the overlap given by ⎧ ⎨2 r2 arccos 1 − w − r − w 2rw − 2r 2 A(r, w) = ⎩ 0
177
(8)
w2 4
if w > 0,
(9)
otherwise.
Same layer: When the two circles are in the same layer, they interact if w > −2d for the particular form of interaction function in Eq. (2). (Note that we need only consider w ≤ 2r, where r is the radius of the circles, due to symmetry.) Thus if w ≤ −2d, the energy is simply 2E(r). For w > −2d, the energy increases with w until w ∼ = 0. As the circles start to overlap (and thus no longer form two circles, but a combined ‘dumbbell’ shape), there is effectively an attractive energy that causes an energy decrease with increasing w until the combined shape, and thus the energy, becomes that of a single circle (w = 2r). More precisely, the energy of two circles is 4D 2(2rπ − L(r, w)) + 2α(2r2 π − A(r, w)) 0.82 θf − 4β dθ1 dθ1 r2 cos(θ1 − θ1 ) G(γ1 (θ1 ) − γ1 (θ1 ))
E(S) (r, w) =
θs θf
− 2β
θs
dθ1 dθ2 r2 cos(θ1 − (π − θ2 )) G(Δ(θ1 , θ2 , w)) ,
(10)
where γ1,2 are two embeddings corresponding to the two circles, parameterized by angles θ1,2 respectively, as shown in Fig. 3. We have taken advantage of symmetry to write the second line in terms of γ1 only. L(r, w) is the arc length of the intersection segment, while Δ(θ1 , θ2 , w) = (r(sin(θ1 ) − sin(θ2 )))2 + (2r − w − r(cos(θ1 ) − cos(θ2 )))2 (11) is the distance between the points γ1 (θ1 ) and γ2 (θ2 ). The limits θs = cos−1 (min(1, 1−w )) and θf = 2π − θs are the radial angles of the two intersection points. 2d The righthand side of Fig. 3 shows plots of E(M ) (r, w) and E(S) (r, w) against w for circles with r = 10. When the overlap is greater than a certain threshold, controlled by κ, the energy of two circles in different layers becomes greater than two partially merged circles in one layer. Below this threshold, the two layer configuration has a lower energy. The stable configuration energy of two circles is given by the lower envelope of the curves in Fig. 3, and thus the repulsive energy that exists in the single-layer MRF GOC model is eliminated in the multi-layer MRF GOC model.
4 Experimental Results In this section, we report on the quantitative evaluation of the behavior and performance of the multi-layer MRF GOC model in object extraction problems involving
178
J. Nemeth, Z. Kato, and I. Jermyn
simulated data and microscope images. Results were obtained as MAP estimates, using the multi-layer MRF GOC model as a prior, combined with a likelihood energy UL ˜ (ω), to be described shortly: ω ˆ = arg maxω P (I|ω)P (ω) = arg minω UL (I, ω) + U where I : S → is the image data. Optimization was performed using Gibbs sampling coupled with simulated annealing [6]. The annealing schedule was exponential, with half-life at least 70 iterations, and a starting temperature of 3.0 for the parameter values used in the experiments. 4.1 Data Likelihood The data likelihood models the image in the interior and exterior regions using Gaussian distributions with constant means, and covariances equal to different multiples of the identity. In addition, we add an image gradient term connecting neighboring pixels, as follows. For each pair of neighboring sites, s and s , let (s, s ) be the unit vector pointing from s to s . Let sˆ = arg maxt∈{s,s } (|∇I(t)|). Let h(s, s ) = |(s, s ) · ∇I(ˆ s)|. Then define (i) (i) h(s, s ) ωs = ωs , gi (s, s ) = (12) |∇I(ˆ s)| − h(s, s ) otherwise. The likelihood energy then becomes 2
Is − μω(i) γ2
1/2 s UL (I, ω) = γ ln (2π) σω(i) + + gi (s, s ) , s 2σ 2 (i) 2 s s i s ∼s ωs (13) where γ and γ2 are positive weights. In practice, the parameters μ±1 and σ±1 of the Gaussian distributions were learned from representative samples.
=1
= 2, κ = 0.4
= 3, κ = 0.4
= 4, κ = 0.4
= 5, κ = 0.05
= 5, κ = 0.4
= 6, κ = 0.05
= 6, κ = 0.4
Fig. 4. Stable configurations of the multi-layer MRF GOC model for different numbers of layers and values of κ
A Multi-Layer ‘Gas of Circles’ Markov Random Field Model
179
Fig. 5. Plots of the relative interior area (left) and shape error (right) of the stable configurations against κ
4.2 Simulation Results with the Multi-Layer MRF GOC Model ˜ . Choosing, wlog, d = 10, In the first experiment, we study the global minima of U with the intra-layer parameters α = 0.18634, D = 0.15451, and β = 0.091137 set according to the stability constraints [8,1] and to ensure that stable circles have negative ˜ was then minimized for different numbers of layers and values of κ. Fig. 4 energy, U shows representative examples of these optimal configurations. The top-left result has = 1: note the spacing of the circles due to the intra-layer repulsive energy. When there are more layers, the intra-layer energies favour a similarly dense ‘gas of circles’ in each layer. For ≤ 3, every layer may contain such a configuration without the circles in different layers overlapping. For > 3, it is not possible to achieve both an optimal configuration in each layer and zero overlap energy. For small κ, the model tries to generate a dense configuration in each layer at the price of having overlaps. For large κ, the situation is the opposite: the model tries to avoid overlaps at the price of having 1 fewer circles in each layer. Fig. 5 shows a plot of the relative interior area N H(ω) against κ, where N = |S|. The value is almost constant for ≤ 3, while for > 3, the value decreases with κ. The circularity of the regions was also evaluated. The righthand plot in Fig. 5 shows the percentage of pixels outside the ideal desired circles. Although for > 3, these errors increase slightly, overall they remain low, meaning that the connected components remain circles to good accuracy for all and κ. 4.3 Quantitative Evaluation on Synthetic Images In this experiment, we demonstrate the efficiency of our model in separating overlapping circles. A series of noisy synthetic images were generated containing two circles of radius 10 with different degrees of overlap. The weights in the likelihood energy were set to γ = 0.1 and γ2 = 0, i.e. no gradient term was used. We used two layers and differing κ values in the range [0.01, 1]. Segmentation error was evaluated as the proportion of incorrectly segmented pixels. A plot of these errors versus the amount of overlap w and κ is shown in Fig. 6. Note that there is a rather clear drop in the segmentation error for κ ∼ = 0.7. When w > 10 (corresponding to an overlap of greater than 50%), a larger κ is required to get an accurate segmentation (κ = 0.88 was needed
180
J. Nemeth, Z. Kato, and I. Jermyn
Noisy image
Small κ
Best κ
Big κ
Fig. 6. Results on noisy synthetic images (SNR= 0dB) containing two circles of radius 10 with different degrees of overlap. Left: typical extraction results. Right: plot of segmentation error as a function of degree of overlap (w) and κ.
in the last case in Fig. 6), and for w > 15, it is hard to get good quality results. In summary, the model performs well for reasonable overlaps and it is not sensitive to the value of κ. On the other hand, there is a performance drop for very large overlaps. 4.4 Application in Biomedical Imaging Biomedical image segmentation aims to find the boundaries of various biological structures, e.g. cells, chromosomes, genes, proteins and other sub-cellular components in various image types [14]. Light microscope techniques are often used, but the resulting images are frequently noisy, blurred, and of low contrast, making accurate segmentation difficult. In many cases, the geometric structures involved are near-circular with many overlaps, so that our model seems well suited to extracting the desired structures. The extraction results shown in Fig. 7 and Fig. 8 demonstrate the effectiveness of the proposed multi-layer MRF GOC model for this type of task. Computation times vary from ∼ 20s to ∼ 1000s for images of size N = 104 . The key factor is the number of layers, with the minimum time corresponding to = 2, the maximum to = 6.
Fig. 7. Extraction of cells from light microscope images using the multi-layer MRF GOC model
A Multi-Layer ‘Gas of Circles’ Markov Random Field Model
181
Fig. 8. Extraction of lipid drops from light microscope images using the multi-layer MRF GOC model
5 Conclusion The multi-layer MRF GOC model enables the representation and modeling of object configurations consisting of an a priori unknown number of approximately circular objects of roughly the same size, which may touch or overlap. Such configurations occur in a number of domains, notably biomedicine and biology (e.g. cell images), and remote sensing (e.g. images of closely planted trees). Experiments show that the model behaves as expected on theoretical grounds, and that, when coupled with an appropriate likelihood model, can successfully extract such object configurations from synthetic and real images. The multi-layer model should also enable the extraction of several sets of approximately circular objects of different sizes, by setting the model parameters differently on different layers of the model.
References 1. Blaskovics, T., Kato, Z., Jermyn, I.: A Markov random field model for extracting nearcircular shapes. In: IEEE Proceedings of International Conference on Image Processing, pp. 1073–1076. IEEE, Cairo (2009) 2. Caselles, V., Kimmel, R., Sapiro, G.: Geodesic active contours. International Journal of Computer Vision 22(1), 61–79 (1997) 3. Cohen, L.: On active contour models and balloons. Computer Vision, Graphics and Image Processing: Image Understanding 53, 211–218 (1991) 4. Cremers, D., Tischhauser, F., Weickert, J., Schnorr, C.: Diffusion snakes: Introducing statistical shape knowledge into the Mumford-Shah functional. International Journal of Computer Vision 50(3), 295–313 (2002) 5. Flach, B., Schlesinger, D.: Combining shape priors and MRF-segmentation. In: da Vitoria Lobo, N., Kasparis, T., Roli, F., Kwok, J.T., Georgiopoulos, M., Anagnostopoulos, G.C., Loog, M. (eds.) S+SSPR 2008. LNCS, vol. 5342, pp. 177–186. Springer, Heidelberg (2008) 6. Geman, S., Geman, D.: Stochastic relaxation, Gibbs distributions and the Bayesian restoration of images. IEEE Transactions on Pattern Analysis and Machine Intelligence 6, 721–741 (1984) 7. Horvath, P., Jermyn, I.H.: A ‘gas of circles’ phase field model and its application to tree crown extraction. In: Proceedings of European Signal Processing Conference (EUSIPCO), Poznan, Poland (September 2007)
182
J. Nemeth, Z. Kato, and I. Jermyn
8. Horvath, P., Jermyn, I., Kato, Z., Zerubia, J.: A higher-order active contour model of a ‘gas of circles’ and its application to tree crown extraction. Pattern Recognition 42(5), 699–709 (2009) 9. Kass, M., Witkin, A., Terzopoulos, D.: Snakes: Active contour models. International Journal of Computer Vision 1(4), 321–331 (1988) 10. Kato, Z., Berthod, M., Zerubia, J.: A hierarchical Markov random field model and multitemperature annealing for parallel image classification. Computer Vision, Graphics and Image Processing: Graphical Models and Image Processing 58(1), 18–37 (1996) 11. Rochery, M., Jermyn, I.H., Zerubia, J.: Higher order active contours. International Journal of Computer Vision 69(1), 27–42 (2006), http://dx.doi.org/10.1007/ s11263-006-6851-y 12. Rochery, M., Jermyn, I.H., Zerubia, J.: Phase field models and higher-order active contours. In: Proc. IEEE International Conference on Computer Vision (ICCV), Beijing, China (October 2005) 13. Rousson, M., Paragios, N.: Shape priors for level set representations. In: Heyden, A., Sparr, G., Nielsen, M., Johansen, P. (eds.) ECCV 2002. LNCS, vol. 2351, pp. 78–92. Springer, Heidelberg (2002) 14. Russell, C., Metaxas, D., Restif, C., Torr, P.: Using the Pn Potts model with learning methods to segment live cell images. In: IEEE 11th International Conference on Computer Vision, pp. 1–8. IEEE, Los Alamitos (2007) 15. Srivastava, A., Joshi, S., Mio, W., Liu, X.: Statistical shape analysis: Clustering, learning, and testing. IEEE Trans. Pattern Analysis and Machine Intelligence 27(4), 590–602 (2005)
Evaluation of Image Segmentation Algorithms from the Perspective of Salient Region Detection Bogdan Popescu, Andreea Iancu, Dumitru Dan Burdescu, Marius Brezovan, and Eugen Ganea University of Craiova, Software Engineering Department, Craiova, Bd. Decebal 107, Romania {bogdan.popescu,andreea.iancu}@itsix.com, {dumitru burdescu,brezovan marius,ganea eugen}@software.ucv.ro
Abstract. The present paper addresses the problem of image segmentation evaluation by comparing seven different approaches.We are presenting a new method of salient object detection with very good results relative to other already known object detection methods. We developed a simple evaluation framework in order to compare the results of our method with other segmentation methods. The results of our experimental work offer good perspectives for our algorithm, in terms of efficiency and precision. Keywords: color segmentation; graph-based segmentation; salient region detection.
1
Introduction
Image segmentation is a very important operation performed on acquired images. The evaluation of this process [5] focuses on two main topics: generality and objectivity. Generality means that the test images in the benchmark should have a large variety so that the evaluation results can be extended to other images and applications. Objectivity means that all the test images in the benchmark should have an unambiguous ground-truth segmentation so that the evaluation can be conducted objectively. The main target of image segmentation process is the domain-independent partition of the image into a set of regions which are visually distinct and uniform with respect to some property, such as grey level, texture or color. The problem of segmentation is an important research field and many segmentation methods have been proposed in the literature so far ([1],[6],[7],[10], [2], [3], [4]). The objective of this paper is to underline the very good results of image segmentation obtained by our segmentation technique, Graph-Based Salient Object Detection, and to compare them with other existing methods. The other methods that we use for comparison are: Efficient Graph-Based Image Segmentation (Local Variation), Normalized Cuts, Unsupervised Segmentation of Natural Images J. Blanc-Talon et al. (Eds.): ACIVS 2011, LNCS 6915, pp. 183–194, 2011. c Springer-Verlag Berlin Heidelberg 2011
184
B. Popescu et al.
via Lossy Data Compression, ROI-SEG: Unsupervised Color Segmentation by Combining Differently Focused Sub-Results, Mean Shift and Multi-Layer Spectral Segmentation. They are all complex and well known algorithms, with very good results and proved efficiency in this area. The experiments were completed using the images and ground-truth segmentations of M SRA Salient Object Database [17]. The paper is organized as follows: in Section 3 we briefly present previous studies in the domain of image segmentation and the segmentation method we propose. The methodology of performance evaluation is presented in Section 4. The experimental results are presented in Section 5. Section 6 concludes the paper and outlines the main directions of the future work.
2
Related Work
Image segmentation evaluation is an open subject in today’s image processing field. The goal of existing studies is to establish the accuracy of each individual approach and find new improvement methods. Some of previous works in this domain do not require ground-truth image segmentation as the reference. In these methods, the segmentation performance is usually measured by some contextual and perceptual properties, such as the homogeneity within the resulting segments and the inhomogeneity across neighboring segments. Most of segmentation methods require ground-truth image segmentations as reference. Since the construction of the ground-truth segmentation for many real images is labor-intensive and sometimes not well or uniquely defined, most prior image segmentation methods are only tested on some special classes of images used in special applications where the ground-truth segmentations are uniquely defined, synthetic images where ground-truth segmentation is also well defined, and/or a small set of real images. The main drawback of providing such reference is represented by the resources that are needed. However, after analyzing the differences between the image under study and the ground-truth segmentation, a performance proof is obtained. Region - based segmentation methods can be broadly classified as either model-based [14] or visual feature - based [15] approaches. A distinct category of region-based segmentation methods that is relevant to our approach is represented by graph-based segmentation methods. Most graph-based segmentation methods attempt to search a certain structure in the associated edge weighted graph constructed on the image pixels, such as minimum spanning tree [6], or minimum cut [16]. Related work on quantitative segmentation evaluation includes both standalone evaluation methods, which do not make use of a reference segmentation, and relative evaluation methods employing ground truth. For standalone evaluation of image segmentations metrics for intra-object homogeneity and inter-object disparity have been proposed in [24]. A combined
Salient Region Detection
185
segmentation and evaluation scheme is developed in [25]. Although standalone evaluation methods can be very useful in such applications, their results do not necessarily coincide with the human perception of the goodness of segmentation. However, when a reference mask is available or can be generated, relative evaluation methods are preferred in order the segmentation results to coincide with the human perception of the goodness of segmentation. Martin et al. [26] propose two metrics that can be used to evaluate the consistency of a pair of segmentations based on human segmentations. The measures are designed to be tolerant to refinement. A methodology for evaluating the performance of boundary detection techniques with a database containing groundtruth data was developed in [27]. It is based in the comparison of the detected boundaries with respect to human-marked boundaries using the Precision-Recall framework. Another evaluation are based on pixel discrepancy measures. In [28], the use of binary edge masks and scalable discrepancy measures is proposed. Odet et al. [29] propose also an evaluation method based on edge pixel discrepancy, but the establishment of a correspondence of regions between the reference mask and the examined one is considered. This paper presents a benchmark for evaluating general-purpose image segmentation methods on a large variety of real images with well defined objects as the ground truth. It is a continued study of our previous work with a more thorough investigation of performance variation resulting from different parameter settings in 7 well-known, region-based segmentation methods.
3
Segmentation Methods
We will compare seven different segmentation techniques: Unsupervised Segmentation of Natural Images via Lossy Data Compression [4], ROI-SEG: Unsupervised Color Segmentation by Combining Differently Focused Sub Results [2], The Mean Shift-Based segmentation algorithm [7], Multi-Layer Spectral Segmentation [3], Efficient Graph-Based segmentation algorithm [6], Normalized Cuts segmentation algorithm [10] and our own contour-based segmentation method [13]. We have chosen Mean Shift-Based segmentation because it is generally effective and has become widely used in the vision community. The Efficient Graph-Based segmentation algorithm was chosen as an interesting comparison to the Mean Shift. Its general approach is similar. However, it excludes the mean shift filtering step itself, thus partially addressing the question of whether the filtering step is useful. Due to its computational efficiency, Normalized Cuts represents a solid reference in our study. The other three methods have been chosen because they combine in a very good manner efficient segmentation algorithms like Mean Shift or N-Cuts. We have used M SRA Salient Object Database as reference for our research. From the many images the database provides, we have chosen a number of 100 representative items to be our experimental basis. We tried an alternative methodology to the well known Berkley segmentation dataset [11].
186
B. Popescu et al.
Our experimental results have shown that [2] offers the most similar approach to the one we are presenting. Our contribution is based on two main aspects: (a) in order to minimize the running time we construct a hexagonal structure based on the image pixels, that is used in both color-based and syntactic-based segmentation algorithms, and (b) we propose an efficient method for segmentation of color images based on spanning trees and both color and syntactic features of regions. 3.1
Graph-Based Salient Object Detection
We introduce an efficient segmentation method that uses color and some geometric features of an image to process it and create a reliable result [13]. The color space we used is RGB because of the color consistency and its computational efficiency. What is particular at this approach is the basic usage of hexagonal structure instead of color pixels. In this way we can represent the structure as a grid-graph G = (V, E) where each hexagon h in the structure has a corresponding vertex v ∈ V , as presented in Figure 1. Every hexagon has six neighbors and each neighborhood connection is represented by an edge in the set E of the graph. For each hexagon on the structure two important attributes are associated: the dominant color and the coordinates of the gravity center. Basically, each hexagonal cell contains eight pixels: six from the frontier and two from the middle. Image segmentation is realized in two distinct steps. The first step represents a pre-segmentation step when only color information is used to determine an initial segmentation. The second step represents a syntactic-based segmentation step when both color and geometric properties of regions are used. The first step of the segmentation algorithm uses a color-based region model and will produce a forest of maximum spanning trees based on a modified form of the Kruskal’s algorithm. In this case the evidence for a boundary between two adjacent regions is based on the difference between the internal contrast and the external contrast between the regions. The color-based segmentation algorithm builds a maximal spanning tree for each salient region of the input image.
Fig. 1. The grid-graph constructed on the hexagonal structure of an image
Salient Region Detection
187
The second step of the segmentation algorithm uses a new graph, which has a vertex for each connected component determined by the color-based segmentation algorithm. In this case the region model contains in addition some geometric properties of regions such as the area of the region and the region boundary. The final segmentation step produces a forest of minimum spanning trees based on a modified form of the Bor˚ uvka’s algorithm. Each determined minimum spanning tree represents a final salient region returned by the segmentation algorithm. 3.2
Unsupervised Segmentation of Natural Images via Lossy Data Compression
The clustering method follows the principle of lossy minimum description length (LM DL) [18] : Principle (Data Segmentation via Lossy Compression). The optimal segmentation minimizes the number of bits needed to code the segmented data, subject to a given distortion. The lossy compression-based method is applied to segmenting natural images. First, a low-level segmentation is applied to partition an image into many small homogeneous patches. The superpixel notion is introduced. The superpixels are used to initialize the mid-level texture-based segmentation, which minimizes the total coding length of all the texture features by repeatedly merging adjacent segments, subject to a distortion. The method studies several simple heuristics for choosing a good ε for each image. 3.3
ROI-SEG: Unsupervised Color Segmentation by Combining Differently Focused Sub Results
This algorithm is used for detection of a set of connected segments in a color image, based on a previously defined region-of-interest (ROI). The detected segments all have approximately the same color distribution as the input ROI. The sub-segmentation algorithm, which gets an arbitrarily shaped ROI and a color image as input, can be roughly divided into four subsequent steps. First, the image is converted into the CIE Luv color space in order to have an isotropic feature space. Second, the color distribution of the ROI is modeled by a Gaussian Mixture Model (GMM), which is initialized by the results of a Mean Shift algorithm. Third, all color pixels are ordered by calculating Bhattacharyya distance values for each pixel. This is done by an efficient integral image based approach. The ordered pixel values are passed to a modified version of the Maximally Stable Extremal Region (MSER) detector to compute the final result - a set of connected regions, which approximately have the same color appearance as the input ROI. 3.4
Multi-Layer Spectral Segmentation
This algorithm introduces an affinity model for image segmentation that uses the relevance scores, learnt from the test image by semi-supervised learning [19], [20], [21], as graph affinities. The first step is to construct a multi-layer graph with
188
B. Popescu et al.
pixels and regions generated by the mean shift algorithm [7]. A semi-supervised strategy is applied on the affinities in order to efficiently estimate them. In a single multi-layer framework of Normalized Cuts, the proposed full affinities are used to simultaneously cluster all pixel and region nodes into visually coherent groups across all layers. The algorithm offers high-quality segmentation results by considering all intra- and inter-layer affinities in the spectral framework. The computation is very efficient by the eigen-decomposition of a sparse matrix. This solution produces much better segmentations with object details than other spectral segmentation methods such as Normalized Cuts (NCut) [10] on natural images. 3.5
Efficient Graph-Based Image Segmentation
Efficient Graph-Based image segmentation [6], is an efficient method of performing image segmentation. The basic principle is to directly process the data points of the image, using a variation of single linkage clustering without any additional filtering. A minimum spanning tree of the data points is used to perform traditional single linkage clustering from which any edges with length greater than a given threshold are removed [9]. Let G = (V, E) be a fully connected graph, with m edges {ei } and n vertices. Each vertex is a pixel, x, represented in the feature space. The final segmentation will be S = (C1 , ..., Cr ), where Ci is a cluster of data points. The algorithm [6] can be shortly presented as follows: 1. Sort E = (e1 , ..., em ) such that |et | ≤ |et |∀t < t 2. Let S 0 =({x1 }, ..., {xn }) in other words each initial cluster contains exactly one vertex. 3. For t = 1, ..., m (a) Let xi and xj be the vertices connected by et . (b) Let Cxt−1 be the connected component containing point xi on iteration i t−1 and li = maxmst Cxt−1 be the longest edge in the minimum spanning i tree of Cxt−1 . Likewise for lj . i (c) Merge Cxt−1 and Cxt−1 if: i j k k |et | < min li + t−1 , lj + t−1 (1) Cxi Cxj 4. S = S m . 3.6
Normalized Cuts
Normalized Cuts method models an image using a graph G = (V, E), where V is a set of vertices corresponding to image pixels and E is a set of edges connecting neighboring pixels. The edge weight w(u, v) describes the affinity between two vertices u and v based on different metrics like proximity and intensity similarity. The algorithm segments an image into two segments that correspond to a graph cut (A, B), where A and B are the vertices in the two resulting subgraphs.
Salient Region Detection
189
The segmentation cost is defined by: N cut(A, B) =
cut(A, B) cut(A, B) + , assoc(A, V ) assoc(B, V )
(2)
where cut(A, B) = u∈A,v∈B w(u, v) is the cut cost of (A, B) and assoc(A, V ) = u∈A,v∈V w(u, v) is the association between A and V . The algorithm finds a graph cut (A, B) with a minimum cost in Eq.(1). Since this is a NP-complete problem, a spectral graph algorithm was developed to find an approximate solution [10]. This algorithm can be recursively applied on the resulting subgraphs to get more segments. For this method, the most important parameter is the number of regions to be segmented. Normalized Cuts is an unbiased measure of dissociation between the subgraphs, and it has the property that minimizing normalized cuts leads directly to maximizing the normalized association relative to the total association within the sub-groups. 3.7
Mean Shift
The Mean Shift-Based segmentation technique [7] is one of many techniques dealing with “feature space analysis”. Advantages of feature-space methods are the global representation of the original data and the excellent tolerance to noise [12]. The algorithm has two important steps: a mean shift filtering of the image data in feature space and a clustering process of the data points already filtered. During the filtering step, segments are processed using the kernel density estimation of the gradient. Details can be found in[7]. A uniform kernel for gradient estimation with radius vector h = [hs , hs , hr , hr , hr ] is used. hs is the radius of the spatial dimensions and hr the radius of the color dimensions. Combining these two parameters, complex analysis can be performed while training on different subjects. Mean shift filtering is only a preprocessing step. Another step is required in the segmentation process: clustering of the filtered data points {x }. During filtering, each data point in the feature space is replaced by its corresponding mode. This suggests a single linkage clustering that converts the filtered points into a segmentation. Another paper that describes the clustering is [8]. A region adjacency graph (RAG) is created to hierarchically cluster the modes. Also, edge information from an edge detector is combined with the color information to better guide the clustering. This is the method used in the available EDISON system, also described in [8]. The EDISON system is the implementation we use in our evaluation system.
4
Saliency Detection Performance Evaluation
We present comparative results of segmentation performance for our contour based segmentation method and the six alternative segmentation methods
190
B. Popescu et al.
mentioned above. Our evaluation measure is based on the performance measure from [22] that we have modified and adapted to fit our needs and better illustrate the results. The image is partitioned into two segments, with one as the foreground and the other as the background. Of course, the segmentation methods produce more than two regions. All the methods partition an image into a set of disjoint segments without labeling the foreground and background. We have used a regionmerging strategy so that they can be fairly evaluated in the benchmark. If the segments in the image I are {R1 , R2 , ..., Rn }, the ground-truth foreground segment corresponds to a subset of the disjoint segments. To evaluate these methods we apply a strategy to merge the segments and then use the merged region as the detected foreground object. For each segment Ri in an image, we count it into the foreground R if it has more than 50% overlap with the ground-truth foreground A in terms of the area. The basic performance measure we use for this analysis is the Jaccard coefficient [23] which measures the region coincidence between the segmentation result and the ground truth. Specifically, let the region A be the ground-truth foreground object and the region R be the merged segments derived from the segmentation result using the region-merging strategy. The segmentation accuracy is: P (R; A) =
|R ∩ A| |R ∩ A| = , |R ∪ A| |R| + |A| − |R ∩ A|
(3)
where |X| represents the cardinal of the set X, which estimates the area of the region X. This measure has no bias to the segmentation that produces overly large or small number of segments. In this equation, R ∩ A measures how much the ground-truth object is detected and R ∪ A is a normalization factor that normalizes the accuracy measure to the range of [0, 1]. Based on the normalization factor, the accuracy measure improves the detection of irrelevant regions as the foreground segments. This region-based measure is insensitive to small variations in the ground-truth construction and incorporates the accuracy and recall measurement into one unified function. We have adapted this metric by dividing the result by the square root of the number of objects in the current segmentation. A P P (R; A) = P (R; A) , (4) R where S represents the number of detected regions of the segmentation S. We use the value P P (R; A) for comparisons because the current segmentation needs to have almost the same number of segments as the ground-truth. We used the MSRA Salient Object Database to perform our analysis and evaluation. This is a public dataset that aims to facilitate research in multimedia information retrieval and related areas. The images in the dataset are collected from a commercial search engine with more than 1000 queries. It contains about 1 million images and 20,000 videos. The surrounding texts that are obtained from
Salient Region Detection
191
Fig. 2. Comparative analysis for: Graph-Based Salient Object Detection - GBSOD, Efficient Graph-Based Image Segmentation (Local Variation) - EGB, Mean Shift - MS, Normalized Cuts - NC, Unsupervised Segmentation of Natural Images via Lossy Data Compression - YA, ROI-SEG: Unsupervised Color Segmentation by Combining Differently Focused Sub-Results - DO, Multi-Layer Spectral Segmentation - KI
more than 1 million Web pages are also provided. The images and videos have been comprehensively annotated, including their relevance levels to corresponding queries, semantic concepts of images, and category and quality information of videos. Six standard tasks are defined on this dataset: (1) image search reranking; (2) image annotation; (3) query-by-example image search; (4) video search reranking; (5) video categorization; and (6) video quality assessment. The segmentation accuracy mentioned above only provides an upper bound of the segmentation performance by assuming an ideal postprocess of region merging for applications without a priori known exact ground truth. For the extreme case where each pixel is partitioned as a segment, the upper-bound performance obtained is a meaningless value of 100%. This is a little similar to the GCE and LCE measures developed in the Berkeley benchmark. But the difference is that GCE and LCE also result in meaningless high accuracy when too few segments are produced, such as the case where the whole image is partitioned as a single segment. In this paper, we always set the segmentation parameters to produce a reasonably small number of segments when applying the strategy to merge the image regions.
5
Experimental Results
Our study of segmentation quality is based on experimental results and uses the MSRA Salient Object Database provided at [17]. In order to obtain the evaluation results, we have performed segmentation of the selected images with all the algorithms using a various set of parameters.
192
B. Popescu et al.
More precisely, by varying some parameters, we have obtained 10 distinct points that define the curve for each approach. For Normalized Cuts [10] we have modified the number of segments in the range of {5, 10, 12, 15, 20, 25, 30, 40, 50, 70}. The variable parameter for Efficient Graph - Based Image Segmentation [6] was the scale of observation, k, in range {100, 200, 300, 400, 500, 600, 700, 800, 900, 1000}. For Mean-Shift [7] we have made 10 combinations from Spatial Bandwidth {8, 16} and Range Bandwidth {4, 7, 10, 13, 16}. For our customized performance measure we have obtained the results illustrated in Figure 2. From the above presented diagram we can see that the PP(R;A) metric for our proposed method, denoted GBSOD - Graph Based Salient Object Detection is situated above the other graphics indicating a better performance result and a balanced algorithm.
6
Conclusion and Future Work
We described in this paper a new graph-method for image segmentation and extraction of visual objects. Starting from a survey of several segmentation strategies, we have performed an image segmentation evaluation experiment. Our segmentation method and other six segmentation methodologies were chosen for the experiment, and the complementary nature of the methods was demonstrated in the results. The study results offer a clear view of the effectiveness of each segmentation algorithm, trying in this way to offer a solid reference for future studies. Future work will be carried out in the direction of integrating syntactic visual information into a semantic level of a semantic image processing and indexing system.
References 1. Fu, K., Mui, J.: A survey on image segmentation. Pattern Recognition (1981) 2. Donoser, M., Bischof, H.: ROI-SEG: Unsupervised Color Segmentation by Combining Differently Focused Sub Results. Institute for Computer Graphics and Vision Graz University of Technology 3. Kim, T.H., Lee, K.M., Lee, S.U.: Learning Full Pairwise Affinities for Spectral Segmentation. Dept. of EECS, ASRI, Seoul National University, 151-742, Seoul, Korea 4. Yang, A.Y., Wright, J., Sastry, S., Ma, Y.: Unsupervised Segmentation of Natural Images via Lossy Data Compression (2007) (preprint submitted to Elsevier) 5. Martin, D., Fowlkes, C., Tal, D., Malik, J.: A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: Proc. Int. Conf. Comp. Vis., vol. 2, pp. 416–425 (2001) 6. Felzenszwalb, P., Huttenlocher, D.: Efficient Graph-Based Image Segmentation. Intl. J. Computer Vision 59(2) (2004)
Salient Region Detection
193
7. Comaniciu, D., Meer, P.: Mean Shift: A Robust Approach toward Feature Space Analysis. IEEE Trans. Pattern Analysis and Machine Intelligence 24, 603–619 (2002) 8. Christoudias, C., Georgescu, B., Meer, P.: Synergism in Low Level Vision. In: Proc. Intl. Conf. Pattern Recognition, vol. 4, pp. 150–156 (2002) 9. Unnikrishnan, R., Pantofaru, C., Hebert, M.: Toward Objective Evaluation of Image Segmentation Algorithms. IEEE Transactions on Pattern Analysis and Machine Inteligence 29(6) (2007) 10. Shi, J., Malik, J.: Normalized Cuts and Image Segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence 22(8) (2000) 11. Berkeley Segmentation and Boundary Detection Benchmark and Dataset (2003), http://www.cs.berkeley.edu/projects/vision/grouping/segbench 12. Duda, R.O., Hart, P.E., Stork, D.G.: Pattern Classification. John Wiley & Sons, New York (2000) 13. Brezovan, M., Burdescu, D.D., Ganea, E., Stanescu, L.: An Adaptive Method for Efficient Detection of Salient Visual Object from Color Images. In: Proc. of the 20th International Conference on Pattern Recognition (ICPR), Istambul, pp. 2346–2349 (2010) 14. Carson, C., Belongie, S., Greenspan, H., Malik, J.: Blobworld: Image segmentation using expectation-maximization and its application to image querying and classification. IEEE Trans. on Pattern Analysis and Machine Intelligence 24(8), 1026–1037 (2002) 15. Fauqueur, J., Boujemaa, N.: Region-based image retrieval: Fast coarse segmentation and fine color description. Journal of Visual Languages and Computing 15(1), 69–95 (2004) 16. Shi, J., Malik, J.: Normalized cuts and image segmentation. In: Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, San Juan, Puerto Rico, pp. 731–737 (1997) 17. MSRA Salient Object Database, http://research.microsoft.com/en-us/um/ people/jiansun/SalientObject/salient_object.htm 18. Dowson, D., Landau, B.: The Frechet distance between multivariate normal distributions. Journal Multivariate Analysis 12(3), 450–455 (1982) 19. Brin, S., Page., L.: The anatomy of a large-scale hypertextual web search engine. In: WWW (1998) 20. Zhou, D., Bousquet, O., Lal, T.N., Weston, J., Scholkopf, B.: Learning with local and global consistency. In: NIPS (2003) 21. Pan, J.-Y., Yang, H.-J., Faloutsos, C., Duygulu, P.: Automatic multimedia crossmodal correlation discovery. In: KDD (2004) 22. Ge, F., Wang, S., Liu, T.: New benchmark for image segmentation evaluation. Journal of Electronic Imaging 16(3), 033011 (JulSep 2007) 23. Cox, T., Cox, M.: Multidimensional Scaling. 2nd edn. Chapman and Hall/CRC Press, Boca Raton, FL (2000) 24. Levine, M., Nazif, A.: Dynamic measurement of computer generated image segmentations. IEEE Trans. on Pattern Analysis and Machine Intelligence 7, 155–164 (1985) 25. Zhang, Y., Wardi, Y.: A recursive segmentation and classification scheme for improving segmentation accuracy and detection rate in realtime machine vision applications. In: Proc. of the Int. Conf. on Digital Signal Processing (DSP 2002), vol. 2 (July, 2002)
194
B. Popescu et al.
26. Martin, D., Fowlkes, C., Tal, D., Malik, J.: A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: Proc. of the IEEE Conference on Computer Vision, pp. 414–425 (2001) 27. Martin, D., Fowlkes, C., Malik, J.: Learning to detect natural image boundaries using local brightness, color and texture cues. IEEE Trans. on Pattern Analysis and Machine Intelligence 26(5), 530–549 (2004) 28. Huang, Q., Dom, B.: Quantitative methods of evaluating image segmentation. In: Proc. of the Int. Conf. on Image Processing (ICIP 1995), vol. 3, pp. 53–56 (1995) 29. Odet, C., Belaroussi, B., Benoit-Cattin, H.: Scalable discrepancy measures for segmentation evaluation. In: Proc. of the Int. Conf. on Image Processing (ICIP 2002), vol. 1, pp. 785–788 (2002)
Robust Active Contour Segmentation with an Efficient Global Optimizer Jonas De Vylder, Jan Aelterman, and Wilfried Philips Department of Telecommunications and Information Processing, IBBT - Image Processing and Interpretation, Ghent University, St-Pietersnieuwstraat 41, B-9000 Ghent, Belgium [email protected] http://telin.ugent.be/~ jdvylder/
Abstract. Active contours or snakes are widely used for segmentation and tracking. Recently a new active contour model was proposed, combining edge and region information. The method has a convex energy function, thus becoming invariant to the initialization of the active contour. This method is promising, but has no regularization term. Therefore segmentation results of this method are highly dependent of the quality of the images. We propose a new active contour model which also uses region and edge information, but which has an extra regularization term. This work provides an efficient optimization scheme based on Split Bregman for the proposed active contour method. It is experimentally shown that the proposed method has significant better results in the presence of noise and clutter. Keywords: Active contours, segmentation, convex optimization, Split Bregman.
1
Introduction
Since Kass et al. [1] introduced there snakes, the active contour framework has become a constant recurring topic in segmentation literature. The framework allows easy tuning to specific segmentation and tracking problems. Prior motion information of objects which need to be tracked [2–4], specific shape models [2, 5, 6], region statistics of objects [7, 8], etc. These are just a small notion of different forms of prior knowledge which have been incorporated in the active contour framework. In the active contour framework, an initial contour is moved and deformed in order to minimize a specific energy function. This energy function should be minimal when the contour is delineating the object of interest, e.g. a leaf. Two main groups can be distinguished in the active contour framework: one group representing the active contour explicitly as a parameterized curve and a second group which represents the contour implicitly using level sets. In the first group, also called snakes, the contour generally converges towards edges in the image [1, 5, 9]. The second group generally has an energy function based on region J. Blanc-Talon et al. (Eds.): ACIVS 2011, LNCS 6915, pp. 195–206, 2011. c Springer-Verlag Berlin Heidelberg 2011
196
J. De Vylder, J. Aelterman, and W. Philips
properties, such as variance of intensity of the enclosed segment [7, 10]. These level set approaches has gained a lot of interest since they have some benefits over snakes. For example, they can easily change their topology, e.g. splitting a segment into multiple unconnected segments. Recently an active contour model has been proposed with a convex energy function, making it possible to define fast global optimizers [11, 12]. These global active contours have the benefit that there result is no longer dependent on the initialization. In [13], Bresson et al. proposed a new type of active contour with a convex energy function, a model which combined edge information and region information. This method combines the original snake model [1] with the active contour model without edges [7]. The proposed model has an energy function which is completely defined by the image, thus eliminating the possibility of regularization. Although the method has some interesting benefits, it lacks robustness to noise and clutter. To tackle this problem, we propose a new active contour model which has the benefits of the model proposed in [13], but which has an extra regularization term. This regularization term enforces smoothness of the boundaries of the segments. This results in segments with a smooth boundary, i.e. avoiding jaggy edges due to noise. This paper is arranged as follows. The next section briefly enumerates the notations and symbols used in this paper. In section 3 the current state of the art of global optimum active contours is summarized and expanded with the proposed method. Section 4 elaborates on a fast optimization method which can be used to calculate the proposed active contour. The next section shows some examples and quantitative results of our technique in comparison to other active contour formulations. Both convergence speed and segmentation result are examined. Section 6 recapitulates and concludes.
2
Notations and Definitions
In the remaining of this paper we will use specific notations some conventional, some more peculiar to this work. Therefore we briefly summarize the notations and symbols used in this work. We will refer to an image, F in its vector notation, i.e. f (i ∗ m + j) = F (i, j), where m × n is the dimension of the image. In a similar way we will represent the contour in vector format, u. If a pixel U (i, j) is part of the segment, it will have a value above a certain threshold, all background pixels will have a value lower than the given threshold. Note that this is similar to level-sets. The way these contours are optimized however is different than with classical level-set active contours, as is explained in the next section. We will use image operators, i.e. gradient, divergence and Laplacian in combination with this vector notation, however the semantics of the image operators remains the same as if it was used with the classical matrix notation: ∇(f (i ∗ m + j)) = F (i + 1, J) − F (i, j), F (i, J + 1) − F (i, j) ∇ · (f (i ∗ m + j)) = F (i + 1, J) − F (i, j) + F (i, J + 1) − F (i, j) ∇2 (f (i ∗ m + j)) = ∇ · ∇(f (i ∗ m + j))
Robust Active Contour Segmentation
197
Further we will use the following inner product and norm notations: f , g = f α,g =
mn
f (i), g(i)
i=1 mn
α α1 g(i) | f (i) |
i=1
If the weights g(i) = 1 for all i, then we will omit g, since we assume this will not cause confusion, but will increase readability.
3
Convex Energy Active Contours
In [11] an active contour model was proposed which has global minimizers. This active contour is calculated by minimizing the following convex energy: E[u] = ∇u1 + μu, r
(1)
r = (mf − f )2 − (mb − f )2
(2)
with Here f represents the intensity values in the image, mf and mb are respectively the mean intensity of the segment and the mean intensity of the background, i.e. every pixel not belonging to the segment. Note that this energy is convex, only if mf and mb are constant. This problem can be solved by iterating between the following two steps: first fix mf and mb and minimize eq. (1), secondly update mf and mb. Chan et al. found that the steady state of the gradient flow corresponding to this energy, i.e. du ∇u =∇· − μr dt |∇u|
(3)
coincides with the steady state of the gradient flow of the original Chan-Vese active contours [7, 11]. So minimizing eq. (1) is equivalent to finding an optimal contour which optimizes the original Chan-Vese energy function. Although the energy in eq. 1 does not have a unique global minimizer, a well defined minimizer can be found within the interval [0, 1]n : u∗ = arg min ∇u1 + μu, r
(4)
u∈[0,1]n
Note that this results in a minimizer which values are between 0 and 1. It is however desirable to have a segmentation result where the values of a minimizer are constrained to (0, 1), i.e. a pixel belongs to a segment or not. Therefore u∗ is tresholded, i.e. 1 if u∗ (x) > α ∗ Φα (u (x)) = (5) 0 otherwise
198
J. De Vylder, J. Aelterman, and W. Philips
with a predefined α ∈ [0, 1]. In [14] it is shown that Φα (u∗ ) is a global minimizer for the energy in eq. (1) and by extension for the energy function of the original Chan-Vese active contour model. In [13] the convex energy function in eq. (1) was generalized in order to incorporate edge information: E[u] = ∇u1,g + μu, r
(6)
1 where g is the result of an edge detector, e.g. g = 1+|∇f | . The active contour minimizing this energy function can be seen as a combination of edge based snake active contours [1] and the region based Chan-Vese active contours [7]. Since this method only minimizes energy terms based on the image, it is highly influenced by the quality of the image. In the presence of noise and clutter the method will find false segments or distorted segment boundaries. In order to make the method more robust we propose to extend the energy function in eq. (6) with an extra regularization term:
E[u] = ∇u1 + γ∇u1,g + μu, r
(7)
Where γ is a weighting parameter defining the influence of the extra regularization therm. This regularization term approximates the length of the segments boundary, thus penalizing small false segments and high curved boundaries due to noise.
4
Optimization
Due to the convexity of the energy function in eq. (7), a wide range of minimizers can be used to find an optimal contour u† . The Split Bregman method is an efficient optimization technique for solving L1-regularized problems and has good convergence properties. In order to find a contour which minimizes eq. (7), the Split Bregman method will ”de-couple” the L1 and L2 norm, by introducing a new variable d and by putting constraints on the problem. This results in the following optimization problem: (u† , d† ) = arg min d1 + γd1,g + μu, r such that d = ∇u u,d
(8)
This optimization problem can be converted to an unconstrained problem by adding a quadratic penalty function, i.e. (u† , d† ) = arg min d1 + γd1,g + μu, r + u,d
λ d − ∇u22 2
(9)
Where λ is a weighting parameter. If γ is heigh, d = ∇u. However setting γ high introduces numerical instability. Note that the quadratic penalty function only approximates the constraint d = ∇u. However, by using a Bregman iteration technique [15], this constraint can be enforced exactly in an efficient way. In
Robust Active Contour Segmentation
199
the Bregman iteration technique an extra vector, bk is added to the penalty function. Then the following two unconstrained steps are iteratively solved. (uk+1 , dk+1 ) = arg min dk 1 + γdk 1,g + μuk , r + uk ,dk
λ dk − ∇uk − bk 22 2 (10)
bk+1 = bk + ∇uk+1 − dk+1
(11)
The first step requires optimizing for two different vectors. We approximate these optimal vectors by alternating between optimizing eq. (10) for u and optimizing eq. (10) for d independently: λ dk − ∇uk − bk 22 2 λ = arg min dk 1 + γdk 1,g + dk − ∇uk+1 − bk 22 2 dk
uk+1 = arg min μuk , r +
(12)
dk+1
(13)
uk
The first problem can be optimized by solving a set of Euler-Lagrange equations. For each element u(i) of the optimal u the following optimality condition should be satisfied: μ ∇2 u(i) = r(i) + ∇ · (d(i) − b(i)) (14) λ Note that this system of equations can be written as Au = w. In [12] they proposed to solve this linear system using the iterative Gauss-Seidel method. In order to guarantee the convergence of this method, A should be strictly diagonally dominant or should be positive semi definite. Unfortunately is A neither. Instead we will optimize eq. (14) using the iterative conjugate residual method, which is a Krylov subspace method for which convergence is guaranteed if A is Hermitian [16]. The solution of eq. (14) is unconstrained, i.e. u(i) does not have to lie in the interval [0, 1]. Note that minimizing eq. (12) for u(i), i.e. all other elements of u remain constant, is equivalent to minimize a quadratic function. If u(i) ∈ / [0, 1] then the constrained optimum is either 0 or 1, since a quadratic function is monotonic in an interval which does not contain its extremum. So the constrained optimum can be calculated as follows: u∗ (i) = max min u(i), 1 , 0 (15) In order to calculate an optimal dk , we can rewrite eq. (13) as follows: dk+1 = arg min dk 1,(1+γg) + dk
λ dk − ∇uk+1 − bk 22 2
(16)
A closed form solution for this optimization step can be calculated using the shrinking operator, i.e. dk+1 (i) = shrink ∇u(i) + bk , 1 + γg(i), λ (17)
200
J. De Vylder, J. Aelterman, and W. Philips
where shrink(τ, θ, λ) =
0 τ−
θ λ
sgn(τ )
if τ ≤ λθ otherwise
(18)
In algorithm 1 we give an overview in pseudo code of the complete optimization algorithm. As an initial value for d[t] and b[t] we chose (0, 0). The initial estimation of mf and mb can be calculated based on Otsu thresholding. The CR function solves eq. (14) using the Conjugate residual method, given the parameters bk , dk and rk . Note that the last line is the update of rk based on the new mean intensity of the foreground/background, which were calculated in the previous two lines of code. Algorithm 1. Split Bregman for active contour segmentation 1 2 3 4 5 6 7 8 9 10
5 5.1
while u∗k+1 − u∗k 2 > do uk+1 = CR(bk , dk , rk ) u∗k+1 = max(min(uk+1 , 1), 0) dk+1 = shrink(∇u∗k+1 + bk , 1 + γg, λ) bk+1 = bk + ∇u∗k+1 − dk+1 sk = φα (u∗k+1 ) mfk+1 = sk ◦ f 1 sk −1 1 mbk+1 = sck ◦ f 1 sck −1 1 rk+1 = (mfk − f )2 − (mbk − f )2 end
Results Examples
A typical application for active contours is segmentation of organs in medical images. As an example Fig. 1 shows the result of segmenting white matter in an MRI image of the brain. The top row shows the RAW MRI image on the left and the segmentation result of the Chan-Vese convex active contour (CVAC); this uses regularization but does not incorporate edge information. The CVAC shows a good segmentation result, although the method does make some small errors near the borders of the contour. These errors generally exists of background pixels with ”high” intensity which are considered to be foreground. Due to their ”high” intensity they resemble the segment, however considering the neighbouring edges; it is unlikely that they actually belong to the segment, e.g. the white matter in this example. Some of these segmentation errors are indicated by the green arrows. The bottom row of Fig. 1fig:exBrain shows the segmentation results of CVAC which incorporate edge information. The left image does not incorporate a regularization term in its energy function, resulting in lots of small segments due to noise. The right image shows the segmentation result of the proposed active contour model. This active contour does not suffer from noise, nor has it the small errors near borders which occur with the original Chan-Vese active contour model.
Robust Active Contour Segmentation
201
Fig. 1. An example of brain white matter segmentation in MRI images. Top left shows an MRI slice of the brain. Top right depicts the segmentation result of the ChanVese convex active contours. Bottom left shows the Chan-Vese convex active contour segmentation using edge information. Bottom right depicts the proposed segmentation method, i.e. active contours using edge information with regularization.
202
J. De Vylder, J. Aelterman, and W. Philips
Fig. 2. An example of segmentation in photos with clutter. Top left shows a gray-scale photo of a squirrel. Top right depicts the segmentation result of the Chan-Vese convex active contours. Bottom left shows the Chan-Vese convex active contour segmentation using edge information. Bottom right depicts the proposed segmentation method, i.e. active contours using edge information with regularization.
A second example is shown in Fig: 2, where a squirrel has to be segmented out of a gray-scale image. The original CVAC, shown in the top row on the right, results in poor segmentation. A part of the head and a piece of the paw are missing in the segmentation result. Incorporating edge information helps to recover these missing parts as can be seen in the bottom row. Due to the clutter in the background the method finds a lot of false segments if there is no regularization, as can be seen on the left. The proposed method however finds the biggest part of the squirrel without adding any background pixels as is shown on the right of Fig. 2. Although the proposed method gives better results in noise images or images with noise, it comes with a cost, i.e. the active contour converges slower. The proposed method does converge slower than the CVAC which incorporates edge information. The convergence speed depends on the amount of regularization needed. If hardly any regularization is needed, i.e. γ in eq. (7) >> 1, the speed approaches the convergence speed of CVAC with edge information [13]. Whereas if the regularization factor is dominant in the energy function, the convergence speed approaches the convergence speed of the original CVAC using Split Bregman optimization [15]. Fig. 3 shows the
Robust Active Contour Segmentation 1.4 1.2
CPU time (s)
1
203
CVAC with edges CVAC without edges CVAC, gamma= 5/1 CVAC, gamma= 5/2 CVAC, gamma= 5/3 CVAC, gamma= 5/4
0.8 0.6 0.4 0.2 0 200
300
400
500
600
700
image size n (pixels)
800
900
1000
Fig. 3. Convergence speed of different active contours in function of the image size. The full lines are active contours from literature, Chan-Vese active contours with and without incorporation of edge information. The dotted lines represent the convergence speed of the proposed method with different γ, i.e. different ratio’s of regulation.
convergence speed in function of the image size. Between these four different CVAC only the γ parameter was changed, μ and λ was constant between all methods, i.e. 0.001 and 0.5 respectively. The full lines depict the CVAC’s from literature, in green the method using edge information, in blue the method using regularization without edge information. The dotted lines show the convergence speed of the proposed method for different γ in eq. (7). 5.2
Error Metric
For the validation of the segmentation, the Dice coefficient is used. If S is the resulting segment from the active contour, i.e. φ0.5 u∗ , and GT the ground truth segment based on manual segmentation, then the Dice coefficient between S and GT is defined as: 2 Area(S ∧ GT ) d(S, GT ) = (19) Area(S) + Area(GT ) where S ∧ GT consist of all pixels which both belong to the detected segment as well as to the ground truth segment. If S and GT are equal, the Dice coefficient is equal to one. The Dice coefficient will approach zero if the regions hardly overlap. 5.3
Noise Robustness
In order to quantitatively validate the proposed method, a dataset of synthetic fluorescent microscopic images was segmented. This dataset has been developed by Ruusuvuri et al. [17] and serves as a benchmark for segmentation algorithms.
204
J. De Vylder, J. Aelterman, and W. Philips
Fig. 4. An example of the micorscopy dataset used for vallidating the robustness of the proposed segmentation technique. The image is contaminated with different levels of white Gaussian noise. The top row has a SNR of 3,5,7 and 9 respectively. The bottom row as a SNR of 11,13,15 and 17 respectively.
1
0.9
Dice coefficient
0.8
0.7
0.6
0.5
0.4
2
CVAC with edges CVAC with edges and regularization 4
6
8
10
12
SNR (dB)
14
16
18
20
Fig. 5. The average Dice coefficient for segmentation of a synthetic dataset contaminated with different noise levels. The Dice coefficients were calculated for the segmentation result coming from the Chan-Vese active contours with edge information as for segmentation using the proposed method.
Robust Active Contour Segmentation
205
The dataset consists of 20 images each containing the 300 fluorescent nuclei. We contaminated this dataset with eight different levels of white Gaussian noise so that we could measure the influence of noise on the proposed segmentation technique. A close up of such an image with different noise levels can be seen in Fig. 4. The segmentation quality of the proposed method was measured using the average Dice coefficient for the full dataset and compared with the average Dice coefficient of segmentation using CVAC with edge information. As can be seen in Fig. 3 is the proposed method significantly more robust. The proposed method still gets a Dice coefficient of 0.7 for an image set with a SNR of only 3 in comparison with a Dice coefficient of 0.4 for the state of the art CVAC.
6
Conclusion
In this paper a new active contour method has been proposed. The method is comparable with the work proposed in [13], both methods are a combination of the original Chan-Vese active contours and snakes. The proposed method allows extra regularization, which was not possible in the method proposed by Bresson. The proposed method uses a convex energy function, allowing the use of global optimizers. This paper proposes a regularization term which enforces a smooth contour. However other convex regularization terms could be used as well, for example regularization using non local self similarity or based on texture such as is done in [18]. An efficient optimizer has been proposed using the Split Bregman optimization scheme. This results in efficient and fast optimization, although not as fast as the active contour proposed by Bresson [13]. However it is experimentally proven that the proposed method is significantly more robust to noise and clutter than the method proposed in [13]. Acknowledgment. This research has been made possible by the Institute for the Promotion of Innovation by Science and Technology in Flanders (IWT).
References 1. Kass, M., Witkin, A., Terzopoulos, D.: Snakes: active contour models. International Journal of Computer Vision, 321–331 (1988) 2. Isard, M., Blake, A.: Active contours. Springer, Heidelberg (1998) 3. Ray, N., Acton, S.: Motion gradient vector flow: An external force for tracking rolling leukocytes with shape and size constrained active contours. IEEE Transaction on Medical Imaging 23, 1466–1478 (2004) 4. Tang, J.: A multi-direction gvf snake for the segmentation of skin cancer images. Pattern Recognition (2008) 5. Charmi, M.A., Derrode, S., Ghorbel, S.: Fourier-based geometric shape prior for snakes. Pattern Recognition Letters 29, 897–904 (2008) 6. Rochery, M., Jermyn, I.H., Zerubia, J.: Higher order active contours. Int. J. Comput. Vision 69(1), 27–42 (2006) 7. Chan, T., Vese, L.: An active contour model without edges. Scale-Space Theories in Computer Vision 1682, 141–151 (1999)
206
J. De Vylder, J. Aelterman, and W. Philips
8. Mille, J.: Narrow band region-based active contours and surfaces for 2d and 3d segmentation. Computer Vision and Image Understanding 113(9), 946–965 (2009) 9. Tsechpenakis, G., Rapantizikos, K., Tsapatsoulis, N., Kollias, S.: A snake model for object tracking in natural sequences. Signal Processing: Image Communication 19, 219–238 (2004) 10. Goldenberg, R., Kimmel, R., Rivlin, E., Rudzsky, M.: Fast geodesic active contours. IEEE Transactions on Image Processing 10(10), 1467–1475 (2001) 11. Chan, T.F., Esedoglu, S., Nikolova, M.: Algorithms for finding global minimizers of image segmentation and denoising models. Siam Journal on Applied Mathematics 66(5), 1632–1648 (2006) 12. Goldstein, T., Bresson, X., Osher, S.: Geometric applications of the split bregman method: Segmentation and surface reconstruction. Journal of Scientific Computing 45(1-3), 272–293 (2010) 13. Bresson, X., Esedoglu, S., Vandergheynst, P., Thiran, J.P., Osher, S.: Fast global minimization of the active contour/snake model. Journal of Mathematical Imaging and Vision 28(2), 151–167 (2007) 14. Bresson, X., Chan, T.F.: Active contours based on chambolle’s mean curvature motion. In: 2007 IEEE International Conference on Image Processing, vol. 1-7, pp. 33–36 (2007) 15. Goldstein, T., Osher, S.: The split bregman method for l1-regularized problems. Siam Journal on Imaging Sciences 2(2), 323–343 (2009) 16. Saad, Y.: Iterative methods for sparse linear systems, 2nd edn. SIAM, Philadelphia (2003) 17. Ruusuvuori, P., Lehmussola, A., Selinummi, J., Rajala, T., Huttunen, H., YliHarja, O.: Set of synthetic images for validating cell image analysis. In: Proc. of the 16th European Signal Processing Conference, EUSIPCO-2008 (2008) 18. Aleman-Flores, M., Alvarez, L., Caselles, V.: Texture-oriented anisotropic filtering and geodesic active contours in breast tumor ultrasound segmentation. Journal of Mathematical Imaging and Vision 28(1), 81–97 (2007)
A Method to Generate Artificial 2D Shape Contour Based in Fourier Transform and Genetic Algorithms Maur´ıcio Falvo1,2 , Jo˜ ao Batista Florindo1 , and Odemir Martinez Bruno1 1
Universidade de S˜ ao Paulo - Instituto de F´ısica de S˜ ao Carlos, S˜ ao Carlos, Brasil [email protected], [email protected], [email protected] 2 Faculdade Adventista de Hortolˆ andia, Hortolˆ andia, Brasil
Abstract. This work presents a simple method to generate 2D contours based in the small number of samples. The method uses the Fourier transform and genetic algorithms. Using crossover and mutation operator news samples were generated. An application case is presented and the samples produced were tested in the classifier construction. The result obtained indicated the method can be a good solution to solve the small sample problem to feature vectors based in shape characteristics. Keywords: small sample problem, Fourier transform, genetic algorithms.
1
Introduction
The building processes of high quality classifiers, fundamentally, are based on a set of training vectors, which are obtained from samples numerically greater than the number of features that make it up. However, when the number of samples is less than the number of features, we have a small sample problem [10]. In the literature there are works that proposed some solutions to the small sample problem. The main idea to fix the small sample problem is complementing the samples set with artificial samples. The challenger of this approach is how to create artificial samples with characteristics capable to improve the statistical model without biasing it. In computer vision Pormelou, in his ALVINN project, used real images to create artificial samples which were applied in to learn of neural networks [19,18]. Examples from object and speech recognition based in artificial samples can be found in [15]. Li and Fang [13] proposed a sample generation method named of non-linear virtual sample generation (NVSG). This same method was used to generate an artificial data-set from Iris Plant Database. Lately this method was applied to facilitate cancer identification for DNA microarray data[14]. However, create artificial samples are not a trivial issue because of difficultly knowledge a function that can generates a statistically consistent set of features from a class of objects. In the construction of classifiers for recognition of biological shapes [2] where the number samples may be a critical factor due to difficult acquisition or from J. Blanc-Talon et al. (Eds.): ACIVS 2011, LNCS 6915, pp. 207–215, 2011. c Springer-Verlag Berlin Heidelberg 2011
208
M. Falvo, J.B. Florindo, and O.M. Bruno
rare occurrence of the researched specie [17], the sample small problem becomes more evident. This article proposes a simple, efficient and statistically consistent method to create artificial shapes based in a small number of real samples, using genetic algorithms and Fourier transform. This was originally designed to produce samples of leafy vegetables but may be used to create any other set of contours. A brief theoretical introduction is discussed in the section 2. In the section 3 the developed method is detailed and study case is showed in the section 4, with a construction of a leaves shape classifier using artificial samples. In the section 5 the results and conclusions are presented.
2
A Brief Theoretical Introduction
This section presents a brief introduction of Fourier transform applied in the contour representation in the complex domain, and basic concepts of genetic algorithms. 2.1
Shape Representation in the Frequency Domain
The parametric signature combine the pairs of set coordinates (X,Y) which can be express by 1D signal ( see equation 1) when represented in complex number domain [3]. c(t) = x(t) + y(t)i (1) From this signal, a set contour descriptors are obtained by the discrete Fourier transform (DFT) [7] apud [3]. These descriptors are represented in the Fourier spectrum by frequency components. The low frequencies define the global contour form while the high frequencies define the minor details. If some high frequencies of descriptor vector are removed (replaced by zeros, by a ideal low-pass filter), and the inverse function of Fourier is applied, a 1D contour complex signal is recovered with its details reduced. This process is described by sequence of equations 2, 3, and 4. Note that ω is the cut frequency of ideal low-pass filter. F (ω) = F {c(t)} ⎧ ⎨ 0, if ωc ≤ ω ≤ −ωc H(F (ω)) = ⎩ F (ω), otherwise.
(2)
c (ω) = F −1 {H(F (ω))}
(4)
(3)
The Figure 1 shows examples of reconstruction signals after a ideal low-pass filter was applied with different cut frequency ω. Therefore, samples that belong to same class are distinguished by its hight frequency components in Fourier spectrum and grouped by its low frequency components. Note that with only a few descriptors, it is possible to characterize the general shape of a contour. This property makes the Fourier descriptors are widely used for differentiation and characterization of boundaries [11], [22], [1], [6].
A Method to Generate Artificial Contour Samples
ω=10
ω=20
ω=30
ω=60
ω=120
ω=180
ω=240
ω=300
209
Fig. 1. Contours of Passiflora caerulea after applied a ideal low-pass filter with different cut frequencies ω
2.2
Genetic Algorithms
Inspired in the evolution and natural selection principles, the genetic algorithms are usually applied in practical problems of optimization and search [9]. Its application involves firstly the coding a finite set of potential feasible solutions, named initial population (see Figure 2). Each solution is ranked by a fitness function which represents a problem to be solved. The solutions have not a good rank are disposed and replaced by new others, in the same quantity. The replacement is done by generation of new solutions. These new ones are generated by data combination of two others solutions that had a good rank. Thus, the initial number of possible solutions in the set is reestablished. The cycle: selection process, disposal, and generation is repeated again, until a optimal solution set is found [5]. Note this mimics the natural selection and biological reproduction, including the shares of crossover and mutation that occur during the reproduction of new solutions. Crossover and mutation are discussed bellow. The Crossover Operator: Considered the primary responsible in imitation of the biological process of heredity, the operator of crossover acts simultaneously on two solutions (like would be in chromosomes), creating two new solutions.
210
M. Falvo, J.B. Florindo, and O.M. Bruno
Fig. 2. Basic process of genetic algorithm
The Figure 2 exemplify the application of a crossover operation. Similar to biological process, the parents have a cut point chosen by which is based the break and swapping between the parts resulting two new solution elements. The intensity of its use is measured by the crossover rate which is defined like as the ratio between the number of offspring generated by the operator and the total number of individuals in the population. A very low rate can induce crossover genetic algorithm to find a false local maximum of a function, but if this rate is too high the algorithm can spend time investigating regions of space solutions not relevant [4] [12]. The Mutation Operator: Applied in solutions generated by the crossover operator, the mutation operator changes the coded information elements of a solution (like would be in genes of a chromosome). In the case of a solution encoded in binary form, this would imply in inversion of the selected bits. The emergence of new coded elements (or genes) that were not changed before in the population, can contribute to solving the problem [12]. Thus, the influence of mutation operator is evaluated by the rate mutation. This is defined as the ratio between the total number of genes altered by total number of genes from the population. When mutation rate is very low, the probability of to find genes that may contribute to solving of a problem is reduced. However, a very high mutation rate causes serious harm to the new younger generations, due the loss of the similarity of child with its parents, reflecting the inability of the algorithm to keep the its search history [4] . So it is very important to make balanced use of this operator in each new generation to keep control over its effects.
A Method to Generate Artificial Contour Samples
211
Fig. 3. Example of application of croosover and mutation operators
3
The Proposed Method
Given a class of objects which are generated samples of artificial boundaries, the whole process basically involves: -Division of real samples in two families (A and B): the aim is to use a family (the A familiy) as an initial source of the Fourier descriptors of the contour, in the generation of descriptors of artificial samples that will be used in learning a classifier. The B family also will be to used to generate artificial samples but these will be used only to test the validity of the classifier trained by the artificial samples of family A. This procedings is need to avoid a biased classifier. -Extraction of the Fourier descriptors from real contours of family A and B: in this process the samples of both families should be binarized and an algorithm extracting contours of the ”chain code” type is applied. A vector of the parametric contour in the domain of complex numbers is assembled according to equation (1). This vector is transformed by the DFT for frequency domain and feature vectors are produced. -Descriptor generation of new individuals through crossover and mutation operators: the generation of new samples has its beginnings in the random selection of two members of the family A. In these, apply the crossover operator, which will produce two new individuals. Sometimes, randomly, a mutation operator is driven to change the value of some frequencies of these new individuals. The points at which the frequency of mutation operator is applied are set randomly. Two operator implementations of crossover were used. The one and two cutoff points [12]. See Figure 3 -Selection of new individuals generated: as occurs in every biological process of reproduction, some times there are people who have severe abnormalities
212
M. Falvo, J.B. Florindo, and O.M. Bruno
Fig. 4. Example of artificial shapes of Passiflora suberosa. In the top of samples is possible to see the parents used in the process,represented by outline.
compared to their parents. In order to produce a representative sample of a class, individuals who showed no consistent characteristics, should be discarded. This disposal may be performed by cluster analysis, using a k-means algorithms[8] with k = 3. For these two classes of groups tend to resemble their parents and one class will contain more samples that are different from them. For this third class is that the samples will be selected for disposal. -Shape reconstruction of new individuals generated: finally the generation of the shapes from new samples is easily obtained by applying of the inverse of discret Fourier transform. All images resulting will have discontinuity problem in its outline which can be solved with a low computational cost through morphological operators of dilation and erosion [20].See Figure 4. In the next section method the evaluated is presented in the construction of leaf shapes classifier.
4
Validation Method
To validate the method developed were used six different species with small number of samples: Passiflora giberti (5 samples), Passifora miiersi (6 samples), Passiflora suberosa (7 samples), Rauwolfia sellowii (8 samples), Cariniana estrellensis (9 samples) and Caesalpinia echinata (25 samples), shown in figure 5. All samples were scanned on a flatbed scanner with a resolution of 300dpi. The binarization of images was performed using the average thresholding algorithms
A Method to Generate Artificial Contour Samples
(a)
(b)
(c)
(d)
(e)
213
(f)
Fig. 5. a. Passiflora giberti, b. Passifora miiersi, c. Passiflora suberosa, d. Rauwolfia sellowii, e. Cariniana estrellensis; f. Caesalpinia echinata
[22] and iterative thresholding [21]. To remove the possible noise in the binary images, morphological operators of opening and closing [20] were applied. The parametric contour was obtained from the contour tracer algorithm [16]. The complex signal 1D was obtained and the DFT was applied resulting in the Fourier descriptors of contour. As described in Section 3 each class of samples went through the process of generating artificial samples and the descriptors of the A family produced 200 artificial samples that were used for training of neural networks while the descriptors of family B were used to generate 50 samples for testing classifier. The features vector was based in 60 Fourier descriptor of curvature (more details see [3]) building from artificial e natural samples The classifier used was a neural network Mult-Layer Perceptron (MLP) with three layers of neurons, each one respectively equal to 30, 25 and 1. So in the case of a binary type of network that was trained specifically to identify whether or not a single class type. Thus were trained and tested six types of classifiers, each corresponding to only one class of vegetal that previously was cited in this section. Finally, these networks were subjected to two types of tests, the first in which each tries to identify all artificial samples created and a second that uses only the real samples (in this case are to reunite the family samples A and B).
5
Results and Conclusions
Table 1 presents the classification results of artificial samples in the first column. With the exception of the species Passiflora miersii the classifier had an accuracy rate 86% while other classes had rates ranging from 92% to 100%. In the same table 1 shows in the second column the results of classification of natural samples whose rates ranged from 71.43% to 100%. As can be seen, the results indicate that the artificial samples generated fulfilled their role in building of the classifiers. The results obtained both in the classification of natural and artificial samples showed consistent values. Also the fact of not having an absolute classification in all classifiers reinforces the consistency of the method of generating artificial samples.
214
M. Falvo, J.B. Florindo, and O.M. Bruno Table 1. Classification rate of artificial and natural samples by class Class Passiflora gibertii Passiflora miersii Passiflora suberosa Rauwolfia sellowii Cariniana estrellensis Caesalpinia echinata
% Artificial samples % Natural samples 100,0% 86,0% 100,0% 92,0% 99,5% 100,0%
71,43% 83,33% 71,43% 100,00% 77,78% 100,00%
References 1. Bandera, A., Urdiales, C., Arrebola, F., Sandoval, F.: 2d object recognition based on curvature functions obtained from local histograms of the contour chain code. Pattern Recognition Letters 20, 49–55 (1999) 2. Casta˜ n´ on, C.A., Fraga, J.S., Fernandez, S., Gruber, A., da Fontoura Costa, L.: Biological shape characterization for automatic image recognition and diagnosis of protozoan parasites of the genus eimeria. Pattern Recognition 40(7), 1899–1910 (2007), http://www.sciencedirect.com/science/article/B6V14-4MMWHFH-2/2/ d21bd74af95fc6439f5e08fc1ca175f7 3. Costa, L.F., Cesar, R.M.J.: Shape Analysis and Classification - Teory and Practice. Image processing series. CRC Press, Boca Raton (2001) 4. Gin, M., Chieng, R.: Genetic algorithms and engineerin design. John Wiley & Sons, Inc., New York (1997) 5. Goldberg, D.E.: Genetic algorithms in search, optimization and machine learning. Addison-Wesley Publishing Company, Inc., Massachusetts (1989) 6. Gonzalez, R.C., Woods, R.E.: Processamento de Imagens Digitais. Edgard Bl¨ ucher Ltda., S˜ ao Paulo (2000) 7. Granlund, G.H.: Fourier preprocessing for hand print character recogniton. IEEE Transactions on Computers 2(C-21), 195–201 (1972) 8. Hair, J.F.J., Black, W.C.: Cluster Analysis. In: Reading and Understanding More Multivariate Statistics, pp. 147–205. American Psychological Association, Washington, D.C (2000) 9. Holland, J.H.: Adapatation in natural and artificial systems. MIT Press, Cambridge (1975) 10. Jain, A.K., Dubes, R.: Feature definition in pattern recognition with small sample size. Pattern Recognition 10(2), 85 – 97 (1978), http://www.sciencedirect.com/ science/article/B6V14-48MPGX5-2S/2/0aff77f32970d8fbecc9104d3e636267 11. Jain, R., Kasturi, R., Schunk, B.G.: Machine Vision. Computer Science. McGrawHill, New York (1995) ´ 12. Lacerda, E.G.M.d., Carvalho, A.C.P.L.F.: Introdu¸c˜ ao aos algoritmos genETicos. ˜ Sistemas Inteligentes - AplicaC ¸ OEs a Recursos H´ıdricos e Ciˆencias Ambientais, 1 edn., pp. 99–150, Cole¸c˜ ao ABRH de Recursos H´ıdricos, Editora da Universidade Universidade do Rio Grande do Sul, Porto Alegre (1999) 13. Li, D.C., Fang, Y.H.: A non-linearly virtual sample generation technique using group discovery and parametric equations of hypersphere. Expert Systems with Applications 36(1), 844–851 (2009), http://www.sciencedirect.com/science/ article/B6V03-4R53W74-6/2/3997723c6618ff44b5847cc7b10dc83f
A Method to Generate Artificial Contour Samples
215
14. Li, D.C., Fang, Y.H., Lai, Y.Y., Hu, S.C.: Utilization of virtual samples to facilitate cancer identification for dna microarray data in the early stages of an investigation. Information Sciences 179(16), 2740–2753 (2009), http://www.sciencedirect.com/science/article/B6V0C-4W3HX9P-2/2/ 8c2f015476973daf78cc54e540c0e70a 15. Niyogi, P., Girosi, F., Poggio, T.: Incorporating prior information in machine learning by creating virtual examples. Proceedings of the IEEE 86(11), 2196–2209 (1998) 16. Pavilids, T.: Algoritms for Graphics and Image Processing. Computer Science, Rockville (1982) 17. Pazoti, M.A., Garcia, R.E., Pessoa, J.D.C., Bruno, O.M.: Comparison of shape analysis methods for guinardia citricarpa ascospore characterization. Electronic Journal of Biotechnology 8(3), 1–6 (2005) 18. Pomerleau, D.A.: Neural network vision for robot driving. In: The Handbook of Brain Theory and Neural Networks, pp. 161–181. University Press (1996) 19. Pormelau, D.A.: Efficient training of artificial neural networks for autonomous navigation. Neural Computation 3(1), 88–97 (1991) 20. Serra, J.: Image Analysis and Mathematical Morphology., vol. 1. Academic Press, London (1982) 21. Sezgin, M., Sankur, B.: Survey over image thresholding techniques and quantitative performance evaluation. Journal of Electronic Imaging 1(13), 146–165 (2004) 22. Sonka, M., Hlavac, V., Boyle, R.: Image Processing, Analysis, and Machine Vision, 2nd edn. PWS Publishing (1998)
Image Segmentation Based on Electrical Proximity in a Resistor-Capacitor Network Jan Gaura, Eduard Sojka, and Michal Krumnikl ˇ - Technical University of Ostrava, VSB Faculty of Electrical Engineering and Computer Science, 17. listopadu 15, 708 33 Ostrava-Poruba, Czech Republic {jan.gaura,eduard.sojka,michal.krumnikl}@vsb.cz
Abstract. Measuring the distances is an important problem in many image-segmentation algorithms. The distance should tell whether two image points belong to a single or, respectively, to two different image segments. The paper deals with the problem of measuring the distance along the manifold that is defined by image. We start from the discussion of difficulties that arise if the geodesic distance, diffusion distance, and some other known metrics are used. Coming from the diffusion equation and inspired by the diffusion distance, we propose to measure the proximity of points as an amount of substance that is transferred in diffusion process. The analogy between the images and electrical circuits is used in the paper, i.e., we measure the proximity as an amount of electrical charge that is transported, during a certain time interval, between two nodes of a resistor-capacitor network. We show how the quantity we introduce can be used in the algorithms for supervised (seeded) and unsupervised image segmentation. We also show that the distance between the areas consisting of more than one point (pixel) can also be easily introduced in a meaningful way. Experimental results are also presented.
1
Introduction
In many image-segmentation algorithms, measuring the distances plays the key role. The distance is usually needed as a quantity the value of which expresses the fact that two points should belong to a single or, respectively, to two different image segments. The use of the Euclidean distance measuring the straight-line lengths is often considered. In the case of image segmentation, however, the input data (images) define certain manifolds in some space (e.g., the brightness function defines such a manifold). The key question then is whether or not it could make some sense to measure the distances along this manifold instead of measuring the straight-line distance. Another natural question is what metric should be used. In the sequel, therefore, we discuss the properties of some metrics from the point of view of how useful they can be for image segmentation, especially the metrics that can be used on the manifolds. Naturally, not all algorithms are immediately prepared for measuring the distances on the manifold since they also deal with some other points arising as new positions or centroids J. Blanc-Talon et al. (Eds.): ACIVS 2011, LNCS 6915, pp. 216–227, 2011. c Springer-Verlag Berlin Heidelberg 2011
Image Segmentation Based on Electrical Proximity
217
that may lie outside the manifold. The works [2] and [8] may be taken as an example of how an algorithm, mean shift in this case, can be modified such that the distance on the manifold could be used instead of the Euclidean distance. Consider two points, denoted by A and B, respectively, on the manifold in the situations as follows (Fig. 1). In Fig. 1a, A and B both lie in one area with a constant brightness. The distance we expect in this case is small since both points should belong to a single resulting image segment. In Fig. 1b, although the points have the same brightness, they apparently should not belong to one image segment since the areas in which they lie are separated by a “dam” created by the manifold. The distance we expect in this case is big. Fig. 1c shows two areas with the same brightness that are connected by a “channel” (the areas are separated by a weak boundary). The width of the channel determines whether or not the areas should be regarded as one segment. Usually, if the channel is narrow, we expect that the distance should be big. Intuitively, the wider the channel is, the shorter distance between A and B should be reported. Preferably, the distance should reflect a relative width of channel in some intelligible way. In Figs. 1d, 1e, two situations are depicted in which we would probably say that the points A and B belong to one area and have similar distances in both cases, in spite of the fact that one area is narrow and the other is wide.
(a)
(b)
(c)
(d)
(e)
Fig. 1. On some problems arising when measuring the distances between two points (A, B) in images: The images are represented by the contour lines (isolines) of brightness similarly as in a map; various situations (a – e) are discussed in the text
Euclidean distance measures the straight-line distances between the points that can be in general positions (not only on the manifold). As a result of the fact that the distance is not measured as a route along the manifold, it can neither reveal the separation by a dam (Fig. 1b) nor quantify the width of a possible channel between the areas with similar brightness (Fig. 1c), which may deteriorate the quality of algorithms. The distance does not depend on the width of the areas as was explained in Figs. 1d, 1e, which is desirable. Geodesic distance measures the length of the shortest path lying entirely on the manifold. The distance reveals the dams (Fig. 1b) correctly. No hint about the channel width (Fig. 1c) is given; even a very thin channel causes that a short distance is reported. In the cases from Figs. 1d, 1e, the same distance is detected, which is correct. The influence of noise is big. Even if the area is theoretically flat, the walker moving from A to B must overcome small hills and depressions
218
J. Gaura, E. Sojka, and M. Krumnikl
on the manifold caused by noise, which integrates along the whole path from A to B. A big distance may be incorrectly reported. If the use of geodesic distance is mentioned in some algorithms, it is probably due to its ability to detect the dams. Other mentioned problems remain. From [3], for example, the possibility of computing k shortest paths follows, which could by used for measuring the width of channel. Nevertheless, the problems with distinguishing the situations depicted in Fig. 1c and 1d still remain. Resistance distance is a metric on graphs introduced by Klein and Randic [5] and further developed also by others [1]. In image segmentation, the use of a resistor network has already been mentioned too [4]. The resistance distance is defined as a resistance between two points in the resistor network that corresponds to the image grid; the resistances between the particular neighbouring nodes increase with the increasing local image contrast. Intuitively, the resistance distance explores many parallel paths whereas the geodesic distance explores only the shortest of them. The resistance distance is able to reveal the dams correctly. It also measures the width of channels. The wider the channel is, the higher number of conductive parallel paths runs through the channel, which decreases the resistance between the points. Unfortunately, the behaviour in the case of measuring the distance in the objects of various width/size (Figs. 1d, 1e) does not seem to be fully satisfactory. In the wide objects, shorter distances are reported than in the objects that are narrow. The resistance distance may introduce an unwanted ordering of distance values, e.g., it may report the distance from Fig. 1e as the shortest, then the distance from Fig. 1c, and, finally, the distance from Fig. 1d. This is a problem since, at the time when the distance is measured, the shape of object is not known and, therefore, it is difficult to judge on the basis of the value whether or not the pixels belong to one object. The resistance distance can be measured not only between two points but also between two sets of points (we can connect all the nodes of one set together and similarly also all the nodes of the other set and measure the resistance between these new two terminals). Diffusion distance is a cumulative squared difference between the fields obtained by solving the diffusion equation for a chosen time and for the impulses placed at the points between which the distance is to be computed. The problem with the diffusion distance is the interpretation of the value that is obtained. From the value itself, it is not clear whether at all some amount of the substance diffusing from one point reached the other point and, therefore, whether the situation along the whole path between both points is taken into account (otherwise, the value can only reflect how the substance spreads in the neighbourhood of both points). The definition of the diffusion distance can be easily extended also for measuring the distance between two set of points (areas). In this case, however, the distance also takes into account the difference in size of both sets, which makes the distance hardly to use in the algorithms based on merging smaller areas into bigger ones. Let it be pointed out that the use of standard diffusion distance has been reported in some contexts, e.g., for histogram comparison [6], but not for segmenting images.
Image Segmentation Based on Electrical Proximity
219
It follows that non of the discussed metrics is problem free. Intuitively, it seems that a tool for measuring the sizes of objects is missing in some of them. The width of the channel from Fig. 1c, for example, should be measured relatively with respect to the sizes of the areas on both its ends. The problem that different values are reported for the distances from Figs. 1d, 1e by some metrics can also be viewed as a problem whose roots lie in the absence of measuring the sizes. The metric should recognise the circumstances and adjust the way in which the distance is measured according to it. The diffusion distance measures the sizes in some way, but it has another disadvantages. In this paper, we propose a new method how the distances on the image manifold can be measured. The method is based on the diffusion equation and reflects the drawbacks of diffusion distance. In contrast with diffusion distance, we measure the amount of substance that is transferred in diffusion process. The analogy between the images and electrical circuits is used for explaining the method (another analogies, e.g., heat could be used too), which means that the amount of electrical charge that is transported between two nodes/areas in an equivalent resistor-capacitor network during a certain time interval is measured. Because of using this analogy, the method resembles the resistance distance discussed before. The difference is that the pure resistive network is modified by adding the “grounding” capacitors to the nodes. Informally speaking, the capacitors are used as a tool for measuring the sizes. Since the capacitors correspond to the pixels and since a certain electrical charge is needed for charging each capacitor/pixel, the electrical charge that is required to charge an area consisting of many pixels depends on the number of pixels, i.e., on the size of the area. If some parts of image are to be regarded as a connected area, it is required that a certain electrical charge, proportional to the size of parts, must be possible to transport between the parts through the resistors of the network in a certain limited time. A big difference is introduced by adding the capacitors since, instead of a steady state of the resistor network, a transient state is solved. The use of the resistor-capacitor network is not new in digital image processing. The well-known Perona-Malik image filtering method [7] is based on solving the diffusion partial differential equation. One of the possible technical interpretations of such an equation is that it solves a resistor-capacitor circuit/network. Since, in order to determine the quantity mentioned in the previous paragraph, we solve the diffusion equation, it is clear that the quantity we introduce has something in common, in this sense, with the diffusion distance. It, however, is not the same since it is defined in a substantially different way and has different properties. The quantity we use may be regarded as a directed proximity/distance (see the next section). At this expense, it can be computed and used effectively in the given context. The paper is organised as follows. In the next section, the resistor-capacitor network is introduced as well as the method how the distance/proximity can be introduced in that network. In Section 3, it is shown how the proximity can be used for image segmentation. The experimental results are presented in Section 4.
220
2
J. Gaura, E. Sojka, and M. Krumnikl
Proximity in a Resistor-Capacitor Network
I this section, we introduce the resistor-capacitor (RC) network and the term proximity. We understand the proximity to be an opposite to the distance (if the proximity is big, the distance is small and vice versa). The problem can be seen as either continuous or discrete. We start from the continuous version since some formulations and expressions can be written and understood more easily in that case. The discrete version will be described later on. Strictly speaking, only the discrete version deserves the name network since, instead of a network, we have a circuit with a continuous resistor and capacitor in the continuous case. In the continuous case, we have a thin two-dimensional sheet of a resistive material. The resistive sheet lies on a dielectric layer and this layer lies on a conductive base, which creates a continuous capacitor (Fig. 2). The current density, denoted by J(x, y), in the resistive sheet is connected with the intensity, denoted by E(x, y), of the electric field by the Ohm law J(x, y) = σ(x, y)E(x, y),
(1)
where σ is a conductivity tensor. The capacity is C(x, y) per area unit. Generally, both σ as well as C may vary at different places (x, y). If the electric potential, denoted by ϕ(x, y), is introduced, the field intensity is E(x, y) = −∇ϕ(x, y).
Fig. 2. A continuous resistor-capacitor circuit: A thin resistive sheet rests on a dielectric layer that lies on a grounded ideally conductive base (for the meaning of A and B, see the further text)
The equation needed for solving the potential in such a circuit is well known. For convenience of the reader, we briefly recall how it can be deduced. Consider an area dS on the resistive sheet; let ∂S stand for its boundary. The capacitor created inside dS is charged by the current flowing into dS along ∂S. Expressing this equilibrium mathematically, applying the Gauss theorem, and setting δϕ/δt ≡ ϕt , we successively obtain the following equations (n, δϕ, δt stand for the normal to ∂S, for the change of ϕ, and for a short time interval, respectively) δϕCdS = J ndl δt, (2)
S
ϕt CdS =
S
∂S
∂S
J ndl =
S
divJdS = −
div (σ∇ϕ) dS, S
(3)
Image Segmentation Based on Electrical Proximity
Cϕt = −div (σ∇ϕ) ,
221
(4)
which is the desired equation (it has the form of the diffusion equation). By making use of the equation, the potential field ϕ can be solved in time, providing that certain initial conditions (e.g., the potential at t = 0) and boundary conditions (potential at a certain set of points) are known. The equation is known for a long time (in image processing, it was probably firstly used in [7]) and the techniques for its solution are well established. In this paper, the equation and the potential are used for measuring the proximity between the areas in image. We introduce the proximity, denoted by p(A, B), of the area A to the area B (A and B do not overlap; Fig. 2) as follows. For all points inside and on the boundary of B, we set C(x, y) = 0, ϕ(x, y) = 1; both C and ϕ are constant over time in B. For all points outside B, we set C(x, y) = 1; ϕ(x, y) evolves in time from ϕ(x, y)|t=0 = 0. (It follows that A is electrically supplied from B.) The proximity p(A, B), i.e., the proximity of A to B, is then measured as a charge, denoted by QA , that is accumulated in A after a certain chosen time, which can be computed by integrating the potential p(A, B) = QA = CϕdS = ϕdS. (5) A
A
The value of proximity is not symmetric in general, i.e., it may happen that p(A, B) = p(B, A), which, however, need not be regarded as an obstacle for its practical use (in the regard of its non-symmetry, it can be compared to the directed Hausdorff distance; the standard diffusion distance is symmetric). In addition to measuring the proximity, the electric circuit described before can also be used for measuring the electrical size of area and for finding its electrical center (generally, diffusion size or diffusion center). Let A be an area that is to be measured. (The reader can imagine A in such a way that inside A, the conductivity is big; ∂A runs around A through the points at which the conductivity is low.) For measuring the size of A, we use a very small area B that is placed into A, and we measure the proximity p(A\B, B) as was described before, i.e., the area A \ B is electrically supplied from B. Naturally, the charge accumulated in A\ B also depends on where B is placed inside A (providing that the size of B is small and constant). We call the position for which the charge accumulated in A \ B is the biggest one the electrical centre of A. The ideas that were introduced before can now be reformulated for the discrete case. In the discrete case, instead of a circuit with a continuous resistor and capacitor, we have a network whose nodes correspond to the image pixels. To each node (x, y), a “grounding” capacitor C(x, y) and the resistors joining the node with its four neighbours (in the case of an inner node) are connected (Fig. 3). Let σ((x1 , y1 ), (x2 , y2 )) stand for the conductivity of the resistor connecting the nodes (x1 , y1 ), (x2 , y2 ) (only the neighbouring nodes are connected). For solving the potential in the network, the following equation can be written for each inner node (for the nodes on the image boundaries, similar equations can be easily deduced)
222
J. Gaura, E. Sojka, and M. Krumnikl
Fig. 3. Measuring the proximity of A to B: From the nodes of B, the capacitors are removed and the nodes are connected to a unit potential; all the capacitors are discharged at the beginning; the proximity is measured as a total charge accumulated in the nodes of A after a chosen time interval
− C(x, y)ϕt (x, y) = σ ((x, y) , (x − 1, y)) [ϕ (x, y) − ϕ (x − 1, y)]
(6)
+ σ ((x, y) , (x, y − 1)) [ϕ (x, y) − ϕ (x, y − 1)] + σ ((x, y) , (x + 1, y)) [ϕ (x, y) − ϕ (x + 1, y)] + σ ((x, y) , (x, y + 1)) [ϕ (x, y) − ϕ (x, y + 1)] . In the discrete case, the areas consist of nodes/pixels. In correspondence to the continuous case, the proximity p(A, B) is determined as follows (Fig. 3). From the nodes creating B, the capacitors are disconnected; instead, the nodes are connected to a unit potential. The potential in the network is then solved by making use of Eq. (6) providing that C(x, y) = 1 and ϕ(x, y)|t=0 = 0 outside B. The value of p(A, B) can then be computed as a sum of charges in capacitors that are connected to the nodes creating A, which is the same as the sum of potentials since C(x, y) = 1. In the RC network, the conductivities of the resistors between the neighbouring nodes correspond to the values of the local contrast in image. We set the conductivity to the value of the sigmoid function 1/(1 + eλ(d−μ)), where d stands for the contrast between the corresponding neighbouring pixels (it can be either the absolute value of the simple difference of brightness or a colour contrast computed in some way), and λ and μ are chosen real constants. The meaning of the constants can be easily understood. The contrast less then μ is regarded more as noise; the contrast greater then this value is regarded more as a sign of an object boundary existing at that place. The value of λ determines how sharp this decision is. The choice of λ and μ should ensure that the value of conductivity will approach to 0 for a big contrast and to 1 for a small contrast. Alternatively, other formulas for determining the conductivity can be also used, 2 e.g., e−λd [4]. A remark should be made regarding the problem of selecting the appropriate time interval for solving the RC network since it is a parameter that determines the behaviour of the method. Say that the proximity p(A, B) is defined as was
Image Segmentation Based on Electrical Proximity
223
explained before. If the time interval is infinitely small, the charge accumulated in A will be zero. If, on the other hand, the time interval is infinitely long, each capacitor in A will be charged to the unit potential (i.e., the same potential that was applied to the nodes of B), regardless the image on the basis of which the conductivities were computed. In the network, a transient process is solved. The time for which the solution is obtained determines how the network measures the distances and sizes. If the time is longer, the more distant (in the x, y plane) pixels are taken into account. From the point of view of practical computing, it is worth mentioning that disconnecting the capacitors and holding the intensity on the unit level as was described before can also be simulated by setting C(x, y) = ∞ and ϕ(x, y)|t=0 = 1 at corresponding nodes. It also further illustrates the difference between the diffusion distance and the distance we introduce. Both the distances can be computed by the spectral matrix decomposition. For the diffusion distance, solving the ordinary eigenvalue problem is sufficient since the values of all capacitors are usually C(x, y) = 1. In the case of the distance we introduce in this paper, solving the generalised eigenvalue problem would be required. Whereas the eigenvalue problem must be solved the only time in the case of diffusion distance, the generalised eigenvalue problem must be solved for every source area in the case of the distance we introduce. This makes the computation through the matrix spectral decomposition ineffective. Therefore, we directly integrate the equation Eq. (6).
3
Using the Proximity for Image Segmentation
In this section, we show how the proximity introduced in Section 2 can be used for image segmentation. We start from the seeded segmentation and continue to the general (unseeded) case. In seeded segmentation, the nuclei (seeds) of the desired areas (objects) in images are given by the operator, which makes the problem easier. Let Si0 stand for the i-th given initial seed. In its essence, the algorithm we propose is iterative, i.e., the areas grow from the seeds in time; Sik stands for the i-th area entering into the k-th iteration; after the iteration, the area Sik+1 is obtained; Si0 is a given seed of the area. At the beginning, the membership of pixels to objects is known for the pixels lying in the seeds. The goal is to decide the membership also for all remaining pixels in image. During the k-th iteration of the algorithm, the following is done. For each pixel Xj whose membership has not been decided yet, its proximity p(Xj , Sik ) for all i is used. The pixel is decided to be a member of a certain area providing that its proximity measured to that area is the highest and, at the same time, higher than a chosen threshold. If no pixel is decided in a certain iteration (i.e., if Sik+1 = Sik for all i), the algorithm stops. The following is worth mentioning: If the threshold mentioned before is chosen to be zero, the algorithm stops after one iteration. It simply only computes the values of p(Xj , Si0 ), i.e., only the proximity measured to the given initial seeds is taken into account. From the computational point of view, it should be noted that the
224
J. Gaura, E. Sojka, and M. Krumnikl
RC network is solved for every seed and for every area arising from this seed during iterations. After solving the network for Sik , the proximities p(Xj , Sik ) are obtained for all Xj lying outside Sik . The problem of general unseeded segmentation can be solved in such a way that the nuclei (seeds) of future areas in image are found automatically instead of determining them by operator. Once a method for determining them is available, the algorithm described in the previous paragraph can be used again for growing the nuclei into the corresponding areas. By making use of the ideas introduced in Section 2, the nuclei can be found as electrical centers. For each pixel Xi , the RC network is solved providing that it is electrically supplied from Xi (i.e., the capacitor at Xi is disconnected and Xi is connected to a unit potential). The electric charge transported from Xi to the rest of the network is computed. If the charge from Xi is greater than the charge computed for the neighbours of Xi and if, at the same time, the charge is greater than a chosen threshold, Xi is declared to be a nucleus of an area. It can be pointed out that the threshold ensures that the future area is big enough; the condition of local maximum ensures that the nucleus lies in a centre of area (in the electrical centre). For big and complicately shaped areas, it may happen that more than one nucleus are found (Fig. 9). A post processing step carried out after the growing described in the previous paragraph may be needed in such cases, during which all particular parts of the bigger areas are connected together along the “invisible boundaries” between the parts (a similar situation is also known from some other segmentation algorithms, for example, from mean shift). From the computational point of view, it should be mentioned that, for finding the seeds as was described, the RC network must be solved many times (one solution for each pixel charging the network). It follows that the method is computationally expensive. Some heuristics may at least partially improve the situation, e.g., the heuristic based on a priori excluding the pixels at which the nuclei cannot be expected.
4
Experimental Results
In this section, we present some experimental results. The experiments are mostly oriented to illustrate practically the claims and expectations that have been pronounced before. Firstly, the problem of measuring the proximity in the presence of a weak boundary or “channel” between two objects (Fig. 1c) is illustrated. In Fig. 4, a source image is depicted as well as the images showing that the values of proximities given by the RC network are useful and lead to the wanted segmentation that was computed by the algorithm described in Sec. 3. In Fig. 4, it can also be seen that the proximities both in the wide as well as in the narrow areas are measured approximately in the same way, which is a problem that was discussed before by making use of Figs. 1d, 1e. The next series of experiments will be presented for the image from Fig. 5. In Fig. 6, the process of growing of Si0 into Sik is shown for one initial seed. The influence of time, which is a parameter that must be chosen, is illustrated in Fig. 7. In Fig. 8, a result of a seeded segmentation is presented. Fig. 9 illustrates
Image Segmentation Based on Electrical Proximity
225
Fig. 4. Measuring the proximity in the presence of a “channel” (weak boundary) between two objects (left image): Contrary to the Euclidean and geodesic distances, the values of proximities given by the RC network are useful and lead to the wanted segmentation. The proximities to a seed (small bright square) in the left area and to a seed in the right area are shown (middle images). The proximities are depicted in gray levels (totally bright means the highest proximity). The final segmentation is also shown (right image); different areas have different brightness.
Fig. 5. The image referred to in the further experiments
Fig. 6. Growing an initial seed (small bright square in the left image) during iterations; the results after the first, third and the fifth iterations are depicted (the left, middle, and the right image, respectively); the grey level shows the proximity of each pixel to the seed or its successive area (the seed as well as the successive areas are depicted as totally bright)
the method of automatically detecting the seeds. For each pixel, the electrical charge that is transported into the network if the network is charged from that pixel is shown. The seeds are detected as the maxima in the charge map created in this way.
226
J. Gaura, E. Sojka, and M. Krumnikl
Fig. 7. On the influence of the length of the time interval: The results after the first, third, and the fifth iteration are shown similarly as in the previous figure; a longer time interval for solving the transient state in the RC network was used in this case
Fig. 8. Supervised segmentation: The seeds were determined manually in this case (left image). The result of segmentation is depicted in the right image; the particular segments are distinguished by different grey levels; for the pixels in the totally black areas, the membership to any segment has not been found.
Fig. 9. A map of input charges and the seeds detected automatically: The grey level at each pixel (except the small squares) depicts the size of charge that is transported into the network if it is charged from that pixel (i.e., the bigger is the area into which the pixel belongs, the bigger is the charge and the brighter is the pixel); the small squares show the seeds that were found automatically as maxima in the charge map
5
Conclusion
A method has been presented for measuring the distance/proximity along the manifold that is defined by image for the purpose of using this method in image segmentation. The information that is contained in image is transformed into an equivalent resistor-capacitor network. Its transient state, which is described by the diffusion equation, is then solved. We measure the proximity as an amount of electrical charge that is transported, during a chosen time interval, between two
Image Segmentation Based on Electrical Proximity
227
nodes of the network. Among the advantages of the method, it also belongs that it can measure the distances not only between the particular points, but also between a set of points and a point, and between two sets of points, which was exploited in the image-segmentation algorithm that has also been proposed. The numerical solution of the diffusion equations is well understood. From the point of view of computational complexity, the proposed segmentation method seems to be well acceptable for seeded segmentation since the number of how many times the resistor-capacitor network must be solved depends on the number of seeds, which usually is not too high. The method of automatically detecting the nuclei of areas that has also been proposed, on the other hand, seems to be extremely computationally expensive since, in essence, the network must be solved for each pixel separately. Moreover, a post processing step is required. Further research is apparently needed in this area. Although the diffusion equation itself is known for a long time and although it was also used many times in the context of image processing, e.g., for filtering, we are not aware of the fact that it would be used for introducing the distance in the way as it was done in this paper. The diffusion distance is defined in another way. Its properties make the diffusion distance not very useful for the use in image segmentation algorithms. Acknowledgements. This work was partially supported by the grant SP 2011/ ˇ - Technical University of Ostrava, Faculty of Electrical Engineering 163 of VSB and Computer Science.
References 1. Babic, D., Klein, D.J., Lukovits, I., Nikolic, S., Trinajstic, N.: Resistance-Distance Matrix: A Computational Algorithm and Its Applications. Int. J. Quant. Chem. 90, 166–176 (2002) 2. Comaniciu, D., Meer, P.: Mean Shift: A Robust Approach toward Feature Space Analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence 24, 1–18 (2002) 3. Eppstein, D.: Finding the k Shortest Paths. SIAM Journal on Computing 28(2), 652–673 (1999) 4. Grady, L.: Random Walks for Image Segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence 28(11), 1768–1783 (2006) 5. Klein, D.J., Randi, M.: Resistance Distance. J. Math. Chem. 12, 81–95 (1993) 6. Ling, H.: Diffusion distance for histogram comparison. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 246–253 (2006) 7. Perona, P., Malik, J.: Scale-space and edge detection using anisotropic diffusion. IEEE Transactions on Pattern Analysis and Machine Intelligence 12(7), 629–639 (1990) 8. Sheikh, Y.A., Khan, E.A., Kanade, T.: Mode-seeking by Medoidshifts. In: International Conference on Computer Vision, pp. 1–8 (2007)
Hierarchical Blurring Mean-Shift ˇ Milan Surkala, Karel Mozdˇreˇ n, Radovan Fusek, and Eduard Sojka Technical University of Ostrava, Faculty of Electrical Engieneering and Informatics, 17. listopadu 15, 708 33 Ostrava-Poruba, Czech Republic {milan.surkala.st,karel.mozdren,radovan.fusek.st,eduard.sojka}@vsb.cz
Abstract. In recent years, various Mean-Shift methods were used for filtration and segmentation of images and other datasets. These methods achieve good segmentation results, but the computational speed is sometimes very low, especially for big images and some specific settings. In this paper, we propose an improved segmentation method that we call Hierarchical Blurring Mean-Shift. The method achieve significant reduction of computation time and minimal influence on segmentation quality. A comparison of our method with traditional Blurring Mean-Shift and Hierarchical Mean-Shift with respect to the quality of segmentation and computational time is demonstrated. Furthermore, we study the influence of parameter settings in various hierarchy depths on computational time and number of segments. Finally, the results promising reliable and fast image segmentation are presented. Keywords: mean-shift, segmentation, filtration, hierarchy, blurring.
1
Introduction
Mean-Shift (MS) is a clustering algorithm that appeared in 1975 [6] and is used for data filtration and segmentation up to now. Mean-Shift seeks for the position with the highest density of data points. These groups of points are called clusters (segments). Mean-shift is not only used as a statistical tool, but also as a method for image segmentation, object tracking [5], etc. Recently, a lot of new Mean-Shift variations have been developed. These methods use various approaches to improve the original idea. In 1995 MS was revised by Cheng [2] and in 1999 and 2002 by Comaniciu [4,3]. One of the most interesting variations of MS is Blurring Mean-Shift (BMS) that appeared in 2006 [1]. It has been proved that BMS has a lower number of iterations per data point and, therefore, a higher speed. As for stopping conditions, it is more difficult to apply the Gaussian kernel because it is not truncated. However, using the truncated kernels as Epanechnikov makes it possible to ignore data that are too far away from its center. Even with such advantages, BMS is still slow and not appropriate for general use. In 2009, P. Vaturri and W.-K. Wong presented Hierarchical Mean-Shift [8] based on an idea of effective application of multiple Mean-Shift filtrations with variable kernel sizes. Computational time of the Mean-Shift method is strongly J. Blanc-Talon et al. (Eds.): ACIVS 2011, LNCS 6915, pp. 228–238, 2011. c Springer-Verlag Berlin Heidelberg 2011
Hierarchical Blurring Mean-Shift
229
dependent on the number of input data points and on the size of the Mean-Shift window (kernel size). The first step is to use MS with a small kernel window to get a lower number of small segments in a short time. These segments are considered as data points with the weight that is equal to the number of data points they represent. In the next hierarchical step (stage) the number of data points is reduced due to preprocessing (previous stage). This allows us to use a greater kernel without the unwanted increase of computational time. The number of hierarchical steps is not limited. In each step, a new level of hierarchy with greater segments is created. The paper is organised as follows. In Section 2, we introduce the reader to the original Mean-Shift method. Section 3 is devoted to the Blurred Mean-Shift version. A new approach (algorithm) is presented in Section 4. The experiments are presented in Section 5.
2
Mean-Shift N
Consider X = {xn }n=1 ⊂ Rd as a dataset of N points in the d-dimensional space. Although the characteristics of these points are not limited, we focus on the images and, therefore, these points are the set of pixels in processed image. We define the kernel density estimator with a kernel K(x) as N x − xn 2 1 p(x) = K (1) σ , N n=1 where x is a processed data point (pixel), xn is a data point in neighbourhood of point x and σ is a bandwidth limiting the size of neighbourhood (radius). We can distinguish between two types of the radii in images. The first one, denoted by σs , is a radius in the spatial domain (for example, a circle with the radius of 20 pixels). The spatial radius is mostly identical for x and y-axis, so we need only one σs . The second radius, denoted by σr , is a radius in the range domain (colour or intensity). If the grey-scale images are processed, only one σr is used, whereas, in the colour images, three different quantities σr may be required, each for one colour channel. In most cases, the same radius is chosen for all three channels, so σr is simplified to one value. We can choose from many types of kernels K(x). The simplest is the uniform flat kernel that takes the value of 1 (full contribution of pixels) for the whole area of searching window (area of kernel). This kernel is defined as follows 1, if x ≤ 1 K(x) = . (2) 0, if x > 1 The second most common kernel is the Epanechnikov kernel. It minimizes the asymptotic mean integrated square error, therefore, it is an optimal kernel. The Epanechnikov kernel is defined by the following equation 1, if 1 − x2 ≤ 1 K(x) = . (3) 0, if x > 1
230
ˇ M. Surkala et al.
The Gaussian kernel is very popular, nevertheless, it has some disadvantages. σs does not limit the size of searching window but it modifies only the shape of Gaussian curve. The computation is carried out for all data points (pixels in the image). Therefore, the use of the Gaussian kernel is very computationally demanding and the only way to achieve a better speed is to truncate the kernel. Fortunately, because of small contribution of distant pixels, the truncation has only a small influence on the quality of segmentation. The Gaussian kernel is represented by the equation 1
2
K(x) = e− 2 x .
(4)
The density gradient estimator, denoted by k(x), is defined as follows k(x) = −K (x). as
(5)
Therefore, we can rewrite the Epanechnikov kernel for mean-shift computation 1 − x, if 0 < x < 1 K(x) = . (6) 0, otherwise
The mean-shift vector represents the change of position for each data point (pixel in the image). Data used for computation remain unchanged in all iterations of mean-shift. The goal is to find an attractor. Attractor is a point to which all “similar” data points from a searching window converge. At such a point, the density of input data points takes its local maximum. All pixels that converge to one attractor form a cluster (segment). The mean-shift vector is represented by the equation N x−xi 2 i−1 xi k σ mσ,k (x) = (7) − x. N x−xi 2 i−1 k σ The first term on the left side is a new position of x computed by all adequate data in the searching window; the second term is the former position of x.
3
Blurring Mean-Shift
Blurring Mean-Shift (BMS) is an another method with a slightly changed equation leading to a significantly different method of computation. MS does not change the original dataset (it moves the data points, however, the computation is done with original data). As a result, it evolves slowly to the local density maxima. BMS changes the dataset in each iteration with computed values. As source data for each iteration, the data modified in the previous stage are used. It is proven that BMS has faster convergence in comparison to MS. The main advantage of BMS is that it reduces the number of iterations, despite that the number of input points remains the same. The equation of BMS can be written as N xm −xi 2 i−1 xi k σ mσ,k (x) = (8) − x. N xm −xi 2 i−1 k σ
Hierarchical Blurring Mean-Shift
231
BMS has several more advantages. After each iteration, for example, there is a visible progress in filtration. If the pixels that should form one cluster are close enough, without any pixel that should not be in the cluster, all these pixels will be grouped in the next iteration. Original MS does not have this feature and it iterates slowly till all the data points reach the common convergence point. This often leads to long computational times. BMS has a tendency to form clusters whose size is similar to the size of searching window, therefore, the number of clusters can be approximately predicted. Nevertheless, the shape of these clusters follows the gradients in the image very well. Many different variants of mean-shift were proposed, such as Evolving Mean-Shift [9], Medoid-Shift [7] and variations of MS and BMS. In next section we are going to describe our MS variation.
4
Hierarchical Blurring Mean-Shift
We propose a new method called Hierarchical Blurring Mean-Shift (HBMS) that is based on one of modifications of Mean-Shift, called Hierarchical Mean-Shift (HMS). It uses the original MS method, which is applied hierarchically with various sizes of kernel. First of all, it uses a very small searching window (σs ), which leads to creation of a large number of little segments representing just few data points. Because of the small searching window size, mean-shift is processed relatively quickly. Consider this as an input to the next stage of mean-shift running with a bigger searching window. Consequently, the segments become new data points. From the first stage, we obtained a filtered dataset which has lower number of data points than the dataset originally contained. In the new dataset, each new data point has a weight that is equal to the number of original data points. Repeatedly running mean-shift on the dataset with larger searching window results in significantly reduced computational time. The main idea is to reduce the number of segments and make them bigger in each iteration. MS and BMS are slow because they use a large searching window from the start. Hierarchical mean-shift processes data with a smaller searching window in the first stage, which results in faster computation. Although, HMS is faster than MS and BMS, it is still too slow for large input images. The main problem is that HMS uses the original MS as a basic algorithm. MS is not well suited for hierarchical processing since it is slower and produces a large number of segments. Therefore, each next iteration of HMS has still a large dataset. Our objective is to use more advanced mean-shift technique, which is more suitable for processing in hierarchical way. Our method uses BMS for each step. This is much more effective and significantly faster, due to a lower number of iterations per pixel. Another advantage is that BMS forms more uniform clusters. They are also bigger and less numerous. Knowing that, we can say that BMS decreases the number of iterations per data point and, moreover, it greatly reduces the number of segments for each next stages of HBMS, so we can expect a high reduction of computational time. We will show this in Section 5. The following algorithm shows an example of implementation in a pseudo code.
232
ˇ M. Surkala et al.
(a) The first stage filtration (b) The second stage filtra- (c) The third stage filtration tion
(d) The first stage segmen- (e) The second stage seg- (f) The third stage segmentation mentation tation Fig. 1. Example of filtration and segmentation evolving after each stage of HBMS
Algorithm 1. Hierarchical Blurring Mean Shift 1: σ ← {σ1 , ..., σj } , where j = number of stages 2: for i ∈ {1, ..., j} do 3: repeat 4: for mi ∈ {1, ..., Ni } do
5:
∀n : p(n|xmi ) ←
xm −xn 2 i k σi
xm −x 2 Ni i n k σi n =1
6: ymi ← N n =1 p(n|xmi )xn 7: end for 8: ∀mi : xmi ← ymi 9: until stop 10: end for
The problem is to find an accurate resizing of the searching window for each iteration. If we enlarge window too much, and if the number of segments from the previous stage is still high, the processing in the next stages will take too long. The greater reduction of data points is, the greater enlargement of searching window could be done. The size of initial window is discussed in Section 5.
Hierarchical Blurring Mean-Shift
5
233
Experiments
This section presents the experiments with HBMS algorithm. The first example is focused on the comparison between the computational speed of HBMS and HMS. We examine the idea that the HBMS approach is much faster than the hierarchical approach to MS (HMS) because BMS is faster than MS. We also study the effect of acceleration on image filtration quality. We also compare how the increasing amount of noise influences quality of image filtering. As a source image, the famous Lena gray-scale photo in 256 × 256 pixels was used and for speed tests, original 512 × 512 version and resized versions were used too. To prove segmentation quality of our method we also used additional images. The algorithms were tested on a computer with Intel Core i5-M520, dualcore processor, with 2.4GHz core frequency, 4 GB of DDR3-1333 memory and a 64-bit operating system. All tests were run in the single core configuration. The first test examined the speed of HMS and our HBMS implementation using two and three-stage hierarchy. In the last run, the value of σs was always set to 40. The two-stage configuration started with σs1 = 5, the three-stage configuration used σs1 = 3 and σs2 = 9. In all stages, σr was set to 24. Table 1 has two columns for each tested resolution. The first column represents the values of the mean square error (MSE) between the final and the original image. MSE shows the intensity difference between the original and the filtered image. The smaller the error is, the better is the correspondence between the original and the filtered image. A good segmentation has small granularity (number of segments) and concurrently small intra-segment intensity variance (MSE error). The second column represents the time taken to complete the computation. MSE is computed as follows M SE =
M N 1 (G1 (i, j) − G2 (i, j))2 , MN
(9)
i=0 j=0
where G1 and G2 are images to be compared, indexes i and j stand for pixel coordinates and M and N represents width and height of image. The results were surprising. We achieved smaller MSE error especially for higher resolutions if BMS was used as a base for hierarchical segmentation. Table 1. Comparison of speed and MSE depending on resolution 64 × 64 128 × 128 256 × 256 512 × 512 1024 × 1024 1536 × 1536 MSE t[s] MSE t[s] MSE t[s] MSE t[s] MSE t[s] MSE t[s] MS HM S2 HM S3 BM S HBM S2 HBM S3
145 200 317 209 232 284
15 0,7 0,45 4,5 0,4 0,3
149 206 263 212 197 216
100 4,6 1,7 35,5 1,4 0,9
307 1516 252 12207 228 50 219 425 172 196 10,5 162 60 127 161 138 152 1085 203 6,2 162 33 111 171 3,6 151 20 111
4310 313 156 76
91 95
358 176
234
ˇ M. Surkala et al.
Fig. 2. Computational time with respect to image resolution
(a) M S, t = 1588s
(b) HM S2 , t = 50s
(c) HM S3 , t = 10.5s
(d) BM S, t = 161s
(e) HBM S2 , t = 6.2s
(f) HBM S3 , t = 3.6s
Fig. 3. Segmentation output from each method we compared
Hierarchical Blurring Mean-Shift
235
The authors of BMS [1] proved that this method has cubic convergence while the original MS has only linear convergence, which means that the use of BMS should lead to faster processing. Table 1 shows that the computation time increases approximately linearly with the size of the image. This means that an image with a 4-times bigger area is computed in an approximately 4-times longer time. From the table, it follows that the computation time increases approximately 8-times (or even more) if HMS is used. Therefore, the difference in performance is relatively small in the case of small images but it is significantly higher for larger images. For example, a 1 megapixel image is processed in 3 minutes with HBMS, whereas HMS needs more than one hour (2-stage configuration). We use our own implementation of HMS and HBMS. It might have seemed that the hierarchical approach is helping more to original HMS than HBMS, however, the difference is not that significant. HMS benefits from a higher number of stages while HBMS is quite fast even with a lower number of stages. The speed-up is big enough even with two stages in HBMS, and it is not necessary to use three or more stages. The more stages used, the better filtration ability and computational speed we get. The drawback is that the granularity increases and the segmentation does not seem to be that good. Using more stages causes creating larger segments with a higher intensity difference of the distant attractors. This increases the probability that the segment will not be included in the computation because the attractor does not fit in the searching window even though a large part of the segment is inside this window. It creates small segments on the borders of large segments, which would not happen if a larger searching window in the spacial domain was used. We can say that all of these methods achieved a good segmentation quality, but the main difference of using the HBMS is in significantly higher speed. The next test examines the influence of noise on segmentation quality. Twostage HMS and HBMS is used on the 256 × 256 image. The noise modifies each pixel in the intensity channel. The noise intensities are ranging from 5 to 70 for each pixel intensity value. We study the filtration quality (mean square error in a comparison with the original noise-free image) and the number of segments. In Table 2, we can see that both HMS and HBMS have an increasing mean square error with the increasing amount of noise applied to the image. It means that the images with a high level of noise cannot be filtered with a low spatial radius. In HBMS, the number of segments does not depend much on the noise level, although the segments are scattered. The noise is still easily visible in the Table 2. Influence of noise on filtration error and number of segments
HMS MSE HMS segm HBMS MSE HBMS segm
0 237 60 187 57
5 213 73 187 64
10 221 71 197 51
15 301 54 201 50
20 255 64 213 66
25 243 60 220 51
30 312 69 244 52
35 398 90 287 48
40 458 77 349 67
45 555 76 497 60
50 617 112 613 69
55 745 113 746 61
60 993 166 942 86
65 1163 183 1135 67
70 1299 152 1306 71
236
ˇ M. Surkala et al.
Fig. 4. Influence of noise on number of segments
T he original image
Af ter the f irst stage
Af ter the second stage
Fig. 5. Segmentation output after each stage for the brain image (σs1 = 7, σs2 = 70) and the images from the Berkeley image database (σs1 = 4, σs2 = 50)
Hierarchical Blurring Mean-Shift
237
processed image but it is obvious that the pixels darkened by noise, are grouped together. The same applies on lightened pixels. In HMS, a stronger noise does lead to an increased number of segments. For the illustration of HBMS method capabilities, we provide three additional images. The first image is a medical scan of brain with the size of 550 × 650 pixels. The bandwidth σs1 was set to 7, and σs2 was set to 70 pixels. The range bandwidth σr remains unchanged with the value of 24 in pixel intensity. After that, we have chosen two images from the Berkeley image database with the size of 481 × 321 pixels. The mountains image is the most simple one. The tree image is more difficult one because it has a very large number of small leaves and so it could be difficult to merge them into one segment. Both images from the Berkeley database were segmented with values set to σs1 = 4, σs2 = 50. The original brain image consists of 357,500 pixels. The dataset was reduced to 3,594 segments after the first stage and 72 segments after the second one. The computation required 53.8 seconds for the first stage and 26.1 second for the second one. In the mountains image, HBMS reduced the number of pixels from 154,401 to 2,749 in the first stage and 29 segments in the final stage. The computation lasted 9.1 seconds (first stage) and 6.5 seconds (second stage). The computation took 16.3 seconds. For comparison, the original BMS with the same 50-pixel searching window lasted 894 seconds and the final segmentation had 62 segments. For the tree image, the number of pixels was reduced from 154,401 to 3,710 segments after the first stage and 65 segments after the second stage. For this case, the execution time was 17.6 seconds.
6
Conclusion
In this paper, we have proposed a new segmentation method based on hierarchical Mean-Shift. Blurring Mean-Shift was used instead of the original MS that has many disadvantages. The proposed method decreases computational time due to the use of BMS on which HBMS is based on, and decreases the number of segments. This is why each succeeding stage is faster. The computational time can reach tens of minutes or hours with large images using the original HMS while our HBMS method needs only a few minutes and quality is still comparable to the original HMS, BMS and MS methods.
References 1. Carreira-Perpi˜ na ´n, M.: Fast nonparametric clustering with Gaussian blurring meanshift. In: Proceedings of the 23rd International Conference on Machine Learning, ICML 2006, pp. 153–160. ACM, New York (2006) 2. Cheng, Y.: Mean shift, mode seeking, and clustering. IEEE Transactions on Pattern Analysis and Machine Intelligence 17, 790–799 (1995) 3. Comaniciu, D., Meer, P.: Mean shift analysis and applications. In: The Proceedings of the Seventh IEEE International Conference on Computer Vision, vol. 2, p. 1197 (1999)
238
ˇ M. Surkala et al.
4. Comaniciu, D., Meer, P.: Mean shift: a robust approach toward feature space analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence 24(5), 603–619 (2002) 5. Comaniciu, D., Ramesh, V., Meer, P.: Kernel-based object tracking. IEEE Trans. Pattern Anal. Mach. Intell. 25, 564–575 (2003) 6. Fukunaga, K., Hostetler, L.: The estimation of the gradient of a density function, with applications in pattern recognition. IEEE Transactions on Information Theory 21(1), 32–40 (1975) 7. Sheikh, Y.A., Khan, E.A., Kanade, T.: Mode-seeking by medoidshifts. In: IEEE International Conference on Computer Vision, pp. 1–8 (2007) 8. Vatturi, P., Wong, W.K.: Category detection using hierarchical mean shift. In: Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 2009, pp. 847–856. ACM, New York (2009) 9. Zhao, Q., Yang, Z., Tao, H., Liu, W.: Evolving mean shift with adaptive bandwidth: A fast and noise robust approach. In: ACCV (1) 2009, pp. 258–268 (2009)
Curve-Skeletons Based on the Fat Graph Approximation Denis Khromov Moscow State University [email protected]
Abstract. We present a new definition of the 3D curve-skeleton. This definition provides a mathematically strict way to compare and evaluate various approaches to the skeletonization of 3D shapes. The definition is based on the usage of fat curves. A fat curve is a 3D object which allows to approximate tubular fragments of the shape. A set of fat curves is used to approximate the entire shape; such a set can be considered as a generalization of the 2D medial axis. We also present an algorithm which allows to build curve-skeletons according to the given definition. The algorithm is robust and efficient. Keywords: curve-skeleton, medial axis, shape analysis.
1
Introduction
The medial axis, first introduced in [1], has been proved to be very useful for 2D shape analysis. The medial axis of a closed bounded set Ω ⊂ R2 is the set of points having more than one closest point on the boundary; or, equivalently, the medial axis is the set of centers of the maximum inscribed in Ω circles. The medial axis is a graph embedded in R2 . This graph emphasizes geometrical and topological properties of the shape Ω. Such graphs are usually called skeletons. There are efficient algorithms for 2D medial axis computation. It would be natural to try use the same approach for 3D shape analysis. A medial axis of a 3D shape Ω ⊂ R3 is a set of points having more than one closest point on the boundary. But such an object is not a graph since it may contain 2D sheets [6]. Those sheets may be very complex, and there are some methods which try to simplify the inner structure of the medial axis in 3d [2]. Therefore 3D medial axis is as difficult for the processing as the initial shape Ω. So there is a problem: how to define a skeleton of a 3D shape as a graph embedded in R3 so that this graph would have all of the advantages of the 2D medial axis? Such graphs are called curve-skeletons. To date, there are lots of publications on curve-skeletons. However, unlike 2D case, where the strict mathematical definition of the medial axis was given decades ago, the definition of a 3D curveskeleton still hasn’t been presented. Usually, curve-skeleton is defined as the result of applying some algorithm to the 3D shape. There is no way to compare these algorithms with each other because their working principles and results J. Blanc-Talon et al. (Eds.): ACIVS 2011, LNCS 6915, pp. 239–248, 2011. c Springer-Verlag Berlin Heidelberg 2011
240
D. Khromov
may have totally different nature. It’s very difficult to evaluate the quality of the skeletons produced by those algorithms, since there is no formal criterion for such an evaluation. There’s only visual evaluation, which is subjective and not mathematical at all. In [4][8] the authors presented the classification of curve-skeleton algorithms. The curve-skeleton is intuitively defined as a 1D thinning of the 3D object. The authors also made a list of possible properties of a curve-skeleton. Some of these properties are strict (for example, topological equivalency between curveskeleton and the original shape), others are intuitive and should be formalized (centeredness of the skeleton and possibility of reconstruction of the original 3D object). Almost every published algorithm computes skeletons which have some of these properties. In such cases, these properties are considered to be advantages of the algorithm. One of the popular approaches is based on the thinning of voxel images. Such thinning may be done directly (deleting boundary voxels step-by-step, [7] [11]) or with the distance function [10]. The skeletons produced by such methods are not continuous but discrete objects. Algorithms of this class are not universal because they’re applicable to the voxel images only. Finally, there is no mathematical criterion to evaluate and compare different techniques of thinning. It seems natural to try to extract 1D curve-skeleton from the 2D medial axis. The medial axis itself is a very complicated object, which consists of quadratic surfaces, so it’s usually replaced by some approximation. However, extraction of 1D piece from the medial axis usually based on some successfully found heuristic. For example, in [5] such an extraction is done with the help of the geodesics on the boundary surface. As in the previous case, there’s no strict criterion for evaluation and comparison of various heuristics of a 1D curve-skeleton extraction. There are some other techniques used to compute curve-skeleton, such as usage of optimal cut planes [9] or physical interpretation of the problem [3]. However, these methods are also successfully found heuristics which allow to compute some object visually corresponding to the human idea of the curveskeleton. And again, formal mathematical evaluation of these algorithms doesn’t seem to be possible. In this paper, a strict definition of the curve-skeleton is given. The model being proposed 1. allows to evaluate the correspondence between the curve-skeleton and the original object; 2. approximates the given shape with a fixed precision; 3. doesn’t depend on the type of the shape description (polygonal model, voxel image or point cloud). Also, an algorithm which computes the curve-skeleton according to the definition, is presented.
Curve-Skeletons Based on the Fat Graph Approximation
2
241
Definitions
Let C be a set of smooth curves in R3 . For every curve c ∈ C , there is a set Rc of continuous non-negative functions defined on c. Definition 1. A fat curve is a pair (c, r), where c ∈ C , r ∈ Rc . The curve c is said to be an axis of the fat curve, and the function r is its radial function. Definition 2. An image of the fat curve (c, r) is a set of points I(c, r) = {x ∈ R3 |∃y ∈ c : ρ(x, y) ≤ r(y)}.
(1)
Definition 3. A boundary of the fat curve (c, r) is a set of points ∂I(c, r) = {x ∈ I(c, r)|∀y ∈ c : ρ(x, y) ≥ r(y)}.
(2)
The fat curve is an object which is very convenient to approximate tubular 3D shapes (see Fig. 1).
Fig. 1. Fat curve
Definition 4. Let C be a set of fat curves such that axis of these fat curves intersect each other in their endpoints only. A fat graph F over a set C is a graph whose edges are fat curves from C and vertices are endpoints of their axis. Definition 5. A boundary ∂F of a fat graph F is an union of boundaries of the fat curves composing F . Let FC be a set of all possible fat graphs. Consider an embedded in R3 connected 3D manifold Ω with the boundary ∂Ω. We’ll approximate Ω with some fat graph.
242
D. Khromov
Definition 6. A distance between the point x ∈ R3 and the fat graph F is a distance between x and the closest point on FC ’s boundary: ρ(x, G) = min ρ(x, y). y∈∂F
(3)
Definition 7. A distance between a manifold Ω and a fat graph F is a value ε(Ω, F ) = ρ2 (x, F )dS. (4) x∈∂Ω
Approximation quality can be evaluated by two values: distance ε(Ω, F ) and complexity of the fat graph F . A complexity of a fat graph can be evaluated as 1. sum of lengths of axis of fat curves composing the fat graph; 2. number of the fat curves. If the set C is rather wide, it’s better to use the first criterion in order to avoid too crooked curves. However, if C is narrow and doesn’t contain such curves, the second criterion can be used since it’s very simple. In these terms, the problem of approximation with a fat graph can be defined as following 1. compute an approximation with the smallest possible ε(Ω, F ) and a fixed fat graph complexity; 2. compute an approximation with the least possible complexity and ε(Ω, F ) < ε0 ,
(5)
where ε0 is a fixed value. The fat graph can also be defined for planar curves. 2D medial axis would be an example of such a graph if we define the radial function at a point x equal to the distance from x to the boundary. Image of this special graph coincides with the whole shape, so its approximation error is zero.
3 3.1
Fat Graph Fitting General Overview
In this chapter, we present the common scheme which allows to compute and optimize curve-skeletons according to the given definitions. Definition 8. A topological class of fat graphs is an equivalence class of FC by the relation of isomorphism between graphs. Let FC |G be a topological class of graph G (i.e. FC |G is a set of all fat graphs isomorphic to G). Minimization of ε˜(V , F ) on set FC is done through the several stages. First we select a topological class and then consider fat graphs only from this class. Then we find the first approximation of the fat graph and construct a sequence converging to some local minimum of the function ε˜.
Curve-Skeletons Based on the Fat Graph Approximation
3.2
243
Fat Graph in the Fixed Topological Class
Let E be a set of edges of the graph G, and V is a set of its vertices. Since we consider only those fat graphs which are isomorphic to G, E also denotes the set of fat curves of the fat graph F . Definition 9. A skeletal structure S on V in the topological class FC |G, defined by the graph G, is a pair of mappings (t, g): t : V → [0; 1], g : V → E.
(6)
Definition 10. Fat graph F and skeletal structure S are called compatible if for each vertex v ∈ V ρ(v, F ) = ρ(v, c(t(v))) + r(t(v)), (7) where c(t) is a Bezier curve representing the axis of g(v) and r(t) is its radial function. The iterative algorithm, which finds the fat graph in the given topological class, consists of the following stages. 1. Selection of the initial approximation. 2. Fitting of the compatible pair of a skeletal structure and a fat graph with the zero radial function. 3. Radial function fitting, skeletal structure elaboration. 4. If the approximation error ε˜ changed less than some fixed value, then stop; otherwise, go to stage 2. We use the segmentation of the vertices set described above to obtain the initial approximation of the skeletal structure. g(v) is defined due to the one-to-one correspondence between segments S(q) and edges E. Fat Graph with the Zero Radial Function. In this chapter, all of the radial functions are equal to zero: r(t) ≡ 0. Let FC0 be a set of all fat graphs with such radial functions. Let cv be an axis of a fat line g(v). We solve the following optimization problem: ρ2 (v, cv (t(v)) → min . (8) J1 (F ) = 0 v∈V
F ∈FC |G
Bezier curves cv ∈ Bn are defined via their reference points. Each of these curves has (n + 1) reference points. Two of them are endpoints, these are vertices of the fat graph. Those curves which are incident to the same vertex of the fat graph share one reference point. There are |VG | different endpoints. Each curve also has (n − 1) reference points which are not endpoints. Therefore, the set of fat curves C is defined by k1 points from R3 , where k1 = |VG | + |EG |(n − 1). Here and below |A| indicates the number of elements in the set A.
(9)
244
D. Khromov
We minimize function J1 with respect to k1 3D vectors, or, which is the same, 3k1 real numbers. To do so, we equate all of the partial derivatives to zero. That would be a system of 3k1 linear equations with 3k1 variables. The solution of this system defines the desired curves. Having the fat graph F , we use it to elaborate the existing skeletal structure S in order to make it compatible with F . It means recalculation of the t and g values. The solution of an equation (which can be found with the help of Newton’s method)
∇c(t), v = 0, t ∈ [0; 1], (10) gives us the closest to v point on the curve c. That allows us to find the closest to v fat curve g(v) and the corresponding parameter value t(v). Again, we compute an optimal zero fat graph for this new skeletal structure, then elaborate the skeletal structure, and so on. This iterative process converges, and it’s result is a compatible pair of a fat graph and a skeletal structure. This fat graph is optimal for the skeletal structure S (by which we suggest a local minimum of J1 ). Non-Zero Radial Function. Let S be a skeletal structure and F ∈ FC0 |G is a fat graph compatible with S. We choose radial functions for fat curves from F in such manner that the function ρ(v, F ), (11) J2 (F ) = v∈V
reaches its minimum. Each radial function is defined by (n + 1) non-negative value. Therefore, J2 should be minimized with respect to k2 variables, where k2 = |EG |(n + 1).
(12)
Like in the previous case, we reduce this problem to the system of k2 linear equations and then elaborate the skeletal system, considering that now radial functions are not constant and depend on t(v).
4
Implementation
The main issue which wasn’t discussed in the previous chapter but seems to be very important in the practical implementations of the method is how to choose the topological class. It’s possible to use any existing algorithm which produces curve-skeletons. But the proposed scheme has the advantage that the first approximation of the skeleton doesn’t have to be very nice and accurate. It can be easily fitted into the shape afterwards. That means that we can use some algorithm which is inaccurate but very fast, hoping to improve it during the fat graph fitting. One way to do so is to use the 2D medial axis of some Ω’s planar projection. There’re some facts in favor of this decision.
Curve-Skeletons Based on the Fat Graph Approximation
245
– It seems that medial information plays the key role in the human vision and visual perception[8]. But the human vision is planar, so if some 3D graph feels to be a good curve-skeleton, its projection would be also considered as a graph which is very close to the 2D medial axis of the object. – 2D medial axis is a well-defined and examined object. There are fast and robust algorithms for 2D skeletonization. – As mentioned above, the 2D medial axis can be considered as a fat graph which has zero approximation error. It’s possible to try to bring this property in 3D as close as possible. – 2D medial axis of a polygonal shape consists of smooth curves of degree 1 and 2, which can be fitted in the manner described above. An example of the object and its planar projection is shown in the Fig. 2.
Fig. 2. A model of a horse (left), its orthogonal planar projection with the 2D medial axis (center) and the approximating fat graph (right)
Each knot of the 2D skeleton is a projection of at least 2 points on the surface (see Fig. 3). A maximal inscribed ball tangent to the surface ∂Ω at those two points is a good mapping of the knot into the 3D space.
Fig. 3. 2D projection (left) of a 3D cylinder (right); center of the maximum inscribed circle is a projection of points A, B
The main problem of this method is the possibility of occlusions. For example, if one of the legs of the horse in Fig. 2 was occluded by another one, the usage of this particular projection would lead to an incorrect result. But since it’s possible to evaluate the approximation error of the fat graph, we can try a lot of projections and select the best one.
246
5
D. Khromov
Experiments
The described algorithm was successfully implemented. As mentioned above, it’s impossible to compare the quality of the skeletons produced by other methods, since there has been no numerical criterion for evaluation of the difference between the curve-skeleton and the given shape. We’ll prove the capacity of our approach in the way that is common in the literature on the curve-skeletons, which is based on the visual evaluation and experimental proof of the claimed properties. First of all, we demonstrate the examples of curve-skeletons produced by the described algorithm (see Fig. 4). 3D models which have been chosen for the experiment are widely used to evaluate various computer geometry algorithms, so they’re appropriate for the visual comparison with other methods. It’s useful to discuss the properties listed in [4]. Homotopy equivalence between the curve-skeleton and the shape is not guaranteed, since the fat graph
Fig. 4. Examples
Curve-Skeletons Based on the Fat Graph Approximation
(a) ε = 0.0077
(c) ε = 0.0082
247
(b) ε = 0.0080
(d) ε = 0.0089
Fig. 5. Curve-skeletons of the horse with different approximation error
with relatively large amount of edges approximates wide non-tubular fragments of the shape with a loop consisting of a pair of edges. However, this property can be easily provided by more strict requirements for the topological class. Invariance under isometric transformations (i.e. transformations in which the distances between points are preserved) is obvious. The possibility of reconstruction of the original shape is provided by the definition of a fat graph: the image of the fat graph is a 3D manifold which approximates the original object with a known precision. The reliability (which means that every boundary point is visible from at least one curve-skeleton location) is not guaranteed, but it’s achieved when the fat graph has enough edges. The robustness is implied by the robustness of the function ε(Ω, F ). In order to justify the meaningfulness of the function ε(Ω, F ), which is the core part of the described method, we’ve prepared a number of various curveskeletons of the same object. These skeletons were made without any fitting and with badly tuned parameters of the algorithm. The curve-skeletons and their corresponding approximation error values are shown on the Fig. 5. It’s pretty obvious that the greater the value of ε(Ω, F ), the worse the visual quality of the produced curve-skeleton. That implies that the proposed definition is not only theoretically grounded but also has some practical utility.
248
6
D. Khromov
Conclusion
In the paper, a new mathematical model for curve-skeleton formalization was presented. This model allows to compare and research various approaches for the 3D skeletonization. Also, a new algorithm for skeletonization was described, implemented and discussed. The further research involves the following issues: – elaboration of the model, in particular, approximation evaluation via Hausdorff metric; – further development of the algorithm, better selection of the first approximation and formalization of the iterative fitting.
References 1. Blum, H.: A Transformation for Extracting New Descriptors of Shape. In: WathenDunn, W. (ed.) Models for the Perception of Speech and Visual Form, pp. 362–380. MIT Press, Cambridge (1967) 2. Chang, M.C., Kimia, B.B.: Regularizing 3D medial axis using medial scaffold transforms. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. p. Accepted. IEEE Computer Society, Los Alamitos (2008) 3. Chuang, J.H., Tsai, C.H., Ko, M.C.: Skeletonization of three-dimensional object using generalized potential field. IEEE Trans. Pattern Anal. Mach. Intell. 22, 1241– 1251 (2000), http://dx.doi.org/10.1109/34.888709 4. Cornea, N.D., Silver, D.: Curve-skeleton properties, applications, and algorithms. IEEE Transactions on Visualization and Computer Graphics 13, 530–548 (2007) 5. Dey, T.K., Sun, J.: Defining and computing curve-skeletons with medial geodesic function. In: Proceedings of the Fourth Eurographics Symposium on Geometry Processing, pp. 143–152. Eurographics Association, Aire-la-Ville (2006), http://portal.acm.org/citation.cfm?id=1281957.1281975 6. Giblin, P., Kimia, B.B.: A formal classification of 3d medial axis points and their local geometry. IEEE Trans. Pattern Anal. Mach. Intell. 26, 238–251 (2004), http://dx.doi.org/10.1109/TPAMI.2004.1262192 7. Pal´ agyi, K., Kuba, A.: A parallel 12-subiteration 3d thinning algorithm to extract medial lines. In: Sommer, G., Daniilidis, K., Pauli, J. (eds.) CAIP 1997. LNCS, vol. 1296, pp. 400–407. Springer, Heidelberg (1997), http://portal.acm.org/citation.cfm?id=648241.752833 8. Siddiqi, K., Pizer, S.: Medial Representations: Mathematics, Algorithms and Applications, 1st edn. Springer Publishing Company, Incorporated, Heidelberg (2008) 9. Tagliasacchi, A., Zhang, H., Cohen-Or, D.: Curve skeleton extraction from incomplete point cloud. ACM Trans. Graph. 28, 1–71 (2009), http://doi.acm.org/10.1145/1531326.1531377 10. Telea, A., van Wijk, J.J.: An augmented fast marching method for computing skeletons and centerlines. In: Proceedings of the Symposium on Data Visualisation, VISSYM 2002, pp. 251–260. Eurographics Association, Aire-la-Ville (2002), http://portal.acm.org/citation.cfm?id=509740.509782 11. Wang, Y.S., Lee, T.Y.: Curve-skeleton extraction using iterative least squares optimization. IEEE Transactions on Visualization and Computer Graphics 14, 926–936 (2008), http://portal.acm.org/citation.cfm?id=1373109.1373261
DTW for Matching Radon Features: A Pattern Recognition and Retrieval Method Santosh K.C.1 , Bart Lamiroy2 , and Laurent Wendling3 1
INRIA Nancy Grand Est Research Centre, LORIA - Campus Scientifique, BP 239 - 54506 Vandoeuvre-l´es-Nancy Cedex, France [email protected] 2 Nancy Universit´e INPL, LORIA - Campus Scientifique, BP 239 - 54506 Vandoeuvre-l´es-Nancy Cedex, France [email protected] 3 LIPADE, Universit´e Paris Descartes 75270 Paris Cedex 06, France [email protected]
Abstract. In this paper, we present a method for pattern such as graphical symbol and shape recognition and retrieval. It is basically based on dynamic programming for matching the Radon features. The key characteristic of the method is to use DTW algorithm to match corresponding pairs of histograms at every projecting angle. This allows to exploit the Radon property to include both boundary as internal structure of shapes, while avoiding compressing pattern representation into a single vector and thus miss information, thanks to the DTW. Experimental results show that the method is robust to distortion and degradation including affine transformations. Keywords: Radon Transform, DTW, Pattern Recognition and Retrieval.
1
Introduction
Given patterns such as symbols of any kind, cursive characters but also forensic elements like human faces, shoe- or finger-prints, recognition or classification can be done using structural [10], statistical [17] as well as hybrid approaches. In structural approaches, graph based methods have been widely used [10]. They provide a powerful relational representation. However, they suffer from intense computational complexity due to the general NP-hard problem of sub-graph matching resulting from the variation of graph structure with the level of noise, occlusion, distortion etc. In some cases, however, matching optimisation can be obtained, but is strongly based on how the image is described [18]. Besides, structural approaches do not have rich set of mathematical tools [7] unlike in statistical approaches [17]. We therefore emphasise in the rest of this paper on statistical pattern analysis representation and recognition. Shape representation has been an important issue in pattern analysis and recognition [23,31]. In this context, features are often categorised as regionbased as well as contour-based descriptors. Generally, contour-based descriptors J. Blanc-Talon et al. (Eds.): ACIVS 2011, LNCS 6915, pp. 249–260, 2011. c Springer-Verlag Berlin Heidelberg 2011
250
K.C. Santosh, B. Lamiroy, and L. Wendling
include Fourier Descriptors (FD) [32,12]. Contour information can also come from polygonal primitives [1], or curvature information [3,5]. In the case of the latter, the shape is described in the scale space by the maximum number of curvatures. Other methods, like Shape Context (SC) [4] or skeleton approaches [33] are based on contour information. In short, contour-based descriptors are appropriate for silhouette shapes since they cannot capture the interior content as well as disconnected shapes or shapes with holes where boundary information is not available. On the other side, region-based descriptors account all pixels within patterns. Common methods are based on moments [28,2,13] including geometric, Legendre, Zernike, and Pseudo-Zernike moments. Comparative studies [2,28] have demonstrated the interest on improving invariance properties and reducing computational time of the Zernike moments [8]. On the other side, to overcome the drawbacks of contour-based Fourier descriptors, Zhang and Lu [30] have proposed a region-based Generic Fourier Descriptor (GFD). To avoid the problem of rotation in the Fourier spectra, the 2D Fourier Transform (FT) is applied on a polar-raster sampled shape. This approach outperforms common contourbased (classical Fourier and curvature approaches) and region-based (Zernike moments) shape descriptors. Region-based descriptors on the whole, can be generally applied. However, high computational complexity needs to be considered. Besides, use of normalisation in order to satisfy common geometric invariance properties introduces errors as well as they are sensitive to noise, eventually affecting the whole recognition process. Pattern representation must be sufficiently enriched with important information. Moreover, global pattern representation is the premier choice due to its simplicity that avoids extra pre-processing and segmentation process as in local pattern representation. To accomplish recognition, matching is another concern. In other words, feature selection corresponds to the matching techniques, eventually affects the overall performance of the method. For instance, compressing pattern information into a single vector as in global signal based descriptors provide immediate matching, while not offering complete shape information. In those respects, we take advantage of radon transform [11] to represent pattern and DTW is used to match patterns of any size that avoids compressing pattern representation into a single vector unlike the use of R−transform [27], for instance. The work is inspired from previous works such as 2D shape categorisation [22], gait recognition [6], off-line signature verification [9,24] and orientation estimation as in [16]. We have examined the method over two different datasets: graphical symbol [15] and shape [26]. The remaining of the paper is organised as follows. We start with detailing the proposed method in Section 2, which mainly includes pattern representation and matching. Section 3 provides a series of tests. In Section 4, analysis of the results are observed or examined and discussed thoroughly. The paper is concluded in Section 5 along with future perspectives.
DTW for Matching Radon Features
2
251
Method
In this work, we use the Radon transform to represent patterns [11], Radon based descriptors do not only encode contour information ([14,22], for instance), they also encode internal structure. Radon transforms are essentially a set of parametrized histograms. We apply Dynamic Time Warping to align every histogram for each projecting angle to absorb varying histogram sizes resulting from image signal variations. 2.1
Pattern Representation
The Radon transform consists of a collection of projections of a pattern at different angles [11]. This is illustrated in 1. In other words, the radon transform of a pattern f (x, y) and for a given set of angles can be thought of as the projection of all non-zero points. This resulting projection is the sum of the non-zero points for the pattern in each direction, thus forming a matrix. The matrix elements are related to the integral of f over a line L(ρ, θ) defined by ρ = x cos θ + y sin θ and can formally be expressed as, ∞ ∞ R(ρ, θ) = f (x, y)δ(x cos θ + y sin θ − ρ)dxdy −∞
−∞
where δ(.) is the Dirac delta function, δ(x) = 1, if x = 0 and 0 otherwise. Also, θ ∈ [0, π[ and ρ ∈] − ∞, ∞[. For radon transform, Li be in normal form (ρi , θi ). Following Fig. 1 (c), for all θi , the Radon transform now can be described as the length of intersections of all lines Li . Note that the range of ρ i.e., −ρmin < ρ ≤ ρmax is entirely based on the size of pattern. Since the Radon transform itself does not satisfy invariance properties, we consider the following affine transformation properties to adapt it for recognition. In case of translation, we use image centroid (xc , yc ) such that translation vector is u = (xc , yc ): R(ρ − xc cos θ − yc sin θ, θ). Therefore, translation of f results in the shift of its transform in ρ by a distance equal to the projection of translation vector of the line L (see Fig. 1 (c)). For rotation, we approximate rotation angle orientation as implemented in [16]. Orientation can be estimated as, d2 σ 2 α = arg min 2θ θ dθ where σθ2 = P1 ρ (R(ρ, θ) − μθ )2 is the variance of projection at θ, μθ = 1 ρ R(ρ, θ) and P , the number of samples. If angle of rotation is α, then P Rα (ρ, θ) = R(ρ, θ + α). This simply implies a circular shift of the histograms such that it does not require histograms duplication from [0, π[ to [π, 2π[ as in [9] to make rotation invariant. For scaling, we simply normalise histograms into [0, 1] at every projecting angle. Fig. 2 shows radon features for reference, rotation, scaling, as well as degradation instances from a known class of graphical symbol [15]. In all cases, radon histograms from their corresponding sample images are similar to each other.
252
K.C. Santosh, B. Lamiroy, and L. Wendling y
y
y
x
x
x
(b) projection at angle θ
(c) definition
(a) basic projections theory
Fig. 1. Radon transform
0.7
−100
0.6 0.5
100
0.4
0.3
200
0.3
0.2
300
0.1
(b) rotation
0
0.4
0
(a) reference
−200
−200
0.6
200
0 50100150 θ
0.8
−300
0.7
0.5
300
−300
0.8
ρ
ρ
0 100
0.2
400
0.1
500
1
0.9
−400
−200 −100
1
−500
0.9
0.9 0.8 0.7
−100
0.6
0
ρ
1 −300
0.5 0.4
100
0.3 200
0.2 0.1
300
0
0
150 0 50100 θ
(c) scaling
0 50100150 θ
(d) degradation (noise)
Fig. 2. Radon features for ideal (reference), rotation, scaling as well as noisy samples
2.2
Pattern Matching
As explained in Section 2.1, the Radon transform matrix R(ρ, θ) can represent any pattern P. Given two patterns P q and P d , matching can be obtained between corresponding histograms Rq (ρ, θ) and Rd (ρ, θ). Radon transforms generate different ρ sizes depending on the image contents’ size. In order to be able to adapt to these differences in size, we develop the following approach: Dynamic Time Warping [20], allows us to find the dissimilarity between two non-linear sequences potentially having different lengths. In radon matrix R(ρ, θ), column refers to the histogram of radon for each projecting angle. Let us say histograms {Hθi }i=0,...,Θ−1 (see Fig. 3). In this illustration, vertical lines represent radon histograms at every projecting angle. Let us consider two column vector sequences from R(ρ, θ), representing histograms Hq and Hd of length K and L, respectively. Hq = {hqk }k=1,...,K and Hd = {hdl }l=1,...,L . At first, a matrix M of size K × L is constructed. Then for each element in matrix M, local distance metric δ(k, l) between the events ek and el is computed. δ(k, l) can be expressed as, δ(k, l) = (ek − el )2 , where ek = hqk and el = hdl . Let D(k, l) be the global distance up to (k, l), D(k, l) = min [D(k − 1, l − 1), D(k − 1, l), D(k, l − 1)] + δ(k, l) with an initial condition D(1, 1) = δ(1, 1) such that it allows warping path going diagonally from starting node (1, 1) to end (K, L). The main aim is to find the
DTW for Matching Radon Features
253
Fig. 3. Radon histogram at every projecting angle θi
path for which the least cost is associated. The warping path therefore provides the difference cost between the compared sequences. Formally, the warping path is, W = {wt }t=1...T where max(k, l) ≤ T < k + l − 1 and tth element of W is w(k, l)t ∈ [1 : K] × [1 : L] for t ∈ [1 : T ]. The optimised warping path W satisfies the following three conditions. c1. boundary condition: w1 = (1, 1) and wT = (K, L) c2. monotonicity condition: k1 ≤ k2 ≤ · · · ≤ kK and l1 ≤ l2 ≤ · · · ≤ lL c3. continuity condition: wt+1 − wt ∈ {(1, 1)(0, 1), (1, 0)} for t ∈ [1 : T − 1] c1 conveys that the path starts form (1, 1) to (K, L), aligning all elements to each other. c2 restricts allowable steps in the warping path to adjacent cells, never be back. And monotonicity condition forces the path advances one step at a time. Note that c3 implies c2. We then define the global distance between Hq and Hd as, D(K, L) Δ Hq , Hd = . T The last element of the K × L matrix, normalised by the T provides the DTWdistance between two sequences where T is the number of discrete warping steps along the diagonal DTW-matrix. Matching Score. Aggregating distances between histograms in all corresponding projecting angles θi between P q and P d yields a global pattern-matching score, Dist(P q , P d ) =
Θ−1 i=0
Δ(Hθqi , Hθdi ). min.
Dist(.)−Dist (.) Scores are normalised into [0, 1] by, Dist(.) = Dist max. (.)−Distmin. (.) . As shown in Fig. 2, it is important to notice that there may be significant the amplitude differences between radon histograms, from one sample to another. These amplitude differences are very well handled by the DTW algorithm. In addition, Fig. 4 gives an overview of matching score values of the proposed method to affine transformations, noise addition and stretching.
254
K.C. Santosh, B. Lamiroy, and L. Wendling
cow1 (reference)
cow2 (noise)
cow1 cow2 cow3 cow4 cow5
cow3 (translate + scale)
cow4 (rotate)
cow1
cow2
cow3
cow4
cow5
0.0000
0.0033 0.0000
0.0004 0.0023 0.0000
0.0023 0.0034 0.0018 0.0000
0.0030 0.0031 0.0026 0.0035 0.0000
cow5 (stretch)
Fig. 4. Matching scores between samples shown above. It provides the differences exist between the samples due to noise, translation, rotation as well as scaling.
2.3
Pattern Recognition and Retrieval
We can now use the previously described approach as a global pattern matching score. This score expresses the similarity between database patterns and query. Our problem is: given a set of points S in a metric space Ms and a query point q ∈ Ms , find the closest point in S to q. Now, we express similarity as, 1 for the closest candidate q d q d Similarity(P , P ) = 1 − Dist(P , P ) = 0 for the farthest candidate. Ranking can therefore be expressed on the decreasing order of similarity. In our experiments, we will distinguish “recognition” (search for the closest candidate) from “retrieval” (where closest candidates are retrieved for a given short-list).
3 3.1
Experiments Benchmarking Methods
We confront D−Radon method with well-known descriptors: R−signature [27], GFD [30], SC [4] and Zernike [19]. For those descriptors, it is important to fit the best parameters. For GFD, we have tuned the parameters, radial (4 : 12) and angular (6 : 20) frequencies to get the best combinations. For SC, we follow [4]. In case of Zernike, we have used 36 Zernike functions of order less than or equal to 7. For radon, projecting range is [0, π]. 3.2
Experimental Results
We work on several different datasets in different contexts. However, we primarily focus on A. distorted and degraded symbols in document analysis – graphics recognition and retrieval, and then B. shape retrieval as a CBIR application.
DTW for Matching Radon Features
255
In order to test the robustness of the methods, we work on raw data, no prefiltering (de-noising, for instance) has been applied. A. Symbol Recognition and Retrieval GREC2003 dataset symbol recognition contest [15] – In this dataset, we have used the following different categories: ideal, scaling, distortion as well as degradation. Altogether, there are 50 different model symbols. Those symbols are grouped into 3 sets, containing 5, 20 and 50 model symbols. Each model symbol has 5 test images in every category except the ideal one. Ideal test images are directly taken from the set of model symbols and therefore the test is to evaluate the ability of simple shape discrimination, as the number of symbols increases. Since vectorial distortion works only with symbols with straight lines, and not arcs, it is applied to a subset of 15 model symbols. Besides, there are 9 models of degradation, aiming to evaluate the robustness to the scalability with degradation. Fig. 5 shows a few samples of GREC2003 dataset. To evalu-
(a)ideal
m1
m2
(b) scaling
(c) distortion
m3 m4 m5 m6 m7 m8 (d) 9 different degradation models (m1 to m9)
m9
Fig. 5. GREC2003 samples – graphical symbol
ate the method, each test image is matched with the model symbols to get the closest model. Experimental results for all types of aforementioned categories of datasets are shown in Table 1. Note that since there are 9 models of degradations (m1 to m9), there are nine tests in every sample in each set (see Fig. 5 (d)). Based on the results from ideal test images, one cannot judge the superiority of the methods. Only running time comparison would be an alternative. For scaled images, R−signature lags far behind. Our method achieves 100% recognition rate while not offering substantial difference with GFD, SC and Zernike. Also, results from test images with vectorial distortions shows identical behaviour as in scaled as well as rotated samples. However, we receive noteworthy differences in case of binary degradations. Overall, D−Radon performs the best of all. CVC hand-drawn symbol dataset – As in [29], we have used 10 × 300 sample images i.e., 10 different known classes of hand-drawn architectural symbols with 300 instances in each. Samples are having distortions, gaps, overlapping as well as missing parts within the shapes. Fig. 6 shows a few samples of it. In this dataset, we aim to retrieve all 300 instances for every chosen query.
256
K.C. Santosh, B. Lamiroy, and L. Wendling Table 1. Recognition rate in % for GREC2003 dataset
ideal scale
set1 set2 set3
distort
distort1-set1 distort2-set1 distort3-set1 distort1-set2 distort2-set2 distort3-set2
degrade
Test images R-sign. GFD Zernike set1 set2 set3
5×1 20×1 50×1
set1 set2 set3
Average – ideal 5×5 20×5 50×5
Average – scale 5×5 5×5 5×5 15×5 15×5 15×5
Average – distort 5×5× 9 20×5× 9 50×5× 9
Average – degrade
SC
D-Radon
100 100 100
100 100 100
100 100 100
100 100 100
100 100 100
100
100
100
100
100
45 36 28
100 100 98
100 98 96
100 100 98
100 100 100
37
99
98
99
100
20 8 8 8 4 4
100 100 100 100 100 100
100 100 100 100 100 100
100 100 100 100 100 100
100 100 100 100 100 100
7
100
100
100
100
12 07 07
86 93 89
79 79 77
87 76 70
95 96 93
08
89
78
77
95
Table 2 shows the average retrieval rate for all requested short-lists (e.g., top-20, top-40 and so on). Up to top-60, one cannot make decision that which method performs well since there exists no notable differences in retrieval rate. It is only determined after top-60. To be precise, the aim of the test is to evaluate retrieval stability of the methods. D−Radon provides the rate of 86% in top-300 while more than 16% difference (fairly in large amount) has been made with SC. SC lags GFD by approximately 9%. R−signature provides an average results compared to Zernike. Compared to all, D−Radon outperforms.
class 1
class 2 class 3 class 4 class 5 class 6
class 7
class 8
class 9
class 10
Fig. 6. 2 hand-drawn samples from 10 different known classes
Table 2. Average retrieval rate in % for CVC hand-drawn symbol dataset
Zernike R−sign. GFD SC D−Radon
10×300
Test top top top top top top top top top top top top top top top images 20 40 60 80 100 120 140 160 180 200 220 240 260 280 300 48 82 96 98 99
42 75 93 95 99
39 69 90 95 98
37 65 88 92 97
35 62 85 91 97
34 59 83 88 96
33 56 81 87 95
32 54 78 85 94
30 51 76 83 93
30 49 73 81 92
29 48 71 78 91
28 46 68 78 90
28 45 66 75 89
27 43 63 73 87
26 42 61 70 86
DTW for Matching Radon Features
257
(a) shapes99
(b) shapes216
Fig. 7. 2 samples from each class of (a) shapes99 and (b) shapes216 datasets
B. Shape Retrieval We have used two different shape datasets [26]: Kimia’s Shapes99 and Shapes216. Shapes99 dataset consists of 9 classes, each one is having 11 samples. In another dataset, there are 18 classes and each one contains 12 samples. Fig. 7 shows a few samples of both datasets. As in [26], we have used the dataset for recognition purpose. For retrieval rate, since there are N instances from each class, we have increased proximity search from 1 to N with the step of +1. In addition, we have tested Bull’s eye score [21,4]. Bull’s eye score is the ratio of the number of correct matches up to proximity search space of 2N to the possible number of matches in the dataset. Table 3 shows the experimental results for both datasets. Compared to Zernike and R−signature, our method outperforms with significant difference while it goes almost equally with GFD and SC.
4
Discussions
We analyse behaviour of the methods based on the key characteristics as well as major challenges of test images. In graphical symbol datasets, samples are distorted, embedded with different levels of noise and even degraded. In particular, there exists missing parts including severe vectorial distortions in hand-drawn symbol dataset in addition to a significant size variation as well as multi-class similarity between the classes. Besides, occlusion exists more in hand-drawn and shape datasets. Of course, global signal based descriptors are easy to implement as explained in Section 1. However, they are generally not well adapted for capturing precisely small detail changes as well as this tends to make them fault tolerant to image distortions, noise as well as missing parts. We have examined such a behaviour throughout the tests. Considering GREC2003 dataset for instance, for degradation models R−signature is affected from every dot (from noise) due to the square effect (via R−transform). Similarly, SC is affected where the level of noise (see m2 to m7 in Fig. 5) is high since it accounts those pixels while sampling. For Zernike, the disadvantage (in particular if high degree polynomials
258
K.C. Santosh, B. Lamiroy, and L. Wendling
top 1
top top top top top top top top top top top bull’s eye 2 3 4 5 6 7 8 9 10 11 12 score
9×11
Zernike R−sign. GFD SC D−Radon
Test image
100 100 100 100 100
74 83 97 97 97
66 73 95 96 95
59 66 92 92 94
53 60 89 90 92
48 56 87 87 91
45 51 85 83 86
42 49 84 80 81
40 47 82 77 80
37 45 80 73 79
35 43 77 70 77
18×12
shapes99
Zernike R−sign. SC GFD D−Radon
shapes216
Table 3. Average retrieval rate and Bull’s eye score in % for kimia’s dataset
100 100 100 100 100
81 86 97 99 99
71 80 93 97 95
63 76 91 95 93
57 71 87 93 91
53 67 85 91 90
50 65 84 90 89
48 62 81 89 86
45 59 79 88 85
43 57 78 86 84
41 54 76 83 83
48 59 84 86 87 39 52 73 80 81
48 64 83 87 88
Table 4. Average running time for generating features and matching for a single pair
Time (sec.)
R−sign. GFD Zernike SC D−Radon 1.5 15 20 43 77
are involved) is the unequal distribution of nodal lines over the unit disk. We have observed the effect in case of degradation models: m8 and m9 in Fig. 5. Similar situations occur in the CVC hand-drawn symbols. GFD on the whole, provides average results. Those descriptors provide interesting results for silhouette shapes. Considering such datasets, our method performs reasonably better. More specifically, it can optimally handle noisy, degraded as well as distorted samples as well as where internal content is necessary to be considered. Running time complexity is usually high since it uses DTW for matching radon histograms, however it is largely depend on how big the image is. As far as concern to computational cost, the observed average running time for all methods is given in Table 4. We have used MATLAB 7.8.0 in Linux platform.
5
Conclusions
We have presented a method for graphics recognition under document analysis and shape retrieval under CBIR applications. The method is basically based on dynamic programming for matching radon features. The method is quite simple and easy to implement since it is parameter free. Computing radon transform is quite immediate. But the running time complexity lies in matching. However, it could be substantially reduced by using optimised DTW [25] – a step to go further.
References 1. Attalla, E., Siy, P.: Robust shape similarity retrieval based on contour segmentation polygonal multiresolution and elastic matching. PR 38(12), 2229–2241 (2005) 2. Bailey, R.R., Srinath, M.: Orthogonal moment features for use with parametric and non-parametric classifiers. IEEE PAMI 18(4), 389–399 (1996)
DTW for Matching Radon Features
259
3. Bandera, C.U.A., Sandoval, F.: Non-parametric planar shape representation based on adaptive curvature functions. PR 35, 43–53 (2002) 4. Belongie, S., Malik, J., Puzicha, J.: Shape matching and object recognition using shape contexts. IEEE PAMI 24(4), 509–522 (2002) 5. Bernier, T., Landry, J.A.: A new method for representing and matching shapes of natural objects. PR 36(8), 1711–1723 (2003) 6. Boulgouris, N.V., Chi, Z.X.: Gait recognition using radon transform and linear discriminant analysis. IEEE Image Processing 16(3), 731–740 (2007) 7. Bunke, H., Riesen, K.: Recent advances in graph-based pattern recognition with applications in document analysis. PR 44(5), 1057–1067 (2011) 8. Chong, C., Raveendran, P., Mukudan, R.: A comparative analysis of algorithms for fast computation of zernike moments. PR 36, 731–742 (2003) 9. Coetzer, J.: Off-line Signature Verification. Ph.D. thesis, University of Stellenbosch (2005) 10. Conte, D., Foggia, P., Sansone, C., Vento, M.: Thirty years of graph matching in pattern recognition. IJPRAI 18(3), 265–298 (2004) 11. Deans, S.R.: Applications of the Radon Transform. Wiley Interscience Publications, New York (1983) 12. El-ghazal, A., Basir, O., Belkasim, S.: Farthest point distance: A new shape signature for fourier descriptors. Signal Processing: Image Communication 24(7), 572– 586 (2009) 13. Flusser, J.: On the independence of rotation moment invariants. PR 33(9), 1405– 1410 (2000) 14. Fr¨ anti, P., Mednonogov, A., Kyrki, V., K¨ alvi¨ ainen, H.: Content-based matching of line-drawing images using the hough transform. IJDAR 3(2), 117–124 (2000) 15. GREC: International symbol recognition contest at grec2003 (2003), http://www.cvc.uab.es/grec2003/SymRecContest/ 16. Jafari-Khouzani, K., Soltanian-Zadeh, H.: Radon transform orientation estimation for rotation invariant texture analysis. IEEE PAMI 27(6), 1004–1008 (2005) 17. Jain, A.K., Duin, R.P.W., Mao, J.: Statistical pattern recognition: A review. IEEE PAMI 22(1), 4–37 (2000) 18. Santosh,K.C., Wendling, L., Lamiroy, B.: Using spatial relations for graphical symbol description. In: ICPR. pp. 2041–2044 (2010) 19. Kim, W.Y., Kim, Y.S.: A region-based shape descriptor using zernike moments. Signal Processing: Image Communication 16(1-2), 95–102 (2000) 20. Kruskall, J.B., Liberman, M.: The symmetric time warping algorithm: From continuous to discrete. In: Time Warps, String Edits and Macromolecules: The Theory and Practice of String Comparison, pp. 125–161. Addison-Wesley, Reading (1983) 21. Latecki, L.J., Lakmper, R., Eckhardt, U.: Shape descriptors for non-rigid shapes with a single closed contour. In: CVPR, pp. 1424–1429 (2000) 22. Leavers, V.F.: Use of the two-dimensional radon transform to generate a taxonomy of shape for the characterization of abrasive powder particles. IEEE PAMI 22(12), 1411–1423 (2000) 23. Loncaric, S.: A survey of shape analysis techniques. PR 31(8), 983–1001 (1998) 24. Jayadevan, R., Kolhe, S.R., Patil, P.M.: Dynamic time warping based static hand printed signature verification. PRR 4(1), 52–65 (2009) 25. Salvador, S., Chan, P.: Toward accurate dynamic time warping in linear time and space. Intell. Data Anal. 11(5), 561–580 (2007) 26. Sebastian, T.B., Klein, P.N., Kimia, B.B.: Recognition of shapes by editing shock graphs. In: ICCV, pp. 755–762 (2001)
260
K.C. Santosh, B. Lamiroy, and L. Wendling
27. Tabbone, S., Wendling, L., Salmon, J.-P.: A new shape descriptor defined on the radon transform. CVIU 102(1), 42–51 (2006) 28. Teh, C.H., Chin, R.T.: On image analysis by the methods of moments. IEEE PAMI 10(4), 496–513 (1988) 29. Wendling, L., Rendek, J., Matsakis, P.: Selection of suitable set of decision rules using choquet integral. In: SSPR/SPR, pp. 947–955 (2008) 30. Zhang, D., Lu, G.: Shape-based image retrieval using generic fourier descriptor. Signal Processing: Image Communication 17, 825–848 (2002) 31. Zhang, D., Lu, G.: Review of shape representation and description techniques. PR 37(1), 1–19 (2004) 32. Zhang, D., Lu, G.: Study and evaluation of different fourier methods for image retrieval. IVC 23(1), 33–49 (2005) 33. Zhu, S.C., Yuille, A.L.: Forms: A flexible object recognition and modelling system. IJCV 20(3), 187–212 (1996)
Ridges and Valleys Detection in Images Using Difference of Rotating Half Smoothing Filters Baptiste Magnier, Philippe Montesinos, and Daniel Diep Ecole des Mines d’ALES, LGI2P, Site EERIE, Parc Scientifique G.Besse 30035 Nimes Cedex 1 {Baptiste.Magnier,Philippe.Montesinos,Daniel.Diep}@mines-ales.fr
Abstract. In this paper we propose a new ridge/valley detection method in images based on the difference of rotating Gaussian semi filters. The novelty of this approach resides in the mixing of ideas coming both from directional filters and DoG method. We obtain a new ridge/valley anisotropic DoG detector enabling very precise detection of ridge/valley points. Moreover, this detector performs correctly at crest lines even if highly bended, and is precise on junctions. This detector has been tested successfully on various image types presenting difficult problems for classical ridges/valleys detection methods. Keywords: Ridge/valley, directional filter, Gauss filter, difference of Gaussian, anisotropic.
1
Introduction
Anisotropic filters are an important part in image processing. Indeed, anisotropic filters provide good results and are often used in edge detection [5] [10], texture removal [9], image enhancing [12] and restoration [11]. In several domains, anisotropic filters allow for a better robustness than classical method. However, they are seldom used in crest lines finding. Ridges and valleys are formed with the points where the intensity gray level reaches a local extremum in a given direction (illustrated in Fig. 1). This direction is the normal to the curve traced by the ridge or respectively the valley at this point [3]. Crest lines correspond to important features in many images. Ridges and valleys are attached but not limited to roads in aerial images [7] or blood vessels in medical images [1] [6]. Classical edge detection [2] fails to detect ridges or valleys in images. Instead, it results in two edges at each side of the ridge or the valley (illustrated in Fig. 1(e)). Edges can be used to detect straight lines using a Hough transform. It is advisable to compute edges using [5] which creates straight contours [13]. However, this method is adapted only for straight lines [4]. According to [6], crest lines extraction can be divided in three main categories of segmentation algorithms. The first refers to pattern recognition and filtering J. Blanc-Talon et al. (Eds.): ACIVS 2011, LNCS 6915, pp. 261–272, 2011. Springer-Verlag Berlin Heidelberg 2011
262
B. Magnier, P. Montesinos, and D. Diep
1
1
z
y
z
y 0
0
y
x (a)
x (b)
x (c)
y
x (d)
(e)
Fig. 1. Valley and ridge in scalar images. (a) Valley in an image. (b) Surface representation of a valley. (c) Ridge in an image. (d) Surface representation of a ridge. T he z axis corresponds to the intensity gray level. (e) Edge detection on the image in (a).
techniques (for example differential geometry [1] [14] and morphology), the second to model-based approaches (snakes) [8] [7], and the third to tracking-based approaches. Filtering techniques are well adapted in ridge and valley extraction because they are able to smooth the noise and amplify the crest line information by computing surface curvature [1] [14]. However, results obtained by these approaches can present important false detection rate in noisy images. It is mainly because the high pass filtering used for the second derivative is sensitive to the noise level. In this paper, we present a rotating filter (inspired by [9] [1] and [10]) able to detect ridges and valleys. Our ridge/valley detector implements anisotropic directional linear filtering by means of difference of two rotating half smoothing filters. Then, we compute a ridge or valley operator using a local directional maximization or respectively minimization of the response of the filters. These directions correspond to the orientation of a crest line or a junction of ridges/valleys. Contrary to several approaches involving crest lines, this algorithm performs fine even on highly bended ridges or valleys. Moreover, our detector is robust at level of crest lines junctions and bended lines due to these two rotating half smoothing kernels. Finally, due to its strong smoothing in the directions of the crest line, the detection is not sensitive to noise. This paper is organized as follows. In the section 2, we present an anisotropic smoothing Gaussian filter. We present a robust crest line detector using difference of half directional Gaussian filters in the section 3. The section 4 is devoted to experimental results, comparison with an other methods and results evaluation. Finally, the section 5 concludes this paper and presents future works.
2
A Rotating Smoothing Filter
In our method, for each pixel of the original image, we use a rotating smoothing filter in order to build a signal s which is a function of a rotation angle θ and the underlying signal. As shown in [10] and [9], smoothing with rotating filters means that the image is smoothed with a bank of rotated anisotropic Gaussian kernels:
Ridges and Valleys Detection in Images
263
θ
Y
1_1 _ 2 2
X
(a)
Smoothing filter
(b)
Rotating filters
(c)
Points selection on an original image 712×220.
Fig. 2. A smoothing rotating filter and points selection on an original image
(a) θ = 34 degrees, μ = 10, λ = 1
(b) θ = 270 degrees, μ = 10, λ = 1
(c) θ = 34 degrees, μ = 10, λ = 1.5
(d) θ = 270 degrees, μ = 10, λ = 1.5
Fig. 3. Image in Fig. 2(c) smoothed using different parameters and different orientations
− x y x G(μ,λ) (x, y, θ) = C.H Rθ e y
⎛ ⎝ R−1 θ
1 2 μ2
0
0 1 2λ2
⎞
⎛
⎠ Rθ ⎝
⎞
x⎠ y
(1)
where C is a normalization coefficient, Rθ a rotation matrix of angle θ, x and y are pixel coordinates and μ and λ the standard-deviations of the Gaussian filter. As we need only the causal part of the filter (illustrated on figure 2(a)), we simply “cut” the smoothing kernel by the middle, this operation corresponds to the Heaviside function H [10]. By convolution with these rotated kernels (see figure 2(b)), we obtain a collection of directional smoothed images Iθ = I ∗ G(μ,λ) (θ). For computational efficiency, we proceed in a first step to the rotation of the image at some discretized orientations from 0 to 360 degrees (of Δθ = 1, 2, 5, or 10 degrees, depending on the angular precision needed and the smoothing parameters) before applying non rotated smoothing filters. μ and λ define the standard-deviations of the Gaussian filter (illustrated on figure 2(a)). As the image is rotated instead of the filters, the filtering implementation can use efficient recursive approximation of the Gaussian filter. As presented in [10], the implementation is quite straightforward. In a second step, we apply an inverse rotation of the smoothed image and obtain a bank of 360/Δθ images (some examples are available in Fig. 3).
264
3 3.1
B. Magnier, P. Montesinos, and D. Diep
Ridge/Valley Lines Detection Using Difference of Directional Gaussian Filters Difference of Rotated Half Smoothing Filters (DRF)
As presented in Fig. 4(a), we want to estimate at each pixel a smoothed second derivative of the image along a curve crossing this pixel. In one dimension, the second derivative of a signal can be estimated thanks to a DoG operator. For our problem, we have just to apply two filters with two different λ and the same μ to obtain directional derivatives (one example of two discretized filters is available in Fig. 4(d)). Then, we compute the difference of these two filters to obtain the desired smoothed second derivative information in the thin net directions (illustrated in Fig. 4(b)). 3.2
Pixel Classification
Applying by convolution the DRF filter to each pixel of an image (by means of a technique of rotated images, as defined above), we obtain for each pixel a signal which corresponds to a 360/Δθ scan in all directions (see Fig. 5). Our idea is then to characterize pixels which belong to